[RFC v2 00/13] Multi-Key Total Memory Encryption API (MKTME)

[RFC v2 00/13] Multi-Key Total Memory Encryption API (MKTME)

[Alison Schofield]

Hi Thomas, David,

Here is an updated RFC on the API's to support MKTME.
(Multi-Key Total Memory Encryption)

This RFC presents the 2 API additions to support the creation and
usage of memory encryption keys:
 1) Kernel Key Service type "mktme"
 2) System call encrypt_mprotect()

This patchset is built upon Kirill Shutemov's work for the core MKTME
support.

David: Please let me know if the changes made, based on your review,
are reasonable. I don't think that the new changes touch key service
specific areas (much).

Thomas: Please provide feedback on encrypt_mprotect(). If not a
review, then a direction check would be helpful.

I picked up a few more 'CCs this time in get_maintainer!

Thanks!
Alison


Changes in RFC2
Add a preparser to mktme key service. (dhowells)
Replace key serial no. with key struct point in mktme_map. (dhowells)
Remove patch that inserts a special MKTME case in keyctl revoke. (dhowells)
Updated key usage syntax in the documentation (Kai)
Replaced NO_PKEY, NO_KEYID with a single constant NO_KEY. (Jarkko)
Clarified comments in changelog and code. (Jarkko)
Add clear, no-encrypt, and update key support.
Add mktme_savekeys (Patch 12 ) to give kernel permission to save key data.
Add cpu hotplug support. (Patch 13)

Alison Schofield (13):
  x86/mktme: Document the MKTME APIs
  mm: Generalize the mprotect implementation to support extensions
  syscall/x86: Wire up a new system call for memory encryption keys
  x86/mm: Add helper functions for MKTME memory encryption keys
  x86/mm: Set KeyIDs in encrypted VMAs
  mm: Add the encrypt_mprotect() system call
  x86/mm: Add helpers for reference counting encrypted VMAs
  mm: Use reference counting for encrypted VMAs
  mm: Restrict memory encryption to anonymous VMA's
  keys/mktme: Add the MKTME Key Service type for memory encryption
  keys/mktme: Program memory encryption keys on a system wide basis
  keys/mktme: Save MKTME data if kernel cmdline parameter allows
  keys/mktme: Support CPU Hotplug for MKTME keys

 Documentation/admin-guide/kernel-parameters.rst |   1 +
 Documentation/admin-guide/kernel-parameters.txt |  11 +
 Documentation/x86/mktme/index.rst               |  11 +
 Documentation/x86/mktme/mktme_demo.rst          |  53 +++
 Documentation/x86/mktme/mktme_encrypt.rst       |  58 +++
 Documentation/x86/mktme/mktme_keys.rst          | 109 +++++
 Documentation/x86/mktme/mktme_overview.rst      |  60 +++
 arch/x86/Kconfig                                |   1 +
 arch/x86/entry/syscalls/syscall_32.tbl          |   1 +
 arch/x86/entry/syscalls/syscall_64.tbl          |   1 +
 arch/x86/include/asm/mktme.h                    |  25 +
 arch/x86/mm/mktme.c                             | 179 ++++++++
 fs/exec.c                                       |   4 +-
 include/keys/mktme-type.h                       |  41 ++
 include/linux/key.h                             |   2 +
 include/linux/mm.h                              |  11 +-
 include/linux/syscalls.h                        |   2 +
 include/uapi/asm-generic/unistd.h               |   4 +-
 kernel/fork.c                                   |   2 +
 kernel/sys_ni.c                                 |   2 +
 mm/mprotect.c                                   |  91 +++-
 security/keys/Kconfig                           |  11 +
 security/keys/Makefile                          |   1 +
 security/keys/mktme_keys.c                      | 580 ++++++++++++++++++++++++
 24 files changed, 1249 insertions(+), 12 deletions(-)
 create mode 100644 Documentation/x86/mktme/index.rst
 create mode 100644 Documentation/x86/mktme/mktme_demo.rst
 create mode 100644 Documentation/x86/mktme/mktme_encrypt.rst
 create mode 100644 Documentation/x86/mktme/mktme_keys.rst
 create mode 100644 Documentation/x86/mktme/mktme_overview.rst
 create mode 100644 include/keys/mktme-type.h
 create mode 100644 security/keys/mktme_keys.c

-- 
2.14.1

[RFC v2 00/13] Multi-Key Total Memory Encryption API (MKTME)

[Alison Schofield]

Hi Thomas, David,

Here is an updated RFC on the API's to support MKTME.
(Multi-Key Total Memory Encryption)

This RFC presents the 2 API additions to support the creation and
usage of memory encryption keys:
 1) Kernel Key Service type "mktme"
 2) System call encrypt_mprotect()

This patchset is built upon Kirill Shutemov's work for the core MKTME
support.

David: Please let me know if the changes made, based on your review,
are reasonable. I don't think that the new changes touch key service
specific areas (much).

Thomas: Please provide feedback on encrypt_mprotect(). If not a
review, then a direction check would be helpful.

I picked up a few more 'CCs this time in get_maintainer!

Thanks!
Alison


Changes in RFC2
Add a preparser to mktme key service. (dhowells)
Replace key serial no. with key struct point in mktme_map. (dhowells)
Remove patch that inserts a special MKTME case in keyctl revoke. (dhowells)
Updated key usage syntax in the documentation (Kai)
Replaced NO_PKEY, NO_KEYID with a single constant NO_KEY. (Jarkko)
Clarified comments in changelog and code. (Jarkko)
Add clear, no-encrypt, and update key support.
Add mktme_savekeys (Patch 12 ) to give kernel permission to save key data.
Add cpu hotplug support. (Patch 13)

Alison Schofield (13):
  x86/mktme: Document the MKTME APIs
  mm: Generalize the mprotect implementation to support extensions
  syscall/x86: Wire up a new system call for memory encryption keys
  x86/mm: Add helper functions for MKTME memory encryption keys
  x86/mm: Set KeyIDs in encrypted VMAs
  mm: Add the encrypt_mprotect() system call
  x86/mm: Add helpers for reference counting encrypted VMAs
  mm: Use reference counting for encrypted VMAs
  mm: Restrict memory encryption to anonymous VMA's
  keys/mktme: Add the MKTME Key Service type for memory encryption
  keys/mktme: Program memory encryption keys on a system wide basis
  keys/mktme: Save MKTME data if kernel cmdline parameter allows
  keys/mktme: Support CPU Hotplug for MKTME keys

 Documentation/admin-guide/kernel-parameters.rst |   1 +
 Documentation/admin-guide/kernel-parameters.txt |  11 +
 Documentation/x86/mktme/index.rst               |  11 +
 Documentation/x86/mktme/mktme_demo.rst          |  53 +++
 Documentation/x86/mktme/mktme_encrypt.rst       |  58 +++
 Documentation/x86/mktme/mktme_keys.rst          | 109 +++++
 Documentation/x86/mktme/mktme_overview.rst      |  60 +++
 arch/x86/Kconfig                                |   1 +
 arch/x86/entry/syscalls/syscall_32.tbl          |   1 +
 arch/x86/entry/syscalls/syscall_64.tbl          |   1 +
 arch/x86/include/asm/mktme.h                    |  25 +
 arch/x86/mm/mktme.c                             | 179 ++++++++
 fs/exec.c                                       |   4 +-
 include/keys/mktme-type.h                       |  41 ++
 include/linux/key.h                             |   2 +
 include/linux/mm.h                              |  11 +-
 include/linux/syscalls.h                        |   2 +
 include/uapi/asm-generic/unistd.h               |   4 +-
 kernel/fork.c                                   |   2 +
 kernel/sys_ni.c                                 |   2 +
 mm/mprotect.c                                   |  91 +++-
 security/keys/Kconfig                           |  11 +
 security/keys/Makefile                          |   1 +
 security/keys/mktme_keys.c                      | 580 ++++++++++++++++++++++++
 24 files changed, 1249 insertions(+), 12 deletions(-)
 create mode 100644 Documentation/x86/mktme/index.rst
 create mode 100644 Documentation/x86/mktme/mktme_demo.rst
 create mode 100644 Documentation/x86/mktme/mktme_encrypt.rst
 create mode 100644 Documentation/x86/mktme/mktme_keys.rst
 create mode 100644 Documentation/x86/mktme/mktme_overview.rst
 create mode 100644 include/keys/mktme-type.h
 create mode 100644 security/keys/mktme_keys.c

-- 
2.14.1

[RFC v2 01/13] x86/mktme: Document the MKTME APIs

[Alison Schofield]

This includes an overview, a section on each API: MTKME Keys and
system call encrypt_mprotect(), and a demonstration program.

(Some of this info is destined for man pages.)

Change-Id: I34dc9ff1a1308c057ec4bb3e652c4d7ce6995606
Signed-off-by: Alison Schofield <alison.schofield@intel.com>
Signed-off-by: Kirill A. Shutemov <kirill.shutemov@linux.intel.com>
---
 Documentation/x86/mktme/index.rst          |  11 +++
 Documentation/x86/mktme/mktme_demo.rst     |  53 ++++++++++++++
 Documentation/x86/mktme/mktme_encrypt.rst  |  58 +++++++++++++++
 Documentation/x86/mktme/mktme_keys.rst     | 109 +++++++++++++++++++++++++++++
 Documentation/x86/mktme/mktme_overview.rst |  60 ++++++++++++++++
 5 files changed, 291 insertions(+)
 create mode 100644 Documentation/x86/mktme/index.rst
 create mode 100644 Documentation/x86/mktme/mktme_demo.rst
 create mode 100644 Documentation/x86/mktme/mktme_encrypt.rst
 create mode 100644 Documentation/x86/mktme/mktme_keys.rst
 create mode 100644 Documentation/x86/mktme/mktme_overview.rst

diff --git a/Documentation/x86/mktme/index.rst b/Documentation/x86/mktme/index.rst
new file mode 100644
index 000000000000..8c556d04cbc4
--- /dev/null
+++ b/Documentation/x86/mktme/index.rst
@@ -0,0 +1,11 @@
+
+======================+Multi-Key Total Memory Encryption (MKTME) API
+======================+
+.. toctree::
+
+   mktme_overview
+   mktme_keys
+   mktme_encrypt
+   mktme_demo
diff --git a/Documentation/x86/mktme/mktme_demo.rst b/Documentation/x86/mktme/mktme_demo.rst
new file mode 100644
index 000000000000..afd50772e65d
--- /dev/null
+++ b/Documentation/x86/mktme/mktme_demo.rst
@@ -0,0 +1,53 @@
+Demonstration Program using MKTME API's
+===================+
+/* Compile with the keyutils library: cc -o mdemo mdemo.c -lkeyutils */
+
+#include <sys/mman.h>
+#include <sys/syscall.h>
+#include <sys/types.h>
+#include <keyutils.h>
+#include <stdio.h>
+#include <string.h>
+#include <unistd.h>
+
+#define PAGE_SIZE sysconf(_SC_PAGE_SIZE)
+#define sys_encrypt_mprotect 335
+
+void main(void)
+{
+	char *options_CPU = "algorithmŪs-xts-128 type=cpu";
+	long size = PAGE_SIZE;
+        key_serial_t key;
+	void *ptra;
+	int ret;
+
+        /* Allocate an MKTME Key */
+	key = add_key("mktme", "testkey", options_CPU, strlen(options_CPU),
+                      KEY_SPEC_THREAD_KEYRING);
+
+	if (key = -1) {
+		printf("addkey FAILED\n");
+		return;
+	}
+        /* Map a page of ANONYMOUS memory */
+	ptra = mmap(NULL, size, PROT_NONE, MAP_ANONYMOUS|MAP_PRIVATE, -1, 0);
+	if (!ptra) {
+		printf("failed to mmap");
+		goto inval_key;
+	}
+        /* Encrypt that page of memory with the MKTME Key */
+	ret = syscall(sys_encrypt_mprotect, ptra, size, PROT_NONE, key);
+	if (ret)
+		printf("mprotect error [%d]\n", ret);
+
+        /* Enjoy that page of encrypted memory */
+
+        /* Free the memory */
+	ret = munmap(ptra, size);
+
+inval_key:
+        /* Free the Key */
+	if (keyctl(KEYCTL_INVALIDATE, key) = -1)
+		printf("invalidate failed on key [%d]\n", key);
+}
diff --git a/Documentation/x86/mktme/mktme_encrypt.rst b/Documentation/x86/mktme/mktme_encrypt.rst
new file mode 100644
index 000000000000..ede5237183fc
--- /dev/null
+++ b/Documentation/x86/mktme/mktme_encrypt.rst
@@ -0,0 +1,58 @@
+MKTME API: system call encrypt_mprotect()
+====================+
+Synopsis
+--------
+int encrypt_mprotect(void \*addr, size_t len, int prot, key_serial_t serial);
+
+Where *key_serial_t serial* is the serial number of a key allocated
+using the MKTME Key Service.
+
+Description
+-----------
+    encrypt_mprotect() encrypts the memory pages containing any part
+    of the address range in the interval specified by addr and len.
+
+    encrypt_mprotect() supports the legacy mprotect() behavior plus
+    the enabling of memory encryption. That means that in addition
+    to encrypting the memory, the protection flags will be updated
+    as requested in the call.
+
+    The *addr* and *len* must be aligned to a page boundary.
+
+    The caller must have *KEY_NEED_VIEW* permission on the key.
+
+    The range of memory that is to be protected must be mapped as
+    *ANONYMOUS*.
+
+Errors
+------
+    In addition to the Errors returned from legacy mprotect()
+    encrypt_mprotect will return:
+
+    ENOKEY *serial* parameter does not represent a valid key.
+
+    EINVAL *len* parameter is not page aligned.
+
+    EACCES Caller does not have *KEY_NEED_VIEW* permission on the key.
+
+EXAMPLE
+--------
+  Allocate an MKTME Key::
+        serial = add_key("mktme", "name", "type=cpu algorithmŪs-xts-128" @u
+
+  Map ANONYMOUS memory::
+        ptr = mmap(NULL, size, PROT_NONE, MAP_ANONYMOUS|MAP_PRIVATE, -1, 0);
+
+  Protect memory::
+        ret = syscall(SYS_encrypt_mprotect, ptr, size, PROT_READ|PROT_WRITE,
+                      serial);
+
+  Use the encrypted memory
+
+  Free memory::
+        ret = munmap(ptr, size);
+
+  Free the key resource::
+        ret = keyctl(KEYCTL_INVALIDATE, serial);
+
diff --git a/Documentation/x86/mktme/mktme_keys.rst b/Documentation/x86/mktme/mktme_keys.rst
new file mode 100644
index 000000000000..5837909b2c54
--- /dev/null
+++ b/Documentation/x86/mktme/mktme_keys.rst
@@ -0,0 +1,109 @@
+MKTME Key Service API
+==========+MKTME is a new key service type added to the Linux Kernel Key Service.
+
+The MKTME Key Service type is available when CONFIG_X86_INTEL_MKTME is
+turned on in Intel platforms that support the MKTME feature.
+
+The MKTME Key Service type manages the allocation of hardware encryption
+keys. Users can request an MKTME type key and then use that key to
+encrypt memory with the encrypt_mprotect() system call.
+
+Usage
+-----
+    When using the Kernel Key Service to request an *mktme* key,
+    specify the *payload* as follows:
+
+    type+        *user*	User will supply the encryption key data. Use this
+                type to directly program a hardware encryption key.
+
+        *cpu*	User requests a CPU generated encryption key.
+                The CPU generates and assigns an ephemeral key.
+
+        *clear* User requests that a hardware encryption key be
+                cleared. This will clear the encryption key from
+                the hardware. On execution this hardware key gets
+                TME behavior.
+
+        *no-encrypt*
+                 User requests that hardware does not encrypt
+                 memory when this key is in use.
+
+    algorithm+        When type=user or type=cpu the algorithm field must be
+        *aes-xts-128*
+
+        When type=clear or type=no-encrypt the algorithm field
+        must not be present in the payload.
+
+    key+        When type=user the user must supply a 128 bit encryption
+        key as exactly 32 ASCII hexadecimal characters.
+
+	When type=cpu the user may optionally supply 128 bits of
+        entropy for the CPU generated encryption key in this field.
+        It must be exactly 32 ASCII hexadecimal characters.
+
+	When type=clear or type=no-encrypt this key field must
+        not be present in the payload.
+
+    tweak+	When type=user the user must supply a 128 bit tweak key
+        as exactly 32 ASCII hexadecimal characters.
+
+	When type=cpu the user may optionally supply 128 bits of
+        entropy for the CPU generated tweak key in this field. It
+        must be exactly 32 ASCII hexadecimal characters.
+
+        When type=clear or type=no-encrypt the tweak field must
+	not be present in the payload.
+
+ERRORS
+------
+    In addition to the Errors returned from the Kernel Key Service,
+    add_key(2) or keyctl(1) commands, the MKTME Key Service type may
+    return the following errors:
+
+    EINVAL for any payload specification that does not match the
+           MKTME type payload as defined above.
+    EACCES for access denied. MKTME key type uses capabilities to
+           restrict the allocation of keys. CAP_SYS_RESOURCE is
+           required, but it will accept the broader capability of
+           CAP_SYS_ADMIN.  See capabilities(7).
+
+    ENOKEY if a hardware key cannot be allocated. Additional error
+           messages will describe the hardware programming errors.
+
+EXAMPLES
+--------
+    Add a 'user' type key::
+
+        char \*options_USER = "type=user
+                               algorithmŪs-xts-128
+                               key\x12345678912345671234567891234567
+                               tweak\x12345678912345671234567891234567";
+
+        key = add_key("mktme", "name", options_USER, strlen(options_USER),
+                      KEY_SPEC_THREAD_KEYRING);
+
+    Add a 'cpu' type key::
+
+        char \*options_USER = "type=cpu algorithmŪs-xts-128";
+
+        key = add_key("mktme", "name", options_CPU, strlen(options_CPU),
+                      KEY_SPEC_THREAD_KEYRING);
+
+    Update a key to 'Clear' type::
+
+        Note: This has the effect of clearing out the previously programmed
+        encryption data in the hardware. Use this to clear the hardware slot
+        prior to invalidating the key.
+
+        ret = keyctl(KEYCTL_UPDATE, key, "type=clear", strlen(options_CLEAR);
+
+    Add a "no-encrypt' type key::
+
+	key = add_key("mktme", "name", "no-encrypt", strlen(options_CPU),
+		      KEY_SPEC_THREAD_KEYRING);
+
diff --git a/Documentation/x86/mktme/mktme_overview.rst b/Documentation/x86/mktme/mktme_overview.rst
new file mode 100644
index 000000000000..cc2c4a8320e7
--- /dev/null
+++ b/Documentation/x86/mktme/mktme_overview.rst
@@ -0,0 +1,60 @@
+Overview
+====
+MKTME (Multi-Key Total Memory Encryption) is a technology that allows
+memory encryption on Intel platforms. The main use case for the feature
+is virtual machine isolation. The API should apply to a wide range of
+use cases.
+
+Find the Intel Architecture Specification for MKTME here:
+https://software.intel.com/sites/default/files/managed/a5/16/Multi-Key-Total-Memory-Encryption-Spec.pdf
+
+The Encryption Process
+----------------------
+Userspace will see MKTME encryption as a Step Process.
+
+Step 1: Use the MKTME Key Service API to allocate an encryption key.
+
+Step 2: Use the encrypt_mprotect() system call to protect memory
+        with the encryption key obtained in Step 1.
+
+Definitions
+-----------
+Keys:	References to Keys in this document are to Userspace Keys.
+	These keys are requested by users and jointly managed by the
+	MKTME Key Service Type, and more broadly by the Kernel Key
+        Service of which MKTME is a part.
+
+	This document does not intend to document KKS, but only the
+	MKTME type of the KKS. The options of the KKS can be grouped
+	into 2 classes for purposes of understanding how MKTME operates
+	within the broader KKS.
+
+KeyIDs: References to KeyIDs in this document are to the hardware KeyID
+	slots that are available on Intel Platforms. A KeyID is a
+	numerical index into a software programmable slot in the Intel
+	hardware. Refer to the Intel specification linked above for
+	details on the implementation of MKTME in Intel platforms.
+
+Key<-->KeyID Mapping:
+	The MKTME Key Service maintains a mapping between Keys and KeyIDS.
+	This mapping is known only to the kernel. Userspace does not need
+	to know which hardware KeyID slot it's Userspace Key has been
+	assigned.
+
+Configuration
+-------------
+
+CONFIG_X86_INTEL_MKTME
+        MKTME is enabled by selecting CONFIG_X86_INTEL_MKTME on Intel
+        platforms supporting the MKTME feature.
+
+mktme_savekeys
+        mktme_savekeys is a kernel cmdline parameter.
+
+        This parameter allows the kernel to save the user specified
+        MKTME key payload. Saving this payload means that the MKTME
+        Key Service can always allow the addition of new physical
+        packages. If the mktme_savekeys parameter is not present,
+        users key data will not be saved, and new physical packages
+        may only be added to the system if no user type MKTME keys
+        are in use.
-- 
2.14.1

[RFC v2 01/13] x86/mktme: Document the MKTME APIs

[Alison Schofield]

This includes an overview, a section on each API: MTKME Keys and
system call encrypt_mprotect(), and a demonstration program.

(Some of this info is destined for man pages.)

Change-Id: I34dc9ff1a1308c057ec4bb3e652c4d7ce6995606
Signed-off-by: Alison Schofield <alison.schofield@intel.com>
Signed-off-by: Kirill A. Shutemov <kirill.shutemov@linux.intel.com>
---
 Documentation/x86/mktme/index.rst          |  11 +++
 Documentation/x86/mktme/mktme_demo.rst     |  53 ++++++++++++++
 Documentation/x86/mktme/mktme_encrypt.rst  |  58 +++++++++++++++
 Documentation/x86/mktme/mktme_keys.rst     | 109 +++++++++++++++++++++++++++++
 Documentation/x86/mktme/mktme_overview.rst |  60 ++++++++++++++++
 5 files changed, 291 insertions(+)
 create mode 100644 Documentation/x86/mktme/index.rst
 create mode 100644 Documentation/x86/mktme/mktme_demo.rst
 create mode 100644 Documentation/x86/mktme/mktme_encrypt.rst
 create mode 100644 Documentation/x86/mktme/mktme_keys.rst
 create mode 100644 Documentation/x86/mktme/mktme_overview.rst

diff --git a/Documentation/x86/mktme/index.rst b/Documentation/x86/mktme/index.rst
new file mode 100644
index 000000000000..8c556d04cbc4
--- /dev/null
+++ b/Documentation/x86/mktme/index.rst
@@ -0,0 +1,11 @@
+
+=============================================
+Multi-Key Total Memory Encryption (MKTME) API
+=============================================
+
+.. toctree::
+
+   mktme_overview
+   mktme_keys
+   mktme_encrypt
+   mktme_demo
diff --git a/Documentation/x86/mktme/mktme_demo.rst b/Documentation/x86/mktme/mktme_demo.rst
new file mode 100644
index 000000000000..afd50772e65d
--- /dev/null
+++ b/Documentation/x86/mktme/mktme_demo.rst
@@ -0,0 +1,53 @@
+Demonstration Program using MKTME API's
+=======================================
+
+/* Compile with the keyutils library: cc -o mdemo mdemo.c -lkeyutils */
+
+#include <sys/mman.h>
+#include <sys/syscall.h>
+#include <sys/types.h>
+#include <keyutils.h>
+#include <stdio.h>
+#include <string.h>
+#include <unistd.h>
+
+#define PAGE_SIZE sysconf(_SC_PAGE_SIZE)
+#define sys_encrypt_mprotect 335
+
+void main(void)
+{
+	char *options_CPU = "algorithm=aes-xts-128 type=cpu";
+	long size = PAGE_SIZE;
+        key_serial_t key;
+	void *ptra;
+	int ret;
+
+        /* Allocate an MKTME Key */
+	key = add_key("mktme", "testkey", options_CPU, strlen(options_CPU),
+                      KEY_SPEC_THREAD_KEYRING);
+
+	if (key == -1) {
+		printf("addkey FAILED\n");
+		return;
+	}
+        /* Map a page of ANONYMOUS memory */
+	ptra = mmap(NULL, size, PROT_NONE, MAP_ANONYMOUS|MAP_PRIVATE, -1, 0);
+	if (!ptra) {
+		printf("failed to mmap");
+		goto inval_key;
+	}
+        /* Encrypt that page of memory with the MKTME Key */
+	ret = syscall(sys_encrypt_mprotect, ptra, size, PROT_NONE, key);
+	if (ret)
+		printf("mprotect error [%d]\n", ret);
+
+        /* Enjoy that page of encrypted memory */
+
+        /* Free the memory */
+	ret = munmap(ptra, size);
+
+inval_key:
+        /* Free the Key */
+	if (keyctl(KEYCTL_INVALIDATE, key) == -1)
+		printf("invalidate failed on key [%d]\n", key);
+}
diff --git a/Documentation/x86/mktme/mktme_encrypt.rst b/Documentation/x86/mktme/mktme_encrypt.rst
new file mode 100644
index 000000000000..ede5237183fc
--- /dev/null
+++ b/Documentation/x86/mktme/mktme_encrypt.rst
@@ -0,0 +1,58 @@
+MKTME API: system call encrypt_mprotect()
+=========================================
+
+Synopsis
+--------
+int encrypt_mprotect(void \*addr, size_t len, int prot, key_serial_t serial);
+
+Where *key_serial_t serial* is the serial number of a key allocated
+using the MKTME Key Service.
+
+Description
+-----------
+    encrypt_mprotect() encrypts the memory pages containing any part
+    of the address range in the interval specified by addr and len.
+
+    encrypt_mprotect() supports the legacy mprotect() behavior plus
+    the enabling of memory encryption. That means that in addition
+    to encrypting the memory, the protection flags will be updated
+    as requested in the call.
+
+    The *addr* and *len* must be aligned to a page boundary.
+
+    The caller must have *KEY_NEED_VIEW* permission on the key.
+
+    The range of memory that is to be protected must be mapped as
+    *ANONYMOUS*.
+
+Errors
+------
+    In addition to the Errors returned from legacy mprotect()
+    encrypt_mprotect will return:
+
+    ENOKEY *serial* parameter does not represent a valid key.
+
+    EINVAL *len* parameter is not page aligned.
+
+    EACCES Caller does not have *KEY_NEED_VIEW* permission on the key.
+
+EXAMPLE
+--------
+  Allocate an MKTME Key::
+        serial = add_key("mktme", "name", "type=cpu algorithm=aes-xts-128" @u
+
+  Map ANONYMOUS memory::
+        ptr = mmap(NULL, size, PROT_NONE, MAP_ANONYMOUS|MAP_PRIVATE, -1, 0);
+
+  Protect memory::
+        ret = syscall(SYS_encrypt_mprotect, ptr, size, PROT_READ|PROT_WRITE,
+                      serial);
+
+  Use the encrypted memory
+
+  Free memory::
+        ret = munmap(ptr, size);
+
+  Free the key resource::
+        ret = keyctl(KEYCTL_INVALIDATE, serial);
+
diff --git a/Documentation/x86/mktme/mktme_keys.rst b/Documentation/x86/mktme/mktme_keys.rst
new file mode 100644
index 000000000000..5837909b2c54
--- /dev/null
+++ b/Documentation/x86/mktme/mktme_keys.rst
@@ -0,0 +1,109 @@
+MKTME Key Service API
+=====================
+MKTME is a new key service type added to the Linux Kernel Key Service.
+
+The MKTME Key Service type is available when CONFIG_X86_INTEL_MKTME is
+turned on in Intel platforms that support the MKTME feature.
+
+The MKTME Key Service type manages the allocation of hardware encryption
+keys. Users can request an MKTME type key and then use that key to
+encrypt memory with the encrypt_mprotect() system call.
+
+Usage
+-----
+    When using the Kernel Key Service to request an *mktme* key,
+    specify the *payload* as follows:
+
+    type=
+        *user*	User will supply the encryption key data. Use this
+                type to directly program a hardware encryption key.
+
+        *cpu*	User requests a CPU generated encryption key.
+                The CPU generates and assigns an ephemeral key.
+
+        *clear* User requests that a hardware encryption key be
+                cleared. This will clear the encryption key from
+                the hardware. On execution this hardware key gets
+                TME behavior.
+
+        *no-encrypt*
+                 User requests that hardware does not encrypt
+                 memory when this key is in use.
+
+    algorithm=
+        When type=user or type=cpu the algorithm field must be
+        *aes-xts-128*
+
+        When type=clear or type=no-encrypt the algorithm field
+        must not be present in the payload.
+
+    key=
+        When type=user the user must supply a 128 bit encryption
+        key as exactly 32 ASCII hexadecimal characters.
+
+	When type=cpu the user may optionally supply 128 bits of
+        entropy for the CPU generated encryption key in this field.
+        It must be exactly 32 ASCII hexadecimal characters.
+
+	When type=clear or type=no-encrypt this key field must
+        not be present in the payload.
+
+    tweak=
+	When type=user the user must supply a 128 bit tweak key
+        as exactly 32 ASCII hexadecimal characters.
+
+	When type=cpu the user may optionally supply 128 bits of
+        entropy for the CPU generated tweak key in this field. It
+        must be exactly 32 ASCII hexadecimal characters.
+
+        When type=clear or type=no-encrypt the tweak field must
+	not be present in the payload.
+
+ERRORS
+------
+    In addition to the Errors returned from the Kernel Key Service,
+    add_key(2) or keyctl(1) commands, the MKTME Key Service type may
+    return the following errors:
+
+    EINVAL for any payload specification that does not match the
+           MKTME type payload as defined above.
+    EACCES for access denied. MKTME key type uses capabilities to
+           restrict the allocation of keys. CAP_SYS_RESOURCE is
+           required, but it will accept the broader capability of
+           CAP_SYS_ADMIN.  See capabilities(7).
+
+    ENOKEY if a hardware key cannot be allocated. Additional error
+           messages will describe the hardware programming errors.
+
+EXAMPLES
+--------
+    Add a 'user' type key::
+
+        char \*options_USER = "type=user
+                               algorithm=aes-xts-128
+                               key=12345678912345671234567891234567
+                               tweak=12345678912345671234567891234567";
+
+        key = add_key("mktme", "name", options_USER, strlen(options_USER),
+                      KEY_SPEC_THREAD_KEYRING);
+
+    Add a 'cpu' type key::
+
+        char \*options_USER = "type=cpu algorithm=aes-xts-128";
+
+        key = add_key("mktme", "name", options_CPU, strlen(options_CPU),
+                      KEY_SPEC_THREAD_KEYRING);
+
+    Update a key to 'Clear' type::
+
+        Note: This has the effect of clearing out the previously programmed
+        encryption data in the hardware. Use this to clear the hardware slot
+        prior to invalidating the key.
+
+        ret = keyctl(KEYCTL_UPDATE, key, "type=clear", strlen(options_CLEAR);
+
+    Add a "no-encrypt' type key::
+
+	key = add_key("mktme", "name", "no-encrypt", strlen(options_CPU),
+		      KEY_SPEC_THREAD_KEYRING);
+
diff --git a/Documentation/x86/mktme/mktme_overview.rst b/Documentation/x86/mktme/mktme_overview.rst
new file mode 100644
index 000000000000..cc2c4a8320e7
--- /dev/null
+++ b/Documentation/x86/mktme/mktme_overview.rst
@@ -0,0 +1,60 @@
+Overview
+========
+MKTME (Multi-Key Total Memory Encryption) is a technology that allows
+memory encryption on Intel platforms. The main use case for the feature
+is virtual machine isolation. The API should apply to a wide range of
+use cases.
+
+Find the Intel Architecture Specification for MKTME here:
+https://software.intel.com/sites/default/files/managed/a5/16/Multi-Key-Total-Memory-Encryption-Spec.pdf
+
+The Encryption Process
+----------------------
+Userspace will see MKTME encryption as a Step Process.
+
+Step 1: Use the MKTME Key Service API to allocate an encryption key.
+
+Step 2: Use the encrypt_mprotect() system call to protect memory
+        with the encryption key obtained in Step 1.
+
+Definitions
+-----------
+Keys:	References to Keys in this document are to Userspace Keys.
+	These keys are requested by users and jointly managed by the
+	MKTME Key Service Type, and more broadly by the Kernel Key
+        Service of which MKTME is a part.
+
+	This document does not intend to document KKS, but only the
+	MKTME type of the KKS. The options of the KKS can be grouped
+	into 2 classes for purposes of understanding how MKTME operates
+	within the broader KKS.
+
+KeyIDs: References to KeyIDs in this document are to the hardware KeyID
+	slots that are available on Intel Platforms. A KeyID is a
+	numerical index into a software programmable slot in the Intel
+	hardware. Refer to the Intel specification linked above for
+	details on the implementation of MKTME in Intel platforms.
+
+Key<-->KeyID Mapping:
+	The MKTME Key Service maintains a mapping between Keys and KeyIDS.
+	This mapping is known only to the kernel. Userspace does not need
+	to know which hardware KeyID slot it's Userspace Key has been
+	assigned.
+
+Configuration
+-------------
+
+CONFIG_X86_INTEL_MKTME
+        MKTME is enabled by selecting CONFIG_X86_INTEL_MKTME on Intel
+        platforms supporting the MKTME feature.
+
+mktme_savekeys
+        mktme_savekeys is a kernel cmdline parameter.
+
+        This parameter allows the kernel to save the user specified
+        MKTME key payload. Saving this payload means that the MKTME
+        Key Service can always allow the addition of new physical
+        packages. If the mktme_savekeys parameter is not present,
+        users key data will not be saved, and new physical packages
+        may only be added to the system if no user type MKTME keys
+        are in use.
-- 
2.14.1

[RFC v2 02/13] mm: Generalize the mprotect implementation to support extensions

[Alison Schofield]

Today mprotect is implemented to support legacy mprotect behavior
plus an extension for memory protection keys. Make it more generic
so that it can support additional extensions in the future.

This is done is preparation for adding a new system call for memory
encyption keys. The intent is that the new encrypted mprotect will be
another extension to legacy mprotect.

Change-Id: Ib09b9d1b605b12d0254d7fb4968dfcc8e3c79dd7
Signed-off-by: Alison Schofield <alison.schofield@intel.com>
Signed-off-by: Kirill A. Shutemov <kirill.shutemov@linux.intel.com>
---
 mm/mprotect.c | 10 ++++++----
 1 file changed, 6 insertions(+), 4 deletions(-)

diff --git a/mm/mprotect.c b/mm/mprotect.c
index df408956dccc..b57075e278fb 100644
--- a/mm/mprotect.c
+++ b/mm/mprotect.c
@@ -35,6 +35,8 @@
 
 #include "internal.h"
 
+#define NO_KEY	-1
+
 static unsigned long change_pte_range(struct vm_area_struct *vma, pmd_t *pmd,
 		unsigned long addr, unsigned long end, pgprot_t newprot,
 		int dirty_accountable, int prot_numa)
@@ -451,9 +453,9 @@ mprotect_fixup(struct vm_area_struct *vma, struct vm_area_struct **pprev,
 }
 
 /*
- * pkey=-1 when doing a legacy mprotect()
+ * When pkey=NO_KEY we get legacy mprotect behavior here.
  */
-static int do_mprotect_pkey(unsigned long start, size_t len,
+static int do_mprotect_ext(unsigned long start, size_t len,
 		unsigned long prot, int pkey)
 {
 	unsigned long nstart, end, tmp, reqprot;
@@ -577,7 +579,7 @@ static int do_mprotect_pkey(unsigned long start, size_t len,
 SYSCALL_DEFINE3(mprotect, unsigned long, start, size_t, len,
 		unsigned long, prot)
 {
-	return do_mprotect_pkey(start, len, prot, -1);
+	return do_mprotect_ext(start, len, prot, NO_KEY);
 }
 
 #ifdef CONFIG_ARCH_HAS_PKEYS
@@ -585,7 +587,7 @@ SYSCALL_DEFINE3(mprotect, unsigned long, start, size_t, len,
 SYSCALL_DEFINE4(pkey_mprotect, unsigned long, start, size_t, len,
 		unsigned long, prot, int, pkey)
 {
-	return do_mprotect_pkey(start, len, prot, pkey);
+	return do_mprotect_ext(start, len, prot, pkey);
 }
 
 SYSCALL_DEFINE2(pkey_alloc, unsigned long, flags, unsigned long, init_val)
-- 
2.14.1

[RFC v2 02/13] mm: Generalize the mprotect implementation to support extensions

[Alison Schofield]

Today mprotect is implemented to support legacy mprotect behavior
plus an extension for memory protection keys. Make it more generic
so that it can support additional extensions in the future.

This is done is preparation for adding a new system call for memory
encyption keys. The intent is that the new encrypted mprotect will be
another extension to legacy mprotect.

Change-Id: Ib09b9d1b605b12d0254d7fb4968dfcc8e3c79dd7
Signed-off-by: Alison Schofield <alison.schofield@intel.com>
Signed-off-by: Kirill A. Shutemov <kirill.shutemov@linux.intel.com>
---
 mm/mprotect.c | 10 ++++++----
 1 file changed, 6 insertions(+), 4 deletions(-)

diff --git a/mm/mprotect.c b/mm/mprotect.c
index df408956dccc..b57075e278fb 100644
--- a/mm/mprotect.c
+++ b/mm/mprotect.c
@@ -35,6 +35,8 @@
 
 #include "internal.h"
 
+#define NO_KEY	-1
+
 static unsigned long change_pte_range(struct vm_area_struct *vma, pmd_t *pmd,
 		unsigned long addr, unsigned long end, pgprot_t newprot,
 		int dirty_accountable, int prot_numa)
@@ -451,9 +453,9 @@ mprotect_fixup(struct vm_area_struct *vma, struct vm_area_struct **pprev,
 }
 
 /*
- * pkey==-1 when doing a legacy mprotect()
+ * When pkey==NO_KEY we get legacy mprotect behavior here.
  */
-static int do_mprotect_pkey(unsigned long start, size_t len,
+static int do_mprotect_ext(unsigned long start, size_t len,
 		unsigned long prot, int pkey)
 {
 	unsigned long nstart, end, tmp, reqprot;
@@ -577,7 +579,7 @@ static int do_mprotect_pkey(unsigned long start, size_t len,
 SYSCALL_DEFINE3(mprotect, unsigned long, start, size_t, len,
 		unsigned long, prot)
 {
-	return do_mprotect_pkey(start, len, prot, -1);
+	return do_mprotect_ext(start, len, prot, NO_KEY);
 }
 
 #ifdef CONFIG_ARCH_HAS_PKEYS
@@ -585,7 +587,7 @@ SYSCALL_DEFINE3(mprotect, unsigned long, start, size_t, len,
 SYSCALL_DEFINE4(pkey_mprotect, unsigned long, start, size_t, len,
 		unsigned long, prot, int, pkey)
 {
-	return do_mprotect_pkey(start, len, prot, pkey);
+	return do_mprotect_ext(start, len, prot, pkey);
 }
 
 SYSCALL_DEFINE2(pkey_alloc, unsigned long, flags, unsigned long, init_val)
-- 
2.14.1

[RFC v2 03/13] syscall/x86: Wire up a new system call for memory encryption keys

[Alison Schofield]

encrypt_mprotect() is a new system call to support memory encryption.

It takes the same parameters as legacy mprotect, plus an additional
key serial number that is mapped to an encryption keyid.

Signed-off-by: Alison Schofield <alison.schofield@intel.com>
Signed-off-by: Kirill A. Shutemov <kirill.shutemov@linux.intel.com>
---
 arch/x86/entry/syscalls/syscall_32.tbl | 1 +
 arch/x86/entry/syscalls/syscall_64.tbl | 1 +
 include/linux/syscalls.h               | 2 ++
 include/uapi/asm-generic/unistd.h      | 4 +++-
 kernel/sys_ni.c                        | 2 ++
 5 files changed, 9 insertions(+), 1 deletion(-)

diff --git a/arch/x86/entry/syscalls/syscall_32.tbl b/arch/x86/entry/syscalls/syscall_32.tbl
index 3cf7b533b3d1..f41ad857d5c6 100644
--- a/arch/x86/entry/syscalls/syscall_32.tbl
+++ b/arch/x86/entry/syscalls/syscall_32.tbl
@@ -398,3 +398,4 @@
 384	i386	arch_prctl		sys_arch_prctl			__ia32_compat_sys_arch_prctl
 385	i386	io_pgetevents		sys_io_pgetevents		__ia32_compat_sys_io_pgetevents
 386	i386	rseq			sys_rseq			__ia32_sys_rseq
+387	i386	encrypt_mprotect	sys_encrypt_mprotect		__ia32_sys_encrypt_mprotect
diff --git a/arch/x86/entry/syscalls/syscall_64.tbl b/arch/x86/entry/syscalls/syscall_64.tbl
index f0b1709a5ffb..cf2decfa6119 100644
--- a/arch/x86/entry/syscalls/syscall_64.tbl
+++ b/arch/x86/entry/syscalls/syscall_64.tbl
@@ -343,6 +343,7 @@
 332	common	statx			__x64_sys_statx
 333	common	io_pgetevents		__x64_sys_io_pgetevents
 334	common	rseq			__x64_sys_rseq
+335	common	encrypt_mprotect	__x64_sys_encrypt_mprotect
 
 #
 # x32-specific system call numbers start at 512 to avoid cache impact
diff --git a/include/linux/syscalls.h b/include/linux/syscalls.h
index 2ac3d13a915b..c728b47e9004 100644
--- a/include/linux/syscalls.h
+++ b/include/linux/syscalls.h
@@ -907,6 +907,8 @@ asmlinkage long sys_statx(int dfd, const char __user *path, unsigned flags,
 			  unsigned mask, struct statx __user *buffer);
 asmlinkage long sys_rseq(struct rseq __user *rseq, uint32_t rseq_len,
 			 int flags, uint32_t sig);
+asmlinkage long sys_encrypt_mprotect(unsigned long start, size_t len,
+				     unsigned long prot, key_serial_t serial);
 
 /*
  * Architecture-specific system calls
diff --git a/include/uapi/asm-generic/unistd.h b/include/uapi/asm-generic/unistd.h
index 538546edbfbd..696c222ebe40 100644
--- a/include/uapi/asm-generic/unistd.h
+++ b/include/uapi/asm-generic/unistd.h
@@ -738,9 +738,11 @@ __SYSCALL(__NR_statx,     sys_statx)
 __SC_COMP(__NR_io_pgetevents, sys_io_pgetevents, compat_sys_io_pgetevents)
 #define __NR_rseq 293
 __SYSCALL(__NR_rseq, sys_rseq)
+#define __NR_encrypt_mprotect 294
+__SYSCALL(__NR_encrypt_mprotect, sys_encrypt_mprotect)
 
 #undef __NR_syscalls
-#define __NR_syscalls 294
+#define __NR_syscalls 295
 
 /*
  * 32 bit systems traditionally used different
diff --git a/kernel/sys_ni.c b/kernel/sys_ni.c
index df556175be50..1b48f709c265 100644
--- a/kernel/sys_ni.c
+++ b/kernel/sys_ni.c
@@ -336,6 +336,8 @@ COND_SYSCALL(pkey_mprotect);
 COND_SYSCALL(pkey_alloc);
 COND_SYSCALL(pkey_free);
 
+/* multi-key total memory encryption keys */
+COND_SYSCALL(encrypt_mprotect);
 
 /*
  * Architecture specific weak syscall entries.
-- 
2.14.1

[RFC v2 03/13] syscall/x86: Wire up a new system call for memory encryption keys

[Alison Schofield]

encrypt_mprotect() is a new system call to support memory encryption.

It takes the same parameters as legacy mprotect, plus an additional
key serial number that is mapped to an encryption keyid.

Signed-off-by: Alison Schofield <alison.schofield@intel.com>
Signed-off-by: Kirill A. Shutemov <kirill.shutemov@linux.intel.com>
---
 arch/x86/entry/syscalls/syscall_32.tbl | 1 +
 arch/x86/entry/syscalls/syscall_64.tbl | 1 +
 include/linux/syscalls.h               | 2 ++
 include/uapi/asm-generic/unistd.h      | 4 +++-
 kernel/sys_ni.c                        | 2 ++
 5 files changed, 9 insertions(+), 1 deletion(-)

diff --git a/arch/x86/entry/syscalls/syscall_32.tbl b/arch/x86/entry/syscalls/syscall_32.tbl
index 3cf7b533b3d1..f41ad857d5c6 100644
--- a/arch/x86/entry/syscalls/syscall_32.tbl
+++ b/arch/x86/entry/syscalls/syscall_32.tbl
@@ -398,3 +398,4 @@
 384	i386	arch_prctl		sys_arch_prctl			__ia32_compat_sys_arch_prctl
 385	i386	io_pgetevents		sys_io_pgetevents		__ia32_compat_sys_io_pgetevents
 386	i386	rseq			sys_rseq			__ia32_sys_rseq
+387	i386	encrypt_mprotect	sys_encrypt_mprotect		__ia32_sys_encrypt_mprotect
diff --git a/arch/x86/entry/syscalls/syscall_64.tbl b/arch/x86/entry/syscalls/syscall_64.tbl
index f0b1709a5ffb..cf2decfa6119 100644
--- a/arch/x86/entry/syscalls/syscall_64.tbl
+++ b/arch/x86/entry/syscalls/syscall_64.tbl
@@ -343,6 +343,7 @@
 332	common	statx			__x64_sys_statx
 333	common	io_pgetevents		__x64_sys_io_pgetevents
 334	common	rseq			__x64_sys_rseq
+335	common	encrypt_mprotect	__x64_sys_encrypt_mprotect
 
 #
 # x32-specific system call numbers start at 512 to avoid cache impact
diff --git a/include/linux/syscalls.h b/include/linux/syscalls.h
index 2ac3d13a915b..c728b47e9004 100644
--- a/include/linux/syscalls.h
+++ b/include/linux/syscalls.h
@@ -907,6 +907,8 @@ asmlinkage long sys_statx(int dfd, const char __user *path, unsigned flags,
 			  unsigned mask, struct statx __user *buffer);
 asmlinkage long sys_rseq(struct rseq __user *rseq, uint32_t rseq_len,
 			 int flags, uint32_t sig);
+asmlinkage long sys_encrypt_mprotect(unsigned long start, size_t len,
+				     unsigned long prot, key_serial_t serial);
 
 /*
  * Architecture-specific system calls
diff --git a/include/uapi/asm-generic/unistd.h b/include/uapi/asm-generic/unistd.h
index 538546edbfbd..696c222ebe40 100644
--- a/include/uapi/asm-generic/unistd.h
+++ b/include/uapi/asm-generic/unistd.h
@@ -738,9 +738,11 @@ __SYSCALL(__NR_statx,     sys_statx)
 __SC_COMP(__NR_io_pgetevents, sys_io_pgetevents, compat_sys_io_pgetevents)
 #define __NR_rseq 293
 __SYSCALL(__NR_rseq, sys_rseq)
+#define __NR_encrypt_mprotect 294
+__SYSCALL(__NR_encrypt_mprotect, sys_encrypt_mprotect)
 
 #undef __NR_syscalls
-#define __NR_syscalls 294
+#define __NR_syscalls 295
 
 /*
  * 32 bit systems traditionally used different
diff --git a/kernel/sys_ni.c b/kernel/sys_ni.c
index df556175be50..1b48f709c265 100644
--- a/kernel/sys_ni.c
+++ b/kernel/sys_ni.c
@@ -336,6 +336,8 @@ COND_SYSCALL(pkey_mprotect);
 COND_SYSCALL(pkey_alloc);
 COND_SYSCALL(pkey_free);
 
+/* multi-key total memory encryption keys */
+COND_SYSCALL(encrypt_mprotect);
 
 /*
  * Architecture specific weak syscall entries.
-- 
2.14.1

[RFC v2 04/13] x86/mm: Add helper functions for MKTME memory encryption keys

[Alison Schofield]

Define a global mapping structure to manage the mapping of userspace
Keys to hardware KeyIDs in MKTME (Multi-Key Total Memory Encryption).
Implement helper functions that access this mapping structure.

The helpers will be used by these MKTME API's:
 >  Key Service API: security/keys/mktme_keys.c
 >  encrypt_mprotect() system call: mm/mprotect.c

Signed-off-by: Alison Schofield <alison.schofield@intel.com>
Signed-off-by: Kirill A. Shutemov <kirill.shutemov@linux.intel.com>
---
 arch/x86/include/asm/mktme.h | 12 ++++++
 arch/x86/mm/mktme.c          | 91 ++++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 103 insertions(+)

diff --git a/arch/x86/include/asm/mktme.h b/arch/x86/include/asm/mktme.h
index f05baa15e6f6..dbb49909d665 100644
--- a/arch/x86/include/asm/mktme.h
+++ b/arch/x86/include/asm/mktme.h
@@ -12,6 +12,18 @@ extern phys_addr_t mktme_keyid_mask;
 extern int mktme_nr_keyids;
 extern int mktme_keyid_shift;
 
+/* Manage mappings between hardware KeyIDs and userspace Keys */
+extern int mktme_map_alloc(void);
+extern void mktme_map_free(void);
+extern void mktme_map_lock(void);
+extern void mktme_map_unlock(void);
+extern int mktme_map_mapped_keyids(void);
+extern void mktme_map_set_keyid(int keyid, void *key);
+extern void mktme_map_free_keyid(int keyid);
+extern int mktme_map_keyid_from_key(void *key);
+extern void *mktme_map_key_from_keyid(int keyid);
+extern int mktme_map_get_free_keyid(void);
+
 DECLARE_STATIC_KEY_FALSE(mktme_enabled_key);
 static inline bool mktme_enabled(void)
 {
diff --git a/arch/x86/mm/mktme.c b/arch/x86/mm/mktme.c
index c81727540e7c..34224d4e3f45 100644
--- a/arch/x86/mm/mktme.c
+++ b/arch/x86/mm/mktme.c
@@ -40,6 +40,97 @@ int __vma_keyid(struct vm_area_struct *vma)
 	return (prot & mktme_keyid_mask) >> mktme_keyid_shift;
 }
 
+/*
+ * struct mktme_map and the mktme_map_* functions manage the mapping
+ * of userspace Keys to hardware KeyIDs. These are used by the MKTME Key
+ * Service API and the encrypt_mprotect() system call.
+ */
+
+struct mktme_mapping {
+	struct mutex	lock;		/* protect this map & HW state */
+	unsigned int	mapped_keyids;
+	void		*key[];
+};
+
+struct mktme_mapping *mktme_map;
+
+static inline long mktme_map_size(void)
+{
+	long size = 0;
+
+	size += sizeof(*mktme_map);
+	size += sizeof(mktme_map->key[0]) * (mktme_nr_keyids + 1);
+	return size;
+}
+
+int mktme_map_alloc(void)
+{
+	mktme_map = kvzalloc(mktme_map_size(), GFP_KERNEL);
+	if (!mktme_map)
+		return 0;
+	mutex_init(&mktme_map->lock);
+	return 1;
+}
+
+void mktme_map_free(void)
+{
+	kvfree(mktme_map);
+}
+
+void mktme_map_lock(void)
+{
+	mutex_lock(&mktme_map->lock);
+}
+
+void mktme_map_unlock(void)
+{
+	mutex_unlock(&mktme_map->lock);
+}
+
+int mktme_map_mapped_keyids(void)
+{
+	return mktme_map->mapped_keyids;
+}
+
+void mktme_map_set_keyid(int keyid, void *key)
+{
+	mktme_map->key[keyid] = key;
+	mktme_map->mapped_keyids++;
+}
+
+void mktme_map_free_keyid(int keyid)
+{
+	mktme_map->key[keyid] = 0;
+	mktme_map->mapped_keyids--;
+}
+
+int mktme_map_keyid_from_key(void *key)
+{
+	int i;
+
+	for (i = 1; i <= mktme_nr_keyids; i++)
+		if (mktme_map->key[i] = key)
+			return i;
+	return 0;
+}
+
+void *mktme_map_key_from_keyid(int keyid)
+{
+	return mktme_map->key[keyid];
+}
+
+int mktme_map_get_free_keyid(void)
+{
+	int i;
+
+	if (mktme_map->mapped_keyids < mktme_nr_keyids) {
+		for (i = 1; i <= mktme_nr_keyids; i++)
+			if (mktme_map->key[i] = 0)
+				return i;
+	}
+	return 0;
+}
+
 /* Prepare page to be used for encryption. Called from page allocator. */
 void __prep_encrypted_page(struct page *page, int order, int keyid, bool zero)
 {
-- 
2.14.1

[RFC v2 04/13] x86/mm: Add helper functions for MKTME memory encryption keys

[Alison Schofield]

Define a global mapping structure to manage the mapping of userspace
Keys to hardware KeyIDs in MKTME (Multi-Key Total Memory Encryption).
Implement helper functions that access this mapping structure.

The helpers will be used by these MKTME API's:
 >  Key Service API: security/keys/mktme_keys.c
 >  encrypt_mprotect() system call: mm/mprotect.c

Signed-off-by: Alison Schofield <alison.schofield@intel.com>
Signed-off-by: Kirill A. Shutemov <kirill.shutemov@linux.intel.com>
---
 arch/x86/include/asm/mktme.h | 12 ++++++
 arch/x86/mm/mktme.c          | 91 ++++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 103 insertions(+)

diff --git a/arch/x86/include/asm/mktme.h b/arch/x86/include/asm/mktme.h
index f05baa15e6f6..dbb49909d665 100644
--- a/arch/x86/include/asm/mktme.h
+++ b/arch/x86/include/asm/mktme.h
@@ -12,6 +12,18 @@ extern phys_addr_t mktme_keyid_mask;
 extern int mktme_nr_keyids;
 extern int mktme_keyid_shift;
 
+/* Manage mappings between hardware KeyIDs and userspace Keys */
+extern int mktme_map_alloc(void);
+extern void mktme_map_free(void);
+extern void mktme_map_lock(void);
+extern void mktme_map_unlock(void);
+extern int mktme_map_mapped_keyids(void);
+extern void mktme_map_set_keyid(int keyid, void *key);
+extern void mktme_map_free_keyid(int keyid);
+extern int mktme_map_keyid_from_key(void *key);
+extern void *mktme_map_key_from_keyid(int keyid);
+extern int mktme_map_get_free_keyid(void);
+
 DECLARE_STATIC_KEY_FALSE(mktme_enabled_key);
 static inline bool mktme_enabled(void)
 {
diff --git a/arch/x86/mm/mktme.c b/arch/x86/mm/mktme.c
index c81727540e7c..34224d4e3f45 100644
--- a/arch/x86/mm/mktme.c
+++ b/arch/x86/mm/mktme.c
@@ -40,6 +40,97 @@ int __vma_keyid(struct vm_area_struct *vma)
 	return (prot & mktme_keyid_mask) >> mktme_keyid_shift;
 }
 
+/*
+ * struct mktme_map and the mktme_map_* functions manage the mapping
+ * of userspace Keys to hardware KeyIDs. These are used by the MKTME Key
+ * Service API and the encrypt_mprotect() system call.
+ */
+
+struct mktme_mapping {
+	struct mutex	lock;		/* protect this map & HW state */
+	unsigned int	mapped_keyids;
+	void		*key[];
+};
+
+struct mktme_mapping *mktme_map;
+
+static inline long mktme_map_size(void)
+{
+	long size = 0;
+
+	size += sizeof(*mktme_map);
+	size += sizeof(mktme_map->key[0]) * (mktme_nr_keyids + 1);
+	return size;
+}
+
+int mktme_map_alloc(void)
+{
+	mktme_map = kvzalloc(mktme_map_size(), GFP_KERNEL);
+	if (!mktme_map)
+		return 0;
+	mutex_init(&mktme_map->lock);
+	return 1;
+}
+
+void mktme_map_free(void)
+{
+	kvfree(mktme_map);
+}
+
+void mktme_map_lock(void)
+{
+	mutex_lock(&mktme_map->lock);
+}
+
+void mktme_map_unlock(void)
+{
+	mutex_unlock(&mktme_map->lock);
+}
+
+int mktme_map_mapped_keyids(void)
+{
+	return mktme_map->mapped_keyids;
+}
+
+void mktme_map_set_keyid(int keyid, void *key)
+{
+	mktme_map->key[keyid] = key;
+	mktme_map->mapped_keyids++;
+}
+
+void mktme_map_free_keyid(int keyid)
+{
+	mktme_map->key[keyid] = 0;
+	mktme_map->mapped_keyids--;
+}
+
+int mktme_map_keyid_from_key(void *key)
+{
+	int i;
+
+	for (i = 1; i <= mktme_nr_keyids; i++)
+		if (mktme_map->key[i] == key)
+			return i;
+	return 0;
+}
+
+void *mktme_map_key_from_keyid(int keyid)
+{
+	return mktme_map->key[keyid];
+}
+
+int mktme_map_get_free_keyid(void)
+{
+	int i;
+
+	if (mktme_map->mapped_keyids < mktme_nr_keyids) {
+		for (i = 1; i <= mktme_nr_keyids; i++)
+			if (mktme_map->key[i] == 0)
+				return i;
+	}
+	return 0;
+}
+
 /* Prepare page to be used for encryption. Called from page allocator. */
 void __prep_encrypted_page(struct page *page, int order, int keyid, bool zero)
 {
-- 
2.14.1

[RFC v2 05/13] x86/mm: Set KeyIDs in encrypted VMAs

[Alison Schofield]

MKTME architecture requires the KeyID to be placed in PTE bits 51:46.
To create an encrypted VMA, place the KeyID in the upper bits of
vm_page_prot that matches the position of those PTE bits.

When the VMA is assigned a KeyID it is always considered a KeyID
change. The VMA is either going from not encrypted to encrypted,
or from encrypted with any KeyID to encrypted with any other KeyID.
To make the change safely, remove the user pages held by the VMA
and unlink the VMA's anonymous chain.

Change-Id: I676056525c49c8803898315a10b196ef5a5c5415
Signed-off-by: Alison Schofield <alison.schofield@intel.com>
Signed-off-by: Kirill A. Shutemov <kirill.shutemov@linux.intel.com>
---
 arch/x86/include/asm/mktme.h |  4 ++++
 arch/x86/mm/mktme.c          | 26 ++++++++++++++++++++++++++
 include/linux/mm.h           |  6 ++++++
 3 files changed, 36 insertions(+)

diff --git a/arch/x86/include/asm/mktme.h b/arch/x86/include/asm/mktme.h
index dbb49909d665..de3e529f3ab0 100644
--- a/arch/x86/include/asm/mktme.h
+++ b/arch/x86/include/asm/mktme.h
@@ -24,6 +24,10 @@ extern int mktme_map_keyid_from_key(void *key);
 extern void *mktme_map_key_from_keyid(int keyid);
 extern int mktme_map_get_free_keyid(void);
 
+/* Set the encryption keyid bits in a VMA */
+extern void mprotect_set_encrypt(struct vm_area_struct *vma, int newkeyid,
+				unsigned long start, unsigned long end);
+
 DECLARE_STATIC_KEY_FALSE(mktme_enabled_key);
 static inline bool mktme_enabled(void)
 {
diff --git a/arch/x86/mm/mktme.c b/arch/x86/mm/mktme.c
index 34224d4e3f45..e3fdf7b48173 100644
--- a/arch/x86/mm/mktme.c
+++ b/arch/x86/mm/mktme.c
@@ -1,5 +1,6 @@
 #include <linux/mm.h>
 #include <linux/highmem.h>
+#include <linux/rmap.h>
 #include <asm/mktme.h>
 #include <asm/set_memory.h>
 
@@ -131,6 +132,31 @@ int mktme_map_get_free_keyid(void)
 	return 0;
 }
 
+/* Set the encryption keyid bits in a VMA */
+void mprotect_set_encrypt(struct vm_area_struct *vma, int newkeyid,
+			  unsigned long start, unsigned long end)
+{
+	int oldkeyid = vma_keyid(vma);
+	pgprotval_t newprot;
+
+	/* Unmap pages with old KeyID if there's any. */
+	zap_page_range(vma, start, end - start);
+
+	if (oldkeyid = newkeyid)
+		return;
+
+	newprot = pgprot_val(vma->vm_page_prot);
+	newprot &= ~mktme_keyid_mask;
+	newprot |= (unsigned long)newkeyid << mktme_keyid_shift;
+	vma->vm_page_prot = __pgprot(newprot);
+
+	/*
+	 * The VMA doesn't have any inherited pages.
+	 * Start anon VMA tree from scratch.
+	 */
+	unlink_anon_vmas(vma);
+}
+
 /* Prepare page to be used for encryption. Called from page allocator. */
 void __prep_encrypted_page(struct page *page, int order, int keyid, bool zero)
 {
diff --git a/include/linux/mm.h b/include/linux/mm.h
index 1309761bb6d0..e2d87e92ca74 100644
--- a/include/linux/mm.h
+++ b/include/linux/mm.h
@@ -2806,5 +2806,11 @@ void __init setup_nr_node_ids(void);
 static inline void setup_nr_node_ids(void) {}
 #endif
 
+#ifndef CONFIG_X86_INTEL_MKTME
+static inline void mprotect_set_encrypt(struct vm_area_struct *vma,
+					int newkeyid,
+					unsigned long start,
+					unsigned long end) {}
+#endif /* CONFIG_X86_INTEL_MKTME */
 #endif /* __KERNEL__ */
 #endif /* _LINUX_MM_H */
-- 
2.14.1

[RFC v2 05/13] x86/mm: Set KeyIDs in encrypted VMAs

[Alison Schofield]

MKTME architecture requires the KeyID to be placed in PTE bits 51:46.
To create an encrypted VMA, place the KeyID in the upper bits of
vm_page_prot that matches the position of those PTE bits.

When the VMA is assigned a KeyID it is always considered a KeyID
change. The VMA is either going from not encrypted to encrypted,
or from encrypted with any KeyID to encrypted with any other KeyID.
To make the change safely, remove the user pages held by the VMA
and unlink the VMA's anonymous chain.

Change-Id: I676056525c49c8803898315a10b196ef5a5c5415
Signed-off-by: Alison Schofield <alison.schofield@intel.com>
Signed-off-by: Kirill A. Shutemov <kirill.shutemov@linux.intel.com>
---
 arch/x86/include/asm/mktme.h |  4 ++++
 arch/x86/mm/mktme.c          | 26 ++++++++++++++++++++++++++
 include/linux/mm.h           |  6 ++++++
 3 files changed, 36 insertions(+)

diff --git a/arch/x86/include/asm/mktme.h b/arch/x86/include/asm/mktme.h
index dbb49909d665..de3e529f3ab0 100644
--- a/arch/x86/include/asm/mktme.h
+++ b/arch/x86/include/asm/mktme.h
@@ -24,6 +24,10 @@ extern int mktme_map_keyid_from_key(void *key);
 extern void *mktme_map_key_from_keyid(int keyid);
 extern int mktme_map_get_free_keyid(void);
 
+/* Set the encryption keyid bits in a VMA */
+extern void mprotect_set_encrypt(struct vm_area_struct *vma, int newkeyid,
+				unsigned long start, unsigned long end);
+
 DECLARE_STATIC_KEY_FALSE(mktme_enabled_key);
 static inline bool mktme_enabled(void)
 {
diff --git a/arch/x86/mm/mktme.c b/arch/x86/mm/mktme.c
index 34224d4e3f45..e3fdf7b48173 100644
--- a/arch/x86/mm/mktme.c
+++ b/arch/x86/mm/mktme.c
@@ -1,5 +1,6 @@
 #include <linux/mm.h>
 #include <linux/highmem.h>
+#include <linux/rmap.h>
 #include <asm/mktme.h>
 #include <asm/set_memory.h>
 
@@ -131,6 +132,31 @@ int mktme_map_get_free_keyid(void)
 	return 0;
 }
 
+/* Set the encryption keyid bits in a VMA */
+void mprotect_set_encrypt(struct vm_area_struct *vma, int newkeyid,
+			  unsigned long start, unsigned long end)
+{
+	int oldkeyid = vma_keyid(vma);
+	pgprotval_t newprot;
+
+	/* Unmap pages with old KeyID if there's any. */
+	zap_page_range(vma, start, end - start);
+
+	if (oldkeyid == newkeyid)
+		return;
+
+	newprot = pgprot_val(vma->vm_page_prot);
+	newprot &= ~mktme_keyid_mask;
+	newprot |= (unsigned long)newkeyid << mktme_keyid_shift;
+	vma->vm_page_prot = __pgprot(newprot);
+
+	/*
+	 * The VMA doesn't have any inherited pages.
+	 * Start anon VMA tree from scratch.
+	 */
+	unlink_anon_vmas(vma);
+}
+
 /* Prepare page to be used for encryption. Called from page allocator. */
 void __prep_encrypted_page(struct page *page, int order, int keyid, bool zero)
 {
diff --git a/include/linux/mm.h b/include/linux/mm.h
index 1309761bb6d0..e2d87e92ca74 100644
--- a/include/linux/mm.h
+++ b/include/linux/mm.h
@@ -2806,5 +2806,11 @@ void __init setup_nr_node_ids(void);
 static inline void setup_nr_node_ids(void) {}
 #endif
 
+#ifndef CONFIG_X86_INTEL_MKTME
+static inline void mprotect_set_encrypt(struct vm_area_struct *vma,
+					int newkeyid,
+					unsigned long start,
+					unsigned long end) {}
+#endif /* CONFIG_X86_INTEL_MKTME */
 #endif /* __KERNEL__ */
 #endif /* _LINUX_MM_H */
-- 
2.14.1

[RFC v2 06/13] mm: Add the encrypt_mprotect() system call

[Alison Schofield]

Implement memory encryption with a new system call that is an
extension of the legacy mprotect() system call.

In encrypt_mprotect the caller must pass a handle to a previously
allocated and programmed encryption key. Validate the key and store
the keyid bits in the vm_page_prot for each VMA in the protection
range.

Signed-off-by: Alison Schofield <alison.schofield@intel.com>
Signed-off-by: Kirill A. Shutemov <kirill.shutemov@linux.intel.com>
---
 fs/exec.c           |  4 ++--
 include/linux/key.h |  2 ++
 include/linux/mm.h  |  3 ++-
 mm/mprotect.c       | 63 +++++++++++++++++++++++++++++++++++++++++++++++------
 4 files changed, 62 insertions(+), 10 deletions(-)

diff --git a/fs/exec.c b/fs/exec.c
index fc281b738a98..a0946b23e2c5 100644
--- a/fs/exec.c
+++ b/fs/exec.c
@@ -752,8 +752,8 @@ int setup_arg_pages(struct linux_binprm *bprm,
 	vm_flags |= mm->def_flags;
 	vm_flags |= VM_STACK_INCOMPLETE_SETUP;
 
-	ret = mprotect_fixup(vma, &prev, vma->vm_start, vma->vm_end,
-			vm_flags);
+	ret = mprotect_fixup(vma, &prev, vma->vm_start, vma->vm_end, vm_flags,
+			     -1);
 	if (ret)
 		goto out_unlock;
 	BUG_ON(prev != vma);
diff --git a/include/linux/key.h b/include/linux/key.h
index e58ee10f6e58..fb8a7d5f6149 100644
--- a/include/linux/key.h
+++ b/include/linux/key.h
@@ -346,6 +346,8 @@ static inline key_serial_t key_serial(const struct key *key)
 
 extern void key_set_timeout(struct key *, unsigned);
 
+extern key_ref_t lookup_user_key(key_serial_t id, unsigned long lflags,
+				 key_perm_t perm);
 /*
  * The permissions required on a key that we're looking up.
  */
diff --git a/include/linux/mm.h b/include/linux/mm.h
index e2d87e92ca74..09182d78e7b7 100644
--- a/include/linux/mm.h
+++ b/include/linux/mm.h
@@ -1607,7 +1607,8 @@ extern unsigned long change_protection(struct vm_area_struct *vma, unsigned long
 			      int dirty_accountable, int prot_numa);
 extern int mprotect_fixup(struct vm_area_struct *vma,
 			  struct vm_area_struct **pprev, unsigned long start,
-			  unsigned long end, unsigned long newflags);
+			  unsigned long end, unsigned long newflags,
+			  int newkeyid);
 
 /*
  * doesn't attempt to fault and will return short.
diff --git a/mm/mprotect.c b/mm/mprotect.c
index b57075e278fb..ad8127dc9aac 100644
--- a/mm/mprotect.c
+++ b/mm/mprotect.c
@@ -28,6 +28,7 @@
 #include <linux/ksm.h>
 #include <linux/uaccess.h>
 #include <linux/mm_inline.h>
+#include <linux/key.h>
 #include <asm/pgtable.h>
 #include <asm/cacheflush.h>
 #include <asm/mmu_context.h>
@@ -346,7 +347,8 @@ static int prot_none_walk(struct vm_area_struct *vma, unsigned long start,
 
 int
 mprotect_fixup(struct vm_area_struct *vma, struct vm_area_struct **pprev,
-	unsigned long start, unsigned long end, unsigned long newflags)
+	       unsigned long start, unsigned long end, unsigned long newflags,
+	       int newkeyid)
 {
 	struct mm_struct *mm = vma->vm_mm;
 	unsigned long oldflags = vma->vm_flags;
@@ -356,7 +358,14 @@ mprotect_fixup(struct vm_area_struct *vma, struct vm_area_struct **pprev,
 	int error;
 	int dirty_accountable = 0;
 
-	if (newflags = oldflags) {
+	/*
+	 * Flags match and Keyids match or we have NO_KEY.
+	 * This _fixup is usually called from do_mprotect_ext() except
+	 * for one special case: caller fs/exec.c/setup_arg_pages()
+	 * In that case, newkeyid is passed as -1 (NO_KEY).
+	 */
+	if (newflags = oldflags &&
+	    (newkeyid = vma_keyid(vma) || newkeyid = NO_KEY)) {
 		*pprev = vma;
 		return 0;
 	}
@@ -422,6 +431,8 @@ mprotect_fixup(struct vm_area_struct *vma, struct vm_area_struct **pprev,
 	}
 
 success:
+	if (newkeyid != NO_KEY)
+		mprotect_set_encrypt(vma, newkeyid, start, end);
 	/*
 	 * vm_flags and vm_page_prot are protected by the mmap_sem
 	 * held in write mode.
@@ -453,10 +464,15 @@ mprotect_fixup(struct vm_area_struct *vma, struct vm_area_struct **pprev,
 }
 
 /*
- * When pkey=NO_KEY we get legacy mprotect behavior here.
+ * do_mprotect_ext() supports the legacy mprotect behavior plus extensions
+ * for Protection Keys and Memory Encryption Keys. These extensions are
+ * mutually exclusive and the behavior is:
+ *	(pkey=NO_KEY && keyid=NO_KEY) => legacy mprotect
+ *	(pkey is valid)  => legacy mprotect plus Protection Key extensions
+ *	(keyid is valid) => legacy mprotect plus Encryption Key extensions
  */
 static int do_mprotect_ext(unsigned long start, size_t len,
-		unsigned long prot, int pkey)
+			   unsigned long prot, int pkey, int keyid)
 {
 	unsigned long nstart, end, tmp, reqprot;
 	struct vm_area_struct *vma, *prev;
@@ -554,7 +570,8 @@ static int do_mprotect_ext(unsigned long start, size_t len,
 		tmp = vma->vm_end;
 		if (tmp > end)
 			tmp = end;
-		error = mprotect_fixup(vma, &prev, nstart, tmp, newflags);
+		error = mprotect_fixup(vma, &prev, nstart, tmp, newflags,
+				       keyid);
 		if (error)
 			goto out;
 		nstart = tmp;
@@ -579,7 +596,7 @@ static int do_mprotect_ext(unsigned long start, size_t len,
 SYSCALL_DEFINE3(mprotect, unsigned long, start, size_t, len,
 		unsigned long, prot)
 {
-	return do_mprotect_ext(start, len, prot, NO_KEY);
+	return do_mprotect_ext(start, len, prot, NO_KEY, NO_KEY);
 }
 
 #ifdef CONFIG_ARCH_HAS_PKEYS
@@ -587,7 +604,7 @@ SYSCALL_DEFINE3(mprotect, unsigned long, start, size_t, len,
 SYSCALL_DEFINE4(pkey_mprotect, unsigned long, start, size_t, len,
 		unsigned long, prot, int, pkey)
 {
-	return do_mprotect_ext(start, len, prot, pkey);
+	return do_mprotect_ext(start, len, prot, pkey, NO_KEY);
 }
 
 SYSCALL_DEFINE2(pkey_alloc, unsigned long, flags, unsigned long, init_val)
@@ -636,3 +653,35 @@ SYSCALL_DEFINE1(pkey_free, int, pkey)
 }
 
 #endif /* CONFIG_ARCH_HAS_PKEYS */
+
+#ifdef CONFIG_X86_INTEL_MKTME
+
+SYSCALL_DEFINE4(encrypt_mprotect, unsigned long, start, size_t, len,
+		unsigned long, prot, key_serial_t, serial)
+{
+	key_ref_t key_ref;
+	struct key *key;
+	int ret, keyid;
+
+	if (!PAGE_ALIGNED(len))
+		return -EINVAL;
+
+	key_ref = lookup_user_key(serial, 0, KEY_NEED_VIEW);
+	if (IS_ERR(key_ref))
+		return PTR_ERR(key_ref);
+
+	key = key_ref_to_ptr(key_ref);
+	mktme_map_lock();
+	keyid = mktme_map_keyid_from_key(key);
+	if (!keyid) {
+		mktme_map_unlock();
+		key_ref_put(key_ref);
+		return -EINVAL;
+	}
+	ret = do_mprotect_ext(start, len, prot, NO_KEY, keyid);
+	mktme_map_unlock();
+	key_ref_put(key_ref);
+	return ret;
+}
+
+#endif /* CONFIG_X86_INTEL_MKTME */
-- 
2.14.1

[RFC v2 06/13] mm: Add the encrypt_mprotect() system call

[Alison Schofield]

Implement memory encryption with a new system call that is an
extension of the legacy mprotect() system call.

In encrypt_mprotect the caller must pass a handle to a previously
allocated and programmed encryption key. Validate the key and store
the keyid bits in the vm_page_prot for each VMA in the protection
range.

Signed-off-by: Alison Schofield <alison.schofield@intel.com>
Signed-off-by: Kirill A. Shutemov <kirill.shutemov@linux.intel.com>
---
 fs/exec.c           |  4 ++--
 include/linux/key.h |  2 ++
 include/linux/mm.h  |  3 ++-
 mm/mprotect.c       | 63 +++++++++++++++++++++++++++++++++++++++++++++++------
 4 files changed, 62 insertions(+), 10 deletions(-)

diff --git a/fs/exec.c b/fs/exec.c
index fc281b738a98..a0946b23e2c5 100644
--- a/fs/exec.c
+++ b/fs/exec.c
@@ -752,8 +752,8 @@ int setup_arg_pages(struct linux_binprm *bprm,
 	vm_flags |= mm->def_flags;
 	vm_flags |= VM_STACK_INCOMPLETE_SETUP;
 
-	ret = mprotect_fixup(vma, &prev, vma->vm_start, vma->vm_end,
-			vm_flags);
+	ret = mprotect_fixup(vma, &prev, vma->vm_start, vma->vm_end, vm_flags,
+			     -1);
 	if (ret)
 		goto out_unlock;
 	BUG_ON(prev != vma);
diff --git a/include/linux/key.h b/include/linux/key.h
index e58ee10f6e58..fb8a7d5f6149 100644
--- a/include/linux/key.h
+++ b/include/linux/key.h
@@ -346,6 +346,8 @@ static inline key_serial_t key_serial(const struct key *key)
 
 extern void key_set_timeout(struct key *, unsigned);
 
+extern key_ref_t lookup_user_key(key_serial_t id, unsigned long lflags,
+				 key_perm_t perm);
 /*
  * The permissions required on a key that we're looking up.
  */
diff --git a/include/linux/mm.h b/include/linux/mm.h
index e2d87e92ca74..09182d78e7b7 100644
--- a/include/linux/mm.h
+++ b/include/linux/mm.h
@@ -1607,7 +1607,8 @@ extern unsigned long change_protection(struct vm_area_struct *vma, unsigned long
 			      int dirty_accountable, int prot_numa);
 extern int mprotect_fixup(struct vm_area_struct *vma,
 			  struct vm_area_struct **pprev, unsigned long start,
-			  unsigned long end, unsigned long newflags);
+			  unsigned long end, unsigned long newflags,
+			  int newkeyid);
 
 /*
  * doesn't attempt to fault and will return short.
diff --git a/mm/mprotect.c b/mm/mprotect.c
index b57075e278fb..ad8127dc9aac 100644
--- a/mm/mprotect.c
+++ b/mm/mprotect.c
@@ -28,6 +28,7 @@
 #include <linux/ksm.h>
 #include <linux/uaccess.h>
 #include <linux/mm_inline.h>
+#include <linux/key.h>
 #include <asm/pgtable.h>
 #include <asm/cacheflush.h>
 #include <asm/mmu_context.h>
@@ -346,7 +347,8 @@ static int prot_none_walk(struct vm_area_struct *vma, unsigned long start,
 
 int
 mprotect_fixup(struct vm_area_struct *vma, struct vm_area_struct **pprev,
-	unsigned long start, unsigned long end, unsigned long newflags)
+	       unsigned long start, unsigned long end, unsigned long newflags,
+	       int newkeyid)
 {
 	struct mm_struct *mm = vma->vm_mm;
 	unsigned long oldflags = vma->vm_flags;
@@ -356,7 +358,14 @@ mprotect_fixup(struct vm_area_struct *vma, struct vm_area_struct **pprev,
 	int error;
 	int dirty_accountable = 0;
 
-	if (newflags == oldflags) {
+	/*
+	 * Flags match and Keyids match or we have NO_KEY.
+	 * This _fixup is usually called from do_mprotect_ext() except
+	 * for one special case: caller fs/exec.c/setup_arg_pages()
+	 * In that case, newkeyid is passed as -1 (NO_KEY).
+	 */
+	if (newflags == oldflags &&
+	    (newkeyid == vma_keyid(vma) || newkeyid == NO_KEY)) {
 		*pprev = vma;
 		return 0;
 	}
@@ -422,6 +431,8 @@ mprotect_fixup(struct vm_area_struct *vma, struct vm_area_struct **pprev,
 	}
 
 success:
+	if (newkeyid != NO_KEY)
+		mprotect_set_encrypt(vma, newkeyid, start, end);
 	/*
 	 * vm_flags and vm_page_prot are protected by the mmap_sem
 	 * held in write mode.
@@ -453,10 +464,15 @@ mprotect_fixup(struct vm_area_struct *vma, struct vm_area_struct **pprev,
 }
 
 /*
- * When pkey==NO_KEY we get legacy mprotect behavior here.
+ * do_mprotect_ext() supports the legacy mprotect behavior plus extensions
+ * for Protection Keys and Memory Encryption Keys. These extensions are
+ * mutually exclusive and the behavior is:
+ *	(pkey==NO_KEY && keyid==NO_KEY) ==> legacy mprotect
+ *	(pkey is valid)  ==> legacy mprotect plus Protection Key extensions
+ *	(keyid is valid) ==> legacy mprotect plus Encryption Key extensions
  */
 static int do_mprotect_ext(unsigned long start, size_t len,
-		unsigned long prot, int pkey)
+			   unsigned long prot, int pkey, int keyid)
 {
 	unsigned long nstart, end, tmp, reqprot;
 	struct vm_area_struct *vma, *prev;
@@ -554,7 +570,8 @@ static int do_mprotect_ext(unsigned long start, size_t len,
 		tmp = vma->vm_end;
 		if (tmp > end)
 			tmp = end;
-		error = mprotect_fixup(vma, &prev, nstart, tmp, newflags);
+		error = mprotect_fixup(vma, &prev, nstart, tmp, newflags,
+				       keyid);
 		if (error)
 			goto out;
 		nstart = tmp;
@@ -579,7 +596,7 @@ static int do_mprotect_ext(unsigned long start, size_t len,
 SYSCALL_DEFINE3(mprotect, unsigned long, start, size_t, len,
 		unsigned long, prot)
 {
-	return do_mprotect_ext(start, len, prot, NO_KEY);
+	return do_mprotect_ext(start, len, prot, NO_KEY, NO_KEY);
 }
 
 #ifdef CONFIG_ARCH_HAS_PKEYS
@@ -587,7 +604,7 @@ SYSCALL_DEFINE3(mprotect, unsigned long, start, size_t, len,
 SYSCALL_DEFINE4(pkey_mprotect, unsigned long, start, size_t, len,
 		unsigned long, prot, int, pkey)
 {
-	return do_mprotect_ext(start, len, prot, pkey);
+	return do_mprotect_ext(start, len, prot, pkey, NO_KEY);
 }
 
 SYSCALL_DEFINE2(pkey_alloc, unsigned long, flags, unsigned long, init_val)
@@ -636,3 +653,35 @@ SYSCALL_DEFINE1(pkey_free, int, pkey)
 }
 
 #endif /* CONFIG_ARCH_HAS_PKEYS */
+
+#ifdef CONFIG_X86_INTEL_MKTME
+
+SYSCALL_DEFINE4(encrypt_mprotect, unsigned long, start, size_t, len,
+		unsigned long, prot, key_serial_t, serial)
+{
+	key_ref_t key_ref;
+	struct key *key;
+	int ret, keyid;
+
+	if (!PAGE_ALIGNED(len))
+		return -EINVAL;
+
+	key_ref = lookup_user_key(serial, 0, KEY_NEED_VIEW);
+	if (IS_ERR(key_ref))
+		return PTR_ERR(key_ref);
+
+	key = key_ref_to_ptr(key_ref);
+	mktme_map_lock();
+	keyid = mktme_map_keyid_from_key(key);
+	if (!keyid) {
+		mktme_map_unlock();
+		key_ref_put(key_ref);
+		return -EINVAL;
+	}
+	ret = do_mprotect_ext(start, len, prot, NO_KEY, keyid);
+	mktme_map_unlock();
+	key_ref_put(key_ref);
+	return ret;
+}
+
+#endif /* CONFIG_X86_INTEL_MKTME */
-- 
2.14.1

[RFC v2 07/13] x86/mm: Add helpers for reference counting encrypted VMAs

[Alison Schofield]

In order to safely manage the usage of memory encryption keys, VMAs
using each KeyID need to be counted. This count allows the MKTME
(Multi-Key Total Memory Encryption) Key Service to know when the KeyID
resource is actually in use, or when it is idle and may be considered
for reuse.

Define a global refcount_t array and provide helper functions to
manipulate the counts.

Signed-off-by: Alison Schofield <alison.schofield@intel.com>
Signed-off-by: Kirill A. Shutemov <kirill.shutemov@linux.intel.com>
---
 arch/x86/include/asm/mktme.h |  9 +++++++
 arch/x86/mm/mktme.c          | 58 ++++++++++++++++++++++++++++++++++++++++++++
 include/linux/mm.h           |  2 ++
 3 files changed, 69 insertions(+)

diff --git a/arch/x86/include/asm/mktme.h b/arch/x86/include/asm/mktme.h
index de3e529f3ab0..22d52635562c 100644
--- a/arch/x86/include/asm/mktme.h
+++ b/arch/x86/include/asm/mktme.h
@@ -28,6 +28,15 @@ extern int mktme_map_get_free_keyid(void);
 extern void mprotect_set_encrypt(struct vm_area_struct *vma, int newkeyid,
 				unsigned long start, unsigned long end);
 
+/* Manage the MTKME encrypt_count references */
+extern int mktme_alloc_encrypt_array(void);
+extern void mktme_free_encrypt_array(void);
+extern int mktme_read_encrypt_ref(int keyid);
+extern void vma_get_encrypt_ref(struct vm_area_struct *vma);
+extern void vma_put_encrypt_ref(struct vm_area_struct *vma);
+extern void key_get_encrypt_ref(int keyid);
+extern void key_put_encrypt_ref(int keyid);
+
 DECLARE_STATIC_KEY_FALSE(mktme_enabled_key);
 static inline bool mktme_enabled(void)
 {
diff --git a/arch/x86/mm/mktme.c b/arch/x86/mm/mktme.c
index e3fdf7b48173..facf08f9cb74 100644
--- a/arch/x86/mm/mktme.c
+++ b/arch/x86/mm/mktme.c
@@ -157,6 +157,64 @@ void mprotect_set_encrypt(struct vm_area_struct *vma, int newkeyid,
 	unlink_anon_vmas(vma);
 }
 
+/*
+ *  Helper functions manage the encrypt_count[] array that counts
+ *  references on each MKTME hardware keyid. The gets & puts are
+ *  used in core mm code that allocates and free's VMA's. The alloc,
+ *  free, and read functions are used by the MKTME key service to
+ *  manage key allocation and programming.
+ */
+refcount_t *encrypt_count;
+
+int mktme_alloc_encrypt_array(void)
+{
+	encrypt_count = kvcalloc(mktme_nr_keyids, sizeof(refcount_t),
+				 GFP_KERNEL);
+	if (!encrypt_count)
+		return -ENOMEM;
+	return 0;
+}
+
+void mktme_free_encrypt_array(void)
+{
+	kvfree(encrypt_count);
+}
+
+int mktme_read_encrypt_ref(int keyid)
+{
+	return refcount_read(&encrypt_count[keyid]);
+}
+
+void vma_get_encrypt_ref(struct vm_area_struct *vma)
+{
+	if (vma_keyid(vma))
+		refcount_inc(&encrypt_count[vma_keyid(vma)]);
+}
+
+void vma_put_encrypt_ref(struct vm_area_struct *vma)
+{
+	if (vma_keyid(vma))
+		if (refcount_dec_and_test(&encrypt_count[vma_keyid(vma)])) {
+			mktme_map_lock();
+			mktme_map_free_keyid(vma_keyid(vma));
+			mktme_map_unlock();
+		}
+}
+
+void key_get_encrypt_ref(int keyid)
+{
+	refcount_inc(&encrypt_count[keyid]);
+}
+
+void key_put_encrypt_ref(int keyid)
+{
+	if (refcount_dec_and_test(&encrypt_count[keyid])) {
+		mktme_map_lock();
+		mktme_map_free_keyid(keyid);
+		mktme_map_unlock();
+	}
+}
+
 /* Prepare page to be used for encryption. Called from page allocator. */
 void __prep_encrypted_page(struct page *page, int order, int keyid, bool zero)
 {
diff --git a/include/linux/mm.h b/include/linux/mm.h
index 09182d78e7b7..453d675dd116 100644
--- a/include/linux/mm.h
+++ b/include/linux/mm.h
@@ -2812,6 +2812,8 @@ static inline void mprotect_set_encrypt(struct vm_area_struct *vma,
 					int newkeyid,
 					unsigned long start,
 					unsigned long end) {}
+static inline void vma_get_encrypt_ref(struct vm_area_struct *vma) {}
+static inline void vma_put_encrypt_ref(struct vm_area_struct *vma) {}
 #endif /* CONFIG_X86_INTEL_MKTME */
 #endif /* __KERNEL__ */
 #endif /* _LINUX_MM_H */
-- 
2.14.1

[RFC v2 07/13] x86/mm: Add helpers for reference counting encrypted VMAs

[Alison Schofield]

In order to safely manage the usage of memory encryption keys, VMAs
using each KeyID need to be counted. This count allows the MKTME
(Multi-Key Total Memory Encryption) Key Service to know when the KeyID
resource is actually in use, or when it is idle and may be considered
for reuse.

Define a global refcount_t array and provide helper functions to
manipulate the counts.

Signed-off-by: Alison Schofield <alison.schofield@intel.com>
Signed-off-by: Kirill A. Shutemov <kirill.shutemov@linux.intel.com>
---
 arch/x86/include/asm/mktme.h |  9 +++++++
 arch/x86/mm/mktme.c          | 58 ++++++++++++++++++++++++++++++++++++++++++++
 include/linux/mm.h           |  2 ++
 3 files changed, 69 insertions(+)

diff --git a/arch/x86/include/asm/mktme.h b/arch/x86/include/asm/mktme.h
index de3e529f3ab0..22d52635562c 100644
--- a/arch/x86/include/asm/mktme.h
+++ b/arch/x86/include/asm/mktme.h
@@ -28,6 +28,15 @@ extern int mktme_map_get_free_keyid(void);
 extern void mprotect_set_encrypt(struct vm_area_struct *vma, int newkeyid,
 				unsigned long start, unsigned long end);
 
+/* Manage the MTKME encrypt_count references */
+extern int mktme_alloc_encrypt_array(void);
+extern void mktme_free_encrypt_array(void);
+extern int mktme_read_encrypt_ref(int keyid);
+extern void vma_get_encrypt_ref(struct vm_area_struct *vma);
+extern void vma_put_encrypt_ref(struct vm_area_struct *vma);
+extern void key_get_encrypt_ref(int keyid);
+extern void key_put_encrypt_ref(int keyid);
+
 DECLARE_STATIC_KEY_FALSE(mktme_enabled_key);
 static inline bool mktme_enabled(void)
 {
diff --git a/arch/x86/mm/mktme.c b/arch/x86/mm/mktme.c
index e3fdf7b48173..facf08f9cb74 100644
--- a/arch/x86/mm/mktme.c
+++ b/arch/x86/mm/mktme.c
@@ -157,6 +157,64 @@ void mprotect_set_encrypt(struct vm_area_struct *vma, int newkeyid,
 	unlink_anon_vmas(vma);
 }
 
+/*
+ *  Helper functions manage the encrypt_count[] array that counts
+ *  references on each MKTME hardware keyid. The gets & puts are
+ *  used in core mm code that allocates and free's VMA's. The alloc,
+ *  free, and read functions are used by the MKTME key service to
+ *  manage key allocation and programming.
+ */
+refcount_t *encrypt_count;
+
+int mktme_alloc_encrypt_array(void)
+{
+	encrypt_count = kvcalloc(mktme_nr_keyids, sizeof(refcount_t),
+				 GFP_KERNEL);
+	if (!encrypt_count)
+		return -ENOMEM;
+	return 0;
+}
+
+void mktme_free_encrypt_array(void)
+{
+	kvfree(encrypt_count);
+}
+
+int mktme_read_encrypt_ref(int keyid)
+{
+	return refcount_read(&encrypt_count[keyid]);
+}
+
+void vma_get_encrypt_ref(struct vm_area_struct *vma)
+{
+	if (vma_keyid(vma))
+		refcount_inc(&encrypt_count[vma_keyid(vma)]);
+}
+
+void vma_put_encrypt_ref(struct vm_area_struct *vma)
+{
+	if (vma_keyid(vma))
+		if (refcount_dec_and_test(&encrypt_count[vma_keyid(vma)])) {
+			mktme_map_lock();
+			mktme_map_free_keyid(vma_keyid(vma));
+			mktme_map_unlock();
+		}
+}
+
+void key_get_encrypt_ref(int keyid)
+{
+	refcount_inc(&encrypt_count[keyid]);
+}
+
+void key_put_encrypt_ref(int keyid)
+{
+	if (refcount_dec_and_test(&encrypt_count[keyid])) {
+		mktme_map_lock();
+		mktme_map_free_keyid(keyid);
+		mktme_map_unlock();
+	}
+}
+
 /* Prepare page to be used for encryption. Called from page allocator. */
 void __prep_encrypted_page(struct page *page, int order, int keyid, bool zero)
 {
diff --git a/include/linux/mm.h b/include/linux/mm.h
index 09182d78e7b7..453d675dd116 100644
--- a/include/linux/mm.h
+++ b/include/linux/mm.h
@@ -2812,6 +2812,8 @@ static inline void mprotect_set_encrypt(struct vm_area_struct *vma,
 					int newkeyid,
 					unsigned long start,
 					unsigned long end) {}
+static inline void vma_get_encrypt_ref(struct vm_area_struct *vma) {}
+static inline void vma_put_encrypt_ref(struct vm_area_struct *vma) {}
 #endif /* CONFIG_X86_INTEL_MKTME */
 #endif /* __KERNEL__ */
 #endif /* _LINUX_MM_H */
-- 
2.14.1

[RFC v2 08/13] mm: Use reference counting for encrypted VMAs

[Alison Schofield]

The MKTME (Multi-Key Total Memory Encryption) Key Service needs
a reference count on encrypted VMAs. This reference count is used
to determine when a hardware encryption keyid is in use, which in
turn, tells the key service what operations can be safely performed
with this keyid.

The approach is:
1) Increment/decrement the reference count during encrypt_mprotect()
system call for initial or updated encryption on a VMA.

2) Piggy back on the new vm_area_dup/free() helpers. If the VMAs being
duplicated, or freed are encrypted, adjust the reference count.

Signed-off-by: Alison Schofield <alison.schofield@intel.com>
Signed-off-by: Kirill A. Shutemov <kirill.shutemov@linux.intel.com>
---
 arch/x86/mm/mktme.c | 2 ++
 kernel/fork.c       | 2 ++
 2 files changed, 4 insertions(+)

diff --git a/arch/x86/mm/mktme.c b/arch/x86/mm/mktme.c
index facf08f9cb74..55d34beb9b81 100644
--- a/arch/x86/mm/mktme.c
+++ b/arch/x86/mm/mktme.c
@@ -145,10 +145,12 @@ void mprotect_set_encrypt(struct vm_area_struct *vma, int newkeyid,
 	if (oldkeyid = newkeyid)
 		return;
 
+	vma_put_encrypt_ref(vma);
 	newprot = pgprot_val(vma->vm_page_prot);
 	newprot &= ~mktme_keyid_mask;
 	newprot |= (unsigned long)newkeyid << mktme_keyid_shift;
 	vma->vm_page_prot = __pgprot(newprot);
+	vma_get_encrypt_ref(vma);
 
 	/*
 	 * The VMA doesn't have any inherited pages.
diff --git a/kernel/fork.c b/kernel/fork.c
index 07cddff89c7b..d12d27b50966 100644
--- a/kernel/fork.c
+++ b/kernel/fork.c
@@ -341,12 +341,14 @@ struct vm_area_struct *vm_area_dup(struct vm_area_struct *orig)
 	if (new) {
 		*new = *orig;
 		INIT_LIST_HEAD(&new->anon_vma_chain);
+		vma_get_encrypt_ref(new);
 	}
 	return new;
 }
 
 void vm_area_free(struct vm_area_struct *vma)
 {
+	vma_put_encrypt_ref(vma);
 	kmem_cache_free(vm_area_cachep, vma);
 }
 
-- 
2.14.1

[RFC v2 08/13] mm: Use reference counting for encrypted VMAs

[Alison Schofield]

The MKTME (Multi-Key Total Memory Encryption) Key Service needs
a reference count on encrypted VMAs. This reference count is used
to determine when a hardware encryption keyid is in use, which in
turn, tells the key service what operations can be safely performed
with this keyid.

The approach is:
1) Increment/decrement the reference count during encrypt_mprotect()
system call for initial or updated encryption on a VMA.

2) Piggy back on the new vm_area_dup/free() helpers. If the VMAs being
duplicated, or freed are encrypted, adjust the reference count.

Signed-off-by: Alison Schofield <alison.schofield@intel.com>
Signed-off-by: Kirill A. Shutemov <kirill.shutemov@linux.intel.com>
---
 arch/x86/mm/mktme.c | 2 ++
 kernel/fork.c       | 2 ++
 2 files changed, 4 insertions(+)

diff --git a/arch/x86/mm/mktme.c b/arch/x86/mm/mktme.c
index facf08f9cb74..55d34beb9b81 100644
--- a/arch/x86/mm/mktme.c
+++ b/arch/x86/mm/mktme.c
@@ -145,10 +145,12 @@ void mprotect_set_encrypt(struct vm_area_struct *vma, int newkeyid,
 	if (oldkeyid == newkeyid)
 		return;
 
+	vma_put_encrypt_ref(vma);
 	newprot = pgprot_val(vma->vm_page_prot);
 	newprot &= ~mktme_keyid_mask;
 	newprot |= (unsigned long)newkeyid << mktme_keyid_shift;
 	vma->vm_page_prot = __pgprot(newprot);
+	vma_get_encrypt_ref(vma);
 
 	/*
 	 * The VMA doesn't have any inherited pages.
diff --git a/kernel/fork.c b/kernel/fork.c
index 07cddff89c7b..d12d27b50966 100644
--- a/kernel/fork.c
+++ b/kernel/fork.c
@@ -341,12 +341,14 @@ struct vm_area_struct *vm_area_dup(struct vm_area_struct *orig)
 	if (new) {
 		*new = *orig;
 		INIT_LIST_HEAD(&new->anon_vma_chain);
+		vma_get_encrypt_ref(new);
 	}
 	return new;
 }
 
 void vm_area_free(struct vm_area_struct *vma)
 {
+	vma_put_encrypt_ref(vma);
 	kmem_cache_free(vm_area_cachep, vma);
 }
 
-- 
2.14.1

[RFC v2 09/13] mm: Restrict memory encryption to anonymous VMA's

[Alison Schofield]

Memory encryption is only supported for mappings that are ANONYMOUS.
Test the entire range of VMA's in an encrypt_mprotect() request to
make sure they all meet that requirement before encrypting any.

The encrypt_mprotect syscall will return -EINVAL and will not encrypt
any VMA's if this check fails.

Signed-off-by: Alison Schofield <alison.schofield@intel.com>
Signed-off-by: Kirill A. Shutemov <kirill.shutemov@linux.intel.com>
---
 mm/mprotect.c | 24 ++++++++++++++++++++++++
 1 file changed, 24 insertions(+)

diff --git a/mm/mprotect.c b/mm/mprotect.c
index ad8127dc9aac..f1c009409134 100644
--- a/mm/mprotect.c
+++ b/mm/mprotect.c
@@ -345,6 +345,24 @@ static int prot_none_walk(struct vm_area_struct *vma, unsigned long start,
 	return walk_page_range(start, end, &prot_none_walk);
 }
 
+/*
+ * Encrypted mprotect is only supported on anonymous mappings.
+ * All VMA's in the requested range must be anonymous. If this
+ * test fails on any single VMA, the entire mprotect request fails.
+ */
+bool mem_supports_encryption(struct vm_area_struct *vma, unsigned long end)
+{
+	struct vm_area_struct *test_vma = vma;
+
+	do {
+		if (!vma_is_anonymous(test_vma))
+			return false;
+
+		test_vma = test_vma->vm_next;
+	} while (test_vma && test_vma->vm_start < end);
+	return true;
+}
+
 int
 mprotect_fixup(struct vm_area_struct *vma, struct vm_area_struct **pprev,
 	       unsigned long start, unsigned long end, unsigned long newflags,
@@ -531,6 +549,12 @@ static int do_mprotect_ext(unsigned long start, size_t len,
 				goto out;
 		}
 	}
+
+	if (keyid > 0 && !mem_supports_encryption(vma, end)) {
+		error = -EINVAL;
+		goto out;
+	}
+
 	if (start > vma->vm_start)
 		prev = vma;
 
-- 
2.14.1

[RFC v2 09/13] mm: Restrict memory encryption to anonymous VMA's

[Alison Schofield]

Memory encryption is only supported for mappings that are ANONYMOUS.
Test the entire range of VMA's in an encrypt_mprotect() request to
make sure they all meet that requirement before encrypting any.

The encrypt_mprotect syscall will return -EINVAL and will not encrypt
any VMA's if this check fails.

Signed-off-by: Alison Schofield <alison.schofield@intel.com>
Signed-off-by: Kirill A. Shutemov <kirill.shutemov@linux.intel.com>
---
 mm/mprotect.c | 24 ++++++++++++++++++++++++
 1 file changed, 24 insertions(+)

diff --git a/mm/mprotect.c b/mm/mprotect.c
index ad8127dc9aac..f1c009409134 100644
--- a/mm/mprotect.c
+++ b/mm/mprotect.c
@@ -345,6 +345,24 @@ static int prot_none_walk(struct vm_area_struct *vma, unsigned long start,
 	return walk_page_range(start, end, &prot_none_walk);
 }
 
+/*
+ * Encrypted mprotect is only supported on anonymous mappings.
+ * All VMA's in the requested range must be anonymous. If this
+ * test fails on any single VMA, the entire mprotect request fails.
+ */
+bool mem_supports_encryption(struct vm_area_struct *vma, unsigned long end)
+{
+	struct vm_area_struct *test_vma = vma;
+
+	do {
+		if (!vma_is_anonymous(test_vma))
+			return false;
+
+		test_vma = test_vma->vm_next;
+	} while (test_vma && test_vma->vm_start < end);
+	return true;
+}
+
 int
 mprotect_fixup(struct vm_area_struct *vma, struct vm_area_struct **pprev,
 	       unsigned long start, unsigned long end, unsigned long newflags,
@@ -531,6 +549,12 @@ static int do_mprotect_ext(unsigned long start, size_t len,
 				goto out;
 		}
 	}
+
+	if (keyid > 0 && !mem_supports_encryption(vma, end)) {
+		error = -EINVAL;
+		goto out;
+	}
+
 	if (start > vma->vm_start)
 		prev = vma;
 
-- 
2.14.1

[RFC v2 10/13] keys/mktme: Add the MKTME Key Service type for memory encryption

[Alison Schofield]

MKTME (Multi-Key Total Memory Encryption) is a technology that allows
transparent memory encryption in upcoming Intel platforms. MKTME will
support mulitple encryption domains, each having their own key. The main
use case for the feature is virtual machine isolation. The API needs the
flexibility to work for a wide range of uses.

The MKTME key service type manages the addition and removal of the memory
encryption keys. It maps Userspace Keys to hardware KeyIDs. It programs
the hardware with the user requested encryption options.

The only supported encryption algorithm is AES-XTS 128.

The MKTME key service is half of the MKTME API level solution. It pairs
with a new memory encryption system call: encrypt_mprotect() that uses
the keys to encrypt memory.

See Documentation/x86/mktme/mktme.rst

Change-Id: Idda4af2beabb739c77719897affff183ee9fa716
Signed-off-by: Alison Schofield <alison.schofield@intel.com>
Signed-off-by: Kirill A. Shutemov <kirill.shutemov@linux.intel.com>
---
 arch/x86/Kconfig           |   1 +
 include/keys/mktme-type.h  |  41 ++++++
 security/keys/Kconfig      |  11 ++
 security/keys/Makefile     |   1 +
 security/keys/mktme_keys.c | 339 +++++++++++++++++++++++++++++++++++++++++++++
 5 files changed, 393 insertions(+)
 create mode 100644 include/keys/mktme-type.h
 create mode 100644 security/keys/mktme_keys.c

diff --git a/arch/x86/Kconfig b/arch/x86/Kconfig
index 7ac78e2856c7..c2e3bb5af077 100644
--- a/arch/x86/Kconfig
+++ b/arch/x86/Kconfig
@@ -1531,6 +1531,7 @@ config X86_INTEL_MKTME
 	bool "Intel Multi-Key Total Memory Encryption"
 	select DYNAMIC_PHYSICAL_MASK
 	select PAGE_EXTENSION
+	select MKTME_KEYS
 	depends on X86_64 && CPU_SUP_INTEL
 	---help---
 	  Say yes to enable support for Multi-Key Total Memory Encryption.
diff --git a/include/keys/mktme-type.h b/include/keys/mktme-type.h
new file mode 100644
index 000000000000..c63c6568087f
--- /dev/null
+++ b/include/keys/mktme-type.h
@@ -0,0 +1,41 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+
+/* Key service for Multi-KEY Total Memory Encryption */
+
+#ifndef _KEYS_MKTME_TYPE_H
+#define _KEYS_MKTME_TYPE_H
+
+#include <linux/key.h>
+
+/*
+ * The AES-XTS 128 encryption algorithm requires 128 bits for each
+ * user supplied data key and tweak key.
+ */
+#define MKTME_AES_XTS_SIZE	16	/* 16 bytes, 128 bits */
+
+enum mktme_alg {
+	MKTME_ALG_AES_XTS_128,
+};
+
+const char *const mktme_alg_names[] = {
+	[MKTME_ALG_AES_XTS_128]	= "aes-xts-128",
+};
+
+enum mktme_type {
+	MKTME_TYPE_ERROR = -1,
+	MKTME_TYPE_USER,
+	MKTME_TYPE_CPU,
+	MKTME_TYPE_CLEAR,
+	MKTME_TYPE_NO_ENCRYPT,
+};
+
+const char *const mktme_type_names[] = {
+	[MKTME_TYPE_USER]	= "user",
+	[MKTME_TYPE_CPU]	= "cpu",
+	[MKTME_TYPE_CLEAR]	= "clear",
+	[MKTME_TYPE_NO_ENCRYPT]	= "no-encrypt",
+};
+
+extern struct key_type key_type_mktme;
+
+#endif /* _KEYS_MKTME_TYPE_H */
diff --git a/security/keys/Kconfig b/security/keys/Kconfig
index 6462e6654ccf..c36972113e67 100644
--- a/security/keys/Kconfig
+++ b/security/keys/Kconfig
@@ -101,3 +101,14 @@ config KEY_DH_OPERATIONS
 	 in the kernel.
 
 	 If you are unsure as to whether this is required, answer N.
+
+config MKTME_KEYS
+	bool "Multi-Key Total Memory Encryption Keys"
+	depends on KEYS && X86_INTEL_MKTME
+	help
+	  This option provides support for Multi-Key Total Memory
+	  Encryption (MKTME) on Intel platforms offering the feature.
+	  MKTME allows userspace to manage the hardware encryption
+	  keys through the kernel key services.
+
+	  If you are unsure as to whether this is required, answer N.
diff --git a/security/keys/Makefile b/security/keys/Makefile
index 9cef54064f60..94c84f10a857 100644
--- a/security/keys/Makefile
+++ b/security/keys/Makefile
@@ -30,3 +30,4 @@ obj-$(CONFIG_ASYMMETRIC_KEY_TYPE) += keyctl_pkey.o
 obj-$(CONFIG_BIG_KEYS) += big_key.o
 obj-$(CONFIG_TRUSTED_KEYS) += trusted.o
 obj-$(CONFIG_ENCRYPTED_KEYS) += encrypted-keys/
+obj-$(CONFIG_MKTME_KEYS) += mktme_keys.o
diff --git a/security/keys/mktme_keys.c b/security/keys/mktme_keys.c
new file mode 100644
index 000000000000..e615eb58e600
--- /dev/null
+++ b/security/keys/mktme_keys.c
@@ -0,0 +1,339 @@
+// SPDX-License-Identifier: GPL-3.0
+
+/* Documentation/x86/mktme/mktme_keys.rst */
+
+#include <linux/cred.h>
+#include <linux/cpu.h>
+#include <linux/err.h>
+#include <linux/init.h>
+#include <linux/key.h>
+#include <linux/key-type.h>
+#include <linux/init.h>
+#include <linux/parser.h>
+#include <linux/random.h>
+#include <linux/slab.h>
+#include <linux/string.h>
+#include <asm/intel_pconfig.h>
+#include <asm/mktme.h>
+#include <keys/mktme-type.h>
+#include <keys/user-type.h>
+
+#include "internal.h"
+
+struct kmem_cache *mktme_prog_cache;	/* Hardware programming cache */
+
+static const char * const mktme_program_err[] = {
+	"KeyID was successfully programmed",	/* 0 */
+	"Invalid KeyID programming command",	/* 1 */
+	"Insufficient entropy",			/* 2 */
+	"KeyID not valid",			/* 3 */
+	"Invalid encryption algorithm chosen",	/* 4 */
+	"Failure to access key table",		/* 5 */
+};
+
+enum mktme_opt_id {
+	OPT_ERROR = -1,
+	OPT_TYPE,
+	OPT_KEY,
+	OPT_TWEAK,
+	OPT_ALGORITHM,
+};
+
+static const match_table_t mktme_token = {
+	{OPT_TYPE, "type=%s"},
+	{OPT_KEY, "key=%s"},
+	{OPT_TWEAK, "tweak=%s"},
+	{OPT_ALGORITHM, "algorithm=%s"},
+	{OPT_ERROR, NULL}
+};
+
+struct mktme_payload {
+	u32		keyid_ctrl;	/* Command & Encryption Algorithm */
+	u8		data_key[MKTME_AES_XTS_SIZE];
+	u8		tweak_key[MKTME_AES_XTS_SIZE];
+};
+
+/* Key Service Method called when Key is garbage collected. */
+static void mktme_destroy_key(struct key *key)
+{
+	key_put_encrypt_ref(mktme_map_keyid_from_key(key));
+}
+
+/* Copy the payload to the HW programming structure and program this KeyID */
+static int mktme_program_keyid(int keyid, struct mktme_payload *payload)
+{
+	struct mktme_key_program *kprog = NULL;
+	u8 kern_entropy[MKTME_AES_XTS_SIZE];
+	int i, ret;
+
+	kprog = kmem_cache_zalloc(mktme_prog_cache, GFP_KERNEL);
+	if (!kprog)
+		return -ENOMEM;
+
+	/* Hardware programming requires cached aligned struct */
+	kprog->keyid = keyid;
+	kprog->keyid_ctrl = payload->keyid_ctrl;
+	memcpy(kprog->key_field_1, payload->data_key, MKTME_AES_XTS_SIZE);
+	memcpy(kprog->key_field_2, payload->tweak_key, MKTME_AES_XTS_SIZE);
+
+	/* Strengthen the entropy fields for CPU generated keys */
+	if ((payload->keyid_ctrl & 0xff) = MKTME_KEYID_SET_KEY_RANDOM) {
+		get_random_bytes(&kern_entropy, sizeof(kern_entropy));
+		for (i = 0; i < (MKTME_AES_XTS_SIZE); i++) {
+			kprog->key_field_1[i] ^= kern_entropy[i];
+			kprog->key_field_2[i] ^= kern_entropy[i];
+		}
+	}
+	ret = mktme_key_program(kprog);
+	kmem_cache_free(mktme_prog_cache, kprog);
+	return ret;
+}
+
+/* Key Service Method to update an existing key. */
+static int mktme_update_key(struct key *key,
+			    struct key_preparsed_payload *prep)
+{
+	struct mktme_payload *payload = prep->payload.data[0];
+	int keyid, ref_count;
+	int ret;
+
+	mktme_map_lock();
+	keyid = mktme_map_keyid_from_key(key);
+	if (keyid <= 0)
+		return -EINVAL;
+	/*
+	 * ref_count will be at least one when we get here because the
+	 * key already exists. If ref_count is not > 1, it is safe to
+	 * update the key while holding the mktme_map_lock.
+	 */
+	ref_count = mktme_read_encrypt_ref(keyid);
+	if (ref_count > 1) {
+		pr_debug("mktme not updating keyid[%d] encrypt_count[%d]\n",
+			 keyid, ref_count);
+		return -EBUSY;
+	}
+	ret = mktme_program_keyid(keyid, payload);
+	if (ret != MKTME_PROG_SUCCESS) {
+		pr_debug("%s: %s\n", __func__, mktme_program_err[ret]);
+		ret = -ENOKEY;
+	}
+	mktme_map_unlock();
+	return ret;
+}
+
+/* Key Service Method to create a new key. Payload is preparsed. */
+int mktme_instantiate_key(struct key *key, struct key_preparsed_payload *prep)
+{
+	struct mktme_payload *payload = prep->payload.data[0];
+	int keyid, ret;
+
+	mktme_map_lock();
+	keyid = mktme_map_get_free_keyid();
+	if (keyid = 0) {
+		ret = -ENOKEY;
+		goto out;
+	}
+	ret = mktme_program_keyid(keyid, payload);
+	if (ret != MKTME_PROG_SUCCESS) {
+		pr_debug("%s: %s\n", __func__, mktme_program_err[ret]);
+		ret = -ENOKEY;
+		goto out;
+	}
+	mktme_map_set_keyid(keyid, key);
+	key_get_encrypt_ref(keyid);
+out:
+	mktme_map_unlock();
+	return ret;
+}
+
+/* Verify the user provided the needed arguments for the TYPE of Key */
+static int mktme_check_options(struct mktme_payload *payload,
+			       unsigned long token_mask, enum mktme_type type)
+{
+	if (!token_mask)
+		return -EINVAL;
+
+	switch (type) {
+	case MKTME_TYPE_USER:
+		if (test_bit(OPT_ALGORITHM, &token_mask))
+			payload->keyid_ctrl |= MKTME_AES_XTS_128;
+		else
+			return -EINVAL;
+
+		if ((test_bit(OPT_KEY, &token_mask)) &&
+		    (test_bit(OPT_TWEAK, &token_mask)))
+			payload->keyid_ctrl |= MKTME_KEYID_SET_KEY_DIRECT;
+		else
+			return -EINVAL;
+		break;
+
+	case MKTME_TYPE_CPU:
+		if (test_bit(OPT_ALGORITHM, &token_mask))
+			payload->keyid_ctrl |= MKTME_AES_XTS_128;
+		else
+			return -EINVAL;
+
+		payload->keyid_ctrl |= MKTME_KEYID_SET_KEY_RANDOM;
+		break;
+
+	case MKTME_TYPE_CLEAR:
+		payload->keyid_ctrl |= MKTME_KEYID_CLEAR_KEY;
+		break;
+
+	case MKTME_TYPE_NO_ENCRYPT:
+		payload->keyid_ctrl |= MKTME_KEYID_NO_ENCRYPT;
+		break;
+
+	default:
+		return -EINVAL;
+	}
+	return 0;
+}
+
+/* Parse the options and store the key programming data in the payload. */
+static int mktme_get_options(char *options, struct mktme_payload *payload)
+{
+	enum mktme_type type = MKTME_TYPE_ERROR;
+	substring_t args[MAX_OPT_ARGS];
+	unsigned long token_mask = 0;
+	char *p = options;
+	int ret, token;
+
+	while ((p = strsep(&options, " \t"))) {
+		if (*p = '\0' || *p = ' ' || *p = '\t')
+			continue;
+		token = match_token(p, mktme_token, args);
+		if (test_and_set_bit(token, &token_mask))
+			return -EINVAL;
+
+		switch (token) {
+		case OPT_KEY:
+			ret = hex2bin(payload->data_key, args[0].from,
+				      MKTME_AES_XTS_SIZE);
+			if (ret < 0)
+				return -EINVAL;
+			break;
+
+		case OPT_TWEAK:
+			ret = hex2bin(payload->tweak_key, args[0].from,
+				      MKTME_AES_XTS_SIZE);
+			if (ret < 0)
+				return -EINVAL;
+			break;
+
+		case OPT_TYPE:
+			type = match_string(mktme_type_names,
+					    ARRAY_SIZE(mktme_type_names),
+					    args[0].from);
+			if (type < 0)
+				return -EINVAL;
+			break;
+
+		case OPT_ALGORITHM:
+			ret = match_string(mktme_alg_names,
+					   ARRAY_SIZE(mktme_alg_names),
+					   args[0].from);
+			if (ret < 0)
+				return -EINVAL;
+			break;
+
+		default:
+			return -EINVAL;
+		}
+	}
+	return mktme_check_options(payload, token_mask, type);
+}
+
+void mktme_free_preparsed_key(struct key_preparsed_payload *prep)
+{
+	kzfree(prep->payload.data[0]);
+}
+
+/*
+ * Key Service Method to preparse a payload before a key is created.
+ * Check permissions and the options. Load the proposed key field
+ * data into the payload for use by instantiate and update methods.
+ */
+int mktme_preparse_key(struct key_preparsed_payload *prep)
+{
+	struct mktme_payload *mktme_payload;
+	size_t datalen = prep->datalen;
+	char *options;
+	int ret;
+
+	if (!capable(CAP_SYS_RESOURCE) && !capable(CAP_SYS_ADMIN))
+		return -EACCES;
+
+	if (datalen <= 0 || datalen > 1024 || !prep->data)
+		return -EINVAL;
+
+	options = kmemdup(prep->data, datalen + 1, GFP_KERNEL);
+	if (!options)
+		return -ENOMEM;
+
+	options[datalen] = '\0';
+
+	mktme_payload = kzalloc(sizeof(*mktme_payload), GFP_KERNEL);
+	if (!mktme_payload) {
+		ret = -ENOMEM;
+		goto out;
+	}
+	ret = mktme_get_options(options, mktme_payload);
+	if (ret < 0)
+		goto out;
+
+	prep->quotalen = sizeof(mktme_payload);
+	prep->payload.data[0] = mktme_payload;
+out:
+	kzfree(options);
+	return ret;
+}
+
+struct key_type key_type_mktme = {
+	.name		= "mktme",
+	.preparse	= mktme_preparse_key,
+	.free_preparse	= mktme_free_preparsed_key,
+	.instantiate	= mktme_instantiate_key,
+	.update		= mktme_update_key,
+	.describe	= user_describe,
+	.destroy	= mktme_destroy_key,
+};
+
+/*
+ * Allocate the global mktme_map structure based on the available keyids.
+ * Create a cache for the hardware structure. Initialize the encrypt_count
+ * array to track * VMA's per keyid. Once all that succeeds, register the
+ * 'mktme' key type.
+ */
+static int __init init_mktme(void)
+{
+	int ret;
+
+	/* Verify keys are present */
+	if (!(mktme_nr_keyids > 0))
+		return -EINVAL;
+
+	if (!mktme_map_alloc())
+		return -ENOMEM;
+
+	mktme_prog_cache = KMEM_CACHE(mktme_key_program, SLAB_PANIC);
+	if (!mktme_prog_cache)
+		goto free_map;
+
+	if (mktme_alloc_encrypt_array() < 0)
+		goto free_cache;
+
+	ret = register_key_type(&key_type_mktme);
+	if (!ret)
+		return ret;			/* SUCCESS */
+
+	mktme_free_encrypt_array();
+free_cache:
+	kmem_cache_destroy(mktme_prog_cache);
+free_map:
+	mktme_map_free();
+
+	return -ENOMEM;
+}
+
+late_initcall(init_mktme);
-- 
2.14.1

[RFC v2 10/13] keys/mktme: Add the MKTME Key Service type for memory encryption

[Alison Schofield]

MKTME (Multi-Key Total Memory Encryption) is a technology that allows
transparent memory encryption in upcoming Intel platforms. MKTME will
support mulitple encryption domains, each having their own key. The main
use case for the feature is virtual machine isolation. The API needs the
flexibility to work for a wide range of uses.

The MKTME key service type manages the addition and removal of the memory
encryption keys. It maps Userspace Keys to hardware KeyIDs. It programs
the hardware with the user requested encryption options.

The only supported encryption algorithm is AES-XTS 128.

The MKTME key service is half of the MKTME API level solution. It pairs
with a new memory encryption system call: encrypt_mprotect() that uses
the keys to encrypt memory.

See Documentation/x86/mktme/mktme.rst

Change-Id: Idda4af2beabb739c77719897affff183ee9fa716
Signed-off-by: Alison Schofield <alison.schofield@intel.com>
Signed-off-by: Kirill A. Shutemov <kirill.shutemov@linux.intel.com>
---
 arch/x86/Kconfig           |   1 +
 include/keys/mktme-type.h  |  41 ++++++
 security/keys/Kconfig      |  11 ++
 security/keys/Makefile     |   1 +
 security/keys/mktme_keys.c | 339 +++++++++++++++++++++++++++++++++++++++++++++
 5 files changed, 393 insertions(+)
 create mode 100644 include/keys/mktme-type.h
 create mode 100644 security/keys/mktme_keys.c

diff --git a/arch/x86/Kconfig b/arch/x86/Kconfig
index 7ac78e2856c7..c2e3bb5af077 100644
--- a/arch/x86/Kconfig
+++ b/arch/x86/Kconfig
@@ -1531,6 +1531,7 @@ config X86_INTEL_MKTME
 	bool "Intel Multi-Key Total Memory Encryption"
 	select DYNAMIC_PHYSICAL_MASK
 	select PAGE_EXTENSION
+	select MKTME_KEYS
 	depends on X86_64 && CPU_SUP_INTEL
 	---help---
 	  Say yes to enable support for Multi-Key Total Memory Encryption.
diff --git a/include/keys/mktme-type.h b/include/keys/mktme-type.h
new file mode 100644
index 000000000000..c63c6568087f
--- /dev/null
+++ b/include/keys/mktme-type.h
@@ -0,0 +1,41 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+
+/* Key service for Multi-KEY Total Memory Encryption */
+
+#ifndef _KEYS_MKTME_TYPE_H
+#define _KEYS_MKTME_TYPE_H
+
+#include <linux/key.h>
+
+/*
+ * The AES-XTS 128 encryption algorithm requires 128 bits for each
+ * user supplied data key and tweak key.
+ */
+#define MKTME_AES_XTS_SIZE	16	/* 16 bytes, 128 bits */
+
+enum mktme_alg {
+	MKTME_ALG_AES_XTS_128,
+};
+
+const char *const mktme_alg_names[] = {
+	[MKTME_ALG_AES_XTS_128]	= "aes-xts-128",
+};
+
+enum mktme_type {
+	MKTME_TYPE_ERROR = -1,
+	MKTME_TYPE_USER,
+	MKTME_TYPE_CPU,
+	MKTME_TYPE_CLEAR,
+	MKTME_TYPE_NO_ENCRYPT,
+};
+
+const char *const mktme_type_names[] = {
+	[MKTME_TYPE_USER]	= "user",
+	[MKTME_TYPE_CPU]	= "cpu",
+	[MKTME_TYPE_CLEAR]	= "clear",
+	[MKTME_TYPE_NO_ENCRYPT]	= "no-encrypt",
+};
+
+extern struct key_type key_type_mktme;
+
+#endif /* _KEYS_MKTME_TYPE_H */
diff --git a/security/keys/Kconfig b/security/keys/Kconfig
index 6462e6654ccf..c36972113e67 100644
--- a/security/keys/Kconfig
+++ b/security/keys/Kconfig
@@ -101,3 +101,14 @@ config KEY_DH_OPERATIONS
 	 in the kernel.
 
 	 If you are unsure as to whether this is required, answer N.
+
+config MKTME_KEYS
+	bool "Multi-Key Total Memory Encryption Keys"
+	depends on KEYS && X86_INTEL_MKTME
+	help
+	  This option provides support for Multi-Key Total Memory
+	  Encryption (MKTME) on Intel platforms offering the feature.
+	  MKTME allows userspace to manage the hardware encryption
+	  keys through the kernel key services.
+
+	  If you are unsure as to whether this is required, answer N.
diff --git a/security/keys/Makefile b/security/keys/Makefile
index 9cef54064f60..94c84f10a857 100644
--- a/security/keys/Makefile
+++ b/security/keys/Makefile
@@ -30,3 +30,4 @@ obj-$(CONFIG_ASYMMETRIC_KEY_TYPE) += keyctl_pkey.o
 obj-$(CONFIG_BIG_KEYS) += big_key.o
 obj-$(CONFIG_TRUSTED_KEYS) += trusted.o
 obj-$(CONFIG_ENCRYPTED_KEYS) += encrypted-keys/
+obj-$(CONFIG_MKTME_KEYS) += mktme_keys.o
diff --git a/security/keys/mktme_keys.c b/security/keys/mktme_keys.c
new file mode 100644
index 000000000000..e615eb58e600
--- /dev/null
+++ b/security/keys/mktme_keys.c
@@ -0,0 +1,339 @@
+// SPDX-License-Identifier: GPL-3.0
+
+/* Documentation/x86/mktme/mktme_keys.rst */
+
+#include <linux/cred.h>
+#include <linux/cpu.h>
+#include <linux/err.h>
+#include <linux/init.h>
+#include <linux/key.h>
+#include <linux/key-type.h>
+#include <linux/init.h>
+#include <linux/parser.h>
+#include <linux/random.h>
+#include <linux/slab.h>
+#include <linux/string.h>
+#include <asm/intel_pconfig.h>
+#include <asm/mktme.h>
+#include <keys/mktme-type.h>
+#include <keys/user-type.h>
+
+#include "internal.h"
+
+struct kmem_cache *mktme_prog_cache;	/* Hardware programming cache */
+
+static const char * const mktme_program_err[] = {
+	"KeyID was successfully programmed",	/* 0 */
+	"Invalid KeyID programming command",	/* 1 */
+	"Insufficient entropy",			/* 2 */
+	"KeyID not valid",			/* 3 */
+	"Invalid encryption algorithm chosen",	/* 4 */
+	"Failure to access key table",		/* 5 */
+};
+
+enum mktme_opt_id {
+	OPT_ERROR = -1,
+	OPT_TYPE,
+	OPT_KEY,
+	OPT_TWEAK,
+	OPT_ALGORITHM,
+};
+
+static const match_table_t mktme_token = {
+	{OPT_TYPE, "type=%s"},
+	{OPT_KEY, "key=%s"},
+	{OPT_TWEAK, "tweak=%s"},
+	{OPT_ALGORITHM, "algorithm=%s"},
+	{OPT_ERROR, NULL}
+};
+
+struct mktme_payload {
+	u32		keyid_ctrl;	/* Command & Encryption Algorithm */
+	u8		data_key[MKTME_AES_XTS_SIZE];
+	u8		tweak_key[MKTME_AES_XTS_SIZE];
+};
+
+/* Key Service Method called when Key is garbage collected. */
+static void mktme_destroy_key(struct key *key)
+{
+	key_put_encrypt_ref(mktme_map_keyid_from_key(key));
+}
+
+/* Copy the payload to the HW programming structure and program this KeyID */
+static int mktme_program_keyid(int keyid, struct mktme_payload *payload)
+{
+	struct mktme_key_program *kprog = NULL;
+	u8 kern_entropy[MKTME_AES_XTS_SIZE];
+	int i, ret;
+
+	kprog = kmem_cache_zalloc(mktme_prog_cache, GFP_KERNEL);
+	if (!kprog)
+		return -ENOMEM;
+
+	/* Hardware programming requires cached aligned struct */
+	kprog->keyid = keyid;
+	kprog->keyid_ctrl = payload->keyid_ctrl;
+	memcpy(kprog->key_field_1, payload->data_key, MKTME_AES_XTS_SIZE);
+	memcpy(kprog->key_field_2, payload->tweak_key, MKTME_AES_XTS_SIZE);
+
+	/* Strengthen the entropy fields for CPU generated keys */
+	if ((payload->keyid_ctrl & 0xff) == MKTME_KEYID_SET_KEY_RANDOM) {
+		get_random_bytes(&kern_entropy, sizeof(kern_entropy));
+		for (i = 0; i < (MKTME_AES_XTS_SIZE); i++) {
+			kprog->key_field_1[i] ^= kern_entropy[i];
+			kprog->key_field_2[i] ^= kern_entropy[i];
+		}
+	}
+	ret = mktme_key_program(kprog);
+	kmem_cache_free(mktme_prog_cache, kprog);
+	return ret;
+}
+
+/* Key Service Method to update an existing key. */
+static int mktme_update_key(struct key *key,
+			    struct key_preparsed_payload *prep)
+{
+	struct mktme_payload *payload = prep->payload.data[0];
+	int keyid, ref_count;
+	int ret;
+
+	mktme_map_lock();
+	keyid = mktme_map_keyid_from_key(key);
+	if (keyid <= 0)
+		return -EINVAL;
+	/*
+	 * ref_count will be at least one when we get here because the
+	 * key already exists. If ref_count is not > 1, it is safe to
+	 * update the key while holding the mktme_map_lock.
+	 */
+	ref_count = mktme_read_encrypt_ref(keyid);
+	if (ref_count > 1) {
+		pr_debug("mktme not updating keyid[%d] encrypt_count[%d]\n",
+			 keyid, ref_count);
+		return -EBUSY;
+	}
+	ret = mktme_program_keyid(keyid, payload);
+	if (ret != MKTME_PROG_SUCCESS) {
+		pr_debug("%s: %s\n", __func__, mktme_program_err[ret]);
+		ret = -ENOKEY;
+	}
+	mktme_map_unlock();
+	return ret;
+}
+
+/* Key Service Method to create a new key. Payload is preparsed. */
+int mktme_instantiate_key(struct key *key, struct key_preparsed_payload *prep)
+{
+	struct mktme_payload *payload = prep->payload.data[0];
+	int keyid, ret;
+
+	mktme_map_lock();
+	keyid = mktme_map_get_free_keyid();
+	if (keyid == 0) {
+		ret = -ENOKEY;
+		goto out;
+	}
+	ret = mktme_program_keyid(keyid, payload);
+	if (ret != MKTME_PROG_SUCCESS) {
+		pr_debug("%s: %s\n", __func__, mktme_program_err[ret]);
+		ret = -ENOKEY;
+		goto out;
+	}
+	mktme_map_set_keyid(keyid, key);
+	key_get_encrypt_ref(keyid);
+out:
+	mktme_map_unlock();
+	return ret;
+}
+
+/* Verify the user provided the needed arguments for the TYPE of Key */
+static int mktme_check_options(struct mktme_payload *payload,
+			       unsigned long token_mask, enum mktme_type type)
+{
+	if (!token_mask)
+		return -EINVAL;
+
+	switch (type) {
+	case MKTME_TYPE_USER:
+		if (test_bit(OPT_ALGORITHM, &token_mask))
+			payload->keyid_ctrl |= MKTME_AES_XTS_128;
+		else
+			return -EINVAL;
+
+		if ((test_bit(OPT_KEY, &token_mask)) &&
+		    (test_bit(OPT_TWEAK, &token_mask)))
+			payload->keyid_ctrl |= MKTME_KEYID_SET_KEY_DIRECT;
+		else
+			return -EINVAL;
+		break;
+
+	case MKTME_TYPE_CPU:
+		if (test_bit(OPT_ALGORITHM, &token_mask))
+			payload->keyid_ctrl |= MKTME_AES_XTS_128;
+		else
+			return -EINVAL;
+
+		payload->keyid_ctrl |= MKTME_KEYID_SET_KEY_RANDOM;
+		break;
+
+	case MKTME_TYPE_CLEAR:
+		payload->keyid_ctrl |= MKTME_KEYID_CLEAR_KEY;
+		break;
+
+	case MKTME_TYPE_NO_ENCRYPT:
+		payload->keyid_ctrl |= MKTME_KEYID_NO_ENCRYPT;
+		break;
+
+	default:
+		return -EINVAL;
+	}
+	return 0;
+}
+
+/* Parse the options and store the key programming data in the payload. */
+static int mktme_get_options(char *options, struct mktme_payload *payload)
+{
+	enum mktme_type type = MKTME_TYPE_ERROR;
+	substring_t args[MAX_OPT_ARGS];
+	unsigned long token_mask = 0;
+	char *p = options;
+	int ret, token;
+
+	while ((p = strsep(&options, " \t"))) {
+		if (*p == '\0' || *p == ' ' || *p == '\t')
+			continue;
+		token = match_token(p, mktme_token, args);
+		if (test_and_set_bit(token, &token_mask))
+			return -EINVAL;
+
+		switch (token) {
+		case OPT_KEY:
+			ret = hex2bin(payload->data_key, args[0].from,
+				      MKTME_AES_XTS_SIZE);
+			if (ret < 0)
+				return -EINVAL;
+			break;
+
+		case OPT_TWEAK:
+			ret = hex2bin(payload->tweak_key, args[0].from,
+				      MKTME_AES_XTS_SIZE);
+			if (ret < 0)
+				return -EINVAL;
+			break;
+
+		case OPT_TYPE:
+			type = match_string(mktme_type_names,
+					    ARRAY_SIZE(mktme_type_names),
+					    args[0].from);
+			if (type < 0)
+				return -EINVAL;
+			break;
+
+		case OPT_ALGORITHM:
+			ret = match_string(mktme_alg_names,
+					   ARRAY_SIZE(mktme_alg_names),
+					   args[0].from);
+			if (ret < 0)
+				return -EINVAL;
+			break;
+
+		default:
+			return -EINVAL;
+		}
+	}
+	return mktme_check_options(payload, token_mask, type);
+}
+
+void mktme_free_preparsed_key(struct key_preparsed_payload *prep)
+{
+	kzfree(prep->payload.data[0]);
+}
+
+/*
+ * Key Service Method to preparse a payload before a key is created.
+ * Check permissions and the options. Load the proposed key field
+ * data into the payload for use by instantiate and update methods.
+ */
+int mktme_preparse_key(struct key_preparsed_payload *prep)
+{
+	struct mktme_payload *mktme_payload;
+	size_t datalen = prep->datalen;
+	char *options;
+	int ret;
+
+	if (!capable(CAP_SYS_RESOURCE) && !capable(CAP_SYS_ADMIN))
+		return -EACCES;
+
+	if (datalen <= 0 || datalen > 1024 || !prep->data)
+		return -EINVAL;
+
+	options = kmemdup(prep->data, datalen + 1, GFP_KERNEL);
+	if (!options)
+		return -ENOMEM;
+
+	options[datalen] = '\0';
+
+	mktme_payload = kzalloc(sizeof(*mktme_payload), GFP_KERNEL);
+	if (!mktme_payload) {
+		ret = -ENOMEM;
+		goto out;
+	}
+	ret = mktme_get_options(options, mktme_payload);
+	if (ret < 0)
+		goto out;
+
+	prep->quotalen = sizeof(mktme_payload);
+	prep->payload.data[0] = mktme_payload;
+out:
+	kzfree(options);
+	return ret;
+}
+
+struct key_type key_type_mktme = {
+	.name		= "mktme",
+	.preparse	= mktme_preparse_key,
+	.free_preparse	= mktme_free_preparsed_key,
+	.instantiate	= mktme_instantiate_key,
+	.update		= mktme_update_key,
+	.describe	= user_describe,
+	.destroy	= mktme_destroy_key,
+};
+
+/*
+ * Allocate the global mktme_map structure based on the available keyids.
+ * Create a cache for the hardware structure. Initialize the encrypt_count
+ * array to track * VMA's per keyid. Once all that succeeds, register the
+ * 'mktme' key type.
+ */
+static int __init init_mktme(void)
+{
+	int ret;
+
+	/* Verify keys are present */
+	if (!(mktme_nr_keyids > 0))
+		return -EINVAL;
+
+	if (!mktme_map_alloc())
+		return -ENOMEM;
+
+	mktme_prog_cache = KMEM_CACHE(mktme_key_program, SLAB_PANIC);
+	if (!mktme_prog_cache)
+		goto free_map;
+
+	if (mktme_alloc_encrypt_array() < 0)
+		goto free_cache;
+
+	ret = register_key_type(&key_type_mktme);
+	if (!ret)
+		return ret;			/* SUCCESS */
+
+	mktme_free_encrypt_array();
+free_cache:
+	kmem_cache_destroy(mktme_prog_cache);
+free_map:
+	mktme_map_free();
+
+	return -ENOMEM;
+}
+
+late_initcall(init_mktme);
-- 
2.14.1

[RFC v2 11/13] keys/mktme: Program memory encryption keys on a system wide basis

[Alison Schofield]

The kernel manages the MKTME (Multi-Key Total Memory Encryption) Keys
as a system wide single pool of keys. The hardware, however, manages
the keys on a per physical package basis. Each physical package
maintains a Key Table that all CPU's in that package share.

In order to maintain the consistent, system wide view that the kernel
requires, program all physical packages during a key program request.

Change-Id: I0ff46f37fde47a0305842baeb8ea600b6c568639
Signed-off-by: Alison Schofield <alison.schofield@intel.com>
---
 security/keys/mktme_keys.c | 61 +++++++++++++++++++++++++++++++++++++++++++++-
 1 file changed, 60 insertions(+), 1 deletion(-)

diff --git a/security/keys/mktme_keys.c b/security/keys/mktme_keys.c
index e615eb58e600..7f113146acf2 100644
--- a/security/keys/mktme_keys.c
+++ b/security/keys/mktme_keys.c
@@ -21,6 +21,7 @@
 #include "internal.h"
 
 struct kmem_cache *mktme_prog_cache;	/* Hardware programming cache */
+cpumask_var_t mktme_leadcpus;		/* one cpu per pkg to program keys */
 
 static const char * const mktme_program_err[] = {
 	"KeyID was successfully programmed",	/* 0 */
@@ -59,6 +60,37 @@ static void mktme_destroy_key(struct key *key)
 	key_put_encrypt_ref(mktme_map_keyid_from_key(key));
 }
 
+struct mktme_hw_program_info {
+	struct mktme_key_program *key_program;
+	unsigned long status;
+};
+
+/* Program a KeyID on a single package. */
+static void mktme_program_package(void *hw_program_info)
+{
+	struct mktme_hw_program_info *info = hw_program_info;
+	int ret;
+
+	ret = mktme_key_program(info->key_program);
+	if (ret != MKTME_PROG_SUCCESS)
+		WRITE_ONCE(info->status, ret);
+}
+
+/* Program a KeyID across the entire system. */
+static int mktme_program_system(struct mktme_key_program *key_program,
+				cpumask_var_t mktme_cpumask)
+{
+	struct mktme_hw_program_info info = {
+		.key_program = key_program,
+		.status = MKTME_PROG_SUCCESS,
+	};
+	get_online_cpus();
+	on_each_cpu_mask(mktme_cpumask, mktme_program_package, &info, 1);
+	put_online_cpus();
+
+	return info.status;
+}
+
 /* Copy the payload to the HW programming structure and program this KeyID */
 static int mktme_program_keyid(int keyid, struct mktme_payload *payload)
 {
@@ -84,7 +116,7 @@ static int mktme_program_keyid(int keyid, struct mktme_payload *payload)
 			kprog->key_field_2[i] ^= kern_entropy[i];
 		}
 	}
-	ret = mktme_key_program(kprog);
+	ret = mktme_program_system(kprog, mktme_leadcpus);
 	kmem_cache_free(mktme_prog_cache, kprog);
 	return ret;
 }
@@ -299,6 +331,28 @@ struct key_type key_type_mktme = {
 	.destroy	= mktme_destroy_key,
 };
 
+static int mktme_build_leadcpus_mask(void)
+{
+	int online_cpu, mktme_cpu;
+	int online_pkgid, mktme_pkgid = -1;
+
+	if (!zalloc_cpumask_var(&mktme_leadcpus, GFP_KERNEL))
+		return -ENOMEM;
+
+	for_each_online_cpu(online_cpu) {
+		online_pkgid = topology_physical_package_id(online_cpu);
+
+		for_each_cpu(mktme_cpu, mktme_leadcpus) {
+			mktme_pkgid = topology_physical_package_id(mktme_cpu);
+			if (mktme_pkgid = online_pkgid)
+				break;
+		}
+		if (mktme_pkgid != online_pkgid)
+			cpumask_set_cpu(online_cpu, mktme_leadcpus);
+	}
+	return 0;
+}
+
 /*
  * Allocate the global mktme_map structure based on the available keyids.
  * Create a cache for the hardware structure. Initialize the encrypt_count
@@ -323,10 +377,15 @@ static int __init init_mktme(void)
 	if (mktme_alloc_encrypt_array() < 0)
 		goto free_cache;
 
+	if (mktme_build_leadcpus_mask() < 0)
+		goto free_array;
+
 	ret = register_key_type(&key_type_mktme);
 	if (!ret)
 		return ret;			/* SUCCESS */
 
+	free_cpumask_var(mktme_leadcpus);
+free_array:
 	mktme_free_encrypt_array();
 free_cache:
 	kmem_cache_destroy(mktme_prog_cache);
-- 
2.14.1

[RFC v2 11/13] keys/mktme: Program memory encryption keys on a system wide basis

[Alison Schofield]

The kernel manages the MKTME (Multi-Key Total Memory Encryption) Keys
as a system wide single pool of keys. The hardware, however, manages
the keys on a per physical package basis. Each physical package
maintains a Key Table that all CPU's in that package share.

In order to maintain the consistent, system wide view that the kernel
requires, program all physical packages during a key program request.

Change-Id: I0ff46f37fde47a0305842baeb8ea600b6c568639
Signed-off-by: Alison Schofield <alison.schofield@intel.com>
---
 security/keys/mktme_keys.c | 61 +++++++++++++++++++++++++++++++++++++++++++++-
 1 file changed, 60 insertions(+), 1 deletion(-)

diff --git a/security/keys/mktme_keys.c b/security/keys/mktme_keys.c
index e615eb58e600..7f113146acf2 100644
--- a/security/keys/mktme_keys.c
+++ b/security/keys/mktme_keys.c
@@ -21,6 +21,7 @@
 #include "internal.h"
 
 struct kmem_cache *mktme_prog_cache;	/* Hardware programming cache */
+cpumask_var_t mktme_leadcpus;		/* one cpu per pkg to program keys */
 
 static const char * const mktme_program_err[] = {
 	"KeyID was successfully programmed",	/* 0 */
@@ -59,6 +60,37 @@ static void mktme_destroy_key(struct key *key)
 	key_put_encrypt_ref(mktme_map_keyid_from_key(key));
 }
 
+struct mktme_hw_program_info {
+	struct mktme_key_program *key_program;
+	unsigned long status;
+};
+
+/* Program a KeyID on a single package. */
+static void mktme_program_package(void *hw_program_info)
+{
+	struct mktme_hw_program_info *info = hw_program_info;
+	int ret;
+
+	ret = mktme_key_program(info->key_program);
+	if (ret != MKTME_PROG_SUCCESS)
+		WRITE_ONCE(info->status, ret);
+}
+
+/* Program a KeyID across the entire system. */
+static int mktme_program_system(struct mktme_key_program *key_program,
+				cpumask_var_t mktme_cpumask)
+{
+	struct mktme_hw_program_info info = {
+		.key_program = key_program,
+		.status = MKTME_PROG_SUCCESS,
+	};
+	get_online_cpus();
+	on_each_cpu_mask(mktme_cpumask, mktme_program_package, &info, 1);
+	put_online_cpus();
+
+	return info.status;
+}
+
 /* Copy the payload to the HW programming structure and program this KeyID */
 static int mktme_program_keyid(int keyid, struct mktme_payload *payload)
 {
@@ -84,7 +116,7 @@ static int mktme_program_keyid(int keyid, struct mktme_payload *payload)
 			kprog->key_field_2[i] ^= kern_entropy[i];
 		}
 	}
-	ret = mktme_key_program(kprog);
+	ret = mktme_program_system(kprog, mktme_leadcpus);
 	kmem_cache_free(mktme_prog_cache, kprog);
 	return ret;
 }
@@ -299,6 +331,28 @@ struct key_type key_type_mktme = {
 	.destroy	= mktme_destroy_key,
 };
 
+static int mktme_build_leadcpus_mask(void)
+{
+	int online_cpu, mktme_cpu;
+	int online_pkgid, mktme_pkgid = -1;
+
+	if (!zalloc_cpumask_var(&mktme_leadcpus, GFP_KERNEL))
+		return -ENOMEM;
+
+	for_each_online_cpu(online_cpu) {
+		online_pkgid = topology_physical_package_id(online_cpu);
+
+		for_each_cpu(mktme_cpu, mktme_leadcpus) {
+			mktme_pkgid = topology_physical_package_id(mktme_cpu);
+			if (mktme_pkgid == online_pkgid)
+				break;
+		}
+		if (mktme_pkgid != online_pkgid)
+			cpumask_set_cpu(online_cpu, mktme_leadcpus);
+	}
+	return 0;
+}
+
 /*
  * Allocate the global mktme_map structure based on the available keyids.
  * Create a cache for the hardware structure. Initialize the encrypt_count
@@ -323,10 +377,15 @@ static int __init init_mktme(void)
 	if (mktme_alloc_encrypt_array() < 0)
 		goto free_cache;
 
+	if (mktme_build_leadcpus_mask() < 0)
+		goto free_array;
+
 	ret = register_key_type(&key_type_mktme);
 	if (!ret)
 		return ret;			/* SUCCESS */
 
+	free_cpumask_var(mktme_leadcpus);
+free_array:
 	mktme_free_encrypt_array();
 free_cache:
 	kmem_cache_destroy(mktme_prog_cache);
-- 
2.14.1

[RFC v2 12/13] keys/mktme: Save MKTME data if kernel cmdline parameter allows

[Alison Schofield]

MKTME (Multi-Key Total Memory Encryption) key payloads may include
data encryption keys, tweak keys, and additional entropy bits. These
are used to program the MKTME encryption hardware. By default, the
kernel destroys this payload data once the hardware is programmed.

However, in order to fully support CPU Hotplug, saving the key data
becomes important. The MKTME Key Service cannot allow a new physical
package to come online unless it can program the new packages Key Table
to match the Key Tables of all existing physical packages.

With CPU generated keys (a.k.a. random keys or ephemeral keys) the
saving of user key data is not an issue. The kernel and MKTME hardware
can generate strong encryption keys without recalling any user supplied
data.

With USER directed keys (a.k.a. user type) saving the key programming
data (data and tweak key) becomes an issue. The data and tweak keys
are required to program those keys on a new physical package.

In preparation for adding CPU hotplug support:

   Add an 'mktme_vault' where key data is stored.

   Add 'mktme_savekeys' kernel command line parameter that directs
   what key data can be stored. If it is not set, kernel does not
   store users data key or tweak key.

   Add 'mktme_bitmap_user_type' to track when USER type keys are in
   use. If no USER type keys are currently in use, a physical package
   may be brought online, despite the absence of 'mktme_savekeys'.

Change-Id: If57414862f1ac131dd97e29bf4f3937ac33777f6
Signed-off-by: Alison Schofield <alison.schofield@intel.com>
Signed-off-by: Kirill A. Shutemov <kirill.shutemov@linux.intel.com>
---
 Documentation/admin-guide/kernel-parameters.rst |  1 +
 Documentation/admin-guide/kernel-parameters.txt | 11 +++++
 arch/x86/mm/mktme.c                             |  2 +
 security/keys/mktme_keys.c                      | 65 +++++++++++++++++++++++++
 4 files changed, 79 insertions(+)

diff --git a/Documentation/admin-guide/kernel-parameters.rst b/Documentation/admin-guide/kernel-parameters.rst
index b8d0bc07ed0a..1b62b86d0666 100644
--- a/Documentation/admin-guide/kernel-parameters.rst
+++ b/Documentation/admin-guide/kernel-parameters.rst
@@ -120,6 +120,7 @@ parameter is applicable::
 			Documentation/m68k/kernel-options.txt.
 	MDA	MDA console support is enabled.
 	MIPS	MIPS architecture is enabled.
+	MKTME	Multi-Key Total Memory Encryption is enabled.
 	MOUSE	Appropriate mouse support is enabled.
 	MSI	Message Signaled Interrupts (PCI).
 	MTD	MTD (Memory Technology Device) support is enabled.
diff --git a/Documentation/admin-guide/kernel-parameters.txt b/Documentation/admin-guide/kernel-parameters.txt
index 81d1d5a74728..c777dbf0f75c 100644
--- a/Documentation/admin-guide/kernel-parameters.txt
+++ b/Documentation/admin-guide/kernel-parameters.txt
@@ -2497,6 +2497,17 @@
 			in the "bleeding edge" mini2440 support kernel at
 			http://repo.or.cz/w/linux-2.6/mini2440.git
 
+	mktme_savekeys  [X86, MKTME] When CONFIG_X86_INTEL_MKTME is set
+			this parameter allows the kernel to save the user
+			specified MKTME key payload. Saving this payload
+			means that the MKTME Key Service can always allows
+			the addition of new physical packages. If the
+			mktme_savekeys parameter is not present, users key
+			data will not be saved, and new physical packages
+			may only be added to the system if no user type
+			MKTME keys are in use.
+			See Documentation/x86/mktme.rst
+
 	mminit_loglevel 			[KNL] When CONFIG_DEBUG_MEMORY_INIT is set, this
 			parameter allows control of the logging verbosity for
diff --git a/arch/x86/mm/mktme.c b/arch/x86/mm/mktme.c
index 55d34beb9b81..f96f4f2884e8 100644
--- a/arch/x86/mm/mktme.c
+++ b/arch/x86/mm/mktme.c
@@ -99,10 +99,12 @@ void mktme_map_set_keyid(int keyid, void *key)
 	mktme_map->mapped_keyids++;
 }
 
+extern unsigned long *mktme_bitmap_user_type;
 void mktme_map_free_keyid(int keyid)
 {
 	mktme_map->key[keyid] = 0;
 	mktme_map->mapped_keyids--;
+	clear_bit(keyid, mktme_bitmap_user_type);
 }
 
 int mktme_map_keyid_from_key(void *key)
diff --git a/security/keys/mktme_keys.c b/security/keys/mktme_keys.c
index 7f113146acf2..e9c7d306cba1 100644
--- a/security/keys/mktme_keys.c
+++ b/security/keys/mktme_keys.c
@@ -23,6 +23,11 @@
 struct kmem_cache *mktme_prog_cache;	/* Hardware programming cache */
 cpumask_var_t mktme_leadcpus;		/* one cpu per pkg to program keys */
 
+/* Kernel command line parameter allows saving of users key payload. */
+static bool mktme_savekeys;
+/* Track the existence of user type keys to make package hotplug decisions. */
+unsigned long *mktme_bitmap_user_type;
+
 static const char * const mktme_program_err[] = {
 	"KeyID was successfully programmed",	/* 0 */
 	"Invalid KeyID programming command",	/* 1 */
@@ -54,6 +59,9 @@ struct mktme_payload {
 	u8		tweak_key[MKTME_AES_XTS_SIZE];
 };
 
+/* Store keys in this vault if cmdline parameter mktme_savekeys allows */
+struct mktme_payload *mktme_vault;
+
 /* Key Service Method called when Key is garbage collected. */
 static void mktme_destroy_key(struct key *key)
 {
@@ -121,6 +129,23 @@ static int mktme_program_keyid(int keyid, struct mktme_payload *payload)
 	return ret;
 }
 
+static void mktme_load_vault(int keyid, struct mktme_payload *payload)
+{
+	/*
+	 * Always save the control fields to program hotplugged
+	 * packages with RANDOM, CLEAR, or NO_ENCRYPT type keys.
+	 */
+	mktme_vault[keyid].keyid_ctrl = payload->keyid_ctrl;
+
+	/* Only save data and tweak keys if allowed */
+	if (mktme_savekeys) {
+		memcpy(mktme_vault[keyid].data_key, payload->data_key,
+		       MKTME_AES_XTS_SIZE);
+		memcpy(mktme_vault[keyid].tweak_key, payload->tweak_key,
+		       MKTME_AES_XTS_SIZE);
+	}
+}
+
 /* Key Service Method to update an existing key. */
 static int mktme_update_key(struct key *key,
 			    struct key_preparsed_payload *prep)
@@ -144,11 +169,23 @@ static int mktme_update_key(struct key *key,
 			 keyid, ref_count);
 		return -EBUSY;
 	}
+
+	/* Forget if key was user type. */
+	clear_bit(keyid, mktme_bitmap_user_type);
+
 	ret = mktme_program_keyid(keyid, payload);
 	if (ret != MKTME_PROG_SUCCESS) {
 		pr_debug("%s: %s\n", __func__, mktme_program_err[ret]);
 		ret = -ENOKEY;
+		goto out;
 	}
+
+	mktme_load_vault(keyid, payload);
+
+	/* Remember if this key is user type. */
+	if ((payload->keyid_ctrl & 0xff) = MKTME_KEYID_SET_KEY_DIRECT)
+		set_bit(keyid, mktme_bitmap_user_type);
+out:
 	mktme_map_unlock();
 	return ret;
 }
@@ -171,6 +208,13 @@ int mktme_instantiate_key(struct key *key, struct key_preparsed_payload *prep)
 		ret = -ENOKEY;
 		goto out;
 	}
+
+	mktme_load_vault(keyid, payload);
+
+	/* Remember if key is user type. */
+	if ((payload->keyid_ctrl & 0xff) = MKTME_KEYID_SET_KEY_DIRECT)
+		set_bit(keyid, mktme_bitmap_user_type);
+
 	mktme_map_set_keyid(keyid, key);
 	key_get_encrypt_ref(keyid);
 out:
@@ -380,10 +424,23 @@ static int __init init_mktme(void)
 	if (mktme_build_leadcpus_mask() < 0)
 		goto free_array;
 
+	mktme_bitmap_user_type = bitmap_zalloc(mktme_nr_keyids, GFP_KERNEL);
+	if (!mktme_bitmap_user_type)
+		goto free_mask;
+
+	mktme_vault = kzalloc(sizeof(mktme_vault[0]) * (mktme_nr_keyids + 1),
+			      GFP_KERNEL);
+	if (!mktme_vault)
+		goto free_bitmap;
+
 	ret = register_key_type(&key_type_mktme);
 	if (!ret)
 		return ret;			/* SUCCESS */
 
+	kfree(mktme_vault);
+free_bitmap:
+	bitmap_free(mktme_bitmap_user_type);
+free_mask:
 	free_cpumask_var(mktme_leadcpus);
 free_array:
 	mktme_free_encrypt_array();
@@ -396,3 +453,11 @@ static int __init init_mktme(void)
 }
 
 late_initcall(init_mktme);
+
+static int mktme_enable_savekeys(char *__unused)
+{
+	mktme_savekeys = true;
+	return 1;
+}
+__setup("mktme_savekeys", mktme_enable_savekeys);
+
-- 
2.14.1

[RFC v2 12/13] keys/mktme: Save MKTME data if kernel cmdline parameter allows

[Alison Schofield]

MKTME (Multi-Key Total Memory Encryption) key payloads may include
data encryption keys, tweak keys, and additional entropy bits. These
are used to program the MKTME encryption hardware. By default, the
kernel destroys this payload data once the hardware is programmed.

However, in order to fully support CPU Hotplug, saving the key data
becomes important. The MKTME Key Service cannot allow a new physical
package to come online unless it can program the new packages Key Table
to match the Key Tables of all existing physical packages.

With CPU generated keys (a.k.a. random keys or ephemeral keys) the
saving of user key data is not an issue. The kernel and MKTME hardware
can generate strong encryption keys without recalling any user supplied
data.

With USER directed keys (a.k.a. user type) saving the key programming
data (data and tweak key) becomes an issue. The data and tweak keys
are required to program those keys on a new physical package.

In preparation for adding CPU hotplug support:

   Add an 'mktme_vault' where key data is stored.

   Add 'mktme_savekeys' kernel command line parameter that directs
   what key data can be stored. If it is not set, kernel does not
   store users data key or tweak key.

   Add 'mktme_bitmap_user_type' to track when USER type keys are in
   use. If no USER type keys are currently in use, a physical package
   may be brought online, despite the absence of 'mktme_savekeys'.

Change-Id: If57414862f1ac131dd97e29bf4f3937ac33777f6
Signed-off-by: Alison Schofield <alison.schofield@intel.com>
Signed-off-by: Kirill A. Shutemov <kirill.shutemov@linux.intel.com>
---
 Documentation/admin-guide/kernel-parameters.rst |  1 +
 Documentation/admin-guide/kernel-parameters.txt | 11 +++++
 arch/x86/mm/mktme.c                             |  2 +
 security/keys/mktme_keys.c                      | 65 +++++++++++++++++++++++++
 4 files changed, 79 insertions(+)

diff --git a/Documentation/admin-guide/kernel-parameters.rst b/Documentation/admin-guide/kernel-parameters.rst
index b8d0bc07ed0a..1b62b86d0666 100644
--- a/Documentation/admin-guide/kernel-parameters.rst
+++ b/Documentation/admin-guide/kernel-parameters.rst
@@ -120,6 +120,7 @@ parameter is applicable::
 			Documentation/m68k/kernel-options.txt.
 	MDA	MDA console support is enabled.
 	MIPS	MIPS architecture is enabled.
+	MKTME	Multi-Key Total Memory Encryption is enabled.
 	MOUSE	Appropriate mouse support is enabled.
 	MSI	Message Signaled Interrupts (PCI).
 	MTD	MTD (Memory Technology Device) support is enabled.
diff --git a/Documentation/admin-guide/kernel-parameters.txt b/Documentation/admin-guide/kernel-parameters.txt
index 81d1d5a74728..c777dbf0f75c 100644
--- a/Documentation/admin-guide/kernel-parameters.txt
+++ b/Documentation/admin-guide/kernel-parameters.txt
@@ -2497,6 +2497,17 @@
 			in the "bleeding edge" mini2440 support kernel at
 			http://repo.or.cz/w/linux-2.6/mini2440.git
 
+	mktme_savekeys  [X86, MKTME] When CONFIG_X86_INTEL_MKTME is set
+			this parameter allows the kernel to save the user
+			specified MKTME key payload. Saving this payload
+			means that the MKTME Key Service can always allows
+			the addition of new physical packages. If the
+			mktme_savekeys parameter is not present, users key
+			data will not be saved, and new physical packages
+			may only be added to the system if no user type
+			MKTME keys are in use.
+			See Documentation/x86/mktme.rst
+
 	mminit_loglevel=
 			[KNL] When CONFIG_DEBUG_MEMORY_INIT is set, this
 			parameter allows control of the logging verbosity for
diff --git a/arch/x86/mm/mktme.c b/arch/x86/mm/mktme.c
index 55d34beb9b81..f96f4f2884e8 100644
--- a/arch/x86/mm/mktme.c
+++ b/arch/x86/mm/mktme.c
@@ -99,10 +99,12 @@ void mktme_map_set_keyid(int keyid, void *key)
 	mktme_map->mapped_keyids++;
 }
 
+extern unsigned long *mktme_bitmap_user_type;
 void mktme_map_free_keyid(int keyid)
 {
 	mktme_map->key[keyid] = 0;
 	mktme_map->mapped_keyids--;
+	clear_bit(keyid, mktme_bitmap_user_type);
 }
 
 int mktme_map_keyid_from_key(void *key)
diff --git a/security/keys/mktme_keys.c b/security/keys/mktme_keys.c
index 7f113146acf2..e9c7d306cba1 100644
--- a/security/keys/mktme_keys.c
+++ b/security/keys/mktme_keys.c
@@ -23,6 +23,11 @@
 struct kmem_cache *mktme_prog_cache;	/* Hardware programming cache */
 cpumask_var_t mktme_leadcpus;		/* one cpu per pkg to program keys */
 
+/* Kernel command line parameter allows saving of users key payload. */
+static bool mktme_savekeys;
+/* Track the existence of user type keys to make package hotplug decisions. */
+unsigned long *mktme_bitmap_user_type;
+
 static const char * const mktme_program_err[] = {
 	"KeyID was successfully programmed",	/* 0 */
 	"Invalid KeyID programming command",	/* 1 */
@@ -54,6 +59,9 @@ struct mktme_payload {
 	u8		tweak_key[MKTME_AES_XTS_SIZE];
 };
 
+/* Store keys in this vault if cmdline parameter mktme_savekeys allows */
+struct mktme_payload *mktme_vault;
+
 /* Key Service Method called when Key is garbage collected. */
 static void mktme_destroy_key(struct key *key)
 {
@@ -121,6 +129,23 @@ static int mktme_program_keyid(int keyid, struct mktme_payload *payload)
 	return ret;
 }
 
+static void mktme_load_vault(int keyid, struct mktme_payload *payload)
+{
+	/*
+	 * Always save the control fields to program hotplugged
+	 * packages with RANDOM, CLEAR, or NO_ENCRYPT type keys.
+	 */
+	mktme_vault[keyid].keyid_ctrl = payload->keyid_ctrl;
+
+	/* Only save data and tweak keys if allowed */
+	if (mktme_savekeys) {
+		memcpy(mktme_vault[keyid].data_key, payload->data_key,
+		       MKTME_AES_XTS_SIZE);
+		memcpy(mktme_vault[keyid].tweak_key, payload->tweak_key,
+		       MKTME_AES_XTS_SIZE);
+	}
+}
+
 /* Key Service Method to update an existing key. */
 static int mktme_update_key(struct key *key,
 			    struct key_preparsed_payload *prep)
@@ -144,11 +169,23 @@ static int mktme_update_key(struct key *key,
 			 keyid, ref_count);
 		return -EBUSY;
 	}
+
+	/* Forget if key was user type. */
+	clear_bit(keyid, mktme_bitmap_user_type);
+
 	ret = mktme_program_keyid(keyid, payload);
 	if (ret != MKTME_PROG_SUCCESS) {
 		pr_debug("%s: %s\n", __func__, mktme_program_err[ret]);
 		ret = -ENOKEY;
+		goto out;
 	}
+
+	mktme_load_vault(keyid, payload);
+
+	/* Remember if this key is user type. */
+	if ((payload->keyid_ctrl & 0xff) == MKTME_KEYID_SET_KEY_DIRECT)
+		set_bit(keyid, mktme_bitmap_user_type);
+out:
 	mktme_map_unlock();
 	return ret;
 }
@@ -171,6 +208,13 @@ int mktme_instantiate_key(struct key *key, struct key_preparsed_payload *prep)
 		ret = -ENOKEY;
 		goto out;
 	}
+
+	mktme_load_vault(keyid, payload);
+
+	/* Remember if key is user type. */
+	if ((payload->keyid_ctrl & 0xff) == MKTME_KEYID_SET_KEY_DIRECT)
+		set_bit(keyid, mktme_bitmap_user_type);
+
 	mktme_map_set_keyid(keyid, key);
 	key_get_encrypt_ref(keyid);
 out:
@@ -380,10 +424,23 @@ static int __init init_mktme(void)
 	if (mktme_build_leadcpus_mask() < 0)
 		goto free_array;
 
+	mktme_bitmap_user_type = bitmap_zalloc(mktme_nr_keyids, GFP_KERNEL);
+	if (!mktme_bitmap_user_type)
+		goto free_mask;
+
+	mktme_vault = kzalloc(sizeof(mktme_vault[0]) * (mktme_nr_keyids + 1),
+			      GFP_KERNEL);
+	if (!mktme_vault)
+		goto free_bitmap;
+
 	ret = register_key_type(&key_type_mktme);
 	if (!ret)
 		return ret;			/* SUCCESS */
 
+	kfree(mktme_vault);
+free_bitmap:
+	bitmap_free(mktme_bitmap_user_type);
+free_mask:
 	free_cpumask_var(mktme_leadcpus);
 free_array:
 	mktme_free_encrypt_array();
@@ -396,3 +453,11 @@ static int __init init_mktme(void)
 }
 
 late_initcall(init_mktme);
+
+static int mktme_enable_savekeys(char *__unused)
+{
+	mktme_savekeys = true;
+	return 1;
+}
+__setup("mktme_savekeys", mktme_enable_savekeys);
+
-- 
2.14.1

[RFC v2 13/13] keys/mktme: Support CPU Hotplug for MKTME keys

[Alison Schofield]

The MKTME (Multi-Key Memory Encryption Keys) hardware resides on each
physical package. The kernel maintains one Key Table on each physical
package and these Key Tables must all remain in sync.
(From here on, package means physical package.)

Although every CPU on that package has the ability to program the Key
Table, the kernel uses one 'lead' cpu per package to program the Key
Tables. Typically, keys are programmed one at a time, across all
packages, as the 'add key' requests come in from userspace.

Some CPU hotplug scenarios are handled quite simply:
>  Teardown a non lead CPU --> do nothing
>  Teardown a lead CPU --> pick a new lead CPU
>  Teardown a lead/last CPU of a package --> forget this package
>  Startup a CPU in a known package --> do nothing
>  Startup a CPU in a new package and no keys are programmed --> do nothing

Then there is the more interesting case for MKTME: a CPU is starting up
for a new package and keys are programmed on the existing packages. The Key
Table on the new package will need to be programmed to match the Key
Tables on all existing packages.

The issue is whether or not the Key Service has the information it needs
to program the new Key Table. To address this, a new kernel commandline
parameter 'mktme_savekeys' was introduced in a previous patch. It allows
the kernel to save the data needed to program keys, beyond their first
add key request.

When 'mktme_savekeys' is not present, new packages may still be added
if all currently programmed keys are not USER type. This means that
CPU generated keys are an option for users not wanting to save key
data, but who also want to support the addition of new packages.

Change-Id: I219192fc59dd9f433963c4959f33d7f013c9f73a
Signed-off-by: Alison Schofield <alison.schofield@intel.com>
Signed-off-by: Kirill A. Shutemov <kirill.shutemov@linux.intel.com>
---
 security/keys/mktme_keys.c | 135 ++++++++++++++++++++++++++++++++++++++++++---
 1 file changed, 126 insertions(+), 9 deletions(-)

diff --git a/security/keys/mktme_keys.c b/security/keys/mktme_keys.c
index e9c7d306cba1..fb4d4061d2f3 100644
--- a/security/keys/mktme_keys.c
+++ b/security/keys/mktme_keys.c
@@ -86,21 +86,29 @@ static void mktme_program_package(void *hw_program_info)
 
 /* Program a KeyID across the entire system. */
 static int mktme_program_system(struct mktme_key_program *key_program,
-				cpumask_var_t mktme_cpumask)
+				cpumask_var_t mktme_cpumask, int hotplug)
 {
 	struct mktme_hw_program_info info = {
 		.key_program = key_program,
 		.status = MKTME_PROG_SUCCESS,
 	};
-	get_online_cpus();
-	on_each_cpu_mask(mktme_cpumask, mktme_program_package, &info, 1);
-	put_online_cpus();
+
+	if (!hotplug) {
+		get_online_cpus();
+		on_each_cpu_mask(mktme_cpumask, mktme_program_package,
+				 &info, 1);
+		put_online_cpus();
+	} else {
+		on_each_cpu_mask(mktme_cpumask, mktme_program_package,
+				 &info, 1);
+	}
 
 	return info.status;
 }
 
 /* Copy the payload to the HW programming structure and program this KeyID */
-static int mktme_program_keyid(int keyid, struct mktme_payload *payload)
+static int mktme_program_keyid(int keyid, struct mktme_payload *payload,
+			       cpumask_var_t mask, int hotplug)
 {
 	struct mktme_key_program *kprog = NULL;
 	u8 kern_entropy[MKTME_AES_XTS_SIZE];
@@ -124,7 +132,7 @@ static int mktme_program_keyid(int keyid, struct mktme_payload *payload)
 			kprog->key_field_2[i] ^= kern_entropy[i];
 		}
 	}
-	ret = mktme_program_system(kprog, mktme_leadcpus);
+	ret = mktme_program_system(kprog, mktme_leadcpus, hotplug);
 	kmem_cache_free(mktme_prog_cache, kprog);
 	return ret;
 }
@@ -173,7 +181,7 @@ static int mktme_update_key(struct key *key,
 	/* Forget if key was user type. */
 	clear_bit(keyid, mktme_bitmap_user_type);
 
-	ret = mktme_program_keyid(keyid, payload);
+	ret = mktme_program_keyid(keyid, payload, mktme_leadcpus, 0);
 	if (ret != MKTME_PROG_SUCCESS) {
 		pr_debug("%s: %s\n", __func__, mktme_program_err[ret]);
 		ret = -ENOKEY;
@@ -202,7 +210,7 @@ int mktme_instantiate_key(struct key *key, struct key_preparsed_payload *prep)
 		ret = -ENOKEY;
 		goto out;
 	}
-	ret = mktme_program_keyid(keyid, payload);
+	ret = mktme_program_keyid(keyid, payload, mktme_leadcpus, 0);
 	if (ret != MKTME_PROG_SUCCESS) {
 		pr_debug("%s: %s\n", __func__, mktme_program_err[ret]);
 		ret = -ENOKEY;
@@ -375,6 +383,10 @@ struct key_type key_type_mktme = {
 	.destroy	= mktme_destroy_key,
 };
 
+/*
+ * Build mktme_leadcpus mask to include one cpu per physical package.
+ * The mask is used to program the Key Table on each physical package.
+ */
 static int mktme_build_leadcpus_mask(void)
 {
 	int online_cpu, mktme_cpu;
@@ -397,6 +409,102 @@ static int mktme_build_leadcpus_mask(void)
 	return 0;
 }
 
+/* A new packages Key Table is programmed with data saved in mktme_vault. */
+static int mktme_program_new_package(cpumask_var_t mask)
+{
+	struct key *key;
+	int hotplug = 1;
+	int keyid, ret;
+
+	/* When a KeyID slot is freed, it's corresponding Key is 0 */
+	for (keyid = 1; keyid <= mktme_nr_keyids; keyid++) {
+		key = mktme_map_key_from_keyid(keyid);
+		if (!key)
+			continue;
+		/* If one key fails to program, fail the entire package. */
+		ret = mktme_program_keyid(keyid, &mktme_vault[keyid],
+					  mask, hotplug);
+		if (ret != MKTME_PROG_SUCCESS) {
+			pr_debug("%s: %s\n", __func__, mktme_program_err[ret]);
+			ret = -ENOKEY;
+			break;
+		}
+	}
+	return ret;
+}
+
+static int mktme_hotplug_cpu_startup(unsigned int cpu)
+{
+	int lead_cpu, ret = 0;
+	cpumask_var_t newmask;
+	int pkgid = topology_physical_package_id(cpu);
+
+	mktme_map_lock();
+
+	/* Nothing to do if a lead CPU exists for this package. */
+	for_each_cpu(lead_cpu, mktme_leadcpus)
+		if (topology_physical_package_id(lead_cpu) = pkgid)
+			goto out_unlock;
+
+	/* No keys to program. Just add the new lead CPU to mask. */
+	if (!mktme_map_mapped_keyids())
+		goto out_add_cpu;
+
+	/* Keys need to be programmed. Confirm programming can be done. */
+	if (!mktme_savekeys &&
+	    (bitmap_weight(mktme_bitmap_user_type, mktme_nr_keyids))) {
+		ret = -EPERM;
+		goto out_unlock;
+	}
+
+	/* Program only this packages Key Table, not all Key Tables. */
+	if (!zalloc_cpumask_var(&newmask, GFP_KERNEL)) {
+		ret = -ENOMEM;
+		goto out_unlock;
+	}
+	cpumask_set_cpu(cpu, newmask);
+	ret = mktme_program_new_package(newmask);
+	if (ret < 0) {
+		free_cpumask_var(newmask);
+		goto out_unlock;
+	}
+
+	free_cpumask_var(newmask);
+out_add_cpu:
+	/* Make this cpu a lead cpu for all future Key programming requests. */
+	cpumask_set_cpu(cpu, mktme_leadcpus);
+out_unlock:
+	mktme_map_unlock();
+	return ret;
+}
+
+static int mktme_hotplug_cpu_teardown(unsigned int cpu)
+{
+	int pkgid, online_cpu;
+
+	mktme_map_lock();
+	/* Teardown cpu is not a lead cpu, nothing to do. */
+	if (!cpumask_test_and_clear_cpu(cpu, mktme_leadcpus))
+		goto out;
+	/*
+	 * Teardown cpu is a lead cpu. If the physical package
+	 * is still present, pick a new lead cpu. Beware: the
+	 * teardown cpu is still in the online_cpu mask. Do
+	 * not pick it again.
+	 */
+	pkgid = topology_physical_package_id(cpu);
+	for_each_online_cpu(online_cpu)
+		if (online_cpu != cpu &&
+		    pkgid = topology_physical_package_id(online_cpu)) {
+			cpumask_set_cpu(online_cpu, mktme_leadcpus);
+			break;
+	}
+out:
+	mktme_map_unlock();
+	/* Teardowns always succeed. */
+	return 0;
+}
+
 /*
  * Allocate the global mktme_map structure based on the available keyids.
  * Create a cache for the hardware structure. Initialize the encrypt_count
@@ -405,7 +513,7 @@ static int mktme_build_leadcpus_mask(void)
  */
 static int __init init_mktme(void)
 {
-	int ret;
+	int ret, cpuhp;
 
 	/* Verify keys are present */
 	if (!(mktme_nr_keyids > 0))
@@ -433,10 +541,19 @@ static int __init init_mktme(void)
 	if (!mktme_vault)
 		goto free_bitmap;
 
+	cpuhp = cpuhp_setup_state_nocalls(CPUHP_AP_ONLINE_DYN,
+					  "keys/mktme_keys:online",
+					  mktme_hotplug_cpu_startup,
+					  mktme_hotplug_cpu_teardown);
+	if (cpuhp < 0)
+		goto free_vault;
+
 	ret = register_key_type(&key_type_mktme);
 	if (!ret)
 		return ret;			/* SUCCESS */
 
+	cpuhp_remove_state_nocalls(cpuhp);
+free_vault:
 	kfree(mktme_vault);
 free_bitmap:
 	bitmap_free(mktme_bitmap_user_type);
-- 
2.14.1

[RFC v2 13/13] keys/mktme: Support CPU Hotplug for MKTME keys

[Alison Schofield]

The MKTME (Multi-Key Memory Encryption Keys) hardware resides on each
physical package. The kernel maintains one Key Table on each physical
package and these Key Tables must all remain in sync.
(From here on, package means physical package.)

Although every CPU on that package has the ability to program the Key
Table, the kernel uses one 'lead' cpu per package to program the Key
Tables. Typically, keys are programmed one at a time, across all
packages, as the 'add key' requests come in from userspace.

Some CPU hotplug scenarios are handled quite simply:
>  Teardown a non lead CPU --> do nothing
>  Teardown a lead CPU --> pick a new lead CPU
>  Teardown a lead/last CPU of a package --> forget this package
>  Startup a CPU in a known package --> do nothing
>  Startup a CPU in a new package and no keys are programmed --> do nothing

Then there is the more interesting case for MKTME: a CPU is starting up
for a new package and keys are programmed on the existing packages. The Key
Table on the new package will need to be programmed to match the Key
Tables on all existing packages.

The issue is whether or not the Key Service has the information it needs
to program the new Key Table. To address this, a new kernel commandline
parameter 'mktme_savekeys' was introduced in a previous patch. It allows
the kernel to save the data needed to program keys, beyond their first
add key request.

When 'mktme_savekeys' is not present, new packages may still be added
if all currently programmed keys are not USER type. This means that
CPU generated keys are an option for users not wanting to save key
data, but who also want to support the addition of new packages.

Change-Id: I219192fc59dd9f433963c4959f33d7f013c9f73a
Signed-off-by: Alison Schofield <alison.schofield@intel.com>
Signed-off-by: Kirill A. Shutemov <kirill.shutemov@linux.intel.com>
---
 security/keys/mktme_keys.c | 135 ++++++++++++++++++++++++++++++++++++++++++---
 1 file changed, 126 insertions(+), 9 deletions(-)

diff --git a/security/keys/mktme_keys.c b/security/keys/mktme_keys.c
index e9c7d306cba1..fb4d4061d2f3 100644
--- a/security/keys/mktme_keys.c
+++ b/security/keys/mktme_keys.c
@@ -86,21 +86,29 @@ static void mktme_program_package(void *hw_program_info)
 
 /* Program a KeyID across the entire system. */
 static int mktme_program_system(struct mktme_key_program *key_program,
-				cpumask_var_t mktme_cpumask)
+				cpumask_var_t mktme_cpumask, int hotplug)
 {
 	struct mktme_hw_program_info info = {
 		.key_program = key_program,
 		.status = MKTME_PROG_SUCCESS,
 	};
-	get_online_cpus();
-	on_each_cpu_mask(mktme_cpumask, mktme_program_package, &info, 1);
-	put_online_cpus();
+
+	if (!hotplug) {
+		get_online_cpus();
+		on_each_cpu_mask(mktme_cpumask, mktme_program_package,
+				 &info, 1);
+		put_online_cpus();
+	} else {
+		on_each_cpu_mask(mktme_cpumask, mktme_program_package,
+				 &info, 1);
+	}
 
 	return info.status;
 }
 
 /* Copy the payload to the HW programming structure and program this KeyID */
-static int mktme_program_keyid(int keyid, struct mktme_payload *payload)
+static int mktme_program_keyid(int keyid, struct mktme_payload *payload,
+			       cpumask_var_t mask, int hotplug)
 {
 	struct mktme_key_program *kprog = NULL;
 	u8 kern_entropy[MKTME_AES_XTS_SIZE];
@@ -124,7 +132,7 @@ static int mktme_program_keyid(int keyid, struct mktme_payload *payload)
 			kprog->key_field_2[i] ^= kern_entropy[i];
 		}
 	}
-	ret = mktme_program_system(kprog, mktme_leadcpus);
+	ret = mktme_program_system(kprog, mktme_leadcpus, hotplug);
 	kmem_cache_free(mktme_prog_cache, kprog);
 	return ret;
 }
@@ -173,7 +181,7 @@ static int mktme_update_key(struct key *key,
 	/* Forget if key was user type. */
 	clear_bit(keyid, mktme_bitmap_user_type);
 
-	ret = mktme_program_keyid(keyid, payload);
+	ret = mktme_program_keyid(keyid, payload, mktme_leadcpus, 0);
 	if (ret != MKTME_PROG_SUCCESS) {
 		pr_debug("%s: %s\n", __func__, mktme_program_err[ret]);
 		ret = -ENOKEY;
@@ -202,7 +210,7 @@ int mktme_instantiate_key(struct key *key, struct key_preparsed_payload *prep)
 		ret = -ENOKEY;
 		goto out;
 	}
-	ret = mktme_program_keyid(keyid, payload);
+	ret = mktme_program_keyid(keyid, payload, mktme_leadcpus, 0);
 	if (ret != MKTME_PROG_SUCCESS) {
 		pr_debug("%s: %s\n", __func__, mktme_program_err[ret]);
 		ret = -ENOKEY;
@@ -375,6 +383,10 @@ struct key_type key_type_mktme = {
 	.destroy	= mktme_destroy_key,
 };
 
+/*
+ * Build mktme_leadcpus mask to include one cpu per physical package.
+ * The mask is used to program the Key Table on each physical package.
+ */
 static int mktme_build_leadcpus_mask(void)
 {
 	int online_cpu, mktme_cpu;
@@ -397,6 +409,102 @@ static int mktme_build_leadcpus_mask(void)
 	return 0;
 }
 
+/* A new packages Key Table is programmed with data saved in mktme_vault. */
+static int mktme_program_new_package(cpumask_var_t mask)
+{
+	struct key *key;
+	int hotplug = 1;
+	int keyid, ret;
+
+	/* When a KeyID slot is freed, it's corresponding Key is 0 */
+	for (keyid = 1; keyid <= mktme_nr_keyids; keyid++) {
+		key = mktme_map_key_from_keyid(keyid);
+		if (!key)
+			continue;
+		/* If one key fails to program, fail the entire package. */
+		ret = mktme_program_keyid(keyid, &mktme_vault[keyid],
+					  mask, hotplug);
+		if (ret != MKTME_PROG_SUCCESS) {
+			pr_debug("%s: %s\n", __func__, mktme_program_err[ret]);
+			ret = -ENOKEY;
+			break;
+		}
+	}
+	return ret;
+}
+
+static int mktme_hotplug_cpu_startup(unsigned int cpu)
+{
+	int lead_cpu, ret = 0;
+	cpumask_var_t newmask;
+	int pkgid = topology_physical_package_id(cpu);
+
+	mktme_map_lock();
+
+	/* Nothing to do if a lead CPU exists for this package. */
+	for_each_cpu(lead_cpu, mktme_leadcpus)
+		if (topology_physical_package_id(lead_cpu) == pkgid)
+			goto out_unlock;
+
+	/* No keys to program. Just add the new lead CPU to mask. */
+	if (!mktme_map_mapped_keyids())
+		goto out_add_cpu;
+
+	/* Keys need to be programmed. Confirm programming can be done. */
+	if (!mktme_savekeys &&
+	    (bitmap_weight(mktme_bitmap_user_type, mktme_nr_keyids))) {
+		ret = -EPERM;
+		goto out_unlock;
+	}
+
+	/* Program only this packages Key Table, not all Key Tables. */
+	if (!zalloc_cpumask_var(&newmask, GFP_KERNEL)) {
+		ret = -ENOMEM;
+		goto out_unlock;
+	}
+	cpumask_set_cpu(cpu, newmask);
+	ret = mktme_program_new_package(newmask);
+	if (ret < 0) {
+		free_cpumask_var(newmask);
+		goto out_unlock;
+	}
+
+	free_cpumask_var(newmask);
+out_add_cpu:
+	/* Make this cpu a lead cpu for all future Key programming requests. */
+	cpumask_set_cpu(cpu, mktme_leadcpus);
+out_unlock:
+	mktme_map_unlock();
+	return ret;
+}
+
+static int mktme_hotplug_cpu_teardown(unsigned int cpu)
+{
+	int pkgid, online_cpu;
+
+	mktme_map_lock();
+	/* Teardown cpu is not a lead cpu, nothing to do. */
+	if (!cpumask_test_and_clear_cpu(cpu, mktme_leadcpus))
+		goto out;
+	/*
+	 * Teardown cpu is a lead cpu. If the physical package
+	 * is still present, pick a new lead cpu. Beware: the
+	 * teardown cpu is still in the online_cpu mask. Do
+	 * not pick it again.
+	 */
+	pkgid = topology_physical_package_id(cpu);
+	for_each_online_cpu(online_cpu)
+		if (online_cpu != cpu &&
+		    pkgid == topology_physical_package_id(online_cpu)) {
+			cpumask_set_cpu(online_cpu, mktme_leadcpus);
+			break;
+	}
+out:
+	mktme_map_unlock();
+	/* Teardowns always succeed. */
+	return 0;
+}
+
 /*
  * Allocate the global mktme_map structure based on the available keyids.
  * Create a cache for the hardware structure. Initialize the encrypt_count
@@ -405,7 +513,7 @@ static int mktme_build_leadcpus_mask(void)
  */
 static int __init init_mktme(void)
 {
-	int ret;
+	int ret, cpuhp;
 
 	/* Verify keys are present */
 	if (!(mktme_nr_keyids > 0))
@@ -433,10 +541,19 @@ static int __init init_mktme(void)
 	if (!mktme_vault)
 		goto free_bitmap;
 
+	cpuhp = cpuhp_setup_state_nocalls(CPUHP_AP_ONLINE_DYN,
+					  "keys/mktme_keys:online",
+					  mktme_hotplug_cpu_startup,
+					  mktme_hotplug_cpu_teardown);
+	if (cpuhp < 0)
+		goto free_vault;
+
 	ret = register_key_type(&key_type_mktme);
 	if (!ret)
 		return ret;			/* SUCCESS */
 
+	cpuhp_remove_state_nocalls(cpuhp);
+free_vault:
 	kfree(mktme_vault);
 free_bitmap:
 	bitmap_free(mktme_bitmap_user_type);
-- 
2.14.1

Re: [RFC v2 07/13] x86/mm: Add helpers for reference counting encrypted VMAs

[Peter Zijlstra]

On Mon, Dec 03, 2018 at 11:39:54PM -0800, Alison Schofield wrote:

> +void vma_put_encrypt_ref(struct vm_area_struct *vma)
> +{
> +	if (vma_keyid(vma))
> +		if (refcount_dec_and_test(&encrypt_count[vma_keyid(vma)])) {
> +			mktme_map_lock();
> +			mktme_map_free_keyid(vma_keyid(vma));
> +			mktme_map_unlock();
> +		}

This violates CodingStyle

> +}

> +void key_put_encrypt_ref(int keyid)
> +{
> +	if (refcount_dec_and_test(&encrypt_count[keyid])) {
> +		mktme_map_lock();

That smells like it wants to use refcount_dec_and_lock() instead.

> +		mktme_map_free_keyid(keyid);
> +		mktme_map_unlock();
> +	}
> +}

Also, if you write that like:

	if (!refcount_dec_and_lock(&encrypt_count[keyid], &lock))
		return;

you loose an indent level.

Re: [RFC v2 07/13] x86/mm: Add helpers for reference counting encrypted VMAs

[Peter Zijlstra]

On Mon, Dec 03, 2018 at 11:39:54PM -0800, Alison Schofield wrote:

> +void vma_put_encrypt_ref(struct vm_area_struct *vma)
> +{
> +	if (vma_keyid(vma))
> +		if (refcount_dec_and_test(&encrypt_count[vma_keyid(vma)])) {
> +			mktme_map_lock();
> +			mktme_map_free_keyid(vma_keyid(vma));
> +			mktme_map_unlock();
> +		}

This violates CodingStyle

> +}

> +void key_put_encrypt_ref(int keyid)
> +{
> +	if (refcount_dec_and_test(&encrypt_count[keyid])) {
> +		mktme_map_lock();

That smells like it wants to use refcount_dec_and_lock() instead.

> +		mktme_map_free_keyid(keyid);
> +		mktme_map_unlock();
> +	}
> +}

Also, if you write that like:

	if (!refcount_dec_and_lock(&encrypt_count[keyid], &lock))
		return;

you loose an indent level.

Re: [RFC v2 09/13] mm: Restrict memory encryption to anonymous VMA's

[Peter Zijlstra]

On Mon, Dec 03, 2018 at 11:39:56PM -0800, Alison Schofield wrote:
> Memory encryption is only supported for mappings that are ANONYMOUS.
> Test the entire range of VMA's in an encrypt_mprotect() request to
> make sure they all meet that requirement before encrypting any.
> 
> The encrypt_mprotect syscall will return -EINVAL and will not encrypt
> any VMA's if this check fails.
> 
> Signed-off-by: Alison Schofield <alison.schofield@intel.com>
> Signed-off-by: Kirill A. Shutemov <kirill.shutemov@linux.intel.com>

That SoB doesn't make sense; per the From you wrote the patch and signed
off on it, wth is Kirill's SoB doing there?

> diff --git a/mm/mprotect.c b/mm/mprotect.c
> index ad8127dc9aac..f1c009409134 100644
> --- a/mm/mprotect.c
> +++ b/mm/mprotect.c
> @@ -345,6 +345,24 @@ static int prot_none_walk(struct vm_area_struct *vma, unsigned long start,
>  	return walk_page_range(start, end, &prot_none_walk);
>  }
>  
> +/*
> + * Encrypted mprotect is only supported on anonymous mappings.
> + * All VMA's in the requested range must be anonymous. If this
> + * test fails on any single VMA, the entire mprotect request fails.
> + */
> +bool mem_supports_encryption(struct vm_area_struct *vma, unsigned long end)

That's a 'weird' interface and cannot do what the comment says it should
do.

> +{
> +	struct vm_area_struct *test_vma = vma;

That variable is utterly pointless.

> +	do {
> +		if (!vma_is_anonymous(test_vma))
> +			return false;
> +
> +		test_vma = test_vma->vm_next;
> +	} while (test_vma && test_vma->vm_start < end);
> +	return true;
> +}

Re: [RFC v2 09/13] mm: Restrict memory encryption to anonymous VMA's

[Peter Zijlstra]

On Mon, Dec 03, 2018 at 11:39:56PM -0800, Alison Schofield wrote:
> Memory encryption is only supported for mappings that are ANONYMOUS.
> Test the entire range of VMA's in an encrypt_mprotect() request to
> make sure they all meet that requirement before encrypting any.
> 
> The encrypt_mprotect syscall will return -EINVAL and will not encrypt
> any VMA's if this check fails.
> 
> Signed-off-by: Alison Schofield <alison.schofield@intel.com>
> Signed-off-by: Kirill A. Shutemov <kirill.shutemov@linux.intel.com>

That SoB doesn't make sense; per the From you wrote the patch and signed
off on it, wth is Kirill's SoB doing there?

> diff --git a/mm/mprotect.c b/mm/mprotect.c
> index ad8127dc9aac..f1c009409134 100644
> --- a/mm/mprotect.c
> +++ b/mm/mprotect.c
> @@ -345,6 +345,24 @@ static int prot_none_walk(struct vm_area_struct *vma, unsigned long start,
>  	return walk_page_range(start, end, &prot_none_walk);
>  }
>  
> +/*
> + * Encrypted mprotect is only supported on anonymous mappings.
> + * All VMA's in the requested range must be anonymous. If this
> + * test fails on any single VMA, the entire mprotect request fails.
> + */
> +bool mem_supports_encryption(struct vm_area_struct *vma, unsigned long end)

That's a 'weird' interface and cannot do what the comment says it should
do.

> +{
> +	struct vm_area_struct *test_vma = vma;

That variable is utterly pointless.

> +	do {
> +		if (!vma_is_anonymous(test_vma))
> +			return false;
> +
> +		test_vma = test_vma->vm_next;
> +	} while (test_vma && test_vma->vm_start < end);
> +	return true;
> +}

Re: [RFC v2 04/13] x86/mm: Add helper functions for MKTME memory encryption keys

[Peter Zijlstra]

On Mon, Dec 03, 2018 at 11:39:51PM -0800, Alison Schofield wrote:
> +int mktme_map_keyid_from_key(void *key)
> +{
> +	int i;
> +
> +	for (i = 1; i <= mktme_nr_keyids; i++)
> +		if (mktme_map->key[i] = key)
> +			return i;

CodingStyle

> +	return 0;
> +}
> +int mktme_map_get_free_keyid(void)
> +{
> +	int i;
> +
> +	if (mktme_map->mapped_keyids < mktme_nr_keyids) {
> +		for (i = 1; i <= mktme_nr_keyids; i++)
> +			if (mktme_map->key[i] = 0)
> +				return i;

CodingStyle

> +	}
> +	return 0;
> +}

Re: [RFC v2 04/13] x86/mm: Add helper functions for MKTME memory encryption keys

[Peter Zijlstra]

On Mon, Dec 03, 2018 at 11:39:51PM -0800, Alison Schofield wrote:
> +int mktme_map_keyid_from_key(void *key)
> +{
> +	int i;
> +
> +	for (i = 1; i <= mktme_nr_keyids; i++)
> +		if (mktme_map->key[i] == key)
> +			return i;

CodingStyle

> +	return 0;
> +}
> +int mktme_map_get_free_keyid(void)
> +{
> +	int i;
> +
> +	if (mktme_map->mapped_keyids < mktme_nr_keyids) {
> +		for (i = 1; i <= mktme_nr_keyids; i++)
> +			if (mktme_map->key[i] == 0)
> +				return i;

CodingStyle

> +	}
> +	return 0;
> +}

Re: [RFC v2 11/13] keys/mktme: Program memory encryption keys on a system wide basis

[Peter Zijlstra]

On Mon, Dec 03, 2018 at 11:39:58PM -0800, Alison Schofield wrote:

> +struct mktme_hw_program_info {
> +	struct mktme_key_program *key_program;
> +	unsigned long status;
> +};
> +
> +/* Program a KeyID on a single package. */
> +static void mktme_program_package(void *hw_program_info)
> +{
> +	struct mktme_hw_program_info *info = hw_program_info;
> +	int ret;
> +
> +	ret = mktme_key_program(info->key_program);
> +	if (ret != MKTME_PROG_SUCCESS)
> +		WRITE_ONCE(info->status, ret);

What's the purpose of that WRITE_ONCE()?

> +}
> +
> +/* Program a KeyID across the entire system. */
> +static int mktme_program_system(struct mktme_key_program *key_program,
> +				cpumask_var_t mktme_cpumask)
> +{
> +	struct mktme_hw_program_info info = {
> +		.key_program = key_program,
> +		.status = MKTME_PROG_SUCCESS,
> +	};
> +	get_online_cpus();
> +	on_each_cpu_mask(mktme_cpumask, mktme_program_package, &info, 1);
> +	put_online_cpus();
> +
> +	return info.status;
> +}
> +
>  /* Copy the payload to the HW programming structure and program this KeyID */
>  static int mktme_program_keyid(int keyid, struct mktme_payload *payload)
>  {
> @@ -84,7 +116,7 @@ static int mktme_program_keyid(int keyid, struct mktme_payload *payload)
>  			kprog->key_field_2[i] ^= kern_entropy[i];
>  		}
>  	}
> -	ret = mktme_key_program(kprog);
> +	ret = mktme_program_system(kprog, mktme_leadcpus);
>  	kmem_cache_free(mktme_prog_cache, kprog);
>  	return ret;
>  }
> @@ -299,6 +331,28 @@ struct key_type key_type_mktme = {
>  	.destroy	= mktme_destroy_key,
>  };
>  
> +static int mktme_build_leadcpus_mask(void)
> +{
> +	int online_cpu, mktme_cpu;
> +	int online_pkgid, mktme_pkgid = -1;
> +
> +	if (!zalloc_cpumask_var(&mktme_leadcpus, GFP_KERNEL))
> +		return -ENOMEM;
> +
> +	for_each_online_cpu(online_cpu) {
> +		online_pkgid = topology_physical_package_id(online_cpu);
> +
> +		for_each_cpu(mktme_cpu, mktme_leadcpus) {
> +			mktme_pkgid = topology_physical_package_id(mktme_cpu);
> +			if (mktme_pkgid = online_pkgid)
> +				break;
> +		}
> +		if (mktme_pkgid != online_pkgid)
> +			cpumask_set_cpu(online_cpu, mktme_leadcpus);

Do you really need LOCK prefixed bit set here?

> +	}
> +	return 0;
> +}

How is that serialized and kept relevant in the face of hotplug?

Also, do you really need O(n^2) to find the first occurence of a value
in an array?

Re: [RFC v2 11/13] keys/mktme: Program memory encryption keys on a system wide basis

[Peter Zijlstra]

On Mon, Dec 03, 2018 at 11:39:58PM -0800, Alison Schofield wrote:

> +struct mktme_hw_program_info {
> +	struct mktme_key_program *key_program;
> +	unsigned long status;
> +};
> +
> +/* Program a KeyID on a single package. */
> +static void mktme_program_package(void *hw_program_info)
> +{
> +	struct mktme_hw_program_info *info = hw_program_info;
> +	int ret;
> +
> +	ret = mktme_key_program(info->key_program);
> +	if (ret != MKTME_PROG_SUCCESS)
> +		WRITE_ONCE(info->status, ret);

What's the purpose of that WRITE_ONCE()?

> +}
> +
> +/* Program a KeyID across the entire system. */
> +static int mktme_program_system(struct mktme_key_program *key_program,
> +				cpumask_var_t mktme_cpumask)
> +{
> +	struct mktme_hw_program_info info = {
> +		.key_program = key_program,
> +		.status = MKTME_PROG_SUCCESS,
> +	};
> +	get_online_cpus();
> +	on_each_cpu_mask(mktme_cpumask, mktme_program_package, &info, 1);
> +	put_online_cpus();
> +
> +	return info.status;
> +}
> +
>  /* Copy the payload to the HW programming structure and program this KeyID */
>  static int mktme_program_keyid(int keyid, struct mktme_payload *payload)
>  {
> @@ -84,7 +116,7 @@ static int mktme_program_keyid(int keyid, struct mktme_payload *payload)
>  			kprog->key_field_2[i] ^= kern_entropy[i];
>  		}
>  	}
> -	ret = mktme_key_program(kprog);
> +	ret = mktme_program_system(kprog, mktme_leadcpus);
>  	kmem_cache_free(mktme_prog_cache, kprog);
>  	return ret;
>  }
> @@ -299,6 +331,28 @@ struct key_type key_type_mktme = {
>  	.destroy	= mktme_destroy_key,
>  };
>  
> +static int mktme_build_leadcpus_mask(void)
> +{
> +	int online_cpu, mktme_cpu;
> +	int online_pkgid, mktme_pkgid = -1;
> +
> +	if (!zalloc_cpumask_var(&mktme_leadcpus, GFP_KERNEL))
> +		return -ENOMEM;
> +
> +	for_each_online_cpu(online_cpu) {
> +		online_pkgid = topology_physical_package_id(online_cpu);
> +
> +		for_each_cpu(mktme_cpu, mktme_leadcpus) {
> +			mktme_pkgid = topology_physical_package_id(mktme_cpu);
> +			if (mktme_pkgid == online_pkgid)
> +				break;
> +		}
> +		if (mktme_pkgid != online_pkgid)
> +			cpumask_set_cpu(online_cpu, mktme_leadcpus);

Do you really need LOCK prefixed bit set here?

> +	}
> +	return 0;
> +}

How is that serialized and kept relevant in the face of hotplug?

Also, do you really need O(n^2) to find the first occurence of a value
in an array?

Re: [RFC v2 12/13] keys/mktme: Save MKTME data if kernel cmdline parameter allows

[Peter Zijlstra]

On Mon, Dec 03, 2018 at 11:39:59PM -0800, Alison Schofield wrote:
> Change-Id: If57414862f1ac131dd97e29bf4f3937ac33777f6

Does not belong in patches..

Re: [RFC v2 12/13] keys/mktme: Save MKTME data if kernel cmdline parameter allows

[Peter Zijlstra]

On Mon, Dec 03, 2018 at 11:39:59PM -0800, Alison Schofield wrote:
> Change-Id: If57414862f1ac131dd97e29bf4f3937ac33777f6

Does not belong in patches..

Re: [RFC v2 00/13] Multi-Key Total Memory Encryption API (MKTME)

[Peter Zijlstra]

On Mon, Dec 03, 2018 at 11:39:47PM -0800, Alison Schofield wrote:
> (Multi-Key Total Memory Encryption)

I think that MKTME is a horrible name, and doesn't appear to accurately
describe what it does either. Specifically the 'total' seems out of
place, it doesn't require all memory to be encrypted.

Re: [RFC v2 00/13] Multi-Key Total Memory Encryption API (MKTME)

[Peter Zijlstra]

On Mon, Dec 03, 2018 at 11:39:47PM -0800, Alison Schofield wrote:
> (Multi-Key Total Memory Encryption)

I think that MKTME is a horrible name, and doesn't appear to accurately
describe what it does either. Specifically the 'total' seems out of
place, it doesn't require all memory to be encrypted.

Re: [RFC v2 13/13] keys/mktme: Support CPU Hotplug for MKTME keys

[Peter Zijlstra]

On Mon, Dec 03, 2018 at 11:40:00PM -0800, Alison Schofield wrote:
> +	for_each_online_cpu(online_cpu)
> +		if (online_cpu != cpu &&
> +		    pkgid = topology_physical_package_id(online_cpu)) {
> +			cpumask_set_cpu(online_cpu, mktme_leadcpus);
> +			break;
> +	}

That's a capital offence right there.

Re: [RFC v2 13/13] keys/mktme: Support CPU Hotplug for MKTME keys

[Peter Zijlstra]

On Mon, Dec 03, 2018 at 11:40:00PM -0800, Alison Schofield wrote:
> +	for_each_online_cpu(online_cpu)
> +		if (online_cpu != cpu &&
> +		    pkgid == topology_physical_package_id(online_cpu)) {
> +			cpumask_set_cpu(online_cpu, mktme_leadcpus);
> +			break;
> +	}

That's a capital offence right there.

Re: [RFC v2 13/13] keys/mktme: Support CPU Hotplug for MKTME keys

[Peter Zijlstra]

On Mon, Dec 03, 2018 at 11:40:00PM -0800, Alison Schofield wrote:
>  static int mktme_program_system(struct mktme_key_program *key_program,
> -				cpumask_var_t mktme_cpumask)
> +				cpumask_var_t mktme_cpumask, int hotplug)
>  {
>  	struct mktme_hw_program_info info = {
>  		.key_program = key_program,
>  		.status = MKTME_PROG_SUCCESS,
>  	};
> -	get_online_cpus();
> -	on_each_cpu_mask(mktme_cpumask, mktme_program_package, &info, 1);
> -	put_online_cpus();
> +
> +	if (!hotplug) {
> +		get_online_cpus();
> +		on_each_cpu_mask(mktme_cpumask, mktme_program_package,
> +				 &info, 1);
> +		put_online_cpus();
> +	} else {
> +		on_each_cpu_mask(mktme_cpumask, mktme_program_package,
> +				 &info, 1);
> +	}
>  
>  	return info.status;
>  }

That is pretty horrible; and I think easily avoided.

Re: [RFC v2 13/13] keys/mktme: Support CPU Hotplug for MKTME keys

[Peter Zijlstra]

On Mon, Dec 03, 2018 at 11:40:00PM -0800, Alison Schofield wrote:
>  static int mktme_program_system(struct mktme_key_program *key_program,
> -				cpumask_var_t mktme_cpumask)
> +				cpumask_var_t mktme_cpumask, int hotplug)
>  {
>  	struct mktme_hw_program_info info = {
>  		.key_program = key_program,
>  		.status = MKTME_PROG_SUCCESS,
>  	};
> -	get_online_cpus();
> -	on_each_cpu_mask(mktme_cpumask, mktme_program_package, &info, 1);
> -	put_online_cpus();
> +
> +	if (!hotplug) {
> +		get_online_cpus();
> +		on_each_cpu_mask(mktme_cpumask, mktme_program_package,
> +				 &info, 1);
> +		put_online_cpus();
> +	} else {
> +		on_each_cpu_mask(mktme_cpumask, mktme_program_package,
> +				 &info, 1);
> +	}
>  
>  	return info.status;
>  }

That is pretty horrible; and I think easily avoided.

Re: [RFC v2 00/13] Multi-Key Total Memory Encryption API (MKTME)

[Kirill A. Shutemov]

On Tue, Dec 04, 2018 at 09:25:50AM +0000, Peter Zijlstra wrote:
> On Mon, Dec 03, 2018 at 11:39:47PM -0800, Alison Schofield wrote:
> > (Multi-Key Total Memory Encryption)
> 
> I think that MKTME is a horrible name, and doesn't appear to accurately
> describe what it does either. Specifically the 'total' seems out of
> place, it doesn't require all memory to be encrypted.

MKTME implies TME. TME is enabled by BIOS and it encrypts all memory with
CPU-generated key. MKTME allows to use other keys or disable encryption
for a page.

But, yes, name is not good.

-- 
 Kirill A. Shutemov

Re: [RFC v2 00/13] Multi-Key Total Memory Encryption API (MKTME)

[Kirill A. Shutemov]

On Tue, Dec 04, 2018 at 09:25:50AM +0000, Peter Zijlstra wrote:
> On Mon, Dec 03, 2018 at 11:39:47PM -0800, Alison Schofield wrote:
> > (Multi-Key Total Memory Encryption)
> 
> I think that MKTME is a horrible name, and doesn't appear to accurately
> describe what it does either. Specifically the 'total' seems out of
> place, it doesn't require all memory to be encrypted.

MKTME implies TME. TME is enabled by BIOS and it encrypts all memory with
CPU-generated key. MKTME allows to use other keys or disable encryption
for a page.

But, yes, name is not good.

-- 
 Kirill A. Shutemov

Re: [RFC v2 11/13] keys/mktme: Program memory encryption keys on a system wide basis

[Kirill A. Shutemov]

On Tue, Dec 04, 2018 at 09:21:45AM +0000, Peter Zijlstra wrote:
> On Mon, Dec 03, 2018 at 11:39:58PM -0800, Alison Schofield wrote:
> 
> > +struct mktme_hw_program_info {
> > +	struct mktme_key_program *key_program;
> > +	unsigned long status;
> > +};
> > +
> > +/* Program a KeyID on a single package. */
> > +static void mktme_program_package(void *hw_program_info)
> > +{
> > +	struct mktme_hw_program_info *info = hw_program_info;
> > +	int ret;
> > +
> > +	ret = mktme_key_program(info->key_program);
> > +	if (ret != MKTME_PROG_SUCCESS)
> > +		WRITE_ONCE(info->status, ret);
> 
> What's the purpose of that WRITE_ONCE()?

[I suggested the code to Alison.]

Yes, you're right. Simple assignment will do.

-- 
 Kirill A. Shutemov

Re: [RFC v2 11/13] keys/mktme: Program memory encryption keys on a system wide basis

[Kirill A. Shutemov]

On Tue, Dec 04, 2018 at 09:21:45AM +0000, Peter Zijlstra wrote:
> On Mon, Dec 03, 2018 at 11:39:58PM -0800, Alison Schofield wrote:
> 
> > +struct mktme_hw_program_info {
> > +	struct mktme_key_program *key_program;
> > +	unsigned long status;
> > +};
> > +
> > +/* Program a KeyID on a single package. */
> > +static void mktme_program_package(void *hw_program_info)
> > +{
> > +	struct mktme_hw_program_info *info = hw_program_info;
> > +	int ret;
> > +
> > +	ret = mktme_key_program(info->key_program);
> > +	if (ret != MKTME_PROG_SUCCESS)
> > +		WRITE_ONCE(info->status, ret);
> 
> What's the purpose of that WRITE_ONCE()?

[I suggested the code to Alison.]

Yes, you're right. Simple assignment will do.

-- 
 Kirill A. Shutemov

Re: [RFC v2 04/13] x86/mm: Add helper functions for MKTME memory encryption keys

[Andy Lutomirski]

> On Dec 3, 2018, at 11:39 PM, Alison Schofield <alison.schofield@intel.com> wrote:
> 
> Define a global mapping structure to manage the mapping of userspace
> Keys to hardware KeyIDs in MKTME (Multi-Key Total Memory Encryption).
> Implement helper functions that access this mapping structure.
> 

Why is a key “void *”?  Who owns the memory?  Can a real type be used?

Re: [RFC v2 04/13] x86/mm: Add helper functions for MKTME memory encryption keys

[Andy Lutomirski]

> On Dec 3, 2018, at 11:39 PM, Alison Schofield <alison.schofield@intel.com> wrote:
> 
> Define a global mapping structure to manage the mapping of userspace
> Keys to hardware KeyIDs in MKTME (Multi-Key Total Memory Encryption).
> Implement helper functions that access this mapping structure.
> 

Why is a key “void *”?  Who owns the memory?  Can a real type be used?

Re: [RFC v2 00/13] Multi-Key Total Memory Encryption API (MKTME)

[Andy Lutomirski]

On Mon, Dec 3, 2018 at 11:37 PM Alison Schofield
<alison.schofield@intel.com> wrote:
>
> Hi Thomas, David,
>
> Here is an updated RFC on the API's to support MKTME.
> (Multi-Key Total Memory Encryption)
>
> This RFC presents the 2 API additions to support the creation and
> usage of memory encryption keys:
>  1) Kernel Key Service type "mktme"
>  2) System call encrypt_mprotect()
>
> This patchset is built upon Kirill Shutemov's work for the core MKTME
> support.
>
> David: Please let me know if the changes made, based on your review,
> are reasonable. I don't think that the new changes touch key service
> specific areas (much).
>
> Thomas: Please provide feedback on encrypt_mprotect(). If not a
> review, then a direction check would be helpful.
>

I'm not Thomas, but I think it's the wrong direction.  As it stands,
encrypt_mprotect() is an incomplete version of mprotect() (since it's
missing the protection key support), and it's also functionally just
MADV_DONTNEED.  In other words, the sole user-visible effect appears
to be that the existing pages are blown away.  The fact that it
changes the key in use doesn't seem terribly useful, since it's
anonymous memory, and the most secure choice is to use CPU-managed
keying, which appears to be the default anyway on TME systems.  It
also has totally unclear semantics WRT swap, and, off the top of my
head, it looks like it may have serious cache-coherency issues and
like swapping the pages might corrupt them, both because there are no
flushes and because the direct-map alias looks like it will use the
default key and therefore appear to contain the wrong data.

I would propose a very different direction: don't try to support MKTME
at all for anonymous memory, and instead figure out the important use
cases and support them directly.  The use cases that I can think of
off the top of my head are:

1. pmem.  This should probably use a very different API.

2. Some kind of VM hardening, where a VM's memory can be protected a
little tiny bit from the main kernel.  But I don't see why this is any
better than XPO (eXclusive Page-frame Ownership), which brings to
mind:

The main implementation concern I have with this patch set is cache
coherency and handling of the direct map.  Unless I missed something,
you're not doing anything about the direct map, which means that you
have RW aliases of the same memory with different keys.  For use case
#2, this probably means that you need to either get rid of the direct
map and make get_user_pages() fail, or you need to change the key on
the direct map as well, probably using the pageattr.c code.

As for caching, As far as I can tell from reading the preliminary
docs, Intel's MKTME, much like AMD's SME, is basically invisible to
the hardware cache coherency mechanism.  So, if you modify a physical
address with one key (or SME-enable bit), and you read it with
another, you get garbage unless you flush.  And, if you modify memory
with one key then remap it with a different key without flushing in
the mean time, you risk corruption.  And, what's worse, if I'm reading
between the lines in the docs correctly, if you use PCONFIG to change
a key, you may need to do a bunch of cache flushing to ensure you get
reasonable effects.  (If you have dirty cache lines for some (PA, key)
and you PCONFIG to change the underlying key, you get different
results depending on whether the writeback happens before or after the
package doing the writeback notices the PCONFIG.)

Finally, If you're going to teach the kernel how to have some user
pages that aren't in the direct map, you've essentially done XPO,
which is nifty but expensive.  And I think that doing this gets you
essentially all the benefit of MKTME for the non-pmem use case.  Why
exactly would any software want to use anything other than a
CPU-managed key for anything other than pmem?

--Andy

Re: [RFC v2 00/13] Multi-Key Total Memory Encryption API (MKTME)

[Andy Lutomirski]

On Mon, Dec 3, 2018 at 11:37 PM Alison Schofield
<alison.schofield@intel.com> wrote:
>
> Hi Thomas, David,
>
> Here is an updated RFC on the API's to support MKTME.
> (Multi-Key Total Memory Encryption)
>
> This RFC presents the 2 API additions to support the creation and
> usage of memory encryption keys:
>  1) Kernel Key Service type "mktme"
>  2) System call encrypt_mprotect()
>
> This patchset is built upon Kirill Shutemov's work for the core MKTME
> support.
>
> David: Please let me know if the changes made, based on your review,
> are reasonable. I don't think that the new changes touch key service
> specific areas (much).
>
> Thomas: Please provide feedback on encrypt_mprotect(). If not a
> review, then a direction check would be helpful.
>

I'm not Thomas, but I think it's the wrong direction.  As it stands,
encrypt_mprotect() is an incomplete version of mprotect() (since it's
missing the protection key support), and it's also functionally just
MADV_DONTNEED.  In other words, the sole user-visible effect appears
to be that the existing pages are blown away.  The fact that it
changes the key in use doesn't seem terribly useful, since it's
anonymous memory, and the most secure choice is to use CPU-managed
keying, which appears to be the default anyway on TME systems.  It
also has totally unclear semantics WRT swap, and, off the top of my
head, it looks like it may have serious cache-coherency issues and
like swapping the pages might corrupt them, both because there are no
flushes and because the direct-map alias looks like it will use the
default key and therefore appear to contain the wrong data.

I would propose a very different direction: don't try to support MKTME
at all for anonymous memory, and instead figure out the important use
cases and support them directly.  The use cases that I can think of
off the top of my head are:

1. pmem.  This should probably use a very different API.

2. Some kind of VM hardening, where a VM's memory can be protected a
little tiny bit from the main kernel.  But I don't see why this is any
better than XPO (eXclusive Page-frame Ownership), which brings to
mind:

The main implementation concern I have with this patch set is cache
coherency and handling of the direct map.  Unless I missed something,
you're not doing anything about the direct map, which means that you
have RW aliases of the same memory with different keys.  For use case
#2, this probably means that you need to either get rid of the direct
map and make get_user_pages() fail, or you need to change the key on
the direct map as well, probably using the pageattr.c code.

As for caching, As far as I can tell from reading the preliminary
docs, Intel's MKTME, much like AMD's SME, is basically invisible to
the hardware cache coherency mechanism.  So, if you modify a physical
address with one key (or SME-enable bit), and you read it with
another, you get garbage unless you flush.  And, if you modify memory
with one key then remap it with a different key without flushing in
the mean time, you risk corruption.  And, what's worse, if I'm reading
between the lines in the docs correctly, if you use PCONFIG to change
a key, you may need to do a bunch of cache flushing to ensure you get
reasonable effects.  (If you have dirty cache lines for some (PA, key)
and you PCONFIG to change the underlying key, you get different
results depending on whether the writeback happens before or after the
package doing the writeback notices the PCONFIG.)

Finally, If you're going to teach the kernel how to have some user
pages that aren't in the direct map, you've essentially done XPO,
which is nifty but expensive.  And I think that doing this gets you
essentially all the benefit of MKTME for the non-pmem use case.  Why
exactly would any software want to use anything other than a
CPU-managed key for anything other than pmem?

--Andy

Re: [RFC v2 00/13] Multi-Key Total Memory Encryption API (MKTME)

[Andy Lutomirski]

On Tue, Dec 4, 2018 at 11:19 AM Andy Lutomirski <luto@kernel.org> wrote:
>
> On Mon, Dec 3, 2018 at 11:37 PM Alison Schofield
> <alison.schofield@intel.com> wrote:
> >

> Finally, If you're going to teach the kernel how to have some user
> pages that aren't in the direct map, you've essentially done XPO,
> which is nifty but expensive.  And I think that doing this gets you
> essentially all the benefit of MKTME for the non-pmem use case.  Why
> exactly would any software want to use anything other than a
> CPU-managed key for anything other than pmem?
>

Let me say this less abstractly.  Here's a somewhat concrete actual
proposal.  Make a new memfd_create() flag like MEMFD_ISOLATED.  The
semantics are that the underlying pages are made not-present in the
direct map when they're allocated (which is hideously slow, but so be
it), and that anything that tries to get_user_pages() the resulting
pages fails.  And then make sure we have all the required APIs so that
QEMU can still map this stuff into a VM.

If there is indeed a situation in which MKTME-ifying the memory adds
some value, then we can consider doing that.

And maybe we get fancy and encrypt this memory when it's swapped, but
maybe we should just encrypt everything when it's swapped.

Re: [RFC v2 00/13] Multi-Key Total Memory Encryption API (MKTME)

[Andy Lutomirski]

On Tue, Dec 4, 2018 at 11:19 AM Andy Lutomirski <luto@kernel.org> wrote:
>
> On Mon, Dec 3, 2018 at 11:37 PM Alison Schofield
> <alison.schofield@intel.com> wrote:
> >

> Finally, If you're going to teach the kernel how to have some user
> pages that aren't in the direct map, you've essentially done XPO,
> which is nifty but expensive.  And I think that doing this gets you
> essentially all the benefit of MKTME for the non-pmem use case.  Why
> exactly would any software want to use anything other than a
> CPU-managed key for anything other than pmem?
>

Let me say this less abstractly.  Here's a somewhat concrete actual
proposal.  Make a new memfd_create() flag like MEMFD_ISOLATED.  The
semantics are that the underlying pages are made not-present in the
direct map when they're allocated (which is hideously slow, but so be
it), and that anything that tries to get_user_pages() the resulting
pages fails.  And then make sure we have all the required APIs so that
QEMU can still map this stuff into a VM.

If there is indeed a situation in which MKTME-ifying the memory adds
some value, then we can consider doing that.

And maybe we get fancy and encrypt this memory when it's swapped, but
maybe we should just encrypt everything when it's swapped.

Re: [RFC v2 00/13] Multi-Key Total Memory Encryption API (MKTME)

[Dave Hansen]

On 12/4/18 12:00 PM, Andy Lutomirski wrote:
> On Tue, Dec 4, 2018 at 11:19 AM Andy Lutomirski <luto@kernel.org> wrote:
>> On Mon, Dec 3, 2018 at 11:37 PM Alison Schofield <alison.schofield@intel.com> wrote:
>> Finally, If you're going to teach the kernel how to have some user
>> pages that aren't in the direct map, you've essentially done XPO,
>> which is nifty but expensive.  And I think that doing this gets you
>> essentially all the benefit of MKTME for the non-pmem use case.  Why
>> exactly would any software want to use anything other than a
>> CPU-managed key for anything other than pmem?
> 
> Let me say this less abstractly.  Here's a somewhat concrete actual
> proposal.  Make a new memfd_create() flag like MEMFD_ISOLATED.  The
> semantics are that the underlying pages are made not-present in the
> direct map when they're allocated (which is hideously slow, but so be
> it), and that anything that tries to get_user_pages() the resulting
> pages fails.  And then make sure we have all the required APIs so that
> QEMU can still map this stuff into a VM.

I think we need get_user_pages().  We want direct I/O to work, *and* we
really want direct device assignment into VMs.

> And maybe we get fancy and encrypt this memory when it's swapped, but
> maybe we should just encrypt everything when it's swapped.

We decided long ago (and this should be in the patches somewhere) that
we wouldn't force memory to be encrypted in swap.  We would just
recommend it in the documentation as a best practice, especially when
using MKTME.

We can walk that back, of course, but that's what we're doing at the moment.

Re: [RFC v2 00/13] Multi-Key Total Memory Encryption API (MKTME)

[Dave Hansen]

On 12/4/18 12:00 PM, Andy Lutomirski wrote:
> On Tue, Dec 4, 2018 at 11:19 AM Andy Lutomirski <luto@kernel.org> wrote:
>> On Mon, Dec 3, 2018 at 11:37 PM Alison Schofield <alison.schofield@intel.com> wrote:
>> Finally, If you're going to teach the kernel how to have some user
>> pages that aren't in the direct map, you've essentially done XPO,
>> which is nifty but expensive.  And I think that doing this gets you
>> essentially all the benefit of MKTME for the non-pmem use case.  Why
>> exactly would any software want to use anything other than a
>> CPU-managed key for anything other than pmem?
> 
> Let me say this less abstractly.  Here's a somewhat concrete actual
> proposal.  Make a new memfd_create() flag like MEMFD_ISOLATED.  The
> semantics are that the underlying pages are made not-present in the
> direct map when they're allocated (which is hideously slow, but so be
> it), and that anything that tries to get_user_pages() the resulting
> pages fails.  And then make sure we have all the required APIs so that
> QEMU can still map this stuff into a VM.

I think we need get_user_pages().  We want direct I/O to work, *and* we
really want direct device assignment into VMs.

> And maybe we get fancy and encrypt this memory when it's swapped, but
> maybe we should just encrypt everything when it's swapped.

We decided long ago (and this should be in the patches somewhere) that
we wouldn't force memory to be encrypted in swap.  We would just
recommend it in the documentation as a best practice, especially when
using MKTME.

We can walk that back, of course, but that's what we're doing at the moment.

Re: [RFC v2 07/13] x86/mm: Add helpers for reference counting encrypted VMAs

[Alison Schofield]

On Tue, Dec 04, 2018 at 09:58:35AM +0100, Peter Zijlstra wrote:
> On Mon, Dec 03, 2018 at 11:39:54PM -0800, Alison Schofield wrote:
> 
> > +void vma_put_encrypt_ref(struct vm_area_struct *vma)
> > +{
> > +	if (vma_keyid(vma))
> > +		if (refcount_dec_and_test(&encrypt_count[vma_keyid(vma)])) {
> > +			mktme_map_lock();
> > +			mktme_map_free_keyid(vma_keyid(vma));
> > +			mktme_map_unlock();
> > +		}
> 
> This violates CodingStyle

Got it!
Will fix this and the other instances where you noticed poorly nested
if statements. 

> > +	if (refcount_dec_and_test(&encrypt_count[keyid])) {
> > +		mktme_map_lock();
> 
> That smells like it wants to use refcount_dec_and_lock() instead.
> 
> Also, if you write that like:
> 
> 	if (!refcount_dec_and_lock(&encrypt_count[keyid], &lock))
> 		return;
> 
> you loose an indent level.
Looks good! I need to make sure it's OK to switch to a spinlock to use
the *_lock functions.

Re: [RFC v2 07/13] x86/mm: Add helpers for reference counting encrypted VMAs

[Alison Schofield]

On Tue, Dec 04, 2018 at 09:58:35AM +0100, Peter Zijlstra wrote:
> On Mon, Dec 03, 2018 at 11:39:54PM -0800, Alison Schofield wrote:
> 
> > +void vma_put_encrypt_ref(struct vm_area_struct *vma)
> > +{
> > +	if (vma_keyid(vma))
> > +		if (refcount_dec_and_test(&encrypt_count[vma_keyid(vma)])) {
> > +			mktme_map_lock();
> > +			mktme_map_free_keyid(vma_keyid(vma));
> > +			mktme_map_unlock();
> > +		}
> 
> This violates CodingStyle

Got it!
Will fix this and the other instances where you noticed poorly nested
if statements. 

> > +	if (refcount_dec_and_test(&encrypt_count[keyid])) {
> > +		mktme_map_lock();
> 
> That smells like it wants to use refcount_dec_and_lock() instead.
> 
> Also, if you write that like:
> 
> 	if (!refcount_dec_and_lock(&encrypt_count[keyid], &lock))
> 		return;
> 
> you loose an indent level.
Looks good! I need to make sure it's OK to switch to a spinlock to use
the *_lock functions.

Re: [RFC v2 09/13] mm: Restrict memory encryption to anonymous VMA's

[Alison Schofield]

On Tue, Dec 04, 2018 at 10:10:44AM +0100, Peter Zijlstra wrote:
> > + * Encrypted mprotect is only supported on anonymous mappings.
> > + * All VMA's in the requested range must be anonymous. If this
> > + * test fails on any single VMA, the entire mprotect request fails.
> > + */
> > +bool mem_supports_encryption(struct vm_area_struct *vma, unsigned long end)
> 
> That's a 'weird' interface and cannot do what the comment says it should
> do.

More please? With MKTME, only anonymous memory supports encryption.
Is it the naming that's weird, or you don't see it doing what it says?

> > +	struct vm_area_struct *test_vma = vma;
> 
> That variable is utterly pointless.
Got it. Will fix.

Thanks

Re: [RFC v2 09/13] mm: Restrict memory encryption to anonymous VMA's

[Alison Schofield]

On Tue, Dec 04, 2018 at 10:10:44AM +0100, Peter Zijlstra wrote:
> > + * Encrypted mprotect is only supported on anonymous mappings.
> > + * All VMA's in the requested range must be anonymous. If this
> > + * test fails on any single VMA, the entire mprotect request fails.
> > + */
> > +bool mem_supports_encryption(struct vm_area_struct *vma, unsigned long end)
> 
> That's a 'weird' interface and cannot do what the comment says it should
> do.

More please? With MKTME, only anonymous memory supports encryption.
Is it the naming that's weird, or you don't see it doing what it says?

> > +	struct vm_area_struct *test_vma = vma;
> 
> That variable is utterly pointless.
Got it. Will fix.

Thanks

Re: [RFC v2 13/13] keys/mktme: Support CPU Hotplug for MKTME keys

[Alison Schofield]

On Tue, Dec 04, 2018 at 10:28:41AM +0100, Peter Zijlstra wrote:
> On Mon, Dec 03, 2018 at 11:40:00PM -0800, Alison Schofield wrote:
> > +	for_each_online_cpu(online_cpu)
> > +		if (online_cpu != cpu &&
> > +		    pkgid = topology_physical_package_id(online_cpu)) {
> > +			cpumask_set_cpu(online_cpu, mktme_leadcpus);
> > +			break;
> > +	}
> 
> That's a capital offence right there.
Got it!

Re: [RFC v2 13/13] keys/mktme: Support CPU Hotplug for MKTME keys

[Alison Schofield]

On Tue, Dec 04, 2018 at 10:28:41AM +0100, Peter Zijlstra wrote:
> On Mon, Dec 03, 2018 at 11:40:00PM -0800, Alison Schofield wrote:
> > +	for_each_online_cpu(online_cpu)
> > +		if (online_cpu != cpu &&
> > +		    pkgid == topology_physical_package_id(online_cpu)) {
> > +			cpumask_set_cpu(online_cpu, mktme_leadcpus);
> > +			break;
> > +	}
> 
> That's a capital offence right there.
Got it!

Re: [RFC v2 13/13] keys/mktme: Support CPU Hotplug for MKTME keys

[Alison Schofield]

On Tue, Dec 04, 2018 at 10:31:16AM +0100, Peter Zijlstra wrote:
> On Mon, Dec 03, 2018 at 11:40:00PM -0800, Alison Schofield wrote:
> >  static int mktme_program_system(struct mktme_key_program *key_program,
> > -				cpumask_var_t mktme_cpumask)
> > +				cpumask_var_t mktme_cpumask, int hotplug)
> >  {
> >  	struct mktme_hw_program_info info = {
> >  		.key_program = key_program,
> >  		.status = MKTME_PROG_SUCCESS,
> >  	};
> > -	get_online_cpus();
> > -	on_each_cpu_mask(mktme_cpumask, mktme_program_package, &info, 1);
> > -	put_online_cpus();
> > +
> > +	if (!hotplug) {
> > +		get_online_cpus();
> > +		on_each_cpu_mask(mktme_cpumask, mktme_program_package,
> > +				 &info, 1);
> > +		put_online_cpus();
> > +	} else {
> > +		on_each_cpu_mask(mktme_cpumask, mktme_program_package,
> > +				 &info, 1);
> > +	}
> >  
> >  	return info.status;
> >  }
> 
> That is pretty horrible; and I think easily avoided.
Agree it's ugly. Not sure we share the same reasoning. I realize that
the hotplug case is on the current cpu and so that whole
one_each_cpu_mask() call is not needed. mktme_program_package() can just
be called on the current cpu.

The ugliness that haunts me is that I wanted to reuse this code path,
and so I passed that 'hotplug' parameter along as a differentiator
between hotplug & 'typical' key programming. 
I'll rework this.

Re: [RFC v2 13/13] keys/mktme: Support CPU Hotplug for MKTME keys

[Alison Schofield]

On Tue, Dec 04, 2018 at 10:31:16AM +0100, Peter Zijlstra wrote:
> On Mon, Dec 03, 2018 at 11:40:00PM -0800, Alison Schofield wrote:
> >  static int mktme_program_system(struct mktme_key_program *key_program,
> > -				cpumask_var_t mktme_cpumask)
> > +				cpumask_var_t mktme_cpumask, int hotplug)
> >  {
> >  	struct mktme_hw_program_info info = {
> >  		.key_program = key_program,
> >  		.status = MKTME_PROG_SUCCESS,
> >  	};
> > -	get_online_cpus();
> > -	on_each_cpu_mask(mktme_cpumask, mktme_program_package, &info, 1);
> > -	put_online_cpus();
> > +
> > +	if (!hotplug) {
> > +		get_online_cpus();
> > +		on_each_cpu_mask(mktme_cpumask, mktme_program_package,
> > +				 &info, 1);
> > +		put_online_cpus();
> > +	} else {
> > +		on_each_cpu_mask(mktme_cpumask, mktme_program_package,
> > +				 &info, 1);
> > +	}
> >  
> >  	return info.status;
> >  }
> 
> That is pretty horrible; and I think easily avoided.
Agree it's ugly. Not sure we share the same reasoning. I realize that
the hotplug case is on the current cpu and so that whole
one_each_cpu_mask() call is not needed. mktme_program_package() can just
be called on the current cpu.

The ugliness that haunts me is that I wanted to reuse this code path,
and so I passed that 'hotplug' parameter along as a differentiator
between hotplug & 'typical' key programming. 
I'll rework this.

Re: [RFC v2 11/13] keys/mktme: Program memory encryption keys on a system wide basis

[Alison Schofield]

On Tue, Dec 04, 2018 at 10:21:45AM +0100, Peter Zijlstra wrote:
> On Mon, Dec 03, 2018 at 11:39:58PM -0800, Alison Schofield wrote:
> 
> > +static int mktme_build_leadcpus_mask(void)
> > +{
> > +	int online_cpu, mktme_cpu;
> > +	int online_pkgid, mktme_pkgid = -1;
> > +
> > +	if (!zalloc_cpumask_var(&mktme_leadcpus, GFP_KERNEL))
> > +		return -ENOMEM;
> > +
> > +	for_each_online_cpu(online_cpu) {
> > +		online_pkgid = topology_physical_package_id(online_cpu);
> > +
> > +		for_each_cpu(mktme_cpu, mktme_leadcpus) {
> > +			mktme_pkgid = topology_physical_package_id(mktme_cpu);
> > +			if (mktme_pkgid = online_pkgid)
> > +				break;
> > +		}
> > +		if (mktme_pkgid != online_pkgid)
> > +			cpumask_set_cpu(online_cpu, mktme_leadcpus);
> 
> Do you really need LOCK prefixed bit set here?
No. Changed to __cpumask_set_cpu(). Will check for other instances
where I can skip LOCK prefix.

> How is that serialized and kept relevant in the face of hotplug?
mktme_leadcpus is updated on hotplug startup and teardowns.

> Also, do you really need O(n^2) to find the first occurence of a value
> in an array?
How about this O(n)?
	
	unsigned long *pkg_map;
	int cpu, pkgid;

	if (!zalloc_cpumask_var(&mktme_leadcpus, GFP_KERNEL))
		return -ENOMEM;

	pkg_map = bitmap_zalloc(topology_max_packages(), GFP_KERNEL);
	if (!pkg_map) {
		free_cpumask_var(mktme_leadcpus);
		return -ENOMEM;
	}
	for_each_online_cpu(cpu) {
		pkgid = topology_physical_package_id(cpu);
		if (!test_and_set_bit(pkgid, pkg_map))
			__cpumask_set_cpu(cpu, mktme_leadcpus);
	}

Re: [RFC v2 11/13] keys/mktme: Program memory encryption keys on a system wide basis

[Alison Schofield]

On Tue, Dec 04, 2018 at 10:21:45AM +0100, Peter Zijlstra wrote:
> On Mon, Dec 03, 2018 at 11:39:58PM -0800, Alison Schofield wrote:
> 
> > +static int mktme_build_leadcpus_mask(void)
> > +{
> > +	int online_cpu, mktme_cpu;
> > +	int online_pkgid, mktme_pkgid = -1;
> > +
> > +	if (!zalloc_cpumask_var(&mktme_leadcpus, GFP_KERNEL))
> > +		return -ENOMEM;
> > +
> > +	for_each_online_cpu(online_cpu) {
> > +		online_pkgid = topology_physical_package_id(online_cpu);
> > +
> > +		for_each_cpu(mktme_cpu, mktme_leadcpus) {
> > +			mktme_pkgid = topology_physical_package_id(mktme_cpu);
> > +			if (mktme_pkgid == online_pkgid)
> > +				break;
> > +		}
> > +		if (mktme_pkgid != online_pkgid)
> > +			cpumask_set_cpu(online_cpu, mktme_leadcpus);
> 
> Do you really need LOCK prefixed bit set here?
No. Changed to __cpumask_set_cpu(). Will check for other instances
where I can skip LOCK prefix.

> How is that serialized and kept relevant in the face of hotplug?
mktme_leadcpus is updated on hotplug startup and teardowns.

> Also, do you really need O(n^2) to find the first occurence of a value
> in an array?
How about this O(n)?
	
	unsigned long *pkg_map;
	int cpu, pkgid;

	if (!zalloc_cpumask_var(&mktme_leadcpus, GFP_KERNEL))
		return -ENOMEM;

	pkg_map = bitmap_zalloc(topology_max_packages(), GFP_KERNEL);
	if (!pkg_map) {
		free_cpumask_var(mktme_leadcpus);
		return -ENOMEM;
	}
	for_each_online_cpu(cpu) {
		pkgid = topology_physical_package_id(cpu);
		if (!test_and_set_bit(pkgid, pkg_map))
			__cpumask_set_cpu(cpu, mktme_leadcpus);
	}

Re: [RFC v2 11/13] keys/mktme: Program memory encryption keys on a system wide basis

[Alison Schofield]

On Tue, Dec 04, 2018 at 12:50:09PM +0300, Kirill A. Shutemov wrote:
> On Tue, Dec 04, 2018 at 09:21:45AM +0000, Peter Zijlstra wrote:
> > On Mon, Dec 03, 2018 at 11:39:58PM -0800, Alison Schofield wrote:
> > 
> > > +struct mktme_hw_program_info {
> > > +	struct mktme_key_program *key_program;
> > > +	unsigned long status;
> > > +};
> > > +
> > > +/* Program a KeyID on a single package. */
> > > +static void mktme_program_package(void *hw_program_info)
> > > +{
> > > +	struct mktme_hw_program_info *info = hw_program_info;
> > > +	int ret;
> > > +
> > > +	ret = mktme_key_program(info->key_program);
> > > +	if (ret != MKTME_PROG_SUCCESS)
> > > +		WRITE_ONCE(info->status, ret);
> > 
> > What's the purpose of that WRITE_ONCE()?
> 
> [I suggested the code to Alison.]
> 
> Yes, you're right. Simple assignment will do.
Will do. Thanks!

> 
> -- 
>  Kirill A. Shutemov

Re: [RFC v2 11/13] keys/mktme: Program memory encryption keys on a system wide basis

[Alison Schofield]

On Tue, Dec 04, 2018 at 12:50:09PM +0300, Kirill A. Shutemov wrote:
> On Tue, Dec 04, 2018 at 09:21:45AM +0000, Peter Zijlstra wrote:
> > On Mon, Dec 03, 2018 at 11:39:58PM -0800, Alison Schofield wrote:
> > 
> > > +struct mktme_hw_program_info {
> > > +	struct mktme_key_program *key_program;
> > > +	unsigned long status;
> > > +};
> > > +
> > > +/* Program a KeyID on a single package. */
> > > +static void mktme_program_package(void *hw_program_info)
> > > +{
> > > +	struct mktme_hw_program_info *info = hw_program_info;
> > > +	int ret;
> > > +
> > > +	ret = mktme_key_program(info->key_program);
> > > +	if (ret != MKTME_PROG_SUCCESS)
> > > +		WRITE_ONCE(info->status, ret);
> > 
> > What's the purpose of that WRITE_ONCE()?
> 
> [I suggested the code to Alison.]
> 
> Yes, you're right. Simple assignment will do.
Will do. Thanks!

> 
> -- 
>  Kirill A. Shutemov

Re: [RFC v2 04/13] x86/mm: Add helper functions for MKTME memory encryption keys

[Alison Schofield]

On Tue, Dec 04, 2018 at 10:14:34AM +0100, Peter Zijlstra wrote:
> On Mon, Dec 03, 2018 at 11:39:51PM -0800, Alison Schofield wrote:
> 
> CodingStyle
> CodingStyle
>
Thanks Peter. I'll repair all the badly nested if statements.

Re: [RFC v2 04/13] x86/mm: Add helper functions for MKTME memory encryption keys

[Alison Schofield]

On Tue, Dec 04, 2018 at 10:14:34AM +0100, Peter Zijlstra wrote:
> On Mon, Dec 03, 2018 at 11:39:51PM -0800, Alison Schofield wrote:
> 
> CodingStyle
> CodingStyle
>
Thanks Peter. I'll repair all the badly nested if statements.

Re: [RFC v2 04/13] x86/mm: Add helper functions for MKTME memory encryption keys

[Alison Schofield]

On Tue, Dec 04, 2018 at 07:35:50AM -0800, Andy Lutomirski wrote:
> 
> 
> > On Dec 3, 2018, at 11:39 PM, Alison Schofield <alison.schofield@intel.com> wrote:
> > 
> > Define a global mapping structure to manage the mapping of userspace
> > Keys to hardware KeyIDs in MKTME (Multi-Key Total Memory Encryption).
> > Implement helper functions that access this mapping structure.
> > 
> 
> Why is a key “void *”?  Who owns the memory?  Can a real type be used?
>
It's of type "struct key" of the kernel key service.
Replacing void w 'struct key'.

Re: [RFC v2 04/13] x86/mm: Add helper functions for MKTME memory encryption keys

[Alison Schofield]

On Tue, Dec 04, 2018 at 07:35:50AM -0800, Andy Lutomirski wrote:
> 
> 
> > On Dec 3, 2018, at 11:39 PM, Alison Schofield <alison.schofield@intel.com> wrote:
> > 
> > Define a global mapping structure to manage the mapping of userspace
> > Keys to hardware KeyIDs in MKTME (Multi-Key Total Memory Encryption).
> > Implement helper functions that access this mapping structure.
> > 
> 
> Why is a key “void *”?  Who owns the memory?  Can a real type be used?
>
It's of type "struct key" of the kernel key service.
Replacing void w 'struct key'.

Re: [RFC v2 09/13] mm: Restrict memory encryption to anonymous VMA's

[Peter Zijlstra]

On Tue, Dec 04, 2018 at 09:30:20PM -0800, Alison Schofield wrote:
> On Tue, Dec 04, 2018 at 10:10:44AM +0100, Peter Zijlstra wrote:
> > > + * Encrypted mprotect is only supported on anonymous mappings.
> > > + * All VMA's in the requested range must be anonymous. If this
> > > + * test fails on any single VMA, the entire mprotect request fails.
> > > + */
> > > +bool mem_supports_encryption(struct vm_area_struct *vma, unsigned long end)
> > 
> > That's a 'weird' interface and cannot do what the comment says it should
> > do.
> 
> More please? With MKTME, only anonymous memory supports encryption.
> Is it the naming that's weird, or you don't see it doing what it says?

It's weird because you don't fully speficy the range -- ie. it cannot
verify the vma argument. It is also weird because the start and end are
not of the same type -- or rather, there is no start at all.

So while the comment talks about a range, there is not in fact a range
(only the implied @start is somewhere inside @vma). The comment also
states all vmas in the range, but again, because of a lack of range
specification it cannot verify this statement.

Now, I don't necessarily object to the function and its implementation,
but that comment is just plain misleading.

Re: [RFC v2 09/13] mm: Restrict memory encryption to anonymous VMA's

[Peter Zijlstra]

On Tue, Dec 04, 2018 at 09:30:20PM -0800, Alison Schofield wrote:
> On Tue, Dec 04, 2018 at 10:10:44AM +0100, Peter Zijlstra wrote:
> > > + * Encrypted mprotect is only supported on anonymous mappings.
> > > + * All VMA's in the requested range must be anonymous. If this
> > > + * test fails on any single VMA, the entire mprotect request fails.
> > > + */
> > > +bool mem_supports_encryption(struct vm_area_struct *vma, unsigned long end)
> > 
> > That's a 'weird' interface and cannot do what the comment says it should
> > do.
> 
> More please? With MKTME, only anonymous memory supports encryption.
> Is it the naming that's weird, or you don't see it doing what it says?

It's weird because you don't fully speficy the range -- ie. it cannot
verify the vma argument. It is also weird because the start and end are
not of the same type -- or rather, there is no start at all.

So while the comment talks about a range, there is not in fact a range
(only the implied @start is somewhere inside @vma). The comment also
states all vmas in the range, but again, because of a lack of range
specification it cannot verify this statement.

Now, I don't necessarily object to the function and its implementation,
but that comment is just plain misleading.

Re: [RFC v2 11/13] keys/mktme: Program memory encryption keys on a system wide basis

[Peter Zijlstra]

On Tue, Dec 04, 2018 at 09:43:53PM -0800, Alison Schofield wrote:
> On Tue, Dec 04, 2018 at 10:21:45AM +0100, Peter Zijlstra wrote:
> > On Mon, Dec 03, 2018 at 11:39:58PM -0800, Alison Schofield wrote:
> > 
> > > +static int mktme_build_leadcpus_mask(void)
> > > +{
> > > +	int online_cpu, mktme_cpu;
> > > +	int online_pkgid, mktme_pkgid = -1;
> > > +
> > > +	if (!zalloc_cpumask_var(&mktme_leadcpus, GFP_KERNEL))
> > > +		return -ENOMEM;
> > > +
> > > +	for_each_online_cpu(online_cpu) {
> > > +		online_pkgid = topology_physical_package_id(online_cpu);
> > > +
> > > +		for_each_cpu(mktme_cpu, mktme_leadcpus) {
> > > +			mktme_pkgid = topology_physical_package_id(mktme_cpu);
> > > +			if (mktme_pkgid = online_pkgid)
> > > +				break;
> > > +		}
> > > +		if (mktme_pkgid != online_pkgid)
> > > +			cpumask_set_cpu(online_cpu, mktme_leadcpus);
> > 
> > Do you really need LOCK prefixed bit set here?
> No. Changed to __cpumask_set_cpu(). Will check for other instances
> where I can skip LOCK prefix.
> 
> > How is that serialized and kept relevant in the face of hotplug?
> mktme_leadcpus is updated on hotplug startup and teardowns.

Not in this patch it is not. That is added in a subsequent patch, which
means that during bisection hotplug is utterly wrecked if you happen to
land between these patches, that is bad.

> > Also, do you really need O(n^2) to find the first occurence of a value
> > in an array?

> How about this O(n)?
> 	
> 	unsigned long *pkg_map;
> 	int cpu, pkgid;
> 
> 	if (!zalloc_cpumask_var(&mktme_leadcpus, GFP_KERNEL))
> 		return -ENOMEM;
> 
> 	pkg_map = bitmap_zalloc(topology_max_packages(), GFP_KERNEL);
> 	if (!pkg_map) {
> 		free_cpumask_var(mktme_leadcpus);
> 		return -ENOMEM;
> 	}
> 	for_each_online_cpu(cpu) {
> 		pkgid = topology_physical_package_id(cpu);
> 		if (!test_and_set_bit(pkgid, pkg_map))

You again don't need that LOCK prefix here.

	__test_and_set_bit() :-)

> 			__cpumask_set_cpu(cpu, mktme_leadcpus);
> 	}

Right.

Re: [RFC v2 11/13] keys/mktme: Program memory encryption keys on a system wide basis

[Peter Zijlstra]

On Tue, Dec 04, 2018 at 09:43:53PM -0800, Alison Schofield wrote:
> On Tue, Dec 04, 2018 at 10:21:45AM +0100, Peter Zijlstra wrote:
> > On Mon, Dec 03, 2018 at 11:39:58PM -0800, Alison Schofield wrote:
> > 
> > > +static int mktme_build_leadcpus_mask(void)
> > > +{
> > > +	int online_cpu, mktme_cpu;
> > > +	int online_pkgid, mktme_pkgid = -1;
> > > +
> > > +	if (!zalloc_cpumask_var(&mktme_leadcpus, GFP_KERNEL))
> > > +		return -ENOMEM;
> > > +
> > > +	for_each_online_cpu(online_cpu) {
> > > +		online_pkgid = topology_physical_package_id(online_cpu);
> > > +
> > > +		for_each_cpu(mktme_cpu, mktme_leadcpus) {
> > > +			mktme_pkgid = topology_physical_package_id(mktme_cpu);
> > > +			if (mktme_pkgid == online_pkgid)
> > > +				break;
> > > +		}
> > > +		if (mktme_pkgid != online_pkgid)
> > > +			cpumask_set_cpu(online_cpu, mktme_leadcpus);
> > 
> > Do you really need LOCK prefixed bit set here?
> No. Changed to __cpumask_set_cpu(). Will check for other instances
> where I can skip LOCK prefix.
> 
> > How is that serialized and kept relevant in the face of hotplug?
> mktme_leadcpus is updated on hotplug startup and teardowns.

Not in this patch it is not. That is added in a subsequent patch, which
means that during bisection hotplug is utterly wrecked if you happen to
land between these patches, that is bad.

> > Also, do you really need O(n^2) to find the first occurence of a value
> > in an array?

> How about this O(n)?
> 	
> 	unsigned long *pkg_map;
> 	int cpu, pkgid;
> 
> 	if (!zalloc_cpumask_var(&mktme_leadcpus, GFP_KERNEL))
> 		return -ENOMEM;
> 
> 	pkg_map = bitmap_zalloc(topology_max_packages(), GFP_KERNEL);
> 	if (!pkg_map) {
> 		free_cpumask_var(mktme_leadcpus);
> 		return -ENOMEM;
> 	}
> 	for_each_online_cpu(cpu) {
> 		pkgid = topology_physical_package_id(cpu);
> 		if (!test_and_set_bit(pkgid, pkg_map))

You again don't need that LOCK prefix here.

	__test_and_set_bit() :-)

> 			__cpumask_set_cpu(cpu, mktme_leadcpus);
> 	}

Right.

Re: [RFC v2 11/13] keys/mktme: Program memory encryption keys on a system wide basis

[Alison Schofield]

On Wed, Dec 05, 2018 at 10:10:29AM +0100, Peter Zijlstra wrote:
> On Tue, Dec 04, 2018 at 09:43:53PM -0800, Alison Schofield wrote:
> > On Tue, Dec 04, 2018 at 10:21:45AM +0100, Peter Zijlstra wrote:
> > > On Mon, Dec 03, 2018 at 11:39:58PM -0800, Alison Schofield wrote:
> > 
> > > How is that serialized and kept relevant in the face of hotplug?
> > mktme_leadcpus is updated on hotplug startup and teardowns.
> 
> Not in this patch it is not. That is added in a subsequent patch, which
> means that during bisection hotplug is utterly wrecked if you happen to
> land between these patches, that is bad.
>
The Key Service support is split between 4 main patches (10-13), but
the dependencies go further back in the patchset.

If the bisect need outweighs any benefit from reviewing in pieces,
then these patches can be squashed to a single patch:

keys/mktme: Add the MKTME Key Service type for memory encryption
keys/mktme: Program memory encryption keys on a system wide basis
keys/mktme: Save MKTME data if kernel cmdline parameter allows
keys/mktme: Support CPU Hotplug for MKTME keys

Am I interpreting your point correctly?
Thanks,
Alison

Re: [RFC v2 11/13] keys/mktme: Program memory encryption keys on a system wide basis

[Alison Schofield]

On Wed, Dec 05, 2018 at 10:10:29AM +0100, Peter Zijlstra wrote:
> On Tue, Dec 04, 2018 at 09:43:53PM -0800, Alison Schofield wrote:
> > On Tue, Dec 04, 2018 at 10:21:45AM +0100, Peter Zijlstra wrote:
> > > On Mon, Dec 03, 2018 at 11:39:58PM -0800, Alison Schofield wrote:
> > 
> > > How is that serialized and kept relevant in the face of hotplug?
> > mktme_leadcpus is updated on hotplug startup and teardowns.
> 
> Not in this patch it is not. That is added in a subsequent patch, which
> means that during bisection hotplug is utterly wrecked if you happen to
> land between these patches, that is bad.
>
The Key Service support is split between 4 main patches (10-13), but
the dependencies go further back in the patchset.

If the bisect need outweighs any benefit from reviewing in pieces,
then these patches can be squashed to a single patch:

keys/mktme: Add the MKTME Key Service type for memory encryption
keys/mktme: Program memory encryption keys on a system wide basis
keys/mktme: Save MKTME data if kernel cmdline parameter allows
keys/mktme: Support CPU Hotplug for MKTME keys

Am I interpreting your point correctly?
Thanks,
Alison

Re: [RFC v2 01/13] x86/mktme: Document the MKTME APIs

[Andy Lutomirski]

> On Dec 3, 2018, at 11:39 PM, Alison Schofield <alison.schofield@intel.com> wrote:

I realize you’re writing code to expose hardware behavior, but I’m not sure this
really makes sense in this context.

> .
> +
> +Usage
> +-----
> +    When using the Kernel Key Service to request an *mktme* key,
> +    specify the *payload* as follows:
> +
> +    type=
> +        *user*    User will supply the encryption key data. Use this
> +                type to directly program a hardware encryption key.
> +

I think that “user” probably sense as a “key service” key, but I don’t think it is at all useful for non-persistent memory.  Even if we take for granted that MKTME for anonymous memory is useful at all, “cpu” seems to be better in all respects.


Perhaps support for “user” should be tabled until there’s a design for how to use this for pmem?  I imagine it would look quite a bit like dm-crypt.  Advanced pmem filesystems could plausibly use different keys for different files, I suppose.

If “user” is dropped, I think a lot of the complexity goes away. Hotplug becomes automatic, right?

> +        *cpu*    User requests a CPU generated encryption key.

Okay, maybe, but it’s still unclear to me exactly what the intended benefit is, though.

> +                The CPU generates and assigns an ephemeral key.
> +
> +        *clear* User requests that a hardware encryption key be
> +                cleared. This will clear the encryption key from
> +                the hardware. On execution this hardware key gets
> +                TME behavior.
> +

Why is this a key type?  Shouldn’t the API to select a key just have an option to ask for no key to be used?

> +        *no-encrypt*
> +                 User requests that hardware does not encrypt
> +                 memory when this key is in use.

Same as above.  If there’s a performance benefit, then there could be a way to ask for cleartext memory.  Similarly, some pmem users may want a way to keep their pmem unencrypted.

—Andy

Re: [RFC v2 01/13] x86/mktme: Document the MKTME APIs

[Andy Lutomirski]

> On Dec 3, 2018, at 11:39 PM, Alison Schofield <alison.schofield@intel.com> wrote:

I realize you’re writing code to expose hardware behavior, but I’m not sure this
really makes sense in this context.

> .
> +
> +Usage
> +-----
> +    When using the Kernel Key Service to request an *mktme* key,
> +    specify the *payload* as follows:
> +
> +    type=
> +        *user*    User will supply the encryption key data. Use this
> +                type to directly program a hardware encryption key.
> +

I think that “user” probably sense as a “key service” key, but I don’t think it is at all useful for non-persistent memory.  Even if we take for granted that MKTME for anonymous memory is useful at all, “cpu” seems to be better in all respects.


Perhaps support for “user” should be tabled until there’s a design for how to use this for pmem?  I imagine it would look quite a bit like dm-crypt.  Advanced pmem filesystems could plausibly use different keys for different files, I suppose.

If “user” is dropped, I think a lot of the complexity goes away. Hotplug becomes automatic, right?

> +        *cpu*    User requests a CPU generated encryption key.

Okay, maybe, but it’s still unclear to me exactly what the intended benefit is, though.

> +                The CPU generates and assigns an ephemeral key.
> +
> +        *clear* User requests that a hardware encryption key be
> +                cleared. This will clear the encryption key from
> +                the hardware. On execution this hardware key gets
> +                TME behavior.
> +

Why is this a key type?  Shouldn’t the API to select a key just have an option to ask for no key to be used?

> +        *no-encrypt*
> +                 User requests that hardware does not encrypt
> +                 memory when this key is in use.

Same as above.  If there’s a performance benefit, then there could be a way to ask for cleartext memory.  Similarly, some pmem users may want a way to keep their pmem unencrypted.

—Andy

Re: [RFC v2 01/13] x86/mktme: Document the MKTME APIs

[Alison Schofield]

On Wed, Dec 05, 2018 at 10:11:18AM -0800, Andy Lutomirski wrote:
> 
> 
> > On Dec 3, 2018, at 11:39 PM, Alison Schofield <alison.schofield@intel.com> wrote:
> 
> I realize you’re writing code to expose hardware behavior, but I’m not sure this
> really makes sense in this context.

Your observation is accurate. The Usage defined here is very closely
aligned to the Intel MKTME Architecture spec. That's a starting point,
but not the ending point. We need to implement the feature set that
makes sense. More below...

> > +
> > +    type> > +        *user*    User will supply the encryption key data. Use this
> > +                type to directly program a hardware encryption key.
> > +
> 
> I think that “user” probably sense as a “key service” key, but I don’t think it is at all useful for non-persistent memory.  Even if we take for granted that MKTME for anonymous memory is useful at all, “cpu” seems to be better in all respects.
> 
> 
> Perhaps support for “user” should be tabled until there’s a design for how to use this for pmem?  I imagine it would look quite a bit like dm-crypt.  Advanced pmem filesystems could plausibly use different keys for different files, I suppose.
> 
> If “user” is dropped, I think a lot of the complexity goes away. Hotplug becomes automatic, right?

Dropping 'user' type removes a great deal of complexity.

Let me follow up in 2 ways:
1) Find out when MKTME support for pmem is required.
2) Go back to the the requirements and get the justification for user
type.

> 
> > +        *cpu*    User requests a CPU generated encryption key.
> 
> Okay, maybe, but it’s still unclear to me exactly what the intended benefit is, though.
*cpu* is the RANDOM key generated by the cpu. If there were no other
options, then this would be default, and go away.

> > +        *clear* User requests that a hardware encryption key be
> > +                cleared. This will clear the encryption key from
> > +                the hardware. On execution this hardware key gets
> > +                TME behavior.
> > +
> 
> Why is this a key type?  Shouldn’t the API to select a key just have an option to ask for no key to be used?

The *clear* key has been requested in order to clear/erase the users
key data that has been programmed into a hardware slot. User does not
want to leave a slot programmed with their encryption data when they
are done with it.

> > +        *no-encrypt*
> > +                 User requests that hardware does not encrypt
> > +                 memory when this key is in use.
> 
> Same as above.  If there’s a performance benefit, then there could be a way to ask for cleartext memory.  Similarly, some pmem users may want a way to keep their pmem unencrypted.

So, this is the way to ask for cleartext memory.
The entire system will be encrypted with the system wide TME Key.
A subset of that will be protected with MKTME Keys.
If user wants, no encrypt, this *no-encrypt* is the way to do it.

Alison
> 
> —Andy

Re: [RFC v2 01/13] x86/mktme: Document the MKTME APIs

[Alison Schofield]

On Wed, Dec 05, 2018 at 10:11:18AM -0800, Andy Lutomirski wrote:
> 
> 
> > On Dec 3, 2018, at 11:39 PM, Alison Schofield <alison.schofield@intel.com> wrote:
> 
> I realize you’re writing code to expose hardware behavior, but I’m not sure this
> really makes sense in this context.

Your observation is accurate. The Usage defined here is very closely
aligned to the Intel MKTME Architecture spec. That's a starting point,
but not the ending point. We need to implement the feature set that
makes sense. More below...

> > +
> > +    type=
> > +        *user*    User will supply the encryption key data. Use this
> > +                type to directly program a hardware encryption key.
> > +
> 
> I think that “user” probably sense as a “key service” key, but I don’t think it is at all useful for non-persistent memory.  Even if we take for granted that MKTME for anonymous memory is useful at all, “cpu” seems to be better in all respects.
> 
> 
> Perhaps support for “user” should be tabled until there’s a design for how to use this for pmem?  I imagine it would look quite a bit like dm-crypt.  Advanced pmem filesystems could plausibly use different keys for different files, I suppose.
> 
> If “user” is dropped, I think a lot of the complexity goes away. Hotplug becomes automatic, right?

Dropping 'user' type removes a great deal of complexity.

Let me follow up in 2 ways:
1) Find out when MKTME support for pmem is required.
2) Go back to the the requirements and get the justification for user
type.

> 
> > +        *cpu*    User requests a CPU generated encryption key.
> 
> Okay, maybe, but it’s still unclear to me exactly what the intended benefit is, though.
*cpu* is the RANDOM key generated by the cpu. If there were no other
options, then this would be default, and go away.

> > +        *clear* User requests that a hardware encryption key be
> > +                cleared. This will clear the encryption key from
> > +                the hardware. On execution this hardware key gets
> > +                TME behavior.
> > +
> 
> Why is this a key type?  Shouldn’t the API to select a key just have an option to ask for no key to be used?

The *clear* key has been requested in order to clear/erase the users
key data that has been programmed into a hardware slot. User does not
want to leave a slot programmed with their encryption data when they
are done with it.

> > +        *no-encrypt*
> > +                 User requests that hardware does not encrypt
> > +                 memory when this key is in use.
> 
> Same as above.  If there’s a performance benefit, then there could be a way to ask for cleartext memory.  Similarly, some pmem users may want a way to keep their pmem unencrypted.

So, this is the way to ask for cleartext memory.
The entire system will be encrypted with the system wide TME Key.
A subset of that will be protected with MKTME Keys.
If user wants, no encrypt, this *no-encrypt* is the way to do it.

Alison
> 
> —Andy

Re: [RFC v2 00/13] Multi-Key Total Memory Encryption API (MKTME)

[Sakkinen, Jarkko]

T24gTW9uLCAyMDE4LTEyLTAzIGF0IDIzOjM5IC0wODAwLCBBbGlzb24gU2Nob2ZpZWxkIHdyb3Rl
Og0KPiBIaSBUaG9tYXMsIERhdmlkLA0KPiANCj4gSGVyZSBpcyBhbiB1cGRhdGVkIFJGQyBvbiB0
aGUgQVBJJ3MgdG8gc3VwcG9ydCBNS1RNRS4NCj4gKE11bHRpLUtleSBUb3RhbCBNZW1vcnkgRW5j
cnlwdGlvbikNCj4gDQo+IFRoaXMgUkZDIHByZXNlbnRzIHRoZSAyIEFQSSBhZGRpdGlvbnMgdG8g
c3VwcG9ydCB0aGUgY3JlYXRpb24gYW5kDQo+IHVzYWdlIG9mIG1lbW9yeSBlbmNyeXB0aW9uIGtl
eXM6DQo+ICAxKSBLZXJuZWwgS2V5IFNlcnZpY2UgdHlwZSAibWt0bWUiDQo+ICAyKSBTeXN0ZW0g
Y2FsbCBlbmNyeXB0X21wcm90ZWN0KCkNCj4gDQo+IFRoaXMgcGF0Y2hzZXQgaXMgYnVpbHQgdXBv
biBLaXJpbGwgU2h1dGVtb3YncyB3b3JrIGZvciB0aGUgY29yZSBNS1RNRQ0KPiBzdXBwb3J0Lg0K
DQpQbGVhc2UsIGV4cGxhaW4gd2hhdCBNS1RNRSBpcyByaWdodCBoZXJlLg0KDQpObyByZWZlcmVu
Y2VzLCBubyBleHBsYW5hdGlvbnMuLi4gRXZlbiB3aXRoIGEgcmVmZXJlbmNlLCBhIHNob3J0DQpz
dW1tYXJ5IHdvdWxkIGJlIHJlYWxseSBuaWNlIHRvIGhhdmUuDQoNCi9KYXJra28NCg=

Re: [RFC v2 00/13] Multi-Key Total Memory Encryption API (MKTME)

[Sakkinen, Jarkko]

On Mon, 2018-12-03 at 23:39 -0800, Alison Schofield wrote:
> Hi Thomas, David,
> 
> Here is an updated RFC on the API's to support MKTME.
> (Multi-Key Total Memory Encryption)
> 
> This RFC presents the 2 API additions to support the creation and
> usage of memory encryption keys:
>  1) Kernel Key Service type "mktme"
>  2) System call encrypt_mprotect()
> 
> This patchset is built upon Kirill Shutemov's work for the core MKTME
> support.

Please, explain what MKTME is right here.

No references, no explanations... Even with a reference, a short
summary would be really nice to have.

/Jarkko

Re: [RFC v2 00/13] Multi-Key Total Memory Encryption API (MKTME)

[Sakkinen, Jarkko]

On Mon, 2018-12-03 at 23:39 -0800, Alison Schofield wrote:
> Hi Thomas, David,
> 
> Here is an updated RFC on the API's to support MKTME.
> (Multi-Key Total Memory Encryption)
> 
> This RFC presents the 2 API additions to support the creation and
> usage of memory encryption keys:
>  1) Kernel Key Service type "mktme"
>  2) System call encrypt_mprotect()
> 
> This patchset is built upon Kirill Shutemov's work for the core MKTME
> support.

Please, explain what MKTME is right here.

No references, no explanations... Even with a reference, a short
summary would be really nice to have.

/Jarkko

Re: [RFC v2 00/13] Multi-Key Total Memory Encryption API (MKTME)

[Sakkinen, Jarkko]

T24gVHVlLCAyMDE4LTEyLTA0IGF0IDEyOjQ2ICswMzAwLCBLaXJpbGwgQS4gU2h1dGVtb3Ygd3Jv
dGU6DQo+IE9uIFR1ZSwgRGVjIDA0LCAyMDE4IGF0IDA5OjI1OjUwQU0gKzAwMDAsIFBldGVyIFpp
amxzdHJhIHdyb3RlOg0KPiA+IE9uIE1vbiwgRGVjIDAzLCAyMDE4IGF0IDExOjM5OjQ3UE0gLTA4
MDAsIEFsaXNvbiBTY2hvZmllbGQgd3JvdGU6DQo+ID4gPiAoTXVsdGktS2V5IFRvdGFsIE1lbW9y
eSBFbmNyeXB0aW9uKQ0KPiA+IA0KPiA+IEkgdGhpbmsgdGhhdCBNS1RNRSBpcyBhIGhvcnJpYmxl
IG5hbWUsIGFuZCBkb2Vzbid0IGFwcGVhciB0byBhY2N1cmF0ZWx5DQo+ID4gZGVzY3JpYmUgd2hh
dCBpdCBkb2VzIGVpdGhlci4gU3BlY2lmaWNhbGx5IHRoZSAndG90YWwnIHNlZW1zIG91dCBvZg0K
PiA+IHBsYWNlLCBpdCBkb2Vzbid0IHJlcXVpcmUgYWxsIG1lbW9yeSB0byBiZSBlbmNyeXB0ZWQu
DQo+IA0KPiBNS1RNRSBpbXBsaWVzIFRNRS4gVE1FIGlzIGVuYWJsZWQgYnkgQklPUyBhbmQgaXQg
ZW5jcnlwdHMgYWxsIG1lbW9yeSB3aXRoDQo+IENQVS1nZW5lcmF0ZWQga2V5LiBNS1RNRSBhbGxv
d3MgdG8gdXNlIG90aGVyIGtleXMgb3IgZGlzYWJsZSBlbmNyeXB0aW9uDQo+IGZvciBhIHBhZ2Uu
DQoNCldoZW4geW91IHNheSAiZGlzYWJsZSBlbmNyeXB0aW9uIHRvIGEgcGFnZSIgZG9lcyB0aGUg
ZW5jcnlwdGlvbiBnZXQNCmFjdHVhbGx5IGRpc2FibGVkIG9yIGRvZXMgdGhlIENQVSBqdXN0IGRl
Y3J5cHQgaXQgdHJhbnNwYXJlbnRseSBpLmUuDQp3aGF0IGhhcHBlbnMgcGh5c2ljYWxseT8NCg0K
PiBCdXQsIHllcywgbmFtZSBpcyBub3QgZ29vZC4NCg0KL0phcmtrbw0K

Re: [RFC v2 00/13] Multi-Key Total Memory Encryption API (MKTME)

[Sakkinen, Jarkko]

On Tue, 2018-12-04 at 12:46 +0300, Kirill A. Shutemov wrote:
> On Tue, Dec 04, 2018 at 09:25:50AM +0000, Peter Zijlstra wrote:
> > On Mon, Dec 03, 2018 at 11:39:47PM -0800, Alison Schofield wrote:
> > > (Multi-Key Total Memory Encryption)
> > 
> > I think that MKTME is a horrible name, and doesn't appear to accurately
> > describe what it does either. Specifically the 'total' seems out of
> > place, it doesn't require all memory to be encrypted.
> 
> MKTME implies TME. TME is enabled by BIOS and it encrypts all memory with
> CPU-generated key. MKTME allows to use other keys or disable encryption
> for a page.

When you say "disable encryption to a page" does the encryption get
actually disabled or does the CPU just decrypt it transparently i.e.
what happens physically?

> But, yes, name is not good.

/Jarkko

Re: [RFC v2 00/13] Multi-Key Total Memory Encryption API (MKTME)

[Sakkinen, Jarkko]

On Tue, 2018-12-04 at 12:46 +0300, Kirill A. Shutemov wrote:
> On Tue, Dec 04, 2018 at 09:25:50AM +0000, Peter Zijlstra wrote:
> > On Mon, Dec 03, 2018 at 11:39:47PM -0800, Alison Schofield wrote:
> > > (Multi-Key Total Memory Encryption)
> > 
> > I think that MKTME is a horrible name, and doesn't appear to accurately
> > describe what it does either. Specifically the 'total' seems out of
> > place, it doesn't require all memory to be encrypted.
> 
> MKTME implies TME. TME is enabled by BIOS and it encrypts all memory with
> CPU-generated key. MKTME allows to use other keys or disable encryption
> for a page.

When you say "disable encryption to a page" does the encryption get
actually disabled or does the CPU just decrypt it transparently i.e.
what happens physically?

> But, yes, name is not good.

/Jarkko

Re: [RFC v2 00/13] Multi-Key Total Memory Encryption API (MKTME)

[Sakkinen, Jarkko]

T24gVHVlLCAyMDE4LTEyLTA0IGF0IDExOjE5IC0wODAwLCBBbmR5IEx1dG9taXJza2kgd3JvdGU6
DQo+IEknbSBub3QgVGhvbWFzLCBidXQgSSB0aGluayBpdCdzIHRoZSB3cm9uZyBkaXJlY3Rpb24u
ICBBcyBpdCBzdGFuZHMsDQo+IGVuY3J5cHRfbXByb3RlY3QoKSBpcyBhbiBpbmNvbXBsZXRlIHZl
cnNpb24gb2YgbXByb3RlY3QoKSAoc2luY2UgaXQncw0KPiBtaXNzaW5nIHRoZSBwcm90ZWN0aW9u
IGtleSBzdXBwb3J0KSwgYW5kIGl0J3MgYWxzbyBmdW5jdGlvbmFsbHkganVzdA0KPiBNQURWX0RP
TlRORUVELiAgSW4gb3RoZXIgd29yZHMsIHRoZSBzb2xlIHVzZXItdmlzaWJsZSBlZmZlY3QgYXBw
ZWFycw0KPiB0byBiZSB0aGF0IHRoZSBleGlzdGluZyBwYWdlcyBhcmUgYmxvd24gYXdheS4gIFRo
ZSBmYWN0IHRoYXQgaXQNCj4gY2hhbmdlcyB0aGUga2V5IGluIHVzZSBkb2Vzbid0IHNlZW0gdGVy
cmlibHkgdXNlZnVsLCBzaW5jZSBpdCdzDQo+IGFub255bW91cyBtZW1vcnksIGFuZCB0aGUgbW9z
dCBzZWN1cmUgY2hvaWNlIGlzIHRvIHVzZSBDUFUtbWFuYWdlZA0KPiBrZXlpbmcsIHdoaWNoIGFw
cGVhcnMgdG8gYmUgdGhlIGRlZmF1bHQgYW55d2F5IG9uIFRNRSBzeXN0ZW1zLiAgSXQNCj4gYWxz
byBoYXMgdG90YWxseSB1bmNsZWFyIHNlbWFudGljcyBXUlQgc3dhcCwgYW5kLCBvZmYgdGhlIHRv
cCBvZiBteQ0KPiBoZWFkLCBpdCBsb29rcyBsaWtlIGl0IG1heSBoYXZlIHNlcmlvdXMgY2FjaGUt
Y29oZXJlbmN5IGlzc3VlcyBhbmQNCj4gbGlrZSBzd2FwcGluZyB0aGUgcGFnZXMgbWlnaHQgY29y
cnVwdCB0aGVtLCBib3RoIGJlY2F1c2UgdGhlcmUgYXJlIG5vDQo+IGZsdXNoZXMgYW5kIGJlY2F1
c2UgdGhlIGRpcmVjdC1tYXAgYWxpYXMgbG9va3MgbGlrZSBpdCB3aWxsIHVzZSB0aGUNCj4gZGVm
YXVsdCBrZXkgYW5kIHRoZXJlZm9yZSBhcHBlYXIgdG8gY29udGFpbiB0aGUgd3JvbmcgZGF0YS4N
Cj4gDQo+IEkgd291bGQgcHJvcG9zZSBhIHZlcnkgZGlmZmVyZW50IGRpcmVjdGlvbjogZG9uJ3Qg
dHJ5IHRvIHN1cHBvcnQgTUtUTUUNCj4gYXQgYWxsIGZvciBhbm9ueW1vdXMgbWVtb3J5LCBhbmQg
aW5zdGVhZCBmaWd1cmUgb3V0IHRoZSBpbXBvcnRhbnQgdXNlDQo+IGNhc2VzIGFuZCBzdXBwb3J0
IHRoZW0gZGlyZWN0bHkuICBUaGUgdXNlIGNhc2VzIHRoYXQgSSBjYW4gdGhpbmsgb2YNCj4gb2Zm
IHRoZSB0b3Agb2YgbXkgaGVhZCBhcmU6DQo+IA0KPiAxLiBwbWVtLiAgVGhpcyBzaG91bGQgcHJv
YmFibHkgdXNlIGEgdmVyeSBkaWZmZXJlbnQgQVBJLg0KPiANCj4gMi4gU29tZSBraW5kIG9mIFZN
IGhhcmRlbmluZywgd2hlcmUgYSBWTSdzIG1lbW9yeSBjYW4gYmUgcHJvdGVjdGVkIGENCj4gbGl0
dGxlIHRpbnkgYml0IGZyb20gdGhlIG1haW4ga2VybmVsLiAgQnV0IEkgZG9uJ3Qgc2VlIHdoeSB0
aGlzIGlzIGFueQ0KPiBiZXR0ZXIgdGhhbiBYUE8gKGVYY2x1c2l2ZSBQYWdlLWZyYW1lIE93bmVy
c2hpcCksIHdoaWNoIGJyaW5ncyB0bw0KPiBtaW5kOg0KDQpXaGF0IGlzIHRoZSB0aHJlYXQgbW9k
ZWwgYW55d2F5IGZvciBBTUQgYW5kIEludGVsIHRlY2hub2xvZ2llcz8NCg0KRm9yIG1lIGl0IGxv
b2tzIGxpa2UgdGhhdCB5b3UgY2FuIHJlYWQsIHdyaXRlIGFuZCBldmVuIHJlcGxheSANCmVuY3J5
cHRlZCBwYWdlcyBib3RoIGluIFNNRSBhbmQgVE1FLiANCg0KL0phcmtrbw0K

Re: [RFC v2 00/13] Multi-Key Total Memory Encryption API (MKTME)

[Sakkinen, Jarkko]

On Tue, 2018-12-04 at 11:19 -0800, Andy Lutomirski wrote:
> I'm not Thomas, but I think it's the wrong direction.  As it stands,
> encrypt_mprotect() is an incomplete version of mprotect() (since it's
> missing the protection key support), and it's also functionally just
> MADV_DONTNEED.  In other words, the sole user-visible effect appears
> to be that the existing pages are blown away.  The fact that it
> changes the key in use doesn't seem terribly useful, since it's
> anonymous memory, and the most secure choice is to use CPU-managed
> keying, which appears to be the default anyway on TME systems.  It
> also has totally unclear semantics WRT swap, and, off the top of my
> head, it looks like it may have serious cache-coherency issues and
> like swapping the pages might corrupt them, both because there are no
> flushes and because the direct-map alias looks like it will use the
> default key and therefore appear to contain the wrong data.
> 
> I would propose a very different direction: don't try to support MKTME
> at all for anonymous memory, and instead figure out the important use
> cases and support them directly.  The use cases that I can think of
> off the top of my head are:
> 
> 1. pmem.  This should probably use a very different API.
> 
> 2. Some kind of VM hardening, where a VM's memory can be protected a
> little tiny bit from the main kernel.  But I don't see why this is any
> better than XPO (eXclusive Page-frame Ownership), which brings to
> mind:

What is the threat model anyway for AMD and Intel technologies?

For me it looks like that you can read, write and even replay 
encrypted pages both in SME and TME. 

/Jarkko

Re: [RFC v2 00/13] Multi-Key Total Memory Encryption API (MKTME)

[Sakkinen, Jarkko]

On Tue, 2018-12-04 at 11:19 -0800, Andy Lutomirski wrote:
> I'm not Thomas, but I think it's the wrong direction.  As it stands,
> encrypt_mprotect() is an incomplete version of mprotect() (since it's
> missing the protection key support), and it's also functionally just
> MADV_DONTNEED.  In other words, the sole user-visible effect appears
> to be that the existing pages are blown away.  The fact that it
> changes the key in use doesn't seem terribly useful, since it's
> anonymous memory, and the most secure choice is to use CPU-managed
> keying, which appears to be the default anyway on TME systems.  It
> also has totally unclear semantics WRT swap, and, off the top of my
> head, it looks like it may have serious cache-coherency issues and
> like swapping the pages might corrupt them, both because there are no
> flushes and because the direct-map alias looks like it will use the
> default key and therefore appear to contain the wrong data.
> 
> I would propose a very different direction: don't try to support MKTME
> at all for anonymous memory, and instead figure out the important use
> cases and support them directly.  The use cases that I can think of
> off the top of my head are:
> 
> 1. pmem.  This should probably use a very different API.
> 
> 2. Some kind of VM hardening, where a VM's memory can be protected a
> little tiny bit from the main kernel.  But I don't see why this is any
> better than XPO (eXclusive Page-frame Ownership), which brings to
> mind:

What is the threat model anyway for AMD and Intel technologies?

For me it looks like that you can read, write and even replay 
encrypted pages both in SME and TME. 

/Jarkko

Re: [RFC v2 01/13] x86/mktme: Document the MKTME APIs

[Andy Lutomirski]

>> On Dec 5, 2018, at 11:22 AM, Alison Schofield <alison.schofield@intel.com> wrote:
>>
>> On Wed, Dec 05, 2018 at 10:11:18AM -0800, Andy Lutomirski wrote:
>>
>>
>>> On Dec 3, 2018, at 11:39 PM, Alison Schofield <alison.schofield@intel.com> wrote:
>>
>> I realize you’re writing code to expose hardware behavior, but I’m not sure this
>> really makes sense in this context.
>
> Your observation is accurate. The Usage defined here is very closely
> aligned to the Intel MKTME Architecture spec. That's a starting point,
> but not the ending point. We need to implement the feature set that
> makes sense. More below...
>
>>> +
>>> +    type=
>>> +        *user*    User will supply the encryption key data. Use this
>>> +                type to directly program a hardware encryption key.
>>> +
>>
>> I think that “user” probably sense as a “key service” key, but I don’t think it is at all useful for non-persistent memory.  Even if we take for granted that MKTME for anonymous memory is useful at all, “cpu” seems to be better in all respects.
>>
>>
>> Perhaps support for “user” should be tabled until there’s a design for how to use this for pmem?  I imagine it would look quite a bit like dm-crypt.  Advanced pmem filesystems could plausibly use different keys for different files, I suppose.
>>
>> If “user” is dropped, I think a lot of the complexity goes away. Hotplug becomes automatic, right?
>
> Dropping 'user' type removes a great deal of complexity.
>
> Let me follow up in 2 ways:
> 1) Find out when MKTME support for pmem is required.
> 2) Go back to the the requirements and get the justification for user
> type.
>
>>
>>> +        *cpu*    User requests a CPU generated encryption key.
>>
>> Okay, maybe, but it’s still unclear to me exactly what the intended benefit is, though.
> *cpu* is the RANDOM key generated by the cpu. If there were no other
> options, then this would be default, and go away.
>
>>> +        *clear* User requests that a hardware encryption key be
>>> +                cleared. This will clear the encryption key from
>>> +                the hardware. On execution this hardware key gets
>>> +                TME behavior.
>>> +
>>
>> Why is this a key type?  Shouldn’t the API to select a key just have an option to ask for no key to be used?
>
> The *clear* key has been requested in order to clear/erase the users
> key data that has been programmed into a hardware slot. User does not
> want to leave a slot programmed with their encryption data when they
> are done with it.

Can’t you just clear the key when the key is deleted by the user?
Asking the user to allocate a *new* key and hope that it somehow ends
up in the same spot seems like a poor design, especially if future
hardware gains support for key slot virtualization in some way that
makes the slot allocation more dynamic.

>
>>> +        *no-encrypt*
>>> +                 User requests that hardware does not encrypt
>>> +                 memory when this key is in use.
>>
>> Same as above.  If there’s a performance benefit, then there could be a way to ask for cleartext memory.  Similarly, some pmem users may want a way to keep their pmem unencrypted.
>
> So, this is the way to ask for cleartext memory.
> The entire system will be encrypted with the system wide TME Key.
> A subset of that will be protected with MKTME Keys.
> If user wants, no encrypt, this *no-encrypt* is the way to do it.
>

Understood.  I’m saying that having a *key* (in the add_key sense) for
it seems unnecessary.  Whatever the final API for controlling the use
of keys, adding an option to ask for clear text seems reasonable.
This actually seems more useful for anonymous memory than the
cpu-generates keys are IMO.

I do think that, before you invest too much time in perfecting the
series with the current design, you should identify the use cases,
make sure the use cases are valid, and figure out whether your API
design is appropriate.  After considerable head-scratching, I haven’t
thought of a reason that explicit CPU generated keys are any better
than the default TME key, at least in the absence of additional
hardware support for locking down what code can use what key.  The
sole exception is that a key can be removed, which is probably faster
than directly zeroing large amounts of data.

I understand that it would be very nice to say "hey, cloud customer,
your VM has all its memory encrypted with a key that is unique to your
VM", but that seems to be more or less just a platitude with no actual
effect.  Anyone who snoops the memory bus or steals a DIMM learns
nothing unless they also take control of the CPU and can replay all
the data into the CPU.  On the other hand, anyone who can get the CPU
to read from a given physical address (which seems like the most
likely threat) can just get the CPU to decrypt any tenant's data.  So,
for example, if someone manages to write a couple of words to the EPT
for one VM, then they can easily read another VM's data, MKTME or no
MKTME, because the memory controller has no clue which VM initiated
the access.

I suppose there's some smallish value in rotating the key every now
and then to make old data non-replayable, but an attack that
compromises the memory bus and only later compromises the CPU is a
strange threat model.

Re: [RFC v2 01/13] x86/mktme: Document the MKTME APIs

[Andy Lutomirski]

>> On Dec 5, 2018, at 11:22 AM, Alison Schofield <alison.schofield@intel.com> wrote:
>>
>> On Wed, Dec 05, 2018 at 10:11:18AM -0800, Andy Lutomirski wrote:
>>
>>
>>> On Dec 3, 2018, at 11:39 PM, Alison Schofield <alison.schofield@intel.com> wrote:
>>
>> I realize you’re writing code to expose hardware behavior, but I’m not sure this
>> really makes sense in this context.
>
> Your observation is accurate. The Usage defined here is very closely
> aligned to the Intel MKTME Architecture spec. That's a starting point,
> but not the ending point. We need to implement the feature set that
> makes sense. More below...
>
>>> +
>>> +    type=
>>> +        *user*    User will supply the encryption key data. Use this
>>> +                type to directly program a hardware encryption key.
>>> +
>>
>> I think that “user” probably sense as a “key service” key, but I don’t think it is at all useful for non-persistent memory.  Even if we take for granted that MKTME for anonymous memory is useful at all, “cpu” seems to be better in all respects.
>>
>>
>> Perhaps support for “user” should be tabled until there’s a design for how to use this for pmem?  I imagine it would look quite a bit like dm-crypt.  Advanced pmem filesystems could plausibly use different keys for different files, I suppose.
>>
>> If “user” is dropped, I think a lot of the complexity goes away. Hotplug becomes automatic, right?
>
> Dropping 'user' type removes a great deal of complexity.
>
> Let me follow up in 2 ways:
> 1) Find out when MKTME support for pmem is required.
> 2) Go back to the the requirements and get the justification for user
> type.
>
>>
>>> +        *cpu*    User requests a CPU generated encryption key.
>>
>> Okay, maybe, but it’s still unclear to me exactly what the intended benefit is, though.
> *cpu* is the RANDOM key generated by the cpu. If there were no other
> options, then this would be default, and go away.
>
>>> +        *clear* User requests that a hardware encryption key be
>>> +                cleared. This will clear the encryption key from
>>> +                the hardware. On execution this hardware key gets
>>> +                TME behavior.
>>> +
>>
>> Why is this a key type?  Shouldn’t the API to select a key just have an option to ask for no key to be used?
>
> The *clear* key has been requested in order to clear/erase the users
> key data that has been programmed into a hardware slot. User does not
> want to leave a slot programmed with their encryption data when they
> are done with it.

Can’t you just clear the key when the key is deleted by the user?
Asking the user to allocate a *new* key and hope that it somehow ends
up in the same spot seems like a poor design, especially if future
hardware gains support for key slot virtualization in some way that
makes the slot allocation more dynamic.

>
>>> +        *no-encrypt*
>>> +                 User requests that hardware does not encrypt
>>> +                 memory when this key is in use.
>>
>> Same as above.  If there’s a performance benefit, then there could be a way to ask for cleartext memory.  Similarly, some pmem users may want a way to keep their pmem unencrypted.
>
> So, this is the way to ask for cleartext memory.
> The entire system will be encrypted with the system wide TME Key.
> A subset of that will be protected with MKTME Keys.
> If user wants, no encrypt, this *no-encrypt* is the way to do it.
>

Understood.  I’m saying that having a *key* (in the add_key sense) for
it seems unnecessary.  Whatever the final API for controlling the use
of keys, adding an option to ask for clear text seems reasonable.
This actually seems more useful for anonymous memory than the
cpu-generates keys are IMO.

I do think that, before you invest too much time in perfecting the
series with the current design, you should identify the use cases,
make sure the use cases are valid, and figure out whether your API
design is appropriate.  After considerable head-scratching, I haven’t
thought of a reason that explicit CPU generated keys are any better
than the default TME key, at least in the absence of additional
hardware support for locking down what code can use what key.  The
sole exception is that a key can be removed, which is probably faster
than directly zeroing large amounts of data.

I understand that it would be very nice to say "hey, cloud customer,
your VM has all its memory encrypted with a key that is unique to your
VM", but that seems to be more or less just a platitude with no actual
effect.  Anyone who snoops the memory bus or steals a DIMM learns
nothing unless they also take control of the CPU and can replay all
the data into the CPU.  On the other hand, anyone who can get the CPU
to read from a given physical address (which seems like the most
likely threat) can just get the CPU to decrypt any tenant's data.  So,
for example, if someone manages to write a couple of words to the EPT
for one VM, then they can easily read another VM's data, MKTME or no
MKTME, because the memory controller has no clue which VM initiated
the access.

I suppose there's some smallish value in rotating the key every now
and then to make old data non-replayable, but an attack that
compromises the memory bus and only later compromises the CPU is a
strange threat model.

Re: [RFC v2 00/13] Multi-Key Total Memory Encryption API (MKTME)

[Dave Hansen]

On 12/4/18 11:19 AM, Andy Lutomirski wrote:
> I'm not Thomas, but I think it's the wrong direction.  As it stands,
> encrypt_mprotect() is an incomplete version of mprotect() (since it's
> missing the protection key support),

I thought about this when I added mprotect_pkey().  We start with:

	mprotect(addr, len, prot);

then

	mprotect_pkey(addr, len, prot);

then

	mprotect_pkey_encrypt(addr, len, prot, key);

That doesn't scale because we eventually have
mprotect_and_a_history_of_mm_features(). :)

What I was hoping to see was them do this (apologies for the horrible
indentation:

	ptr = mmap(..., PROT_NONE);
	mprotect_pkey(   addr, len, PROT_NONE, pkey);
	mprotect_encrypt(addr, len, PROT_NONE, keyid);
	mprotect(        addr, len, real_prot);

The point is that you *can* stack these things and don't have to have an
mprotect_kitchen_sink() if you use PROT_NONE for intermediate
permissions during setup.

> and it's also functionally just MADV_DONTNEED.  In other words, the
> sole user-visible effect appears to be that the existing pages are
> blown away.  The fact that it changes the key in use doesn't seem
> terribly useful, since it's anonymous memory,

It's functionally MADV_DONTNEED, plus a future promise that your writes
will never show up as plaintext on the DIMM.

We also haven't settled on the file-backed properties.  For file-backed,
my hope was that you could do:

	ptr = mmap(fd, size, prot);
	printf("ciphertext: %x\n", *ptr);
	mprotect_encrypt(ptr, len, prot, keyid);
	printf("plaintext: %x\n", *ptr);

> and the most secure choice is to use CPU-managed keying, which
> appears to be the default anyway on TME systems.  It also has totally
> unclear semantics WRT swap, and, off the top of my head, it looks
> like it may have serious cache-coherency issues and like swapping the
> pages might corrupt them, both because there are no flushes and
> because the direct-map alias looks like it will use the default key
> and therefore appear to contain the wrong data.

I think we fleshed this out on IRC a bit, but the other part of the
implementation is described here: https://lwn.net/Articles/758313/, and
contains a direct map per keyid.  When you do phys_to_virt() and
friends, you get the correct, decrypted view direct map which is
appropriate for the physical page.  And, yes, this has very
consequential security implications.

> I would propose a very different direction: don't try to support MKTME
> at all for anonymous memory, and instead figure out the important use
> cases and support them directly.  The use cases that I can think of
> off the top of my head are:
> 
> 1. pmem.  This should probably use a very different API.
> 
> 2. Some kind of VM hardening, where a VM's memory can be protected a
> little tiny bit from the main kernel.  But I don't see why this is any
> better than XPO (eXclusive Page-frame Ownership), which brings to
> mind:

The XPO approach is "fun", and would certainly be a way to keep the
direct map from being exploited to get access to plain-text mappings of
ciphertext.

But, it also has massive performance implications and we didn't quite
want to go there quite yet.

> The main implementation concern I have with this patch set is cache
> coherency and handling of the direct map.  Unless I missed something,
> you're not doing anything about the direct map, which means that you
> have RW aliases of the same memory with different keys.  For use case
> #2, this probably means that you need to either get rid of the direct
> map and make get_user_pages() fail, or you need to change the key on
> the direct map as well, probably using the pageattr.c code.

The current, public hardware spec has a description of what's required
to maintain cache coherency.  Basically, you can keep as many mappings
of a physical page as you want, but only write to one mapping at a time,
and clflush the old one when you want to write to a new one.

> As for caching, As far as I can tell from reading the preliminary
> docs, Intel's MKTME, much like AMD's SME, is basically invisible to
> the hardware cache coherency mechanism.  So, if you modify a physical
> address with one key (or SME-enable bit), and you read it with
> another, you get garbage unless you flush.  And, if you modify memory
> with one key then remap it with a different key without flushing in
> the mean time, you risk corruption.

Yes, all true (at least with respect to Intel's implementation).

> And, what's worse, if I'm reading
> between the lines in the docs correctly, if you use PCONFIG to change
> a key, you may need to do a bunch of cache flushing to ensure you get
> reasonable effects.  (If you have dirty cache lines for some (PA, key)
> and you PCONFIG to change the underlying key, you get different
> results depending on whether the writeback happens before or after the
> package doing the writeback notices the PCONFIG.)

We're not going to allow a key to be PCONFIG'd while there are any
physical pages still associated with it.  There are per-VMA refcounts
tied back to the keyid slots, IIRC.  So, before PCONFIG can happen, we
just need to make sure that all the VMAs are gone, all the pages are
freed, and all dirty cachelines have been clflushed.

This is where get_user_pages() is our mortal enemy, though.  I hope we
got that right.  Kirill/Alison, we should chat about this one. :)

> Finally, If you're going to teach the kernel how to have some user
> pages that aren't in the direct map, you've essentially done XPO,
> which is nifty but expensive.  And I think that doing this gets you
> essentially all the benefit of MKTME for the non-pmem use case.  Why
> exactly would any software want to use anything other than a
> CPU-managed key for anything other than pmem?

It is handy, for one, to let you "cluster" key usage.  If you have 5
Pepsi VMs and 5 Coke VMs, each Pepsi one using the same key and each
Coke one using the same key, you can boil it down to only 2 hardware
keyid slots that get used, and do this transparently.

But, I think what you're implying is that the security properties of
user-supplied keys can only be *worse* than using CPU-generated keys
(assuming the CPU does a good job generating it).  So, why bother
allowing user-specified keys in the first place?

It's a good question and I don't have a solid answer for why folks want
this.  I'll find out.

Re: [RFC v2 00/13] Multi-Key Total Memory Encryption API (MKTME)

[Dave Hansen]

On 12/4/18 11:19 AM, Andy Lutomirski wrote:
> I'm not Thomas, but I think it's the wrong direction.  As it stands,
> encrypt_mprotect() is an incomplete version of mprotect() (since it's
> missing the protection key support),

I thought about this when I added mprotect_pkey().  We start with:

	mprotect(addr, len, prot);

then

	mprotect_pkey(addr, len, prot);

then

	mprotect_pkey_encrypt(addr, len, prot, key);

That doesn't scale because we eventually have
mprotect_and_a_history_of_mm_features(). :)

What I was hoping to see was them do this (apologies for the horrible
indentation:

	ptr = mmap(..., PROT_NONE);
	mprotect_pkey(   addr, len, PROT_NONE, pkey);
	mprotect_encrypt(addr, len, PROT_NONE, keyid);
	mprotect(        addr, len, real_prot);

The point is that you *can* stack these things and don't have to have an
mprotect_kitchen_sink() if you use PROT_NONE for intermediate
permissions during setup.

> and it's also functionally just MADV_DONTNEED.  In other words, the
> sole user-visible effect appears to be that the existing pages are
> blown away.  The fact that it changes the key in use doesn't seem
> terribly useful, since it's anonymous memory,

It's functionally MADV_DONTNEED, plus a future promise that your writes
will never show up as plaintext on the DIMM.

We also haven't settled on the file-backed properties.  For file-backed,
my hope was that you could do:

	ptr = mmap(fd, size, prot);
	printf("ciphertext: %x\n", *ptr);
	mprotect_encrypt(ptr, len, prot, keyid);
	printf("plaintext: %x\n", *ptr);

> and the most secure choice is to use CPU-managed keying, which
> appears to be the default anyway on TME systems.  It also has totally
> unclear semantics WRT swap, and, off the top of my head, it looks
> like it may have serious cache-coherency issues and like swapping the
> pages might corrupt them, both because there are no flushes and
> because the direct-map alias looks like it will use the default key
> and therefore appear to contain the wrong data.

I think we fleshed this out on IRC a bit, but the other part of the
implementation is described here: https://lwn.net/Articles/758313/, and
contains a direct map per keyid.  When you do phys_to_virt() and
friends, you get the correct, decrypted view direct map which is
appropriate for the physical page.  And, yes, this has very
consequential security implications.

> I would propose a very different direction: don't try to support MKTME
> at all for anonymous memory, and instead figure out the important use
> cases and support them directly.  The use cases that I can think of
> off the top of my head are:
> 
> 1. pmem.  This should probably use a very different API.
> 
> 2. Some kind of VM hardening, where a VM's memory can be protected a
> little tiny bit from the main kernel.  But I don't see why this is any
> better than XPO (eXclusive Page-frame Ownership), which brings to
> mind:

The XPO approach is "fun", and would certainly be a way to keep the
direct map from being exploited to get access to plain-text mappings of
ciphertext.

But, it also has massive performance implications and we didn't quite
want to go there quite yet.

> The main implementation concern I have with this patch set is cache
> coherency and handling of the direct map.  Unless I missed something,
> you're not doing anything about the direct map, which means that you
> have RW aliases of the same memory with different keys.  For use case
> #2, this probably means that you need to either get rid of the direct
> map and make get_user_pages() fail, or you need to change the key on
> the direct map as well, probably using the pageattr.c code.

The current, public hardware spec has a description of what's required
to maintain cache coherency.  Basically, you can keep as many mappings
of a physical page as you want, but only write to one mapping at a time,
and clflush the old one when you want to write to a new one.

> As for caching, As far as I can tell from reading the preliminary
> docs, Intel's MKTME, much like AMD's SME, is basically invisible to
> the hardware cache coherency mechanism.  So, if you modify a physical
> address with one key (or SME-enable bit), and you read it with
> another, you get garbage unless you flush.  And, if you modify memory
> with one key then remap it with a different key without flushing in
> the mean time, you risk corruption.

Yes, all true (at least with respect to Intel's implementation).

> And, what's worse, if I'm reading
> between the lines in the docs correctly, if you use PCONFIG to change
> a key, you may need to do a bunch of cache flushing to ensure you get
> reasonable effects.  (If you have dirty cache lines for some (PA, key)
> and you PCONFIG to change the underlying key, you get different
> results depending on whether the writeback happens before or after the
> package doing the writeback notices the PCONFIG.)

We're not going to allow a key to be PCONFIG'd while there are any
physical pages still associated with it.  There are per-VMA refcounts
tied back to the keyid slots, IIRC.  So, before PCONFIG can happen, we
just need to make sure that all the VMAs are gone, all the pages are
freed, and all dirty cachelines have been clflushed.

This is where get_user_pages() is our mortal enemy, though.  I hope we
got that right.  Kirill/Alison, we should chat about this one. :)

> Finally, If you're going to teach the kernel how to have some user
> pages that aren't in the direct map, you've essentially done XPO,
> which is nifty but expensive.  And I think that doing this gets you
> essentially all the benefit of MKTME for the non-pmem use case.  Why
> exactly would any software want to use anything other than a
> CPU-managed key for anything other than pmem?

It is handy, for one, to let you "cluster" key usage.  If you have 5
Pepsi VMs and 5 Coke VMs, each Pepsi one using the same key and each
Coke one using the same key, you can boil it down to only 2 hardware
keyid slots that get used, and do this transparently.

But, I think what you're implying is that the security properties of
user-supplied keys can only be *worse* than using CPU-generated keys
(assuming the CPU does a good job generating it).  So, why bother
allowing user-specified keys in the first place?

It's a good question and I don't have a solid answer for why folks want
this.  I'll find out.

Re: [RFC v2 00/13] Multi-Key Total Memory Encryption API (MKTME)

[Andy Lutomirski]

On Wed, Dec 5, 2018 at 3:49 PM Dave Hansen <dave.hansen@intel.com> wrote:
>
> On 12/4/18 11:19 AM, Andy Lutomirski wrote:
> > I'm not Thomas, but I think it's the wrong direction.  As it stands,
> > encrypt_mprotect() is an incomplete version of mprotect() (since it's
> > missing the protection key support),
>
> I thought about this when I added mprotect_pkey().  We start with:
>
>         mprotect(addr, len, prot);
>
> then
>
>         mprotect_pkey(addr, len, prot);
>
> then
>
>         mprotect_pkey_encrypt(addr, len, prot, key);
>
> That doesn't scale because we eventually have
> mprotect_and_a_history_of_mm_features(). :)
>
> What I was hoping to see was them do this (apologies for the horrible
> indentation:
>
>         ptr = mmap(..., PROT_NONE);
>         mprotect_pkey(   addr, len, PROT_NONE, pkey);
>         mprotect_encrypt(addr, len, PROT_NONE, keyid);
>         mprotect(        addr, len, real_prot);
>
> The point is that you *can* stack these things and don't have to have an
> mprotect_kitchen_sink() if you use PROT_NONE for intermediate
> permissions during setup.

Sure, but then why call it mprotect at all?  How about:

mmap(..., PROT_NONE);
mencrypt(..., keyid);
mprotect_pkey(...);

But wouldn't this be much nicer:

int fd = memfd_create(...);
memfd_set_tme_key(fd, keyid);  /* fails if len != 0 */
mmap(fd, ...);

>
> > and it's also functionally just MADV_DONTNEED.  In other words, the
> > sole user-visible effect appears to be that the existing pages are
> > blown away.  The fact that it changes the key in use doesn't seem
> > terribly useful, since it's anonymous memory,
>
> It's functionally MADV_DONTNEED, plus a future promise that your writes
> will never show up as plaintext on the DIMM.

But that's mostly vacuous.  If I read the docs right, MKTME systems
also support TME, so you *already* have that promise, unless the
firmware totally blew it.  If we want a boot option to have the kernel
use MKTME to forcibly encrypt everything regardless of what the TME
MSRs say, I'd be entirely on board.  Heck, the implementation would be
quite simple because we mostly reuse the SME code.

>
> We also haven't settled on the file-backed properties.  For file-backed,
> my hope was that you could do:
>
>         ptr = mmap(fd, size, prot);
>         printf("ciphertext: %x\n", *ptr);
>         mprotect_encrypt(ptr, len, prot, keyid);
>         printf("plaintext: %x\n", *ptr);

Why would you ever want the plaintext?  Also, how does this work on a
normal fs, where relocation of the file would cause the ciphertext to
get lost?  It really seems to be that it should look more like
dm-crypt where you encrypt a filesystem.  Maybe you'd just configure
the pmem device to be encrypted before you mount it, or you'd get a
new pmem-mktme device node instead.  This would also avoid some nasty
multiple-copies-of-the-direct-map issue, since you'd only ever have
one of them mapped.

>
> > The main implementation concern I have with this patch set is cache
> > coherency and handling of the direct map.  Unless I missed something,
> > you're not doing anything about the direct map, which means that you
> > have RW aliases of the same memory with different keys.  For use case
> > #2, this probably means that you need to either get rid of the direct
> > map and make get_user_pages() fail, or you need to change the key on
> > the direct map as well, probably using the pageattr.c code.
>
> The current, public hardware spec has a description of what's required
> to maintain cache coherency.  Basically, you can keep as many mappings
> of a physical page as you want, but only write to one mapping at a time,
> and clflush the old one when you want to write to a new one.

Surely you at least have to clflush the old mapping and then the new
mapping, since the new mapping could have been speculatively read.

> > Finally, If you're going to teach the kernel how to have some user
> > pages that aren't in the direct map, you've essentially done XPO,
> > which is nifty but expensive.  And I think that doing this gets you
> > essentially all the benefit of MKTME for the non-pmem use case.  Why
> > exactly would any software want to use anything other than a
> > CPU-managed key for anything other than pmem?
>
> It is handy, for one, to let you "cluster" key usage.  If you have 5
> Pepsi VMs and 5 Coke VMs, each Pepsi one using the same key and each
> Coke one using the same key, you can boil it down to only 2 hardware
> keyid slots that get used, and do this transparently.

I understand this from a marketing perspective but not a security
perspective.  Say I'm Coke and you've sold me some VMs that are
"encrypted with a Coke-specific key and no other VMs get to use that
key."  I can't think of *any* not-exceedingly-contrived attack in
which this makes the slightest difference.  If Pepsi tries to attack
Coke without MKTME, then they'll either need to get the hypervisor to
leak Coke's data through the direct map or they'll have to find some
way to corrupt a page table or use something like L1TF to read from a
physical address Coke owns.  With MKTME, if they can read through the
host direct map, then they'll get Coke's cleartext, and if they can
corrupt a page table or use L1TF to read from your memory, they'll get
Coke's cleartext.

TME itself provides a ton of protection -- you can't just barge into
the datacenter, refrigerate the DIMMs, walk away with them, and read
off everyone's data.

Am I missing something?

>
> But, I think what you're implying is that the security properties of
> user-supplied keys can only be *worse* than using CPU-generated keys
> (assuming the CPU does a good job generating it).  So, why bother
> allowing user-specified keys in the first place?

That too :)

Re: [RFC v2 00/13] Multi-Key Total Memory Encryption API (MKTME)

[Andy Lutomirski]

On Wed, Dec 5, 2018 at 3:49 PM Dave Hansen <dave.hansen@intel.com> wrote:
>
> On 12/4/18 11:19 AM, Andy Lutomirski wrote:
> > I'm not Thomas, but I think it's the wrong direction.  As it stands,
> > encrypt_mprotect() is an incomplete version of mprotect() (since it's
> > missing the protection key support),
>
> I thought about this when I added mprotect_pkey().  We start with:
>
>         mprotect(addr, len, prot);
>
> then
>
>         mprotect_pkey(addr, len, prot);
>
> then
>
>         mprotect_pkey_encrypt(addr, len, prot, key);
>
> That doesn't scale because we eventually have
> mprotect_and_a_history_of_mm_features(). :)
>
> What I was hoping to see was them do this (apologies for the horrible
> indentation:
>
>         ptr = mmap(..., PROT_NONE);
>         mprotect_pkey(   addr, len, PROT_NONE, pkey);
>         mprotect_encrypt(addr, len, PROT_NONE, keyid);
>         mprotect(        addr, len, real_prot);
>
> The point is that you *can* stack these things and don't have to have an
> mprotect_kitchen_sink() if you use PROT_NONE for intermediate
> permissions during setup.

Sure, but then why call it mprotect at all?  How about:

mmap(..., PROT_NONE);
mencrypt(..., keyid);
mprotect_pkey(...);

But wouldn't this be much nicer:

int fd = memfd_create(...);
memfd_set_tme_key(fd, keyid);  /* fails if len != 0 */
mmap(fd, ...);

>
> > and it's also functionally just MADV_DONTNEED.  In other words, the
> > sole user-visible effect appears to be that the existing pages are
> > blown away.  The fact that it changes the key in use doesn't seem
> > terribly useful, since it's anonymous memory,
>
> It's functionally MADV_DONTNEED, plus a future promise that your writes
> will never show up as plaintext on the DIMM.

But that's mostly vacuous.  If I read the docs right, MKTME systems
also support TME, so you *already* have that promise, unless the
firmware totally blew it.  If we want a boot option to have the kernel
use MKTME to forcibly encrypt everything regardless of what the TME
MSRs say, I'd be entirely on board.  Heck, the implementation would be
quite simple because we mostly reuse the SME code.

>
> We also haven't settled on the file-backed properties.  For file-backed,
> my hope was that you could do:
>
>         ptr = mmap(fd, size, prot);
>         printf("ciphertext: %x\n", *ptr);
>         mprotect_encrypt(ptr, len, prot, keyid);
>         printf("plaintext: %x\n", *ptr);

Why would you ever want the plaintext?  Also, how does this work on a
normal fs, where relocation of the file would cause the ciphertext to
get lost?  It really seems to be that it should look more like
dm-crypt where you encrypt a filesystem.  Maybe you'd just configure
the pmem device to be encrypted before you mount it, or you'd get a
new pmem-mktme device node instead.  This would also avoid some nasty
multiple-copies-of-the-direct-map issue, since you'd only ever have
one of them mapped.

>
> > The main implementation concern I have with this patch set is cache
> > coherency and handling of the direct map.  Unless I missed something,
> > you're not doing anything about the direct map, which means that you
> > have RW aliases of the same memory with different keys.  For use case
> > #2, this probably means that you need to either get rid of the direct
> > map and make get_user_pages() fail, or you need to change the key on
> > the direct map as well, probably using the pageattr.c code.
>
> The current, public hardware spec has a description of what's required
> to maintain cache coherency.  Basically, you can keep as many mappings
> of a physical page as you want, but only write to one mapping at a time,
> and clflush the old one when you want to write to a new one.

Surely you at least have to clflush the old mapping and then the new
mapping, since the new mapping could have been speculatively read.

> > Finally, If you're going to teach the kernel how to have some user
> > pages that aren't in the direct map, you've essentially done XPO,
> > which is nifty but expensive.  And I think that doing this gets you
> > essentially all the benefit of MKTME for the non-pmem use case.  Why
> > exactly would any software want to use anything other than a
> > CPU-managed key for anything other than pmem?
>
> It is handy, for one, to let you "cluster" key usage.  If you have 5
> Pepsi VMs and 5 Coke VMs, each Pepsi one using the same key and each
> Coke one using the same key, you can boil it down to only 2 hardware
> keyid slots that get used, and do this transparently.

I understand this from a marketing perspective but not a security
perspective.  Say I'm Coke and you've sold me some VMs that are
"encrypted with a Coke-specific key and no other VMs get to use that
key."  I can't think of *any* not-exceedingly-contrived attack in
which this makes the slightest difference.  If Pepsi tries to attack
Coke without MKTME, then they'll either need to get the hypervisor to
leak Coke's data through the direct map or they'll have to find some
way to corrupt a page table or use something like L1TF to read from a
physical address Coke owns.  With MKTME, if they can read through the
host direct map, then they'll get Coke's cleartext, and if they can
corrupt a page table or use L1TF to read from your memory, they'll get
Coke's cleartext.

TME itself provides a ton of protection -- you can't just barge into
the datacenter, refrigerate the DIMMs, walk away with them, and read
off everyone's data.

Am I missing something?

>
> But, I think what you're implying is that the security properties of
> user-supplied keys can only be *worse* than using CPU-generated keys
> (assuming the CPU does a good job generating it).  So, why bother
> allowing user-specified keys in the first place?

That too :)

Re: [RFC v2 00/13] Multi-Key Total Memory Encryption API (MKTME)

[Dan Williams]

[ only responding to the pmem side of things... ]

On Wed, Dec 5, 2018 at 5:09 PM Andy Lutomirski <luto@kernel.org> wrote:
>
> On Wed, Dec 5, 2018 at 3:49 PM Dave Hansen <dave.hansen@intel.com> wrote:
[..]
> > We also haven't settled on the file-backed properties.  For file-backed,
> > my hope was that you could do:
> >
> >         ptr = mmap(fd, size, prot);
> >         printf("ciphertext: %x\n", *ptr);
> >         mprotect_encrypt(ptr, len, prot, keyid);
> >         printf("plaintext: %x\n", *ptr);
>
> Why would you ever want the plaintext?  Also, how does this work on a
> normal fs, where relocation of the file would cause the ciphertext to
> get lost?  It really seems to be that it should look more like
> dm-crypt where you encrypt a filesystem.  Maybe you'd just configure
> the pmem device to be encrypted before you mount it, or you'd get a
> new pmem-mktme device node instead.  This would also avoid some nasty
> multiple-copies-of-the-direct-map issue, since you'd only ever have
> one of them mapped.

Yes, this is really the only way it can work. Otherwise you need to
teach the filesystem that "these blocks can't move without the key
because encryption", and have an fs-feature flag to say "you can't
mount this legacy / encryption unaware filesystem from an older kernel
because we're not sure you'll move something and break the
encryption".

So pmem namespaces (volumes) would be encrypted providing something
similar to dm-crypt, although we're looking at following the lead of
the fscrypt key management scheme.

Re: [RFC v2 00/13] Multi-Key Total Memory Encryption API (MKTME)

[Dan Williams]

[ only responding to the pmem side of things... ]

On Wed, Dec 5, 2018 at 5:09 PM Andy Lutomirski <luto@kernel.org> wrote:
>
> On Wed, Dec 5, 2018 at 3:49 PM Dave Hansen <dave.hansen@intel.com> wrote:
[..]
> > We also haven't settled on the file-backed properties.  For file-backed,
> > my hope was that you could do:
> >
> >         ptr = mmap(fd, size, prot);
> >         printf("ciphertext: %x\n", *ptr);
> >         mprotect_encrypt(ptr, len, prot, keyid);
> >         printf("plaintext: %x\n", *ptr);
>
> Why would you ever want the plaintext?  Also, how does this work on a
> normal fs, where relocation of the file would cause the ciphertext to
> get lost?  It really seems to be that it should look more like
> dm-crypt where you encrypt a filesystem.  Maybe you'd just configure
> the pmem device to be encrypted before you mount it, or you'd get a
> new pmem-mktme device node instead.  This would also avoid some nasty
> multiple-copies-of-the-direct-map issue, since you'd only ever have
> one of them mapped.

Yes, this is really the only way it can work. Otherwise you need to
teach the filesystem that "these blocks can't move without the key
because encryption", and have an fs-feature flag to say "you can't
mount this legacy / encryption unaware filesystem from an older kernel
because we're not sure you'll move something and break the
encryption".

So pmem namespaces (volumes) would be encrypted providing something
similar to dm-crypt, although we're looking at following the lead of
the fscrypt key management scheme.

Re: [RFC v2 01/13] x86/mktme: Document the MKTME APIs

[Sakkinen, Jarkko]

SSdsbCBmb2N1cyBteSByZW1hcmtzIG5vdyB0b3dhcmRzIGRvY3VtZW50YXRpb24gYXMgSSBoYXZl
IGxvdHMgb2YNCmNhdGNoaW5nIHVwIHRvIGRvIHdpdGggVE1FIDotKSBJJ2xsIGdpdmUgbW9yZSBm
ZWVkYmFjayBvZiBhY3R1YWwgY29kZQ0KY2hhbmdlcyBvbmNlIHYxOCBvZiB0aGUgU0dYIHBhdGNo
IHNldCBpcyBvdXQsIHB1bGwgcmVxdWVzdCBmb3IgVFBNIDQuMjENCmNoYW5nZXMgaXMgb3V0IGFu
ZCBtYXliZSBhIG5ldyB2ZXJzaW9uIG9mIHRoaXMgcGF0Y2ggc2V0IGhhcyBiZWVuDQpyZWxlYXNl
ZC4NCg0KUmlnaHQgbm93IHRvbyBtdWNoIG9uIG15IHNob3VsZGVycyB0byBnbyB0b28gZGVlcCB3
aXRoIHRoaXMgcGF0Y2ggc2V0Lg0KDQpPbiBNb24sIDIwMTgtMTItMDMgYXQgMjM6MzkgLTA4MDAs
IEFsaXNvbiBTY2hvZmllbGQgd3JvdGU6DQo+IFRoaXMgaW5jbHVkZXMgYW4gb3ZlcnZpZXcsIGEg
c2VjdGlvbiBvbiBlYWNoIEFQSTogTVRLTUUgS2V5cyBhbmQNCj4gc3lzdGVtIGNhbGwgZW5jcnlw
dF9tcHJvdGVjdCgpLCBhbmQgYSBkZW1vbnN0cmF0aW9uIHByb2dyYW0uDQo+IA0KPiAoU29tZSBv
ZiB0aGlzIGluZm8gaXMgZGVzdGluZWQgZm9yIG1hbiBwYWdlcy4pDQo+IA0KPiBDaGFuZ2UtSWQ6
IEkzNGRjOWZmMWExMzA4YzA1N2VjNGJiM2U2NTJjNGQ3Y2U2OTk1NjA2DQo+IFNpZ25lZC1vZmYt
Ynk6IEFsaXNvbiBTY2hvZmllbGQgPGFsaXNvbi5zY2hvZmllbGRAaW50ZWwuY29tPg0KDQpDby1k
ZXZlbG9wZWQtYnk6IEtpcmlsbCBBLiBTaHV0ZW1vdiA8a2lyaWxsLnNodXRlbW92QGxpbnV4Lmlu
dGVsLmNvbT4gPw0KDQpOb3QgbmVlZGVkIGlmIHRoaXMgaXMgZm9yIHRoZSBtb3N0IHBhcnQgd3Jp
dHRlbiBieSB5b3UuDQoNCj4gU2lnbmVkLW9mZi1ieTogS2lyaWxsIEEuIFNodXRlbW92IDxraXJp
bGwuc2h1dGVtb3ZAbGludXguaW50ZWwuY29tPg0KPiAtLS0NCj4gIERvY3VtZW50YXRpb24veDg2
L21rdG1lL2luZGV4LnJzdCAgICAgICAgICB8ICAxMSArKysNCj4gIERvY3VtZW50YXRpb24veDg2
L21rdG1lL21rdG1lX2RlbW8ucnN0ICAgICB8ICA1MyArKysrKysrKysrKysrKw0KPiAgRG9jdW1l
bnRhdGlvbi94ODYvbWt0bWUvbWt0bWVfZW5jcnlwdC5yc3QgIHwgIDU4ICsrKysrKysrKysrKysr
Kw0KPiAgRG9jdW1lbnRhdGlvbi94ODYvbWt0bWUvbWt0bWVfa2V5cy5yc3QgICAgIHwgMTA5DQo+
ICsrKysrKysrKysrKysrKysrKysrKysrKysrKysrDQo+ICBEb2N1bWVudGF0aW9uL3g4Ni9ta3Rt
ZS9ta3RtZV9vdmVydmlldy5yc3QgfCAgNjAgKysrKysrKysrKysrKysrKw0KPiAgNSBmaWxlcyBj
aGFuZ2VkLCAyOTEgaW5zZXJ0aW9ucygrKQ0KPiAgY3JlYXRlIG1vZGUgMTAwNjQ0IERvY3VtZW50
YXRpb24veDg2L21rdG1lL2luZGV4LnJzdA0KPiAgY3JlYXRlIG1vZGUgMTAwNjQ0IERvY3VtZW50
YXRpb24veDg2L21rdG1lL21rdG1lX2RlbW8ucnN0DQo+ICBjcmVhdGUgbW9kZSAxMDA2NDQgRG9j
dW1lbnRhdGlvbi94ODYvbWt0bWUvbWt0bWVfZW5jcnlwdC5yc3QNCj4gIGNyZWF0ZSBtb2RlIDEw
MDY0NCBEb2N1bWVudGF0aW9uL3g4Ni9ta3RtZS9ta3RtZV9rZXlzLnJzdA0KPiAgY3JlYXRlIG1v
ZGUgMTAwNjQ0IERvY3VtZW50YXRpb24veDg2L21rdG1lL21rdG1lX292ZXJ2aWV3LnJzdA0KPiAN
Cj4gZGlmZiAtLWdpdCBhL0RvY3VtZW50YXRpb24veDg2L21rdG1lL2luZGV4LnJzdA0KPiBiL0Rv
Y3VtZW50YXRpb24veDg2L21rdG1lL2luZGV4LnJzdA0KPiBuZXcgZmlsZSBtb2RlIDEwMDY0NA0K
PiBpbmRleCAwMDAwMDAwMDAwMDAuLjhjNTU2ZDA0Y2JjNA0KPiAtLS0gL2Rldi9udWxsDQo+ICsr
KyBiL0RvY3VtZW50YXRpb24veDg2L21rdG1lL2luZGV4LnJzdA0KPiBAQCAtMCwwICsxLDExIEBA
DQo+ICsNCg0KU1BEWD8NCg0KPiArPT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09
PT09PT09PT09DQo+ICtNdWx0aS1LZXkgVG90YWwgTWVtb3J5IEVuY3J5cHRpb24gKE1LVE1FKSBB
UEkNCj4gKz09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PQ0KPiAr
DQo+ICsuLiB0b2N0cmVlOjoNCj4gKw0KPiArICAgbWt0bWVfb3ZlcnZpZXcNCj4gKyAgIG1rdG1l
X2tleXMNCj4gKyAgIG1rdG1lX2VuY3J5cHQNCj4gKyAgIG1rdG1lX2RlbW8NCj4gZGlmZiAtLWdp
dCBhL0RvY3VtZW50YXRpb24veDg2L21rdG1lL21rdG1lX2RlbW8ucnN0DQo+IGIvRG9jdW1lbnRh
dGlvbi94ODYvbWt0bWUvbWt0bWVfZGVtby5yc3QNCj4gbmV3IGZpbGUgbW9kZSAxMDA2NDQNCj4g
aW5kZXggMDAwMDAwMDAwMDAwLi5hZmQ1MDc3MmU2NWQNCj4gLS0tIC9kZXYvbnVsbA0KPiArKysg
Yi9Eb2N1bWVudGF0aW9uL3g4Ni9ta3RtZS9ta3RtZV9kZW1vLnJzdA0KPiBAQCAtMCwwICsxLDUz
IEBADQo+ICtEZW1vbnN0cmF0aW9uIFByb2dyYW0gdXNpbmcgTUtUTUUgQVBJJ3MNCj4gKz09PT09
PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PQ0KDQpQcm9iYWJseSB3b3VsZCBiZSBi
ZXR0ZXIgaWRlYSB0byBwdXQgaW50byB0b29scy90ZXN0aW5ncy9zZWxmdGVzdC94ODYNCmFuZCBu
b3QgYXMgcGFydCBvZiB0aGUgZG9jdW1lbnRhdGlvbi4NCg0KPiArDQo+ICsvKiBDb21waWxlIHdp
dGggdGhlIGtleXV0aWxzIGxpYnJhcnk6IGNjIC1vIG1kZW1vIG1kZW1vLmMgLWxrZXl1dGlscyAq
Lw0KPiArDQo+ICsjaW5jbHVkZSA8c3lzL21tYW4uaD4NCj4gKyNpbmNsdWRlIDxzeXMvc3lzY2Fs
bC5oPg0KPiArI2luY2x1ZGUgPHN5cy90eXBlcy5oPg0KPiArI2luY2x1ZGUgPGtleXV0aWxzLmg+
DQo+ICsjaW5jbHVkZSA8c3RkaW8uaD4NCj4gKyNpbmNsdWRlIDxzdHJpbmcuaD4NCj4gKyNpbmNs
dWRlIDx1bmlzdGQuaD4NCj4gKw0KPiArI2RlZmluZSBQQUdFX1NJWkUgc3lzY29uZihfU0NfUEFH
RV9TSVpFKQ0KPiArI2RlZmluZSBzeXNfZW5jcnlwdF9tcHJvdGVjdCAzMzUNCj4gKw0KPiArdm9p
ZCBtYWluKHZvaWQpDQo+ICt7DQo+ICsJY2hhciAqb3B0aW9uc19DUFUgPSAiYWxnb3JpdGhtPWFl
cy14dHMtMTI4IHR5cGU9Y3B1IjsNCj4gKwlsb25nIHNpemUgPSBQQUdFX1NJWkU7DQo+ICsgICAg
ICAgIGtleV9zZXJpYWxfdCBrZXk7DQo+ICsJdm9pZCAqcHRyYTsNCj4gKwlpbnQgcmV0Ow0KPiAr
DQo+ICsgICAgICAgIC8qIEFsbG9jYXRlIGFuIE1LVE1FIEtleSAqLw0KPiArCWtleSA9IGFkZF9r
ZXkoIm1rdG1lIiwgInRlc3RrZXkiLCBvcHRpb25zX0NQVSwgc3RybGVuKG9wdGlvbnNfQ1BVKSwN
Cj4gKyAgICAgICAgICAgICAgICAgICAgICBLRVlfU1BFQ19USFJFQURfS0VZUklORyk7DQo+ICsN
Cj4gKwlpZiAoa2V5ID09IC0xKSB7DQo+ICsJCXByaW50ZigiYWRka2V5IEZBSUxFRFxuIik7DQo+
ICsJCXJldHVybjsNCj4gKwl9DQo+ICsgICAgICAgIC8qIE1hcCBhIHBhZ2Ugb2YgQU5PTllNT1VT
IG1lbW9yeSAqLw0KPiArCXB0cmEgPSBtbWFwKE5VTEwsIHNpemUsIFBST1RfTk9ORSwgTUFQX0FO
T05ZTU9VU3xNQVBfUFJJVkFURSwgLTEsIDApOw0KPiArCWlmICghcHRyYSkgew0KPiArCQlwcmlu
dGYoImZhaWxlZCB0byBtbWFwIik7DQo+ICsJCWdvdG8gaW52YWxfa2V5Ow0KPiArCX0NCj4gKyAg
ICAgICAgLyogRW5jcnlwdCB0aGF0IHBhZ2Ugb2YgbWVtb3J5IHdpdGggdGhlIE1LVE1FIEtleSAq
Lw0KPiArCXJldCA9IHN5c2NhbGwoc3lzX2VuY3J5cHRfbXByb3RlY3QsIHB0cmEsIHNpemUsIFBS
T1RfTk9ORSwga2V5KTsNCj4gKwlpZiAocmV0KQ0KPiArCQlwcmludGYoIm1wcm90ZWN0IGVycm9y
IFslZF1cbiIsIHJldCk7DQo+ICsNCj4gKyAgICAgICAgLyogRW5qb3kgdGhhdCBwYWdlIG9mIGVu
Y3J5cHRlZCBtZW1vcnkgKi8NCj4gKw0KPiArICAgICAgICAvKiBGcmVlIHRoZSBtZW1vcnkgKi8N
Cj4gKwlyZXQgPSBtdW5tYXAocHRyYSwgc2l6ZSk7DQo+ICsNCj4gK2ludmFsX2tleToNCj4gKyAg
ICAgICAgLyogRnJlZSB0aGUgS2V5ICovDQo+ICsJaWYgKGtleWN0bChLRVlDVExfSU5WQUxJREFU
RSwga2V5KSA9PSAtMSkNCj4gKwkJcHJpbnRmKCJpbnZhbGlkYXRlIGZhaWxlZCBvbiBrZXkgWyVk
XVxuIiwga2V5KTsNCg0KV291bGQgaXQgbWFrZSBzZW5zZSB0byBwcmludCBlcnJvciBtZXNzYWdl
cyB0byBzdGRlcnI/DQoNCj4gK30NCj4gZGlmZiAtLWdpdCBhL0RvY3VtZW50YXRpb24veDg2L21r
dG1lL21rdG1lX2VuY3J5cHQucnN0DQo+IGIvRG9jdW1lbnRhdGlvbi94ODYvbWt0bWUvbWt0bWVf
ZW5jcnlwdC5yc3QNCj4gbmV3IGZpbGUgbW9kZSAxMDA2NDQNCj4gaW5kZXggMDAwMDAwMDAwMDAw
Li5lZGU1MjM3MTgzZmMNCj4gLS0tIC9kZXYvbnVsbA0KPiArKysgYi9Eb2N1bWVudGF0aW9uL3g4
Ni9ta3RtZS9ta3RtZV9lbmNyeXB0LnJzdA0KPiBAQCAtMCwwICsxLDU4IEBADQo+ICtNS1RNRSBB
UEk6IHN5c3RlbSBjYWxsIGVuY3J5cHRfbXByb3RlY3QoKQ0KPiArPT09PT09PT09PT09PT09PT09
PT09PT09PT09PT09PT09PT09PT09PT0NCj4gKw0KPiArU3lub3BzaXMNCj4gKy0tLS0tLS0tDQo+
ICtpbnQgZW5jcnlwdF9tcHJvdGVjdCh2b2lkIFwqYWRkciwgc2l6ZV90IGxlbiwgaW50IHByb3Qs
IGtleV9zZXJpYWxfdCBzZXJpYWwpOw0KPiArDQo+ICtXaGVyZSAqa2V5X3NlcmlhbF90IHNlcmlh
bCogaXMgdGhlIHNlcmlhbCBudW1iZXIgb2YgYSBrZXkgYWxsb2NhdGVkDQo+ICt1c2luZyB0aGUg
TUtUTUUgS2V5IFNlcnZpY2UuDQoNClRoZXJlIGlzIG9ubHkgb25lIGtleSBzZXJ2aWNlIGkuZS4g
dGhlIGtlcm5lbCBrZXlyaW5nLiBTaG91bGQgYmUgcmVwaHJhc2VkDQpzb21laG93Lg0KDQo+ICsN
Cj4gK0Rlc2NyaXB0aW9uDQo+ICstLS0tLS0tLS0tLQ0KPiArICAgIGVuY3J5cHRfbXByb3RlY3Qo
KSBlbmNyeXB0cyB0aGUgbWVtb3J5IHBhZ2VzIGNvbnRhaW5pbmcgYW55IHBhcnQNCj4gKyAgICBv
ZiB0aGUgYWRkcmVzcyByYW5nZSBpbiB0aGUgaW50ZXJ2YWwgc3BlY2lmaWVkIGJ5IGFkZHIgYW5k
IGxlbi4NCg0KV2hhdCBkb2VzIGl0IGFjdHVhbGx5IGRvPyBJIGRvbid0IHRoaW5rIHRoZSBzeXNj
YWxsIGRvZXMgYW55IGVuY3J5cHRpb24sDQpkb2VzIGl0PyBJJ20gbm90IGxvb2tpbmcgU0RNIGxl
dmVsIGRldGFpbHMgYnV0IHNvbWVob3cgYmV0dGVyDQpkZXNjcmlwdGlvbiB3aGF0IGRvZXMgaXQg
ZG8gd291bGQgYmUgbmljZS4NCg0KPiArDQo+ICsgICAgZW5jcnlwdF9tcHJvdGVjdCgpIHN1cHBv
cnRzIHRoZSBsZWdhY3kgbXByb3RlY3QoKSBiZWhhdmlvciBwbHVzDQo+ICsgICAgdGhlIGVuYWJs
aW5nIG9mIG1lbW9yeSBlbmNyeXB0aW9uLiBUaGF0IG1lYW5zIHRoYXQgaW4gYWRkaXRpb24NCj4g
KyAgICB0byBlbmNyeXB0aW5nIHRoZSBtZW1vcnksIHRoZSBwcm90ZWN0aW9uIGZsYWdzIHdpbGwg
YmUgdXBkYXRlZA0KPiArICAgIGFzIHJlcXVlc3RlZCBpbiB0aGUgY2FsbC4NCg0KRGl0dG8uDQoN
Cj4gKw0KPiArICAgIFRoZSAqYWRkciogYW5kICpsZW4qIG11c3QgYmUgYWxpZ25lZCB0byBhIHBh
Z2UgYm91bmRhcnkuDQo+ICsNCj4gKyAgICBUaGUgY2FsbGVyIG11c3QgaGF2ZSAqS0VZX05FRURf
VklFVyogcGVybWlzc2lvbiBvbiB0aGUga2V5Lg0KDQpNYXliZSBtb3JlIHZlcmJvc2UgZGVzY3Jp
cHRpb24sIGVzcGVjaWFsbHkgd2hlbiBpdCBpcyBhIG11c3QuDQoNCj4gKw0KPiArICAgIFRoZSBy
YW5nZSBvZiBtZW1vcnkgdGhhdCBpcyB0byBiZSBwcm90ZWN0ZWQgbXVzdCBiZSBtYXBwZWQgYXMN
Cj4gKyAgICAqQU5PTllNT1VTKi4NCg0KRGl0dG8uDQoNCj4gKw0KPiArRXJyb3JzDQo+ICstLS0t
LS0NCj4gKyAgICBJbiBhZGRpdGlvbiB0byB0aGUgRXJyb3JzIHJldHVybmVkIGZyb20gbGVnYWN5
IG1wcm90ZWN0KCkNCj4gKyAgICBlbmNyeXB0X21wcm90ZWN0IHdpbGwgcmV0dXJuOg0KPiArDQo+
ICsgICAgRU5PS0VZICpzZXJpYWwqIHBhcmFtZXRlciBkb2VzIG5vdCByZXByZXNlbnQgYSB2YWxp
ZCBrZXkuDQo+ICsNCj4gKyAgICBFSU5WQUwgKmxlbiogcGFyYW1ldGVyIGlzIG5vdCBwYWdlIGFs
aWduZWQuDQo+ICsNCj4gKyAgICBFQUNDRVMgQ2FsbGVyIGRvZXMgbm90IGhhdmUgKktFWV9ORUVE
X1ZJRVcqIHBlcm1pc3Npb24gb24gdGhlIGtleS4NCj4gKw0KPiArRVhBTVBMRQ0KPiArLS0tLS0t
LS0NCj4gKyAgQWxsb2NhdGUgYW4gTUtUTUUgS2V5OjoNCj4gKyAgICAgICAgc2VyaWFsID0gYWRk
X2tleSgibWt0bWUiLCAibmFtZSIsICJ0eXBlPWNwdSBhbGdvcml0aG09YWVzLXh0cy0xMjgiIEB1
DQo+ICsNCj4gKyAgTWFwIEFOT05ZTU9VUyBtZW1vcnk6Og0KPiArICAgICAgICBwdHIgPSBtbWFw
KE5VTEwsIHNpemUsIFBST1RfTk9ORSwgTUFQX0FOT05ZTU9VU3xNQVBfUFJJVkFURSwgLTEsIDAp
Ow0KPiArDQo+ICsgIFByb3RlY3QgbWVtb3J5OjoNCj4gKyAgICAgICAgcmV0ID0gc3lzY2FsbChT
WVNfZW5jcnlwdF9tcHJvdGVjdCwgcHRyLCBzaXplLCBQUk9UX1JFQUR8UFJPVF9XUklURSwNCj4g
KyAgICAgICAgICAgICAgICAgICAgICBzZXJpYWwpOw0KPiArDQo+ICsgIFVzZSB0aGUgZW5jcnlw
dGVkIG1lbW9yeQ0KPiArDQo+ICsgIEZyZWUgbWVtb3J5OjoNCj4gKyAgICAgICAgcmV0ID0gbXVu
bWFwKHB0ciwgc2l6ZSk7DQo+ICsNCj4gKyAgRnJlZSB0aGUga2V5IHJlc291cmNlOjoNCj4gKyAg
ICAgICAgcmV0ID0ga2V5Y3RsKEtFWUNUTF9JTlZBTElEQVRFLCBzZXJpYWwpOw0KPiArDQoNCk5v
dCByZWFsbHkgc3VyZSB3aGF0IHRoaXMgZXhhbXBsZSBzZXJ2ZXMgYXMgeW91IGFscmVhZHkgaGFk
IGFuIGV4YW1wbGUNCnByb2dyYW0uLi4NCg0KSGF2ZSB5b3UgcmVhZCBwa2V5cyBtYW4gcGFnZT8g
SXQgaGFzIHJlYWxseSBnb29kIGJhbGFuY2UgZXhwbGFpbmluZyBob3cNCml0IGlzIGltcGxlbWVu
dGVkIGFuZCB1c2VkIChtYW4gcGtleXMsIG5vdCBtYW4gcGtleV9tcHJvdGVjdCgpKS4NCg0KV2hh
dCBpZiB5b3Ugc3VkZGVubHkgY2hhbmdlIGEga2V5IGZvciBWTUE/IEkgZ3Vlc3MgbWVtb3J5IGlz
IHRoZW4NCmNvcnJ1cHRlZD8gTm90IGRvY3VtZW50ZWQgaGVyZS4gU2hvdWxkIGJlLg0KDQpJIGRp
ZCBub3QgZmluZCB0aGUgdGhpbmcgSSB3YXMgbG9va2luZyBmb3IgbW9zdCBpLmUuIHNvbWUgaGln
aCBsZXZlbA0KZGVzY3JpcHRpb24gb2YgdGhlIHRocmVhdCBtb2RlbC4gRW1waGFzaXMgb24gaGln
aC1sZXZlbCBhcyBrZXJuZWwNCmRvY3VtZW50YXRpb24gaXMgbm90IGEgQ1ZFIGRhdGFiYXNlLg0K
DQo+IGRpZmYgLS1naXQgYS9Eb2N1bWVudGF0aW9uL3g4Ni9ta3RtZS9ta3RtZV9rZXlzLnJzdA0K
PiBiL0RvY3VtZW50YXRpb24veDg2L21rdG1lL21rdG1lX2tleXMucnN0DQo+IG5ldyBmaWxlIG1v
ZGUgMTAwNjQ0DQo+IGluZGV4IDAwMDAwMDAwMDAwMC4uNTgzNzkwOWIyYzU0DQo+IC0tLSAvZGV2
L251bGwNCj4gKysrIGIvRG9jdW1lbnRhdGlvbi94ODYvbWt0bWUvbWt0bWVfa2V5cy5yc3QNCj4g
QEAgLTAsMCArMSwxMDkgQEANCj4gK01LVE1FIEtleSBTZXJ2aWNlIEFQSQ0KPiArPT09PT09PT09
PT09PT09PT09PT09DQo+ICtNS1RNRSBpcyBhIG5ldyBrZXkgc2VydmljZSB0eXBlIGFkZGVkIHRv
IHRoZSBMaW51eCBLZXJuZWwgS2V5IFNlcnZpY2UuDQo+ICsNCj4gK1RoZSBNS1RNRSBLZXkgU2Vy
dmljZSB0eXBlIGlzIGF2YWlsYWJsZSB3aGVuIENPTkZJR19YODZfSU5URUxfTUtUTUUgaXMNCj4g
K3R1cm5lZCBvbiBpbiBJbnRlbCBwbGF0Zm9ybXMgdGhhdCBzdXBwb3J0IHRoZSBNS1RNRSBmZWF0
dXJlLg0KPiArDQo+ICtUaGUgTUtUTUUgS2V5IFNlcnZpY2UgdHlwZSBtYW5hZ2VzIHRoZSBhbGxv
Y2F0aW9uIG9mIGhhcmR3YXJlIGVuY3J5cHRpb24NCj4gK2tleXMuIFVzZXJzIGNhbiByZXF1ZXN0
IGFuIE1LVE1FIHR5cGUga2V5IGFuZCB0aGVuIHVzZSB0aGF0IGtleSB0bw0KPiArZW5jcnlwdCBt
ZW1vcnkgd2l0aCB0aGUgZW5jcnlwdF9tcHJvdGVjdCgpIHN5c3RlbSBjYWxsLg0KPiArDQo+ICtV
c2FnZQ0KPiArLS0tLS0NCj4gKyAgICBXaGVuIHVzaW5nIHRoZSBLZXJuZWwgS2V5IFNlcnZpY2Ug
dG8gcmVxdWVzdCBhbiAqbWt0bWUqIGtleSwNCj4gKyAgICBzcGVjaWZ5IHRoZSAqcGF5bG9hZCog
YXMgZm9sbG93czoNCj4gKw0KPiArICAgIHR5cGU9DQo+ICsgICAgICAgICp1c2VyKglVc2VyIHdp
bGwgc3VwcGx5IHRoZSBlbmNyeXB0aW9uIGtleSBkYXRhLiBVc2UgdGhpcw0KPiArICAgICAgICAg
ICAgICAgIHR5cGUgdG8gZGlyZWN0bHkgcHJvZ3JhbSBhIGhhcmR3YXJlIGVuY3J5cHRpb24ga2V5
Lg0KPiArDQo+ICsgICAgICAgICpjcHUqCVVzZXIgcmVxdWVzdHMgYSBDUFUgZ2VuZXJhdGVkIGVu
Y3J5cHRpb24ga2V5Lg0KPiArICAgICAgICAgICAgICAgIFRoZSBDUFUgZ2VuZXJhdGVzIGFuZCBh
c3NpZ25zIGFuIGVwaGVtZXJhbCBrZXkuDQoNCkhvdyBhcmUgdGhlc2UgaW1wbGVtZW50ZWQ/IElz
IHRoZXJlIGFuIG9wY29kZSB0byByZXF1ZXN0IENQVSB0byBnZW5lcmF0ZQ0KYSBrZXksIG9yPyBX
aGF0IGFib3V0IHRoZSB1c2VyIGtleT8gRG9lcyBjcHUga2V5IGV2ZXIgbGVhdmUgb3V0IG9mIHRo
ZQ0KQ1BVIHBhY2thZ2U/DQoNClRoZSB1c2VyIGtleSBzb3VuZHMgbGlrZSBhIHJlYWxseSBiYWQg
aWRlYSBhdCB0aGUgZmlyc3Qgc2lnaHQgYW5kIG1heWJlDQpzaG91bGQgYmUgY29uc2lkZXJlZCB0
byBiZSBsZWZ0IG91dC4gV2hhdCB3b3VsZCBiZSBhIGxlZ2l0IHVzZSBjYXNlIGZvcg0KaXQ/DQoN
CkFyZSB0aGUga2V5cyBwZXItcHJvY2VzcyBvciBpcyBpdCBhIGdsb2JhbCByZXNvdXJjZT8NCg0K
PiArICAgICAgICAqY2xlYXIqIFVzZXIgcmVxdWVzdHMgdGhhdCBhIGhhcmR3YXJlIGVuY3J5cHRp
b24ga2V5IGJlDQo+ICsgICAgICAgICAgICAgICAgY2xlYXJlZC4gVGhpcyB3aWxsIGNsZWFyIHRo
ZSBlbmNyeXB0aW9uIGtleSBmcm9tDQo+ICsgICAgICAgICAgICAgICAgdGhlIGhhcmR3YXJlLiBP
biBleGVjdXRpb24gdGhpcyBoYXJkd2FyZSBrZXkgZ2V0cw0KPiArICAgICAgICAgICAgICAgIFRN
RSBiZWhhdmlvci4NCj4gKw0KPiArICAgICAgICAqbm8tZW5jcnlwdCoNCj4gKyAgICAgICAgICAg
ICAgICAgVXNlciByZXF1ZXN0cyB0aGF0IGhhcmR3YXJlIGRvZXMgbm90IGVuY3J5cHQNCj4gKyAg
ICAgICAgICAgICAgICAgbWVtb3J5IHdoZW4gdGhpcyBrZXkgaXMgaW4gdXNlLg0KDQpOb3Qgc3Vy
ZSBhYm91dCB0aGVzZSB3aXRoIG15IGN1cnJlbnQga25vd2xlZGdlLg0KDQo+ICsNCj4gKyAgICBh
bGdvcml0aG09DQo+ICsgICAgICAgIFdoZW4gdHlwZT11c2VyIG9yIHR5cGU9Y3B1IHRoZSBhbGdv
cml0aG0gZmllbGQgbXVzdCBiZQ0KPiArICAgICAgICAqYWVzLXh0cy0xMjgqDQo+ICsNCj4gKyAg
ICAgICAgV2hlbiB0eXBlPWNsZWFyIG9yIHR5cGU9bm8tZW5jcnlwdCB0aGUgYWxnb3JpdGhtIGZp
ZWxkDQo+ICsgICAgICAgIG11c3Qgbm90IGJlIHByZXNlbnQgaW4gdGhlIHBheWxvYWQuDQoNClRo
aXMgcGFyYW1ldGVyIG11c3QgYmUgcmVtb3ZlZCBhcyBpdCBpcyBhIGZ1bmN0aW9uIG9mIG90aGVy
IHBhcmFtYXRlcnMNCmFuZCBub3RoaW5nIGVsc2UgaS5lLiBjb21wbGV4aXR5IHdpdGhvdXQgZ2Fp
bi4NCg0KPiArCVRoaXMgZG9jdW1lbnQgZG9lcyBub3QgaW50ZW5kIHRvIGRvY3VtZW50IEtLUywg
YnV0IG9ubHkgdGhlDQo+ICsJTUtUTUUgdHlwZSBvZiB0aGUgS0tTLiBUaGUgb3B0aW9ucyBvZiB0
aGUgS0tTIGNhbiBiZSBncm91cGVkDQo+ICsJaW50byAyIGNsYXNzZXMgZm9yIHB1cnBvc2VzIG9m
IHVuZGVyc3RhbmRpbmcgaG93IE1LVE1FIG9wZXJhdGVzDQo+ICsJd2l0aGluIHRoZSBicm9hZGVy
IEtLUy4NCg0KTWF5YmUganVzdCBkZWxldGUgdGhpcyBwYXJhZ3JhcGg/IEkgdGhpbmsgaXQgaXMg
anVzdCBzdGF0aW5nIHRoZQ0Kb2J2aW91cy4NCg0KSSB0aGluayB5b3UgbmVlZCB0aGlzIHBhcmFn
cmFwaCBvbmx5IGJlY2F1c2UgeW91IGhhdmUgZGVwbG95ZWQgdGhpcw0KZG9jdW1lbnQgdG8gd3Jv
bmcgcGxhY2UuIEJldHRlciBwYXRoIHdvdWxkIGJlDQoNCkRvY3VtZW50YXRpb24vc2VjdXJpdHkv
a2V5cy9ta3RtZS5yc3QuDQoNCi9KYXJra28NCg=

Re: [RFC v2 01/13] x86/mktme: Document the MKTME APIs

[Sakkinen, Jarkko]

I'll focus my remarks now towards documentation as I have lots of
catching up to do with TME :-) I'll give more feedback of actual code
changes once v18 of the SGX patch set is out, pull request for TPM 4.21
changes is out and maybe a new version of this patch set has been
released.

Right now too much on my shoulders to go too deep with this patch set.

On Mon, 2018-12-03 at 23:39 -0800, Alison Schofield wrote:
> This includes an overview, a section on each API: MTKME Keys and
> system call encrypt_mprotect(), and a demonstration program.
> 
> (Some of this info is destined for man pages.)
> 
> Change-Id: I34dc9ff1a1308c057ec4bb3e652c4d7ce6995606
> Signed-off-by: Alison Schofield <alison.schofield@intel.com>

Co-developed-by: Kirill A. Shutemov <kirill.shutemov@linux.intel.com> ?

Not needed if this is for the most part written by you.

> Signed-off-by: Kirill A. Shutemov <kirill.shutemov@linux.intel.com>
> ---
>  Documentation/x86/mktme/index.rst          |  11 +++
>  Documentation/x86/mktme/mktme_demo.rst     |  53 ++++++++++++++
>  Documentation/x86/mktme/mktme_encrypt.rst  |  58 +++++++++++++++
>  Documentation/x86/mktme/mktme_keys.rst     | 109
> +++++++++++++++++++++++++++++
>  Documentation/x86/mktme/mktme_overview.rst |  60 ++++++++++++++++
>  5 files changed, 291 insertions(+)
>  create mode 100644 Documentation/x86/mktme/index.rst
>  create mode 100644 Documentation/x86/mktme/mktme_demo.rst
>  create mode 100644 Documentation/x86/mktme/mktme_encrypt.rst
>  create mode 100644 Documentation/x86/mktme/mktme_keys.rst
>  create mode 100644 Documentation/x86/mktme/mktme_overview.rst
> 
> diff --git a/Documentation/x86/mktme/index.rst
> b/Documentation/x86/mktme/index.rst
> new file mode 100644
> index 000000000000..8c556d04cbc4
> --- /dev/null
> +++ b/Documentation/x86/mktme/index.rst
> @@ -0,0 +1,11 @@
> +

SPDX?

> +=============================================
> +Multi-Key Total Memory Encryption (MKTME) API
> +=============================================
> +
> +.. toctree::
> +
> +   mktme_overview
> +   mktme_keys
> +   mktme_encrypt
> +   mktme_demo
> diff --git a/Documentation/x86/mktme/mktme_demo.rst
> b/Documentation/x86/mktme/mktme_demo.rst
> new file mode 100644
> index 000000000000..afd50772e65d
> --- /dev/null
> +++ b/Documentation/x86/mktme/mktme_demo.rst
> @@ -0,0 +1,53 @@
> +Demonstration Program using MKTME API's
> +=======================================

Probably would be better idea to put into tools/testings/selftest/x86
and not as part of the documentation.

> +
> +/* Compile with the keyutils library: cc -o mdemo mdemo.c -lkeyutils */
> +
> +#include <sys/mman.h>
> +#include <sys/syscall.h>
> +#include <sys/types.h>
> +#include <keyutils.h>
> +#include <stdio.h>
> +#include <string.h>
> +#include <unistd.h>
> +
> +#define PAGE_SIZE sysconf(_SC_PAGE_SIZE)
> +#define sys_encrypt_mprotect 335
> +
> +void main(void)
> +{
> +	char *options_CPU = "algorithm=aes-xts-128 type=cpu";
> +	long size = PAGE_SIZE;
> +        key_serial_t key;
> +	void *ptra;
> +	int ret;
> +
> +        /* Allocate an MKTME Key */
> +	key = add_key("mktme", "testkey", options_CPU, strlen(options_CPU),
> +                      KEY_SPEC_THREAD_KEYRING);
> +
> +	if (key == -1) {
> +		printf("addkey FAILED\n");
> +		return;
> +	}
> +        /* Map a page of ANONYMOUS memory */
> +	ptra = mmap(NULL, size, PROT_NONE, MAP_ANONYMOUS|MAP_PRIVATE, -1, 0);
> +	if (!ptra) {
> +		printf("failed to mmap");
> +		goto inval_key;
> +	}
> +        /* Encrypt that page of memory with the MKTME Key */
> +	ret = syscall(sys_encrypt_mprotect, ptra, size, PROT_NONE, key);
> +	if (ret)
> +		printf("mprotect error [%d]\n", ret);
> +
> +        /* Enjoy that page of encrypted memory */
> +
> +        /* Free the memory */
> +	ret = munmap(ptra, size);
> +
> +inval_key:
> +        /* Free the Key */
> +	if (keyctl(KEYCTL_INVALIDATE, key) == -1)
> +		printf("invalidate failed on key [%d]\n", key);

Would it make sense to print error messages to stderr?

> +}
> diff --git a/Documentation/x86/mktme/mktme_encrypt.rst
> b/Documentation/x86/mktme/mktme_encrypt.rst
> new file mode 100644
> index 000000000000..ede5237183fc
> --- /dev/null
> +++ b/Documentation/x86/mktme/mktme_encrypt.rst
> @@ -0,0 +1,58 @@
> +MKTME API: system call encrypt_mprotect()
> +=========================================
> +
> +Synopsis
> +--------
> +int encrypt_mprotect(void \*addr, size_t len, int prot, key_serial_t serial);
> +
> +Where *key_serial_t serial* is the serial number of a key allocated
> +using the MKTME Key Service.

There is only one key service i.e. the kernel keyring. Should be rephrased
somehow.

> +
> +Description
> +-----------
> +    encrypt_mprotect() encrypts the memory pages containing any part
> +    of the address range in the interval specified by addr and len.

What does it actually do? I don't think the syscall does any encryption,
does it? I'm not looking SDM level details but somehow better
description what does it do would be nice.

> +
> +    encrypt_mprotect() supports the legacy mprotect() behavior plus
> +    the enabling of memory encryption. That means that in addition
> +    to encrypting the memory, the protection flags will be updated
> +    as requested in the call.

Ditto.

> +
> +    The *addr* and *len* must be aligned to a page boundary.
> +
> +    The caller must have *KEY_NEED_VIEW* permission on the key.

Maybe more verbose description, especially when it is a must.

> +
> +    The range of memory that is to be protected must be mapped as
> +    *ANONYMOUS*.

Ditto.

> +
> +Errors
> +------
> +    In addition to the Errors returned from legacy mprotect()
> +    encrypt_mprotect will return:
> +
> +    ENOKEY *serial* parameter does not represent a valid key.
> +
> +    EINVAL *len* parameter is not page aligned.
> +
> +    EACCES Caller does not have *KEY_NEED_VIEW* permission on the key.
> +
> +EXAMPLE
> +--------
> +  Allocate an MKTME Key::
> +        serial = add_key("mktme", "name", "type=cpu algorithm=aes-xts-128" @u
> +
> +  Map ANONYMOUS memory::
> +        ptr = mmap(NULL, size, PROT_NONE, MAP_ANONYMOUS|MAP_PRIVATE, -1, 0);
> +
> +  Protect memory::
> +        ret = syscall(SYS_encrypt_mprotect, ptr, size, PROT_READ|PROT_WRITE,
> +                      serial);
> +
> +  Use the encrypted memory
> +
> +  Free memory::
> +        ret = munmap(ptr, size);
> +
> +  Free the key resource::
> +        ret = keyctl(KEYCTL_INVALIDATE, serial);
> +

Not really sure what this example serves as you already had an example
program...

Have you read pkeys man page? It has really good balance explaining how
it is implemented and used (man pkeys, not man pkey_mprotect()).

What if you suddenly change a key for VMA? I guess memory is then
corrupted? Not documented here. Should be.

I did not find the thing I was looking for most i.e. some high level
description of the threat model. Emphasis on high-level as kernel
documentation is not a CVE database.

> diff --git a/Documentation/x86/mktme/mktme_keys.rst
> b/Documentation/x86/mktme/mktme_keys.rst
> new file mode 100644
> index 000000000000..5837909b2c54
> --- /dev/null
> +++ b/Documentation/x86/mktme/mktme_keys.rst
> @@ -0,0 +1,109 @@
> +MKTME Key Service API
> +=====================
> +MKTME is a new key service type added to the Linux Kernel Key Service.
> +
> +The MKTME Key Service type is available when CONFIG_X86_INTEL_MKTME is
> +turned on in Intel platforms that support the MKTME feature.
> +
> +The MKTME Key Service type manages the allocation of hardware encryption
> +keys. Users can request an MKTME type key and then use that key to
> +encrypt memory with the encrypt_mprotect() system call.
> +
> +Usage
> +-----
> +    When using the Kernel Key Service to request an *mktme* key,
> +    specify the *payload* as follows:
> +
> +    type=
> +        *user*	User will supply the encryption key data. Use this
> +                type to directly program a hardware encryption key.
> +
> +        *cpu*	User requests a CPU generated encryption key.
> +                The CPU generates and assigns an ephemeral key.

How are these implemented? Is there an opcode to request CPU to generate
a key, or? What about the user key? Does cpu key ever leave out of the
CPU package?

The user key sounds like a really bad idea at the first sight and maybe
should be considered to be left out. What would be a legit use case for
it?

Are the keys per-process or is it a global resource?

> +        *clear* User requests that a hardware encryption key be
> +                cleared. This will clear the encryption key from
> +                the hardware. On execution this hardware key gets
> +                TME behavior.
> +
> +        *no-encrypt*
> +                 User requests that hardware does not encrypt
> +                 memory when this key is in use.

Not sure about these with my current knowledge.

> +
> +    algorithm=
> +        When type=user or type=cpu the algorithm field must be
> +        *aes-xts-128*
> +
> +        When type=clear or type=no-encrypt the algorithm field
> +        must not be present in the payload.

This parameter must be removed as it is a function of other paramaters
and nothing else i.e. complexity without gain.

> +	This document does not intend to document KKS, but only the
> +	MKTME type of the KKS. The options of the KKS can be grouped
> +	into 2 classes for purposes of understanding how MKTME operates
> +	within the broader KKS.

Maybe just delete this paragraph? I think it is just stating the
obvious.

I think you need this paragraph only because you have deployed this
document to wrong place. Better path would be

Documentation/security/keys/mktme.rst.

/Jarkko

Re: [RFC v2 01/13] x86/mktme: Document the MKTME APIs

[Sakkinen, Jarkko]

I'll focus my remarks now towards documentation as I have lots of
catching up to do with TME :-) I'll give more feedback of actual code
changes once v18 of the SGX patch set is out, pull request for TPM 4.21
changes is out and maybe a new version of this patch set has been
released.

Right now too much on my shoulders to go too deep with this patch set.

On Mon, 2018-12-03 at 23:39 -0800, Alison Schofield wrote:
> This includes an overview, a section on each API: MTKME Keys and
> system call encrypt_mprotect(), and a demonstration program.
> 
> (Some of this info is destined for man pages.)
> 
> Change-Id: I34dc9ff1a1308c057ec4bb3e652c4d7ce6995606
> Signed-off-by: Alison Schofield <alison.schofield@intel.com>

Co-developed-by: Kirill A. Shutemov <kirill.shutemov@linux.intel.com> ?

Not needed if this is for the most part written by you.

> Signed-off-by: Kirill A. Shutemov <kirill.shutemov@linux.intel.com>
> ---
>  Documentation/x86/mktme/index.rst          |  11 +++
>  Documentation/x86/mktme/mktme_demo.rst     |  53 ++++++++++++++
>  Documentation/x86/mktme/mktme_encrypt.rst  |  58 +++++++++++++++
>  Documentation/x86/mktme/mktme_keys.rst     | 109
> +++++++++++++++++++++++++++++
>  Documentation/x86/mktme/mktme_overview.rst |  60 ++++++++++++++++
>  5 files changed, 291 insertions(+)
>  create mode 100644 Documentation/x86/mktme/index.rst
>  create mode 100644 Documentation/x86/mktme/mktme_demo.rst
>  create mode 100644 Documentation/x86/mktme/mktme_encrypt.rst
>  create mode 100644 Documentation/x86/mktme/mktme_keys.rst
>  create mode 100644 Documentation/x86/mktme/mktme_overview.rst
> 
> diff --git a/Documentation/x86/mktme/index.rst
> b/Documentation/x86/mktme/index.rst
> new file mode 100644
> index 000000000000..8c556d04cbc4
> --- /dev/null
> +++ b/Documentation/x86/mktme/index.rst
> @@ -0,0 +1,11 @@
> +

SPDX?

> +=============================================
> +Multi-Key Total Memory Encryption (MKTME) API
> +=============================================
> +
> +.. toctree::
> +
> +   mktme_overview
> +   mktme_keys
> +   mktme_encrypt
> +   mktme_demo
> diff --git a/Documentation/x86/mktme/mktme_demo.rst
> b/Documentation/x86/mktme/mktme_demo.rst
> new file mode 100644
> index 000000000000..afd50772e65d
> --- /dev/null
> +++ b/Documentation/x86/mktme/mktme_demo.rst
> @@ -0,0 +1,53 @@
> +Demonstration Program using MKTME API's
> +=======================================

Probably would be better idea to put into tools/testings/selftest/x86
and not as part of the documentation.

> +
> +/* Compile with the keyutils library: cc -o mdemo mdemo.c -lkeyutils */
> +
> +#include <sys/mman.h>
> +#include <sys/syscall.h>
> +#include <sys/types.h>
> +#include <keyutils.h>
> +#include <stdio.h>
> +#include <string.h>
> +#include <unistd.h>
> +
> +#define PAGE_SIZE sysconf(_SC_PAGE_SIZE)
> +#define sys_encrypt_mprotect 335
> +
> +void main(void)
> +{
> +	char *options_CPU = "algorithm=aes-xts-128 type=cpu";
> +	long size = PAGE_SIZE;
> +        key_serial_t key;
> +	void *ptra;
> +	int ret;
> +
> +        /* Allocate an MKTME Key */
> +	key = add_key("mktme", "testkey", options_CPU, strlen(options_CPU),
> +                      KEY_SPEC_THREAD_KEYRING);
> +
> +	if (key == -1) {
> +		printf("addkey FAILED\n");
> +		return;
> +	}
> +        /* Map a page of ANONYMOUS memory */
> +	ptra = mmap(NULL, size, PROT_NONE, MAP_ANONYMOUS|MAP_PRIVATE, -1, 0);
> +	if (!ptra) {
> +		printf("failed to mmap");
> +		goto inval_key;
> +	}
> +        /* Encrypt that page of memory with the MKTME Key */
> +	ret = syscall(sys_encrypt_mprotect, ptra, size, PROT_NONE, key);
> +	if (ret)
> +		printf("mprotect error [%d]\n", ret);
> +
> +        /* Enjoy that page of encrypted memory */
> +
> +        /* Free the memory */
> +	ret = munmap(ptra, size);
> +
> +inval_key:
> +        /* Free the Key */
> +	if (keyctl(KEYCTL_INVALIDATE, key) == -1)
> +		printf("invalidate failed on key [%d]\n", key);

Would it make sense to print error messages to stderr?

> +}
> diff --git a/Documentation/x86/mktme/mktme_encrypt.rst
> b/Documentation/x86/mktme/mktme_encrypt.rst
> new file mode 100644
> index 000000000000..ede5237183fc
> --- /dev/null
> +++ b/Documentation/x86/mktme/mktme_encrypt.rst
> @@ -0,0 +1,58 @@
> +MKTME API: system call encrypt_mprotect()
> +=========================================
> +
> +Synopsis
> +--------
> +int encrypt_mprotect(void \*addr, size_t len, int prot, key_serial_t serial);
> +
> +Where *key_serial_t serial* is the serial number of a key allocated
> +using the MKTME Key Service.

There is only one key service i.e. the kernel keyring. Should be rephrased
somehow.

> +
> +Description
> +-----------
> +    encrypt_mprotect() encrypts the memory pages containing any part
> +    of the address range in the interval specified by addr and len.

What does it actually do? I don't think the syscall does any encryption,
does it? I'm not looking SDM level details but somehow better
description what does it do would be nice.

> +
> +    encrypt_mprotect() supports the legacy mprotect() behavior plus
> +    the enabling of memory encryption. That means that in addition
> +    to encrypting the memory, the protection flags will be updated
> +    as requested in the call.

Ditto.

> +
> +    The *addr* and *len* must be aligned to a page boundary.
> +
> +    The caller must have *KEY_NEED_VIEW* permission on the key.

Maybe more verbose description, especially when it is a must.

> +
> +    The range of memory that is to be protected must be mapped as
> +    *ANONYMOUS*.

Ditto.

> +
> +Errors
> +------
> +    In addition to the Errors returned from legacy mprotect()
> +    encrypt_mprotect will return:
> +
> +    ENOKEY *serial* parameter does not represent a valid key.
> +
> +    EINVAL *len* parameter is not page aligned.
> +
> +    EACCES Caller does not have *KEY_NEED_VIEW* permission on the key.
> +
> +EXAMPLE
> +--------
> +  Allocate an MKTME Key::
> +        serial = add_key("mktme", "name", "type=cpu algorithm=aes-xts-128" @u
> +
> +  Map ANONYMOUS memory::
> +        ptr = mmap(NULL, size, PROT_NONE, MAP_ANONYMOUS|MAP_PRIVATE, -1, 0);
> +
> +  Protect memory::
> +        ret = syscall(SYS_encrypt_mprotect, ptr, size, PROT_READ|PROT_WRITE,
> +                      serial);
> +
> +  Use the encrypted memory
> +
> +  Free memory::
> +        ret = munmap(ptr, size);
> +
> +  Free the key resource::
> +        ret = keyctl(KEYCTL_INVALIDATE, serial);
> +

Not really sure what this example serves as you already had an example
program...

Have you read pkeys man page? It has really good balance explaining how
it is implemented and used (man pkeys, not man pkey_mprotect()).

What if you suddenly change a key for VMA? I guess memory is then
corrupted? Not documented here. Should be.

I did not find the thing I was looking for most i.e. some high level
description of the threat model. Emphasis on high-level as kernel
documentation is not a CVE database.

> diff --git a/Documentation/x86/mktme/mktme_keys.rst
> b/Documentation/x86/mktme/mktme_keys.rst
> new file mode 100644
> index 000000000000..5837909b2c54
> --- /dev/null
> +++ b/Documentation/x86/mktme/mktme_keys.rst
> @@ -0,0 +1,109 @@
> +MKTME Key Service API
> +=====================
> +MKTME is a new key service type added to the Linux Kernel Key Service.
> +
> +The MKTME Key Service type is available when CONFIG_X86_INTEL_MKTME is
> +turned on in Intel platforms that support the MKTME feature.
> +
> +The MKTME Key Service type manages the allocation of hardware encryption
> +keys. Users can request an MKTME type key and then use that key to
> +encrypt memory with the encrypt_mprotect() system call.
> +
> +Usage
> +-----
> +    When using the Kernel Key Service to request an *mktme* key,
> +    specify the *payload* as follows:
> +
> +    type=
> +        *user*	User will supply the encryption key data. Use this
> +                type to directly program a hardware encryption key.
> +
> +        *cpu*	User requests a CPU generated encryption key.
> +                The CPU generates and assigns an ephemeral key.

How are these implemented? Is there an opcode to request CPU to generate
a key, or? What about the user key? Does cpu key ever leave out of the
CPU package?

The user key sounds like a really bad idea at the first sight and maybe
should be considered to be left out. What would be a legit use case for
it?

Are the keys per-process or is it a global resource?

> +        *clear* User requests that a hardware encryption key be
> +                cleared. This will clear the encryption key from
> +                the hardware. On execution this hardware key gets
> +                TME behavior.
> +
> +        *no-encrypt*
> +                 User requests that hardware does not encrypt
> +                 memory when this key is in use.

Not sure about these with my current knowledge.

> +
> +    algorithm=
> +        When type=user or type=cpu the algorithm field must be
> +        *aes-xts-128*
> +
> +        When type=clear or type=no-encrypt the algorithm field
> +        must not be present in the payload.

This parameter must be removed as it is a function of other paramaters
and nothing else i.e. complexity without gain.

> +	This document does not intend to document KKS, but only the
> +	MKTME type of the KKS. The options of the KKS can be grouped
> +	into 2 classes for purposes of understanding how MKTME operates
> +	within the broader KKS.

Maybe just delete this paragraph? I think it is just stating the
obvious.

I think you need this paragraph only because you have deployed this
document to wrong place. Better path would be

Documentation/security/keys/mktme.rst.

/Jarkko

Re: [RFC v2 02/13] mm: Generalize the mprotect implementation to support extensions

[Sakkinen, Jarkko]

T24gTW9uLCAyMDE4LTEyLTAzIGF0IDIzOjM5IC0wODAwLCBBbGlzb24gU2Nob2ZpZWxkIHdyb3Rl
Og0KPiBUb2RheSBtcHJvdGVjdCBpcyBpbXBsZW1lbnRlZCB0byBzdXBwb3J0IGxlZ2FjeSBtcHJv
dGVjdCBiZWhhdmlvcg0KPiBwbHVzIGFuIGV4dGVuc2lvbiBmb3IgbWVtb3J5IHByb3RlY3Rpb24g
a2V5cy4gTWFrZSBpdCBtb3JlIGdlbmVyaWMNCj4gc28gdGhhdCBpdCBjYW4gc3VwcG9ydCBhZGRp
dGlvbmFsIGV4dGVuc2lvbnMgaW4gdGhlIGZ1dHVyZS4NCj4gDQo+IFRoaXMgaXMgZG9uZSBpcyBw
cmVwYXJhdGlvbiBmb3IgYWRkaW5nIGEgbmV3IHN5c3RlbSBjYWxsIGZvciBtZW1vcnkNCj4gZW5j
eXB0aW9uIGtleXMuIFRoZSBpbnRlbnQgaXMgdGhhdCB0aGUgbmV3IGVuY3J5cHRlZCBtcHJvdGVj
dCB3aWxsIGJlDQo+IGFub3RoZXIgZXh0ZW5zaW9uIHRvIGxlZ2FjeSBtcHJvdGVjdC4NCj4gDQo+
IENoYW5nZS1JZDogSWIwOWI5ZDFiNjA1YjEyZDAyNTRkN2ZiNDk2OGRmY2M4ZTNjNzlkZDcNCg0K
V2hhdCBpcyB0aGlzPz8NCg0KPiBTaWduZWQtb2ZmLWJ5OiBBbGlzb24gU2Nob2ZpZWxkIDxhbGlz
b24uc2Nob2ZpZWxkQGludGVsLmNvbT4NCj4gU2lnbmVkLW9mZi1ieTogS2lyaWxsIEEuIFNodXRl
bW92IDxraXJpbGwuc2h1dGVtb3ZAbGludXguaW50ZWwuY29tPg0KPiAtLS0NCj4gIG1tL21wcm90
ZWN0LmMgfCAxMCArKysrKystLS0tDQo+ICAxIGZpbGUgY2hhbmdlZCwgNiBpbnNlcnRpb25zKCsp
LCA0IGRlbGV0aW9ucygtKQ0KPiANCj4gZGlmZiAtLWdpdCBhL21tL21wcm90ZWN0LmMgYi9tbS9t
cHJvdGVjdC5jDQo+IGluZGV4IGRmNDA4OTU2ZGNjYy4uYjU3MDc1ZTI3OGZiIDEwMDY0NA0KPiAt
LS0gYS9tbS9tcHJvdGVjdC5jDQo+ICsrKyBiL21tL21wcm90ZWN0LmMNCj4gQEAgLTM1LDYgKzM1
LDggQEANCj4gIA0KPiAgI2luY2x1ZGUgImludGVybmFsLmgiDQo+ICANCj4gKyNkZWZpbmUgTk9f
S0VZCS0xDQo+ICsNCj4gIHN0YXRpYyB1bnNpZ25lZCBsb25nIGNoYW5nZV9wdGVfcmFuZ2Uoc3Ry
dWN0IHZtX2FyZWFfc3RydWN0ICp2bWEsIHBtZF90ICpwbWQsDQo+ICAJCXVuc2lnbmVkIGxvbmcg
YWRkciwgdW5zaWduZWQgbG9uZyBlbmQsIHBncHJvdF90IG5ld3Byb3QsDQo+ICAJCWludCBkaXJ0
eV9hY2NvdW50YWJsZSwgaW50IHByb3RfbnVtYSkNCj4gQEAgLTQ1MSw5ICs0NTMsOSBAQCBtcHJv
dGVjdF9maXh1cChzdHJ1Y3Qgdm1fYXJlYV9zdHJ1Y3QgKnZtYSwgc3RydWN0DQo+IHZtX2FyZWFf
c3RydWN0ICoqcHByZXYsDQo+ICB9DQo+ICANCj4gIC8qDQo+IC0gKiBwa2V5PT0tMSB3aGVuIGRv
aW5nIGEgbGVnYWN5IG1wcm90ZWN0KCkNCj4gKyAqIFdoZW4gcGtleT09Tk9fS0VZIHdlIGdldCBs
ZWdhY3kgbXByb3RlY3QgYmVoYXZpb3IgaGVyZS4NCj4gICAqLw0KPiAtc3RhdGljIGludCBkb19t
cHJvdGVjdF9wa2V5KHVuc2lnbmVkIGxvbmcgc3RhcnQsIHNpemVfdCBsZW4sDQo+ICtzdGF0aWMg
aW50IGRvX21wcm90ZWN0X2V4dCh1bnNpZ25lZCBsb25nIHN0YXJ0LCBzaXplX3QgbGVuLA0KPiAg
CQl1bnNpZ25lZCBsb25nIHByb3QsIGludCBwa2V5KQ0KPiAgew0KPiAgCXVuc2lnbmVkIGxvbmcg
bnN0YXJ0LCBlbmQsIHRtcCwgcmVxcHJvdDsNCj4gQEAgLTU3Nyw3ICs1NzksNyBAQCBzdGF0aWMg
aW50IGRvX21wcm90ZWN0X3BrZXkodW5zaWduZWQgbG9uZyBzdGFydCwgc2l6ZV90DQo+IGxlbiwN
Cj4gIFNZU0NBTExfREVGSU5FMyhtcHJvdGVjdCwgdW5zaWduZWQgbG9uZywgc3RhcnQsIHNpemVf
dCwgbGVuLA0KPiAgCQl1bnNpZ25lZCBsb25nLCBwcm90KQ0KPiAgew0KPiAtCXJldHVybiBkb19t
cHJvdGVjdF9wa2V5KHN0YXJ0LCBsZW4sIHByb3QsIC0xKTsNCj4gKwlyZXR1cm4gZG9fbXByb3Rl
Y3RfZXh0KHN0YXJ0LCBsZW4sIHByb3QsIE5PX0tFWSk7DQo+ICB9DQo+ICANCj4gICNpZmRlZiBD
T05GSUdfQVJDSF9IQVNfUEtFWVMNCj4gQEAgLTU4NSw3ICs1ODcsNyBAQCBTWVNDQUxMX0RFRklO
RTMobXByb3RlY3QsIHVuc2lnbmVkIGxvbmcsIHN0YXJ0LCBzaXplX3QsDQo+IGxlbiwNCj4gIFNZ
U0NBTExfREVGSU5FNChwa2V5X21wcm90ZWN0LCB1bnNpZ25lZCBsb25nLCBzdGFydCwgc2l6ZV90
LCBsZW4sDQo+ICAJCXVuc2lnbmVkIGxvbmcsIHByb3QsIGludCwgcGtleSkNCj4gIHsNCj4gLQly
ZXR1cm4gZG9fbXByb3RlY3RfcGtleShzdGFydCwgbGVuLCBwcm90LCBwa2V5KTsNCj4gKwlyZXR1
cm4gZG9fbXByb3RlY3RfZXh0KHN0YXJ0LCBsZW4sIHByb3QsIHBrZXkpOw0KPiAgfQ0KPiAgDQo+
ICBTWVNDQUxMX0RFRklORTIocGtleV9hbGxvYywgdW5zaWduZWQgbG9uZywgZmxhZ3MsIHVuc2ln
bmVkIGxvbmcsIGluaXRfdmFsKQ0KDQpXb3VsZCBzcXVhc2ggdGhpcyB3aGF0ZXZlciB0aGlzIGlz
IHJlcXVpcmVkIGZvci4gVGhpcyBzcGxpdCBtYWtlcw0KcmV2aWV3IG1vcmUgY29tcGxleCAoSU1I
TykuDQoNCi9KYXJra28NCg=

Re: [RFC v2 02/13] mm: Generalize the mprotect implementation to support extensions

[Sakkinen, Jarkko]

On Mon, 2018-12-03 at 23:39 -0800, Alison Schofield wrote:
> Today mprotect is implemented to support legacy mprotect behavior
> plus an extension for memory protection keys. Make it more generic
> so that it can support additional extensions in the future.
> 
> This is done is preparation for adding a new system call for memory
> encyption keys. The intent is that the new encrypted mprotect will be
> another extension to legacy mprotect.
> 
> Change-Id: Ib09b9d1b605b12d0254d7fb4968dfcc8e3c79dd7

What is this??

> Signed-off-by: Alison Schofield <alison.schofield@intel.com>
> Signed-off-by: Kirill A. Shutemov <kirill.shutemov@linux.intel.com>
> ---
>  mm/mprotect.c | 10 ++++++----
>  1 file changed, 6 insertions(+), 4 deletions(-)
> 
> diff --git a/mm/mprotect.c b/mm/mprotect.c
> index df408956dccc..b57075e278fb 100644
> --- a/mm/mprotect.c
> +++ b/mm/mprotect.c
> @@ -35,6 +35,8 @@
>  
>  #include "internal.h"
>  
> +#define NO_KEY	-1
> +
>  static unsigned long change_pte_range(struct vm_area_struct *vma, pmd_t *pmd,
>  		unsigned long addr, unsigned long end, pgprot_t newprot,
>  		int dirty_accountable, int prot_numa)
> @@ -451,9 +453,9 @@ mprotect_fixup(struct vm_area_struct *vma, struct
> vm_area_struct **pprev,
>  }
>  
>  /*
> - * pkey==-1 when doing a legacy mprotect()
> + * When pkey==NO_KEY we get legacy mprotect behavior here.
>   */
> -static int do_mprotect_pkey(unsigned long start, size_t len,
> +static int do_mprotect_ext(unsigned long start, size_t len,
>  		unsigned long prot, int pkey)
>  {
>  	unsigned long nstart, end, tmp, reqprot;
> @@ -577,7 +579,7 @@ static int do_mprotect_pkey(unsigned long start, size_t
> len,
>  SYSCALL_DEFINE3(mprotect, unsigned long, start, size_t, len,
>  		unsigned long, prot)
>  {
> -	return do_mprotect_pkey(start, len, prot, -1);
> +	return do_mprotect_ext(start, len, prot, NO_KEY);
>  }
>  
>  #ifdef CONFIG_ARCH_HAS_PKEYS
> @@ -585,7 +587,7 @@ SYSCALL_DEFINE3(mprotect, unsigned long, start, size_t,
> len,
>  SYSCALL_DEFINE4(pkey_mprotect, unsigned long, start, size_t, len,
>  		unsigned long, prot, int, pkey)
>  {
> -	return do_mprotect_pkey(start, len, prot, pkey);
> +	return do_mprotect_ext(start, len, prot, pkey);
>  }
>  
>  SYSCALL_DEFINE2(pkey_alloc, unsigned long, flags, unsigned long, init_val)

Would squash this whatever this is required for. This split makes
review more complex (IMHO).

/Jarkko

Re: [RFC v2 02/13] mm: Generalize the mprotect implementation to support extensions

[Sakkinen, Jarkko]

On Mon, 2018-12-03 at 23:39 -0800, Alison Schofield wrote:
> Today mprotect is implemented to support legacy mprotect behavior
> plus an extension for memory protection keys. Make it more generic
> so that it can support additional extensions in the future.
> 
> This is done is preparation for adding a new system call for memory
> encyption keys. The intent is that the new encrypted mprotect will be
> another extension to legacy mprotect.
> 
> Change-Id: Ib09b9d1b605b12d0254d7fb4968dfcc8e3c79dd7

What is this??

> Signed-off-by: Alison Schofield <alison.schofield@intel.com>
> Signed-off-by: Kirill A. Shutemov <kirill.shutemov@linux.intel.com>
> ---
>  mm/mprotect.c | 10 ++++++----
>  1 file changed, 6 insertions(+), 4 deletions(-)
> 
> diff --git a/mm/mprotect.c b/mm/mprotect.c
> index df408956dccc..b57075e278fb 100644
> --- a/mm/mprotect.c
> +++ b/mm/mprotect.c
> @@ -35,6 +35,8 @@
>  
>  #include "internal.h"
>  
> +#define NO_KEY	-1
> +
>  static unsigned long change_pte_range(struct vm_area_struct *vma, pmd_t *pmd,
>  		unsigned long addr, unsigned long end, pgprot_t newprot,
>  		int dirty_accountable, int prot_numa)
> @@ -451,9 +453,9 @@ mprotect_fixup(struct vm_area_struct *vma, struct
> vm_area_struct **pprev,
>  }
>  
>  /*
> - * pkey==-1 when doing a legacy mprotect()
> + * When pkey==NO_KEY we get legacy mprotect behavior here.
>   */
> -static int do_mprotect_pkey(unsigned long start, size_t len,
> +static int do_mprotect_ext(unsigned long start, size_t len,
>  		unsigned long prot, int pkey)
>  {
>  	unsigned long nstart, end, tmp, reqprot;
> @@ -577,7 +579,7 @@ static int do_mprotect_pkey(unsigned long start, size_t
> len,
>  SYSCALL_DEFINE3(mprotect, unsigned long, start, size_t, len,
>  		unsigned long, prot)
>  {
> -	return do_mprotect_pkey(start, len, prot, -1);
> +	return do_mprotect_ext(start, len, prot, NO_KEY);
>  }
>  
>  #ifdef CONFIG_ARCH_HAS_PKEYS
> @@ -585,7 +587,7 @@ SYSCALL_DEFINE3(mprotect, unsigned long, start, size_t,
> len,
>  SYSCALL_DEFINE4(pkey_mprotect, unsigned long, start, size_t, len,
>  		unsigned long, prot, int, pkey)
>  {
> -	return do_mprotect_pkey(start, len, prot, pkey);
> +	return do_mprotect_ext(start, len, prot, pkey);
>  }
>  
>  SYSCALL_DEFINE2(pkey_alloc, unsigned long, flags, unsigned long, init_val)

Would squash this whatever this is required for. This split makes
review more complex (IMHO).

/Jarkko

Re: [RFC v2 04/13] x86/mm: Add helper functions for MKTME memory encryption keys

[Sakkinen, Jarkko]

T24gTW9uLCAyMDE4LTEyLTAzIGF0IDIzOjM5IC0wODAwLCBBbGlzb24gU2Nob2ZpZWxkIHdyb3Rl
Og0KPiArZXh0ZXJuIGludCBta3RtZV9tYXBfYWxsb2Modm9pZCk7DQo+ICtleHRlcm4gdm9pZCBt
a3RtZV9tYXBfZnJlZSh2b2lkKTsNCj4gK2V4dGVybiB2b2lkIG1rdG1lX21hcF9sb2NrKHZvaWQp
Ow0KPiArZXh0ZXJuIHZvaWQgbWt0bWVfbWFwX3VubG9jayh2b2lkKTsNCj4gK2V4dGVybiBpbnQg
bWt0bWVfbWFwX21hcHBlZF9rZXlpZHModm9pZCk7DQo+ICtleHRlcm4gdm9pZCBta3RtZV9tYXBf
c2V0X2tleWlkKGludCBrZXlpZCwgdm9pZCAqa2V5KTsNCj4gK2V4dGVybiB2b2lkIG1rdG1lX21h
cF9mcmVlX2tleWlkKGludCBrZXlpZCk7DQo+ICtleHRlcm4gaW50IG1rdG1lX21hcF9rZXlpZF9m
cm9tX2tleSh2b2lkICprZXkpOw0KPiArZXh0ZXJuIHZvaWQgKm1rdG1lX21hcF9rZXlfZnJvbV9r
ZXlpZChpbnQga2V5aWQpOw0KPiArZXh0ZXJuIGludCBta3RtZV9tYXBfZ2V0X2ZyZWVfa2V5aWQo
dm9pZCk7DQoNCk5vIG5lZWQgZm9yIGV4dGVybiBrZXl3b3JkIGZvciBmdW5jdGlvbiBkZWNsYXJh
dGlvbnMuIEl0IGlzDQpvbmx5IG5lZWRlZCBmb3IgdmFyaWFibGUgZGVjbGFyYXRpb25zLg0KDQo+
ICsNCj4gIERFQ0xBUkVfU1RBVElDX0tFWV9GQUxTRShta3RtZV9lbmFibGVkX2tleSk7DQo+ICBz
dGF0aWMgaW5saW5lIGJvb2wgbWt0bWVfZW5hYmxlZCh2b2lkKQ0KPiAgew0KPiBkaWZmIC0tZ2l0
IGEvYXJjaC94ODYvbW0vbWt0bWUuYyBiL2FyY2gveDg2L21tL21rdG1lLmMNCj4gaW5kZXggYzgx
NzI3NTQwZTdjLi4zNDIyNGQ0ZTNmNDUgMTAwNjQ0DQo+IC0tLSBhL2FyY2gveDg2L21tL21rdG1l
LmMNCj4gKysrIGIvYXJjaC94ODYvbW0vbWt0bWUuYw0KPiBAQCAtNDAsNiArNDAsOTcgQEAgaW50
IF9fdm1hX2tleWlkKHN0cnVjdCB2bV9hcmVhX3N0cnVjdCAqdm1hKQ0KPiAgCXJldHVybiAocHJv
dCAmIG1rdG1lX2tleWlkX21hc2spID4+IG1rdG1lX2tleWlkX3NoaWZ0Ow0KPiAgfQ0KPiAgDQo+
ICsvKg0KPiArICogc3RydWN0IG1rdG1lX21hcCBhbmQgdGhlIG1rdG1lX21hcF8qIGZ1bmN0aW9u
cyBtYW5hZ2UgdGhlIG1hcHBpbmcNCj4gKyAqIG9mIHVzZXJzcGFjZSBLZXlzIHRvIGhhcmR3YXJl
IEtleUlEcy4gVGhlc2UgYXJlIHVzZWQgYnkgdGhlIE1LVE1FIEtleQ0KDQpXaGF0IGFyZSAidXNl
cnNwYWNlIEtleXMiIGFueXdheSBhbmQgd2h5IEtleSBhbmQgbm90IGtleT8NCg0KPiArICogU2Vy
dmljZSBBUEkgYW5kIHRoZSBlbmNyeXB0X21wcm90ZWN0KCkgc3lzdGVtIGNhbGwuDQo+ICsgKi8N
Cj4gKw0KPiArc3RydWN0IG1rdG1lX21hcHBpbmcgew0KPiArCXN0cnVjdCBtdXRleAlsb2NrOwkJ
LyogcHJvdGVjdCB0aGlzIG1hcCAmIEhXIHN0YXRlICovDQo+ICsJdW5zaWduZWQgaW50CW1hcHBl
ZF9rZXlpZHM7DQo+ICsJdm9pZAkJKmtleVtdOw0KPiArfTsNCg0KUGVyc29uYWxseSwgSSBwcmVm
ZXIgbm90IHRvIGFsaWduIHN0cnVjdCBmaWVsZHMgKEkgZG8gYWxpZ24gZW51bXMNCmJlY2F1c2Ug
dGhlcmUgaXQgbWFrZXMgbW9yZSBzZW5zZSkgYXMgb2Z0ZW4geW91IGVuZCB1cCByZWFsaWduaW5n
DQpldmVyeXRoaW5nLg0KDQpEb2N1bWVudGF0aW9uIHdvdWxkIGJyaW5nIG1vcmUgY2xhcml0eS4g
Rm9yIGV4YW1wbGUsIHdoYXQgZG9lcyBrZXlbXQ0KY29udGFpbiwgd2h5IHRoZXJlIGlzIGEgbG9j
ayBhbmQgd2hhdCBtYXBwZWRfa2V5aWRzIGZpZWxkIGNvbnRhaW5zPw0KDQovSmFya2tvDQo

Re: [RFC v2 04/13] x86/mm: Add helper functions for MKTME memory encryption keys

[Sakkinen, Jarkko]

On Mon, 2018-12-03 at 23:39 -0800, Alison Schofield wrote:
> +extern int mktme_map_alloc(void);
> +extern void mktme_map_free(void);
> +extern void mktme_map_lock(void);
> +extern void mktme_map_unlock(void);
> +extern int mktme_map_mapped_keyids(void);
> +extern void mktme_map_set_keyid(int keyid, void *key);
> +extern void mktme_map_free_keyid(int keyid);
> +extern int mktme_map_keyid_from_key(void *key);
> +extern void *mktme_map_key_from_keyid(int keyid);
> +extern int mktme_map_get_free_keyid(void);

No need for extern keyword for function declarations. It is
only needed for variable declarations.

> +
>  DECLARE_STATIC_KEY_FALSE(mktme_enabled_key);
>  static inline bool mktme_enabled(void)
>  {
> diff --git a/arch/x86/mm/mktme.c b/arch/x86/mm/mktme.c
> index c81727540e7c..34224d4e3f45 100644
> --- a/arch/x86/mm/mktme.c
> +++ b/arch/x86/mm/mktme.c
> @@ -40,6 +40,97 @@ int __vma_keyid(struct vm_area_struct *vma)
>  	return (prot & mktme_keyid_mask) >> mktme_keyid_shift;
>  }
>  
> +/*
> + * struct mktme_map and the mktme_map_* functions manage the mapping
> + * of userspace Keys to hardware KeyIDs. These are used by the MKTME Key

What are "userspace Keys" anyway and why Key and not key?

> + * Service API and the encrypt_mprotect() system call.
> + */
> +
> +struct mktme_mapping {
> +	struct mutex	lock;		/* protect this map & HW state */
> +	unsigned int	mapped_keyids;
> +	void		*key[];
> +};

Personally, I prefer not to align struct fields (I do align enums
because there it makes more sense) as often you end up realigning
everything.

Documentation would bring more clarity. For example, what does key[]
contain, why there is a lock and what mapped_keyids field contains?

/Jarkko

Re: [RFC v2 04/13] x86/mm: Add helper functions for MKTME memory encryption keys

[Sakkinen, Jarkko]

On Mon, 2018-12-03 at 23:39 -0800, Alison Schofield wrote:
> +extern int mktme_map_alloc(void);
> +extern void mktme_map_free(void);
> +extern void mktme_map_lock(void);
> +extern void mktme_map_unlock(void);
> +extern int mktme_map_mapped_keyids(void);
> +extern void mktme_map_set_keyid(int keyid, void *key);
> +extern void mktme_map_free_keyid(int keyid);
> +extern int mktme_map_keyid_from_key(void *key);
> +extern void *mktme_map_key_from_keyid(int keyid);
> +extern int mktme_map_get_free_keyid(void);

No need for extern keyword for function declarations. It is
only needed for variable declarations.

> +
>  DECLARE_STATIC_KEY_FALSE(mktme_enabled_key);
>  static inline bool mktme_enabled(void)
>  {
> diff --git a/arch/x86/mm/mktme.c b/arch/x86/mm/mktme.c
> index c81727540e7c..34224d4e3f45 100644
> --- a/arch/x86/mm/mktme.c
> +++ b/arch/x86/mm/mktme.c
> @@ -40,6 +40,97 @@ int __vma_keyid(struct vm_area_struct *vma)
>  	return (prot & mktme_keyid_mask) >> mktme_keyid_shift;
>  }
>  
> +/*
> + * struct mktme_map and the mktme_map_* functions manage the mapping
> + * of userspace Keys to hardware KeyIDs. These are used by the MKTME Key

What are "userspace Keys" anyway and why Key and not key?

> + * Service API and the encrypt_mprotect() system call.
> + */
> +
> +struct mktme_mapping {
> +	struct mutex	lock;		/* protect this map & HW state */
> +	unsigned int	mapped_keyids;
> +	void		*key[];
> +};

Personally, I prefer not to align struct fields (I do align enums
because there it makes more sense) as often you end up realigning
everything.

Documentation would bring more clarity. For example, what does key[]
contain, why there is a lock and what mapped_keyids field contains?

/Jarkko

Re: [RFC v2 05/13] x86/mm: Set KeyIDs in encrypted VMAs

[Sakkinen, Jarkko]

T24gTW9uLCAyMDE4LTEyLTAzIGF0IDIzOjM5IC0wODAwLCBBbGlzb24gU2Nob2ZpZWxkIHdyb3Rl
Og0KPiBNS1RNRSBhcmNoaXRlY3R1cmUgcmVxdWlyZXMgdGhlIEtleUlEIHRvIGJlIHBsYWNlZCBp
biBQVEUgYml0cyA1MTo0Ni4NCj4gVG8gY3JlYXRlIGFuIGVuY3J5cHRlZCBWTUEsIHBsYWNlIHRo
ZSBLZXlJRCBpbiB0aGUgdXBwZXIgYml0cyBvZg0KPiB2bV9wYWdlX3Byb3QgdGhhdCBtYXRjaGVz
IHRoZSBwb3NpdGlvbiBvZiB0aG9zZSBQVEUgYml0cy4NCj4gDQo+IFdoZW4gdGhlIFZNQSBpcyBh
c3NpZ25lZCBhIEtleUlEIGl0IGlzIGFsd2F5cyBjb25zaWRlcmVkIGEgS2V5SUQNCj4gY2hhbmdl
LiBUaGUgVk1BIGlzIGVpdGhlciBnb2luZyBmcm9tIG5vdCBlbmNyeXB0ZWQgdG8gZW5jcnlwdGVk
LA0KPiBvciBmcm9tIGVuY3J5cHRlZCB3aXRoIGFueSBLZXlJRCB0byBlbmNyeXB0ZWQgd2l0aCBh
bnkgb3RoZXIgS2V5SUQuDQo+IFRvIG1ha2UgdGhlIGNoYW5nZSBzYWZlbHksIHJlbW92ZSB0aGUg
dXNlciBwYWdlcyBoZWxkIGJ5IHRoZSBWTUENCj4gYW5kIHVubGluayB0aGUgVk1BJ3MgYW5vbnlt
b3VzIGNoYWluLg0KPiANCj4gQ2hhbmdlLUlkOiBJNjc2MDU2NTI1YzQ5Yzg4MDM4OTgzMTVhMTBi
MTk2ZWY1YTVjNTQxNQ0KDQpSZW1vdmUuDQoNCj4gU2lnbmVkLW9mZi1ieTogQWxpc29uIFNjaG9m
aWVsZCA8YWxpc29uLnNjaG9maWVsZEBpbnRlbC5jb20+DQo+IFNpZ25lZC1vZmYtYnk6IEtpcmls
bCBBLiBTaHV0ZW1vdiA8a2lyaWxsLnNodXRlbW92QGxpbnV4LmludGVsLmNvbT4NCj4gLS0tDQo+
ICBhcmNoL3g4Ni9pbmNsdWRlL2FzbS9ta3RtZS5oIHwgIDQgKysrKw0KPiAgYXJjaC94ODYvbW0v
bWt0bWUuYyAgICAgICAgICB8IDI2ICsrKysrKysrKysrKysrKysrKysrKysrKysrDQo+ICBpbmNs
dWRlL2xpbnV4L21tLmggICAgICAgICAgIHwgIDYgKysrKysrDQo+ICAzIGZpbGVzIGNoYW5nZWQs
IDM2IGluc2VydGlvbnMoKykNCj4gDQo+IGRpZmYgLS1naXQgYS9hcmNoL3g4Ni9pbmNsdWRlL2Fz
bS9ta3RtZS5oIGIvYXJjaC94ODYvaW5jbHVkZS9hc20vbWt0bWUuaA0KPiBpbmRleCBkYmI0OTkw
OWQ2NjUuLmRlM2U1MjlmM2FiMCAxMDA2NDQNCj4gLS0tIGEvYXJjaC94ODYvaW5jbHVkZS9hc20v
bWt0bWUuaA0KPiArKysgYi9hcmNoL3g4Ni9pbmNsdWRlL2FzbS9ta3RtZS5oDQo+IEBAIC0yNCw2
ICsyNCwxMCBAQCBleHRlcm4gaW50IG1rdG1lX21hcF9rZXlpZF9mcm9tX2tleSh2b2lkICprZXkp
Ow0KPiAgZXh0ZXJuIHZvaWQgKm1rdG1lX21hcF9rZXlfZnJvbV9rZXlpZChpbnQga2V5aWQpOw0K
PiAgZXh0ZXJuIGludCBta3RtZV9tYXBfZ2V0X2ZyZWVfa2V5aWQodm9pZCk7DQo+ICANCj4gKy8q
IFNldCB0aGUgZW5jcnlwdGlvbiBrZXlpZCBiaXRzIGluIGEgVk1BICovDQo+ICtleHRlcm4gdm9p
ZCBtcHJvdGVjdF9zZXRfZW5jcnlwdChzdHJ1Y3Qgdm1fYXJlYV9zdHJ1Y3QgKnZtYSwgaW50IG5l
d2tleWlkLA0KPiArCQkJCXVuc2lnbmVkIGxvbmcgc3RhcnQsIHVuc2lnbmVkIGxvbmcgZW5kKTsN
Cj4gKw0KPiAgREVDTEFSRV9TVEFUSUNfS0VZX0ZBTFNFKG1rdG1lX2VuYWJsZWRfa2V5KTsNCj4g
IHN0YXRpYyBpbmxpbmUgYm9vbCBta3RtZV9lbmFibGVkKHZvaWQpDQo+ICB7DQo+IGRpZmYgLS1n
aXQgYS9hcmNoL3g4Ni9tbS9ta3RtZS5jIGIvYXJjaC94ODYvbW0vbWt0bWUuYw0KPiBpbmRleCAz
NDIyNGQ0ZTNmNDUuLmUzZmRmN2I0ODE3MyAxMDA2NDQNCj4gLS0tIGEvYXJjaC94ODYvbW0vbWt0
bWUuYw0KPiArKysgYi9hcmNoL3g4Ni9tbS9ta3RtZS5jDQo+IEBAIC0xLDUgKzEsNiBAQA0KPiAg
I2luY2x1ZGUgPGxpbnV4L21tLmg+DQo+ICAjaW5jbHVkZSA8bGludXgvaGlnaG1lbS5oPg0KPiAr
I2luY2x1ZGUgPGxpbnV4L3JtYXAuaD4NCj4gICNpbmNsdWRlIDxhc20vbWt0bWUuaD4NCj4gICNp
bmNsdWRlIDxhc20vc2V0X21lbW9yeS5oPg0KPiAgDQo+IEBAIC0xMzEsNiArMTMyLDMxIEBAIGlu
dCBta3RtZV9tYXBfZ2V0X2ZyZWVfa2V5aWQodm9pZCkNCj4gIAlyZXR1cm4gMDsNCj4gIH0NCj4g
IA0KPiArLyogU2V0IHRoZSBlbmNyeXB0aW9uIGtleWlkIGJpdHMgaW4gYSBWTUEgKi8NCg0KTWF5
YmUgcHJvcGVyIGtkb2M/DQoNCj4gK3ZvaWQgbXByb3RlY3Rfc2V0X2VuY3J5cHQoc3RydWN0IHZt
X2FyZWFfc3RydWN0ICp2bWEsIGludCBuZXdrZXlpZCwNCj4gKwkJCSAgdW5zaWduZWQgbG9uZyBz
dGFydCwgdW5zaWduZWQgbG9uZyBlbmQpDQo+ICt7DQo+ICsJaW50IG9sZGtleWlkID0gdm1hX2tl
eWlkKHZtYSk7DQo+ICsJcGdwcm90dmFsX3QgbmV3cHJvdDsNCj4gKw0KPiArCS8qIFVubWFwIHBh
Z2VzIHdpdGggb2xkIEtleUlEIGlmIHRoZXJlJ3MgYW55LiAqLw0KPiArCXphcF9wYWdlX3Jhbmdl
KHZtYSwgc3RhcnQsIGVuZCAtIHN0YXJ0KTsNCj4gKw0KPiArCWlmIChvbGRrZXlpZCA9PSBuZXdr
ZXlpZCkNCj4gKwkJcmV0dXJuOw0KPiArDQo+ICsJbmV3cHJvdCA9IHBncHJvdF92YWwodm1hLT52
bV9wYWdlX3Byb3QpOw0KPiArCW5ld3Byb3QgJj0gfm1rdG1lX2tleWlkX21hc2s7DQo+ICsJbmV3
cHJvdCB8PSAodW5zaWduZWQgbG9uZyluZXdrZXlpZCA8PCBta3RtZV9rZXlpZF9zaGlmdDsNCj4g
Kwl2bWEtPnZtX3BhZ2VfcHJvdCA9IF9fcGdwcm90KG5ld3Byb3QpOw0KPiArDQo+ICsJLyoNCg0K
Tm8gZW1wdHkgY29tbWVudCBsaW5lLg0KDQo+ICsJICogVGhlIFZNQSBkb2Vzbid0IGhhdmUgYW55
IGluaGVyaXRlZCBwYWdlcy4NCj4gKwkgKiBTdGFydCBhbm9uIFZNQSB0cmVlIGZyb20gc2NyYXRj
aC4NCj4gKwkgKi8NCj4gK30NCj4gKw0KPiAgLyogUHJlcGFyZSBwYWdlIHRvIGJlIHVzZWQgZm9y
IGVuY3J5cHRpb24uIENhbGxlZCBmcm9tIHBhZ2UgYWxsb2NhdG9yLiAqLw0KPiAgdm9pZCBfX3By
ZXBfZW5jcnlwdGVkX3BhZ2Uoc3RydWN0IHBhZ2UgKnBhZ2UsIGludCBvcmRlciwgaW50IGtleWlk
LCBib29sDQo+IHplcm8pDQo+ICB7DQo+IGRpZmYgLS1naXQgYS9pbmNsdWRlL2xpbnV4L21tLmgg
Yi9pbmNsdWRlL2xpbnV4L21tLmgNCj4gaW5kZXggMTMwOTc2MWJiNmQwLi5lMmQ4N2U5MmNhNzQg
MTAwNjQ0DQo+IC0tLSBhL2luY2x1ZGUvbGludXgvbW0uaA0KPiArKysgYi9pbmNsdWRlL2xpbnV4
L21tLmgNCj4gQEAgLTI4MDYsNSArMjgwNiwxMSBAQCB2b2lkIF9faW5pdCBzZXR1cF9ucl9ub2Rl
X2lkcyh2b2lkKTsNCj4gIHN0YXRpYyBpbmxpbmUgdm9pZCBzZXR1cF9ucl9ub2RlX2lkcyh2b2lk
KSB7fQ0KPiAgI2VuZGlmDQo+ICANCj4gKyNpZm5kZWYgQ09ORklHX1g4Nl9JTlRFTF9NS1RNRQ0K
PiArc3RhdGljIGlubGluZSB2b2lkIG1wcm90ZWN0X3NldF9lbmNyeXB0KHN0cnVjdCB2bV9hcmVh
X3N0cnVjdCAqdm1hLA0KPiArCQkJCQlpbnQgbmV3a2V5aWQsDQo+ICsJCQkJCXVuc2lnbmVkIGxv
bmcgc3RhcnQsDQo+ICsJCQkJCXVuc2lnbmVkIGxvbmcgZW5kKSB7fQ0KDQpBZGQgYSBuZXcgbGlu
ZSBhbmQNCg0Kew0KfQ0KDQoNCj4gKyNlbmRpZiAvKiBDT05GSUdfWDg2X0lOVEVMX01LVE1FICov
DQo+ICAjZW5kaWYgLyogX19LRVJORUxfXyAqLw0KPiAgI2VuZGlmIC8qIF9MSU5VWF9NTV9IICov
DQoNCi9KYXJra28NCg=

Re: [RFC v2 05/13] x86/mm: Set KeyIDs in encrypted VMAs

[Sakkinen, Jarkko]

On Mon, 2018-12-03 at 23:39 -0800, Alison Schofield wrote:
> MKTME architecture requires the KeyID to be placed in PTE bits 51:46.
> To create an encrypted VMA, place the KeyID in the upper bits of
> vm_page_prot that matches the position of those PTE bits.
> 
> When the VMA is assigned a KeyID it is always considered a KeyID
> change. The VMA is either going from not encrypted to encrypted,
> or from encrypted with any KeyID to encrypted with any other KeyID.
> To make the change safely, remove the user pages held by the VMA
> and unlink the VMA's anonymous chain.
> 
> Change-Id: I676056525c49c8803898315a10b196ef5a5c5415

Remove.

> Signed-off-by: Alison Schofield <alison.schofield@intel.com>
> Signed-off-by: Kirill A. Shutemov <kirill.shutemov@linux.intel.com>
> ---
>  arch/x86/include/asm/mktme.h |  4 ++++
>  arch/x86/mm/mktme.c          | 26 ++++++++++++++++++++++++++
>  include/linux/mm.h           |  6 ++++++
>  3 files changed, 36 insertions(+)
> 
> diff --git a/arch/x86/include/asm/mktme.h b/arch/x86/include/asm/mktme.h
> index dbb49909d665..de3e529f3ab0 100644
> --- a/arch/x86/include/asm/mktme.h
> +++ b/arch/x86/include/asm/mktme.h
> @@ -24,6 +24,10 @@ extern int mktme_map_keyid_from_key(void *key);
>  extern void *mktme_map_key_from_keyid(int keyid);
>  extern int mktme_map_get_free_keyid(void);
>  
> +/* Set the encryption keyid bits in a VMA */
> +extern void mprotect_set_encrypt(struct vm_area_struct *vma, int newkeyid,
> +				unsigned long start, unsigned long end);
> +
>  DECLARE_STATIC_KEY_FALSE(mktme_enabled_key);
>  static inline bool mktme_enabled(void)
>  {
> diff --git a/arch/x86/mm/mktme.c b/arch/x86/mm/mktme.c
> index 34224d4e3f45..e3fdf7b48173 100644
> --- a/arch/x86/mm/mktme.c
> +++ b/arch/x86/mm/mktme.c
> @@ -1,5 +1,6 @@
>  #include <linux/mm.h>
>  #include <linux/highmem.h>
> +#include <linux/rmap.h>
>  #include <asm/mktme.h>
>  #include <asm/set_memory.h>
>  
> @@ -131,6 +132,31 @@ int mktme_map_get_free_keyid(void)
>  	return 0;
>  }
>  
> +/* Set the encryption keyid bits in a VMA */

Maybe proper kdoc?

> +void mprotect_set_encrypt(struct vm_area_struct *vma, int newkeyid,
> +			  unsigned long start, unsigned long end)
> +{
> +	int oldkeyid = vma_keyid(vma);
> +	pgprotval_t newprot;
> +
> +	/* Unmap pages with old KeyID if there's any. */
> +	zap_page_range(vma, start, end - start);
> +
> +	if (oldkeyid == newkeyid)
> +		return;
> +
> +	newprot = pgprot_val(vma->vm_page_prot);
> +	newprot &= ~mktme_keyid_mask;
> +	newprot |= (unsigned long)newkeyid << mktme_keyid_shift;
> +	vma->vm_page_prot = __pgprot(newprot);
> +
> +	/*

No empty comment line.

> +	 * The VMA doesn't have any inherited pages.
> +	 * Start anon VMA tree from scratch.
> +	 */
> +}
> +
>  /* Prepare page to be used for encryption. Called from page allocator. */
>  void __prep_encrypted_page(struct page *page, int order, int keyid, bool
> zero)
>  {
> diff --git a/include/linux/mm.h b/include/linux/mm.h
> index 1309761bb6d0..e2d87e92ca74 100644
> --- a/include/linux/mm.h
> +++ b/include/linux/mm.h
> @@ -2806,5 +2806,11 @@ void __init setup_nr_node_ids(void);
>  static inline void setup_nr_node_ids(void) {}
>  #endif
>  
> +#ifndef CONFIG_X86_INTEL_MKTME
> +static inline void mprotect_set_encrypt(struct vm_area_struct *vma,
> +					int newkeyid,
> +					unsigned long start,
> +					unsigned long end) {}

Add a new line and

{
}


> +#endif /* CONFIG_X86_INTEL_MKTME */
>  #endif /* __KERNEL__ */
>  #endif /* _LINUX_MM_H */

/Jarkko

Re: [RFC v2 05/13] x86/mm: Set KeyIDs in encrypted VMAs

[Sakkinen, Jarkko]

On Mon, 2018-12-03 at 23:39 -0800, Alison Schofield wrote:
> MKTME architecture requires the KeyID to be placed in PTE bits 51:46.
> To create an encrypted VMA, place the KeyID in the upper bits of
> vm_page_prot that matches the position of those PTE bits.
> 
> When the VMA is assigned a KeyID it is always considered a KeyID
> change. The VMA is either going from not encrypted to encrypted,
> or from encrypted with any KeyID to encrypted with any other KeyID.
> To make the change safely, remove the user pages held by the VMA
> and unlink the VMA's anonymous chain.
> 
> Change-Id: I676056525c49c8803898315a10b196ef5a5c5415

Remove.

> Signed-off-by: Alison Schofield <alison.schofield@intel.com>
> Signed-off-by: Kirill A. Shutemov <kirill.shutemov@linux.intel.com>
> ---
>  arch/x86/include/asm/mktme.h |  4 ++++
>  arch/x86/mm/mktme.c          | 26 ++++++++++++++++++++++++++
>  include/linux/mm.h           |  6 ++++++
>  3 files changed, 36 insertions(+)
> 
> diff --git a/arch/x86/include/asm/mktme.h b/arch/x86/include/asm/mktme.h
> index dbb49909d665..de3e529f3ab0 100644
> --- a/arch/x86/include/asm/mktme.h
> +++ b/arch/x86/include/asm/mktme.h
> @@ -24,6 +24,10 @@ extern int mktme_map_keyid_from_key(void *key);
>  extern void *mktme_map_key_from_keyid(int keyid);
>  extern int mktme_map_get_free_keyid(void);
>  
> +/* Set the encryption keyid bits in a VMA */
> +extern void mprotect_set_encrypt(struct vm_area_struct *vma, int newkeyid,
> +				unsigned long start, unsigned long end);
> +
>  DECLARE_STATIC_KEY_FALSE(mktme_enabled_key);
>  static inline bool mktme_enabled(void)
>  {
> diff --git a/arch/x86/mm/mktme.c b/arch/x86/mm/mktme.c
> index 34224d4e3f45..e3fdf7b48173 100644
> --- a/arch/x86/mm/mktme.c
> +++ b/arch/x86/mm/mktme.c
> @@ -1,5 +1,6 @@
>  #include <linux/mm.h>
>  #include <linux/highmem.h>
> +#include <linux/rmap.h>
>  #include <asm/mktme.h>
>  #include <asm/set_memory.h>
>  
> @@ -131,6 +132,31 @@ int mktme_map_get_free_keyid(void)
>  	return 0;
>  }
>  
> +/* Set the encryption keyid bits in a VMA */

Maybe proper kdoc?

> +void mprotect_set_encrypt(struct vm_area_struct *vma, int newkeyid,
> +			  unsigned long start, unsigned long end)
> +{
> +	int oldkeyid = vma_keyid(vma);
> +	pgprotval_t newprot;
> +
> +	/* Unmap pages with old KeyID if there's any. */
> +	zap_page_range(vma, start, end - start);
> +
> +	if (oldkeyid == newkeyid)
> +		return;
> +
> +	newprot = pgprot_val(vma->vm_page_prot);
> +	newprot &= ~mktme_keyid_mask;
> +	newprot |= (unsigned long)newkeyid << mktme_keyid_shift;
> +	vma->vm_page_prot = __pgprot(newprot);
> +
> +	/*

No empty comment line.

> +	 * The VMA doesn't have any inherited pages.
> +	 * Start anon VMA tree from scratch.
> +	 */
> +}
> +
>  /* Prepare page to be used for encryption. Called from page allocator. */
>  void __prep_encrypted_page(struct page *page, int order, int keyid, bool
> zero)
>  {
> diff --git a/include/linux/mm.h b/include/linux/mm.h
> index 1309761bb6d0..e2d87e92ca74 100644
> --- a/include/linux/mm.h
> +++ b/include/linux/mm.h
> @@ -2806,5 +2806,11 @@ void __init setup_nr_node_ids(void);
>  static inline void setup_nr_node_ids(void) {}
>  #endif
>  
> +#ifndef CONFIG_X86_INTEL_MKTME
> +static inline void mprotect_set_encrypt(struct vm_area_struct *vma,
> +					int newkeyid,
> +					unsigned long start,
> +					unsigned long end) {}

Add a new line and

{
}


> +#endif /* CONFIG_X86_INTEL_MKTME */
>  #endif /* __KERNEL__ */
>  #endif /* _LINUX_MM_H */

/Jarkko

Re: [RFC v2 06/13] mm: Add the encrypt_mprotect() system call

[Sakkinen, Jarkko]

T24gTW9uLCAyMDE4LTEyLTAzIGF0IDIzOjM5IC0wODAwLCBBbGlzb24gU2Nob2ZpZWxkIHdyb3Rl
Og0KPiBJbXBsZW1lbnQgbWVtb3J5IGVuY3J5cHRpb24gd2l0aCBhIG5ldyBzeXN0ZW0gY2FsbCB0
aGF0IGlzIGFuDQo+IGV4dGVuc2lvbiBvZiB0aGUgbGVnYWN5IG1wcm90ZWN0KCkgc3lzdGVtIGNh
bGwuDQo+IA0KPiBJbiBlbmNyeXB0X21wcm90ZWN0IHRoZSBjYWxsZXIgbXVzdCBwYXNzIGEgaGFu
ZGxlIHRvIGEgcHJldmlvdXNseQ0KPiBhbGxvY2F0ZWQgYW5kIHByb2dyYW1tZWQgZW5jcnlwdGlv
biBrZXkuIFZhbGlkYXRlIHRoZSBrZXkgYW5kIHN0b3JlDQo+IHRoZSBrZXlpZCBiaXRzIGluIHRo
ZSB2bV9wYWdlX3Byb3QgZm9yIGVhY2ggVk1BIGluIHRoZSBwcm90ZWN0aW9uDQo+IHJhbmdlLg0K
PiANCj4gU2lnbmVkLW9mZi1ieTogQWxpc29uIFNjaG9maWVsZCA8YWxpc29uLnNjaG9maWVsZEBp
bnRlbC5jb20+DQo+IFNpZ25lZC1vZmYtYnk6IEtpcmlsbCBBLiBTaHV0ZW1vdiA8a2lyaWxsLnNo
dXRlbW92QGxpbnV4LmludGVsLmNvbT4NCg0KV2h5IHlvdSBkb24ndCB1c2UgdGhhdCBOT19LRVkg
aW4gdGhpcyBwYXRjaD8NCg0KL0phcmtrbw0K

Re: [RFC v2 06/13] mm: Add the encrypt_mprotect() system call

[Sakkinen, Jarkko]

On Mon, 2018-12-03 at 23:39 -0800, Alison Schofield wrote:
> Implement memory encryption with a new system call that is an
> extension of the legacy mprotect() system call.
> 
> In encrypt_mprotect the caller must pass a handle to a previously
> allocated and programmed encryption key. Validate the key and store
> the keyid bits in the vm_page_prot for each VMA in the protection
> range.
> 
> Signed-off-by: Alison Schofield <alison.schofield@intel.com>
> Signed-off-by: Kirill A. Shutemov <kirill.shutemov@linux.intel.com>

Why you don't use that NO_KEY in this patch?

/Jarkko

Re: [RFC v2 06/13] mm: Add the encrypt_mprotect() system call

[Sakkinen, Jarkko]

On Mon, 2018-12-03 at 23:39 -0800, Alison Schofield wrote:
> Implement memory encryption with a new system call that is an
> extension of the legacy mprotect() system call.
> 
> In encrypt_mprotect the caller must pass a handle to a previously
> allocated and programmed encryption key. Validate the key and store
> the keyid bits in the vm_page_prot for each VMA in the protection
> range.
> 
> Signed-off-by: Alison Schofield <alison.schofield@intel.com>
> Signed-off-by: Kirill A. Shutemov <kirill.shutemov@linux.intel.com>

Why you don't use that NO_KEY in this patch?

/Jarkko

Re: [RFC v2 10/13] keys/mktme: Add the MKTME Key Service type for memory encryption

[Sakkinen, Jarkko]

T24gTW9uLCAyMDE4LTEyLTAzIGF0IDIzOjM5IC0wODAwLCBBbGlzb24gU2Nob2ZpZWxkIHdyb3Rl
Og0KPiBNS1RNRSAoTXVsdGktS2V5IFRvdGFsIE1lbW9yeSBFbmNyeXB0aW9uKSBpcyBhIHRlY2hu
b2xvZ3kgdGhhdCBhbGxvd3MNCj4gdHJhbnNwYXJlbnQgbWVtb3J5IGVuY3J5cHRpb24gaW4gdXBj
b21pbmcgSW50ZWwgcGxhdGZvcm1zLiBNS1RNRSB3aWxsDQo+IHN1cHBvcnQgbXVsaXRwbGUgZW5j
cnlwdGlvbiBkb21haW5zLCBlYWNoIGhhdmluZyB0aGVpciBvd24ga2V5LiBUaGUgbWFpbg0KPiB1
c2UgY2FzZSBmb3IgdGhlIGZlYXR1cmUgaXMgdmlydHVhbCBtYWNoaW5lIGlzb2xhdGlvbi4gVGhl
IEFQSSBuZWVkcyB0aGUNCj4gZmxleGliaWxpdHkgdG8gd29yayBmb3IgYSB3aWRlIHJhbmdlIG9m
IHVzZXMuDQoNClNvbWUsIG1heWJlIGJydXRhbCwgaG9uZXN0eSAoYXBvbG9naWVzKS4uLg0KDQpI
YXZlIG5ldmVyIHJlYWxseSBnb3QgdGhlIGdyaXAgd2h5IGVpdGhlciBTTUUgb3IgVE1FIHdvdWxk
IG1ha2UNCmlzb2xhdGlvbiBhbnkgYmV0dGVyLiBJZiB5b3UgY2FuIGJyZWFrIGludG8gaHlwZXJ2
aXNvciwgeW91J2xsDQpoYXZlIHRoZXNlIHRvb2xzIGF2YWlsYWJlOg0KDQoxLiBSZWFkIHBhZ2Ug
KGluIGVuY3J5cHRlZCBmb3JtKS4NCjIuIFdyaXRlIHBhZ2UgKGZvciBleGFtcGxlIHJlcGxheSBh
cyBwYWdlcyBhcmUgbm90IHZlcnNpb25lZCkuDQoNCndpdGggYWxsIHRoZSBzaWRlLWNoYW5uZWwg
cG9zc2liaWxpdGllcyBvZiBjb3Vyc2Ugc2luY2UgeW91IGNhbg0KY29udHJvbCB0aGUgVk1zIChp
biB3aGljaCBjb3JlIHRoZXkgZXhlY3V0ZSBldGMuKS4NCg0KSSd2ZSBzZWVuIG5vdyBTTUUgcHJl
c2VudGF0aW9uIHRocmVlIHRpbWVzIGFuZCBpdCBhbHdheXMgbGVhdmVzDQptZSBhbiBlbXB0eSBm
ZWVsaW5nLiBUaGlzIGZlZWxzIHRoZSBzYW1lIHNhbWUuDQoNCj4gVGhlIE1LVE1FIGtleSBzZXJ2
aWNlIHR5cGUgbWFuYWdlcyB0aGUgYWRkaXRpb24gYW5kIHJlbW92YWwgb2YgdGhlIG1lbW9yeQ0K
PiBlbmNyeXB0aW9uIGtleXMuIEl0IG1hcHMgVXNlcnNwYWNlIEtleXMgdG8gaGFyZHdhcmUgS2V5
SURzLiBJdCBwcm9ncmFtcw0KPiB0aGUgaGFyZHdhcmUgd2l0aCB0aGUgdXNlciByZXF1ZXN0ZWQg
ZW5jcnlwdGlvbiBvcHRpb25zLg0KPiANCj4gVGhlIG9ubHkgc3VwcG9ydGVkIGVuY3J5cHRpb24g
YWxnb3JpdGhtIGlzIEFFUy1YVFMgMTI4Lg0KPiANCj4gVGhlIE1LVE1FIGtleSBzZXJ2aWNlIGlz
IGhhbGYgb2YgdGhlIE1LVE1FIEFQSSBsZXZlbCBzb2x1dGlvbi4gSXQgcGFpcnMNCj4gd2l0aCBh
IG5ldyBtZW1vcnkgZW5jcnlwdGlvbiBzeXN0ZW0gY2FsbDogZW5jcnlwdF9tcHJvdGVjdCgpIHRo
YXQgdXNlcw0KPiB0aGUga2V5cyB0byBlbmNyeXB0IG1lbW9yeS4NCj4gDQo+IFNlZSBEb2N1bWVu
dGF0aW9uL3g4Ni9ta3RtZS9ta3RtZS5yc3QNCj4gDQo+IENoYW5nZS1JZDogSWRkYTRhZjJiZWFi
YjczOWM3NzcxOTg5N2FmZmZmMTgzZWU5ZmE3MTYNCj4gU2lnbmVkLW9mZi1ieTogQWxpc29uIFNj
aG9maWVsZCA8YWxpc29uLnNjaG9maWVsZEBpbnRlbC5jb20+DQo+IFNpZ25lZC1vZmYtYnk6IEtp
cmlsbCBBLiBTaHV0ZW1vdiA8a2lyaWxsLnNodXRlbW92QGxpbnV4LmludGVsLmNvbT4NCj4gLS0t
DQo+ICBhcmNoL3g4Ni9LY29uZmlnICAgICAgICAgICB8ICAgMSArDQo+ICBpbmNsdWRlL2tleXMv
bWt0bWUtdHlwZS5oICB8ICA0MSArKysrKysNCj4gIHNlY3VyaXR5L2tleXMvS2NvbmZpZyAgICAg
IHwgIDExICsrDQo+ICBzZWN1cml0eS9rZXlzL01ha2VmaWxlICAgICB8ICAgMSArDQo+ICBzZWN1
cml0eS9rZXlzL21rdG1lX2tleXMuYyB8IDMzOQ0KPiArKysrKysrKysrKysrKysrKysrKysrKysr
KysrKysrKysrKysrKysrKysrKysNCj4gIDUgZmlsZXMgY2hhbmdlZCwgMzkzIGluc2VydGlvbnMo
KykNCj4gIGNyZWF0ZSBtb2RlIDEwMDY0NCBpbmNsdWRlL2tleXMvbWt0bWUtdHlwZS5oDQo+ICBj
cmVhdGUgbW9kZSAxMDA2NDQgc2VjdXJpdHkva2V5cy9ta3RtZV9rZXlzLmMNCj4gDQo+IGRpZmYg
LS1naXQgYS9hcmNoL3g4Ni9LY29uZmlnIGIvYXJjaC94ODYvS2NvbmZpZw0KPiBpbmRleCA3YWM3
OGUyODU2YzcuLmMyZTNiYjVhZjA3NyAxMDA2NDQNCj4gLS0tIGEvYXJjaC94ODYvS2NvbmZpZw0K
PiArKysgYi9hcmNoL3g4Ni9LY29uZmlnDQo+IEBAIC0xNTMxLDYgKzE1MzEsNyBAQCBjb25maWcg
WDg2X0lOVEVMX01LVE1FDQo+ICAJYm9vbCAiSW50ZWwgTXVsdGktS2V5IFRvdGFsIE1lbW9yeSBF
bmNyeXB0aW9uIg0KPiAgCXNlbGVjdCBEWU5BTUlDX1BIWVNJQ0FMX01BU0sNCj4gIAlzZWxlY3Qg
UEFHRV9FWFRFTlNJT04NCj4gKwlzZWxlY3QgTUtUTUVfS0VZUw0KPiAgCWRlcGVuZHMgb24gWDg2
XzY0ICYmIENQVV9TVVBfSU5URUwNCj4gIAktLS1oZWxwLS0tDQo+ICAJICBTYXkgeWVzIHRvIGVu
YWJsZSBzdXBwb3J0IGZvciBNdWx0aS1LZXkgVG90YWwgTWVtb3J5IEVuY3J5cHRpb24uDQo+IGRp
ZmYgLS1naXQgYS9pbmNsdWRlL2tleXMvbWt0bWUtdHlwZS5oIGIvaW5jbHVkZS9rZXlzL21rdG1l
LXR5cGUuaA0KPiBuZXcgZmlsZSBtb2RlIDEwMDY0NA0KPiBpbmRleCAwMDAwMDAwMDAwMDAuLmM2
M2M2NTY4MDg3Zg0KPiAtLS0gL2Rldi9udWxsDQo+ICsrKyBiL2luY2x1ZGUva2V5cy9ta3RtZS10
eXBlLmgNCj4gQEAgLTAsMCArMSw0MSBAQA0KPiArLyogU1BEWC1MaWNlbnNlLUlkZW50aWZpZXI6
IEdQTC0yLjAgKi8NCj4gKw0KPiArLyogS2V5IHNlcnZpY2UgZm9yIE11bHRpLUtFWSBUb3RhbCBN
ZW1vcnkgRW5jcnlwdGlvbiAqLw0KPiArDQo+ICsjaWZuZGVmIF9LRVlTX01LVE1FX1RZUEVfSA0K
PiArI2RlZmluZSBfS0VZU19NS1RNRV9UWVBFX0gNCj4gKw0KPiArI2luY2x1ZGUgPGxpbnV4L2tl
eS5oPg0KPiArDQo+ICsvKg0KPiArICogVGhlIEFFUy1YVFMgMTI4IGVuY3J5cHRpb24gYWxnb3Jp
dGhtIHJlcXVpcmVzIDEyOCBiaXRzIGZvciBlYWNoDQo+ICsgKiB1c2VyIHN1cHBsaWVkIGRhdGEg
a2V5IGFuZCB0d2VhayBrZXkuDQo+ICsgKi8NCj4gKyNkZWZpbmUgTUtUTUVfQUVTX1hUU19TSVpF
CTE2CS8qIDE2IGJ5dGVzLCAxMjggYml0cyAqLw0KPiArDQo+ICtlbnVtIG1rdG1lX2FsZyB7DQo+
ICsJTUtUTUVfQUxHX0FFU19YVFNfMTI4LA0KPiArfTsNCj4gKw0KPiArY29uc3QgY2hhciAqY29u
c3QgbWt0bWVfYWxnX25hbWVzW10gPSB7DQo+ICsJW01LVE1FX0FMR19BRVNfWFRTXzEyOF0JPSAi
YWVzLXh0cy0xMjgiLA0KPiArfTsNCj4gKw0KPiArZW51bSBta3RtZV90eXBlIHsNCj4gKwlNS1RN
RV9UWVBFX0VSUk9SID0gLTEsDQo+ICsJTUtUTUVfVFlQRV9VU0VSLA0KPiArCU1LVE1FX1RZUEVf
Q1BVLA0KPiArCU1LVE1FX1RZUEVfQ0xFQVIsDQo+ICsJTUtUTUVfVFlQRV9OT19FTkNSWVBULA0K
PiArfTsNCj4gKw0KPiArY29uc3QgY2hhciAqY29uc3QgbWt0bWVfdHlwZV9uYW1lc1tdID0gew0K
PiArCVtNS1RNRV9UWVBFX1VTRVJdCT0gInVzZXIiLA0KPiArCVtNS1RNRV9UWVBFX0NQVV0JPSAi
Y3B1IiwNCj4gKwlbTUtUTUVfVFlQRV9DTEVBUl0JPSAiY2xlYXIiLA0KPiArCVtNS1RNRV9UWVBF
X05PX0VOQ1JZUFRdCT0gIm5vLWVuY3J5cHQiLA0KPiArfTsNCj4gKw0KPiArZXh0ZXJuIHN0cnVj
dCBrZXlfdHlwZSBrZXlfdHlwZV9ta3RtZTsNCj4gKw0KPiArI2VuZGlmIC8qIF9LRVlTX01LVE1F
X1RZUEVfSCAqLw0KPiBkaWZmIC0tZ2l0IGEvc2VjdXJpdHkva2V5cy9LY29uZmlnIGIvc2VjdXJp
dHkva2V5cy9LY29uZmlnDQo+IGluZGV4IDY0NjJlNjY1NGNjZi4uYzM2OTcyMTEzZTY3IDEwMDY0
NA0KPiAtLS0gYS9zZWN1cml0eS9rZXlzL0tjb25maWcNCj4gKysrIGIvc2VjdXJpdHkva2V5cy9L
Y29uZmlnDQo+IEBAIC0xMDEsMyArMTAxLDE0IEBAIGNvbmZpZyBLRVlfREhfT1BFUkFUSU9OUw0K
PiAgCSBpbiB0aGUga2VybmVsLg0KPiAgDQo+ICAJIElmIHlvdSBhcmUgdW5zdXJlIGFzIHRvIHdo
ZXRoZXIgdGhpcyBpcyByZXF1aXJlZCwgYW5zd2VyIE4uDQo+ICsNCj4gK2NvbmZpZyBNS1RNRV9L
RVlTDQo+ICsJYm9vbCAiTXVsdGktS2V5IFRvdGFsIE1lbW9yeSBFbmNyeXB0aW9uIEtleXMiDQo+
ICsJZGVwZW5kcyBvbiBLRVlTICYmIFg4Nl9JTlRFTF9NS1RNRQ0KPiArCWhlbHANCj4gKwkgIFRo
aXMgb3B0aW9uIHByb3ZpZGVzIHN1cHBvcnQgZm9yIE11bHRpLUtleSBUb3RhbCBNZW1vcnkNCj4g
KwkgIEVuY3J5cHRpb24gKE1LVE1FKSBvbiBJbnRlbCBwbGF0Zm9ybXMgb2ZmZXJpbmcgdGhlIGZl
YXR1cmUuDQo+ICsJICBNS1RNRSBhbGxvd3MgdXNlcnNwYWNlIHRvIG1hbmFnZSB0aGUgaGFyZHdh
cmUgZW5jcnlwdGlvbg0KPiArCSAga2V5cyB0aHJvdWdoIHRoZSBrZXJuZWwga2V5IHNlcnZpY2Vz
Lg0KPiArDQo+ICsJICBJZiB5b3UgYXJlIHVuc3VyZSBhcyB0byB3aGV0aGVyIHRoaXMgaXMgcmVx
dWlyZWQsIGFuc3dlciBOLg0KPiBkaWZmIC0tZ2l0IGEvc2VjdXJpdHkva2V5cy9NYWtlZmlsZSBi
L3NlY3VyaXR5L2tleXMvTWFrZWZpbGUNCj4gaW5kZXggOWNlZjU0MDY0ZjYwLi45NGM4NGYxMGE4
NTcgMTAwNjQ0DQo+IC0tLSBhL3NlY3VyaXR5L2tleXMvTWFrZWZpbGUNCj4gKysrIGIvc2VjdXJp
dHkva2V5cy9NYWtlZmlsZQ0KPiBAQCAtMzAsMyArMzAsNCBAQCBvYmotJChDT05GSUdfQVNZTU1F
VFJJQ19LRVlfVFlQRSkgKz0ga2V5Y3RsX3BrZXkubw0KPiAgb2JqLSQoQ09ORklHX0JJR19LRVlT
KSArPSBiaWdfa2V5Lm8NCj4gIG9iai0kKENPTkZJR19UUlVTVEVEX0tFWVMpICs9IHRydXN0ZWQu
bw0KPiAgb2JqLSQoQ09ORklHX0VOQ1JZUFRFRF9LRVlTKSArPSBlbmNyeXB0ZWQta2V5cy8NCj4g
K29iai0kKENPTkZJR19NS1RNRV9LRVlTKSArPSBta3RtZV9rZXlzLm8NCj4gZGlmZiAtLWdpdCBh
L3NlY3VyaXR5L2tleXMvbWt0bWVfa2V5cy5jIGIvc2VjdXJpdHkva2V5cy9ta3RtZV9rZXlzLmMN
Cj4gbmV3IGZpbGUgbW9kZSAxMDA2NDQNCj4gaW5kZXggMDAwMDAwMDAwMDAwLi5lNjE1ZWI1OGU2
MDANCj4gLS0tIC9kZXYvbnVsbA0KPiArKysgYi9zZWN1cml0eS9rZXlzL21rdG1lX2tleXMuYw0K
PiBAQCAtMCwwICsxLDMzOSBAQA0KPiArLy8gU1BEWC1MaWNlbnNlLUlkZW50aWZpZXI6IEdQTC0z
LjANCj4gKw0KPiArLyogRG9jdW1lbnRhdGlvbi94ODYvbWt0bWUvbWt0bWVfa2V5cy5yc3QgKi8N
Cj4gKw0KPiArI2luY2x1ZGUgPGxpbnV4L2NyZWQuaD4NCj4gKyNpbmNsdWRlIDxsaW51eC9jcHUu
aD4NCj4gKyNpbmNsdWRlIDxsaW51eC9lcnIuaD4NCj4gKyNpbmNsdWRlIDxsaW51eC9pbml0Lmg+
DQo+ICsjaW5jbHVkZSA8bGludXgva2V5Lmg+DQo+ICsjaW5jbHVkZSA8bGludXgva2V5LXR5cGUu
aD4NCj4gKyNpbmNsdWRlIDxsaW51eC9pbml0Lmg+DQo+ICsjaW5jbHVkZSA8bGludXgvcGFyc2Vy
Lmg+DQo+ICsjaW5jbHVkZSA8bGludXgvcmFuZG9tLmg+DQo+ICsjaW5jbHVkZSA8bGludXgvc2xh
Yi5oPg0KPiArI2luY2x1ZGUgPGxpbnV4L3N0cmluZy5oPg0KPiArI2luY2x1ZGUgPGFzbS9pbnRl
bF9wY29uZmlnLmg+DQo+ICsjaW5jbHVkZSA8YXNtL21rdG1lLmg+DQo+ICsjaW5jbHVkZSA8a2V5
cy9ta3RtZS10eXBlLmg+DQo+ICsjaW5jbHVkZSA8a2V5cy91c2VyLXR5cGUuaD4NCj4gKw0KPiAr
I2luY2x1ZGUgImludGVybmFsLmgiDQo+ICsNCj4gK3N0cnVjdCBrbWVtX2NhY2hlICpta3RtZV9w
cm9nX2NhY2hlOwkvKiBIYXJkd2FyZSBwcm9ncmFtbWluZyBjYWNoZSAqLw0KPiArDQo+ICtzdGF0
aWMgY29uc3QgY2hhciAqIGNvbnN0IG1rdG1lX3Byb2dyYW1fZXJyW10gPSB7DQo+ICsJIktleUlE
IHdhcyBzdWNjZXNzZnVsbHkgcHJvZ3JhbW1lZCIsCS8qIDAgKi8NCj4gKwkiSW52YWxpZCBLZXlJ
RCBwcm9ncmFtbWluZyBjb21tYW5kIiwJLyogMSAqLw0KPiArCSJJbnN1ZmZpY2llbnQgZW50cm9w
eSIsCQkJLyogMiAqLw0KPiArCSJLZXlJRCBub3QgdmFsaWQiLAkJCS8qIDMgKi8NCj4gKwkiSW52
YWxpZCBlbmNyeXB0aW9uIGFsZ29yaXRobSBjaG9zZW4iLAkvKiA0ICovDQo+ICsJIkZhaWx1cmUg
dG8gYWNjZXNzIGtleSB0YWJsZSIsCQkvKiA1ICovDQo+ICt9Ow0KPiArDQo+ICtlbnVtIG1rdG1l
X29wdF9pZCB7DQo+ICsJT1BUX0VSUk9SID0gLTEsDQo+ICsJT1BUX1RZUEUsDQo+ICsJT1BUX0tF
WSwNCj4gKwlPUFRfVFdFQUssDQo+ICsJT1BUX0FMR09SSVRITSwNCj4gK307DQo+ICsNCj4gK3N0
YXRpYyBjb25zdCBtYXRjaF90YWJsZV90IG1rdG1lX3Rva2VuID0gew0KPiArCXtPUFRfVFlQRSwg
InR5cGU9JXMifSwNCj4gKwl7T1BUX0tFWSwgImtleT0lcyJ9LA0KPiArCXtPUFRfVFdFQUssICJ0
d2Vhaz0lcyJ9LA0KPiArCXtPUFRfQUxHT1JJVEhNLCAiYWxnb3JpdGhtPSVzIn0sDQo+ICsJe09Q
VF9FUlJPUiwgTlVMTH0NCj4gK307DQo+ICsNCj4gK3N0cnVjdCBta3RtZV9wYXlsb2FkIHsNCj4g
Kwl1MzIJCWtleWlkX2N0cmw7CS8qIENvbW1hbmQgJiBFbmNyeXB0aW9uIEFsZ29yaXRobSAqLw0K
PiArCXU4CQlkYXRhX2tleVtNS1RNRV9BRVNfWFRTX1NJWkVdOw0KPiArCXU4CQl0d2Vha19rZXlb
TUtUTUVfQUVTX1hUU19TSVpFXTsNCj4gK307DQo+ICsNCj4gKy8qIEtleSBTZXJ2aWNlIE1ldGhv
ZCBjYWxsZWQgd2hlbiBLZXkgaXMgZ2FyYmFnZSBjb2xsZWN0ZWQuICovDQo+ICtzdGF0aWMgdm9p
ZCBta3RtZV9kZXN0cm95X2tleShzdHJ1Y3Qga2V5ICprZXkpDQo+ICt7DQo+ICsJa2V5X3B1dF9l
bmNyeXB0X3JlZihta3RtZV9tYXBfa2V5aWRfZnJvbV9rZXkoa2V5KSk7DQo+ICt9DQo+ICsNCj4g
Ky8qIENvcHkgdGhlIHBheWxvYWQgdG8gdGhlIEhXIHByb2dyYW1taW5nIHN0cnVjdHVyZSBhbmQg
cHJvZ3JhbSB0aGlzIEtleUlEICovDQo+ICtzdGF0aWMgaW50IG1rdG1lX3Byb2dyYW1fa2V5aWQo
aW50IGtleWlkLCBzdHJ1Y3QgbWt0bWVfcGF5bG9hZCAqcGF5bG9hZCkNCj4gK3sNCj4gKwlzdHJ1
Y3QgbWt0bWVfa2V5X3Byb2dyYW0gKmtwcm9nID0gTlVMTDsNCj4gKwl1OCBrZXJuX2VudHJvcHlb
TUtUTUVfQUVTX1hUU19TSVpFXTsNCj4gKwlpbnQgaSwgcmV0Ow0KPiArDQo+ICsJa3Byb2cgPSBr
bWVtX2NhY2hlX3phbGxvYyhta3RtZV9wcm9nX2NhY2hlLCBHRlBfS0VSTkVMKTsNCj4gKwlpZiAo
IWtwcm9nKQ0KPiArCQlyZXR1cm4gLUVOT01FTTsNCj4gKw0KPiArCS8qIEhhcmR3YXJlIHByb2dy
YW1taW5nIHJlcXVpcmVzIGNhY2hlZCBhbGlnbmVkIHN0cnVjdCAqLw0KPiArCWtwcm9nLT5rZXlp
ZCA9IGtleWlkOw0KPiArCWtwcm9nLT5rZXlpZF9jdHJsID0gcGF5bG9hZC0+a2V5aWRfY3RybDsN
Cj4gKwltZW1jcHkoa3Byb2ctPmtleV9maWVsZF8xLCBwYXlsb2FkLT5kYXRhX2tleSwgTUtUTUVf
QUVTX1hUU19TSVpFKTsNCj4gKwltZW1jcHkoa3Byb2ctPmtleV9maWVsZF8yLCBwYXlsb2FkLT50
d2Vha19rZXksIE1LVE1FX0FFU19YVFNfU0laRSk7DQo+ICsNCj4gKwkvKiBTdHJlbmd0aGVuIHRo
ZSBlbnRyb3B5IGZpZWxkcyBmb3IgQ1BVIGdlbmVyYXRlZCBrZXlzICovDQo+ICsJaWYgKChwYXls
b2FkLT5rZXlpZF9jdHJsICYgMHhmZikgPT0gTUtUTUVfS0VZSURfU0VUX0tFWV9SQU5ET00pIHsN
Cj4gKwkJZ2V0X3JhbmRvbV9ieXRlcygma2Vybl9lbnRyb3B5LCBzaXplb2Yoa2Vybl9lbnRyb3B5
KSk7DQo+ICsJCWZvciAoaSA9IDA7IGkgPCAoTUtUTUVfQUVTX1hUU19TSVpFKTsgaSsrKSB7DQo+
ICsJCQlrcHJvZy0+a2V5X2ZpZWxkXzFbaV0gXj0ga2Vybl9lbnRyb3B5W2ldOw0KPiArCQkJa3By
b2ctPmtleV9maWVsZF8yW2ldIF49IGtlcm5fZW50cm9weVtpXTsNCj4gKwkJfQ0KPiArCX0NCj4g
KwlyZXQgPSBta3RtZV9rZXlfcHJvZ3JhbShrcHJvZyk7DQo+ICsJa21lbV9jYWNoZV9mcmVlKG1r
dG1lX3Byb2dfY2FjaGUsIGtwcm9nKTsNCj4gKwlyZXR1cm4gcmV0Ow0KPiArfQ0KPiArDQo+ICsv
KiBLZXkgU2VydmljZSBNZXRob2QgdG8gdXBkYXRlIGFuIGV4aXN0aW5nIGtleS4gKi8NCj4gK3N0
YXRpYyBpbnQgbWt0bWVfdXBkYXRlX2tleShzdHJ1Y3Qga2V5ICprZXksDQo+ICsJCQkgICAgc3Ry
dWN0IGtleV9wcmVwYXJzZWRfcGF5bG9hZCAqcHJlcCkNCj4gK3sNCj4gKwlzdHJ1Y3QgbWt0bWVf
cGF5bG9hZCAqcGF5bG9hZCA9IHByZXAtPnBheWxvYWQuZGF0YVswXTsNCj4gKwlpbnQga2V5aWQs
IHJlZl9jb3VudDsNCj4gKwlpbnQgcmV0Ow0KPiArDQo+ICsJbWt0bWVfbWFwX2xvY2soKTsNCj4g
KwlrZXlpZCA9IG1rdG1lX21hcF9rZXlpZF9mcm9tX2tleShrZXkpOw0KPiArCWlmIChrZXlpZCA8
PSAwKQ0KPiArCQlyZXR1cm4gLUVJTlZBTDsNCj4gKwkvKg0KPiArCSAqIHJlZl9jb3VudCB3aWxs
IGJlIGF0IGxlYXN0IG9uZSB3aGVuIHdlIGdldCBoZXJlIGJlY2F1c2UgdGhlDQo+ICsJICoga2V5
IGFscmVhZHkgZXhpc3RzLiBJZiByZWZfY291bnQgaXMgbm90ID4gMSwgaXQgaXMgc2FmZSB0bw0K
PiArCSAqIHVwZGF0ZSB0aGUga2V5IHdoaWxlIGhvbGRpbmcgdGhlIG1rdG1lX21hcF9sb2NrLg0K
PiArCSAqLw0KPiArCXJlZl9jb3VudCA9IG1rdG1lX3JlYWRfZW5jcnlwdF9yZWYoa2V5aWQpOw0K
PiArCWlmIChyZWZfY291bnQgPiAxKSB7DQo+ICsJCXByX2RlYnVnKCJta3RtZSBub3QgdXBkYXRp
bmcga2V5aWRbJWRdIGVuY3J5cHRfY291bnRbJWRdXG4iLA0KPiArCQkJIGtleWlkLCByZWZfY291
bnQpOw0KPiArCQlyZXR1cm4gLUVCVVNZOw0KPiArCX0NCj4gKwlyZXQgPSBta3RtZV9wcm9ncmFt
X2tleWlkKGtleWlkLCBwYXlsb2FkKTsNCj4gKwlpZiAocmV0ICE9IE1LVE1FX1BST0dfU1VDQ0VT
Uykgew0KPiArCQlwcl9kZWJ1ZygiJXM6ICVzXG4iLCBfX2Z1bmNfXywgbWt0bWVfcHJvZ3JhbV9l
cnJbcmV0XSk7DQo+ICsJCXJldCA9IC1FTk9LRVk7DQo+ICsJfQ0KPiArCW1rdG1lX21hcF91bmxv
Y2soKTsNCj4gKwlyZXR1cm4gcmV0Ow0KPiArfQ0KPiArDQo+ICsvKiBLZXkgU2VydmljZSBNZXRo
b2QgdG8gY3JlYXRlIGEgbmV3IGtleS4gUGF5bG9hZCBpcyBwcmVwYXJzZWQuICovDQo+ICtpbnQg
bWt0bWVfaW5zdGFudGlhdGVfa2V5KHN0cnVjdCBrZXkgKmtleSwgc3RydWN0IGtleV9wcmVwYXJz
ZWRfcGF5bG9hZA0KPiAqcHJlcCkNCj4gK3sNCj4gKwlzdHJ1Y3QgbWt0bWVfcGF5bG9hZCAqcGF5
bG9hZCA9IHByZXAtPnBheWxvYWQuZGF0YVswXTsNCj4gKwlpbnQga2V5aWQsIHJldDsNCj4gKw0K
PiArCW1rdG1lX21hcF9sb2NrKCk7DQo+ICsJa2V5aWQgPSBta3RtZV9tYXBfZ2V0X2ZyZWVfa2V5
aWQoKTsNCj4gKwlpZiAoa2V5aWQgPT0gMCkgew0KPiArCQlyZXQgPSAtRU5PS0VZOw0KPiArCQln
b3RvIG91dDsNCj4gKwl9DQo+ICsJcmV0ID0gbWt0bWVfcHJvZ3JhbV9rZXlpZChrZXlpZCwgcGF5
bG9hZCk7DQo+ICsJaWYgKHJldCAhPSBNS1RNRV9QUk9HX1NVQ0NFU1MpIHsNCj4gKwkJcHJfZGVi
dWcoIiVzOiAlc1xuIiwgX19mdW5jX18sIG1rdG1lX3Byb2dyYW1fZXJyW3JldF0pOw0KPiArCQly
ZXQgPSAtRU5PS0VZOw0KPiArCQlnb3RvIG91dDsNCj4gKwl9DQo+ICsJbWt0bWVfbWFwX3NldF9r
ZXlpZChrZXlpZCwga2V5KTsNCj4gKwlrZXlfZ2V0X2VuY3J5cHRfcmVmKGtleWlkKTsNCj4gK291
dDoNCj4gKwlta3RtZV9tYXBfdW5sb2NrKCk7DQo+ICsJcmV0dXJuIHJldDsNCj4gK30NCj4gKw0K
PiArLyogVmVyaWZ5IHRoZSB1c2VyIHByb3ZpZGVkIHRoZSBuZWVkZWQgYXJndW1lbnRzIGZvciB0
aGUgVFlQRSBvZiBLZXkgKi8NCj4gK3N0YXRpYyBpbnQgbWt0bWVfY2hlY2tfb3B0aW9ucyhzdHJ1
Y3QgbWt0bWVfcGF5bG9hZCAqcGF5bG9hZCwNCj4gKwkJCSAgICAgICB1bnNpZ25lZCBsb25nIHRv
a2VuX21hc2ssIGVudW0gbWt0bWVfdHlwZSB0eXBlKQ0KPiArew0KPiArCWlmICghdG9rZW5fbWFz
aykNCj4gKwkJcmV0dXJuIC1FSU5WQUw7DQo+ICsNCj4gKwlzd2l0Y2ggKHR5cGUpIHsNCj4gKwlj
YXNlIE1LVE1FX1RZUEVfVVNFUjoNCj4gKwkJaWYgKHRlc3RfYml0KE9QVF9BTEdPUklUSE0sICZ0
b2tlbl9tYXNrKSkNCj4gKwkJCXBheWxvYWQtPmtleWlkX2N0cmwgfD0gTUtUTUVfQUVTX1hUU18x
Mjg7DQo+ICsJCWVsc2UNCj4gKwkJCXJldHVybiAtRUlOVkFMOw0KPiArDQo+ICsJCWlmICgodGVz
dF9iaXQoT1BUX0tFWSwgJnRva2VuX21hc2spKSAmJg0KPiArCQkgICAgKHRlc3RfYml0KE9QVF9U
V0VBSywgJnRva2VuX21hc2spKSkNCj4gKwkJCXBheWxvYWQtPmtleWlkX2N0cmwgfD0gTUtUTUVf
S0VZSURfU0VUX0tFWV9ESVJFQ1Q7DQo+ICsJCWVsc2UNCj4gKwkJCXJldHVybiAtRUlOVkFMOw0K
PiArCQlicmVhazsNCj4gKw0KPiArCWNhc2UgTUtUTUVfVFlQRV9DUFU6DQo+ICsJCWlmICh0ZXN0
X2JpdChPUFRfQUxHT1JJVEhNLCAmdG9rZW5fbWFzaykpDQo+ICsJCQlwYXlsb2FkLT5rZXlpZF9j
dHJsIHw9IE1LVE1FX0FFU19YVFNfMTI4Ow0KPiArCQllbHNlDQo+ICsJCQlyZXR1cm4gLUVJTlZB
TDsNCj4gKw0KPiArCQlwYXlsb2FkLT5rZXlpZF9jdHJsIHw9IE1LVE1FX0tFWUlEX1NFVF9LRVlf
UkFORE9NOw0KPiArCQlicmVhazsNCj4gKw0KPiArCWNhc2UgTUtUTUVfVFlQRV9DTEVBUjoNCj4g
KwkJcGF5bG9hZC0+a2V5aWRfY3RybCB8PSBNS1RNRV9LRVlJRF9DTEVBUl9LRVk7DQo+ICsJCWJy
ZWFrOw0KPiArDQo+ICsJY2FzZSBNS1RNRV9UWVBFX05PX0VOQ1JZUFQ6DQo+ICsJCXBheWxvYWQt
PmtleWlkX2N0cmwgfD0gTUtUTUVfS0VZSURfTk9fRU5DUllQVDsNCj4gKwkJYnJlYWs7DQo+ICsN
Cj4gKwlkZWZhdWx0Og0KPiArCQlyZXR1cm4gLUVJTlZBTDsNCj4gKwl9DQo+ICsJcmV0dXJuIDA7
DQo+ICt9DQo+ICsNCj4gKy8qIFBhcnNlIHRoZSBvcHRpb25zIGFuZCBzdG9yZSB0aGUga2V5IHBy
b2dyYW1taW5nIGRhdGEgaW4gdGhlIHBheWxvYWQuICovDQo+ICtzdGF0aWMgaW50IG1rdG1lX2dl
dF9vcHRpb25zKGNoYXIgKm9wdGlvbnMsIHN0cnVjdCBta3RtZV9wYXlsb2FkICpwYXlsb2FkKQ0K
PiArew0KPiArCWVudW0gbWt0bWVfdHlwZSB0eXBlID0gTUtUTUVfVFlQRV9FUlJPUjsNCj4gKwlz
dWJzdHJpbmdfdCBhcmdzW01BWF9PUFRfQVJHU107DQo+ICsJdW5zaWduZWQgbG9uZyB0b2tlbl9t
YXNrID0gMDsNCj4gKwljaGFyICpwID0gb3B0aW9uczsNCj4gKwlpbnQgcmV0LCB0b2tlbjsNCj4g
Kw0KPiArCXdoaWxlICgocCA9IHN0cnNlcCgmb3B0aW9ucywgIiBcdCIpKSkgew0KPiArCQlpZiAo
KnAgPT0gJ1wwJyB8fCAqcCA9PSAnICcgfHwgKnAgPT0gJ1x0JykNCj4gKwkJCWNvbnRpbnVlOw0K
PiArCQl0b2tlbiA9IG1hdGNoX3Rva2VuKHAsIG1rdG1lX3Rva2VuLCBhcmdzKTsNCj4gKwkJaWYg
KHRlc3RfYW5kX3NldF9iaXQodG9rZW4sICZ0b2tlbl9tYXNrKSkNCj4gKwkJCXJldHVybiAtRUlO
VkFMOw0KPiArDQo+ICsJCXN3aXRjaCAodG9rZW4pIHsNCj4gKwkJY2FzZSBPUFRfS0VZOg0KPiAr
CQkJcmV0ID0gaGV4MmJpbihwYXlsb2FkLT5kYXRhX2tleSwgYXJnc1swXS5mcm9tLA0KPiArCQkJ
CSAgICAgIE1LVE1FX0FFU19YVFNfU0laRSk7DQo+ICsJCQlpZiAocmV0IDwgMCkNCj4gKwkJCQly
ZXR1cm4gLUVJTlZBTDsNCj4gKwkJCWJyZWFrOw0KPiArDQo+ICsJCWNhc2UgT1BUX1RXRUFLOg0K
PiArCQkJcmV0ID0gaGV4MmJpbihwYXlsb2FkLT50d2Vha19rZXksIGFyZ3NbMF0uZnJvbSwNCj4g
KwkJCQkgICAgICBNS1RNRV9BRVNfWFRTX1NJWkUpOw0KPiArCQkJaWYgKHJldCA8IDApDQo+ICsJ
CQkJcmV0dXJuIC1FSU5WQUw7DQo+ICsJCQlicmVhazsNCj4gKw0KPiArCQljYXNlIE9QVF9UWVBF
Og0KPiArCQkJdHlwZSA9IG1hdGNoX3N0cmluZyhta3RtZV90eXBlX25hbWVzLA0KPiArCQkJCQkg
ICAgQVJSQVlfU0laRShta3RtZV90eXBlX25hbWVzKSwNCj4gKwkJCQkJICAgIGFyZ3NbMF0uZnJv
bSk7DQo+ICsJCQlpZiAodHlwZSA8IDApDQo+ICsJCQkJcmV0dXJuIC1FSU5WQUw7DQo+ICsJCQli
cmVhazsNCj4gKw0KPiArCQljYXNlIE9QVF9BTEdPUklUSE06DQo+ICsJCQlyZXQgPSBtYXRjaF9z
dHJpbmcobWt0bWVfYWxnX25hbWVzLA0KPiArCQkJCQkgICBBUlJBWV9TSVpFKG1rdG1lX2FsZ19u
YW1lcyksDQo+ICsJCQkJCSAgIGFyZ3NbMF0uZnJvbSk7DQo+ICsJCQlpZiAocmV0IDwgMCkNCj4g
KwkJCQlyZXR1cm4gLUVJTlZBTDsNCj4gKwkJCWJyZWFrOw0KPiArDQo+ICsJCWRlZmF1bHQ6DQo+
ICsJCQlyZXR1cm4gLUVJTlZBTDsNCj4gKwkJfQ0KPiArCX0NCj4gKwlyZXR1cm4gbWt0bWVfY2hl
Y2tfb3B0aW9ucyhwYXlsb2FkLCB0b2tlbl9tYXNrLCB0eXBlKTsNCj4gK30NCj4gKw0KPiArdm9p
ZCBta3RtZV9mcmVlX3ByZXBhcnNlZF9rZXkoc3RydWN0IGtleV9wcmVwYXJzZWRfcGF5bG9hZCAq
cHJlcCkNCj4gK3sNCj4gKwlremZyZWUocHJlcC0+cGF5bG9hZC5kYXRhWzBdKTsNCj4gK30NCj4g
Kw0KPiArLyoNCj4gKyAqIEtleSBTZXJ2aWNlIE1ldGhvZCB0byBwcmVwYXJzZSBhIHBheWxvYWQg
YmVmb3JlIGEga2V5IGlzIGNyZWF0ZWQuDQo+ICsgKiBDaGVjayBwZXJtaXNzaW9ucyBhbmQgdGhl
IG9wdGlvbnMuIExvYWQgdGhlIHByb3Bvc2VkIGtleSBmaWVsZA0KPiArICogZGF0YSBpbnRvIHRo
ZSBwYXlsb2FkIGZvciB1c2UgYnkgaW5zdGFudGlhdGUgYW5kIHVwZGF0ZSBtZXRob2RzLg0KPiAr
ICovDQo+ICtpbnQgbWt0bWVfcHJlcGFyc2Vfa2V5KHN0cnVjdCBrZXlfcHJlcGFyc2VkX3BheWxv
YWQgKnByZXApDQo+ICt7DQo+ICsJc3RydWN0IG1rdG1lX3BheWxvYWQgKm1rdG1lX3BheWxvYWQ7
DQo+ICsJc2l6ZV90IGRhdGFsZW4gPSBwcmVwLT5kYXRhbGVuOw0KPiArCWNoYXIgKm9wdGlvbnM7
DQo+ICsJaW50IHJldDsNCj4gKw0KPiArCWlmICghY2FwYWJsZShDQVBfU1lTX1JFU09VUkNFKSAm
JiAhY2FwYWJsZShDQVBfU1lTX0FETUlOKSkNCj4gKwkJcmV0dXJuIC1FQUNDRVM7DQo+ICsNCj4g
KwlpZiAoZGF0YWxlbiA8PSAwIHx8IGRhdGFsZW4gPiAxMDI0IHx8ICFwcmVwLT5kYXRhKQ0KPiAr
CQlyZXR1cm4gLUVJTlZBTDsNCj4gKw0KPiArCW9wdGlvbnMgPSBrbWVtZHVwKHByZXAtPmRhdGEs
IGRhdGFsZW4gKyAxLCBHRlBfS0VSTkVMKTsNCj4gKwlpZiAoIW9wdGlvbnMpDQo+ICsJCXJldHVy
biAtRU5PTUVNOw0KPiArDQo+ICsJb3B0aW9uc1tkYXRhbGVuXSA9ICdcMCc7DQo+ICsNCj4gKwlt
a3RtZV9wYXlsb2FkID0ga3phbGxvYyhzaXplb2YoKm1rdG1lX3BheWxvYWQpLCBHRlBfS0VSTkVM
KTsNCj4gKwlpZiAoIW1rdG1lX3BheWxvYWQpIHsNCj4gKwkJcmV0ID0gLUVOT01FTTsNCj4gKwkJ
Z290byBvdXQ7DQo+ICsJfQ0KPiArCXJldCA9IG1rdG1lX2dldF9vcHRpb25zKG9wdGlvbnMsIG1r
dG1lX3BheWxvYWQpOw0KPiArCWlmIChyZXQgPCAwKQ0KPiArCQlnb3RvIG91dDsNCj4gKw0KPiAr
CXByZXAtPnF1b3RhbGVuID0gc2l6ZW9mKG1rdG1lX3BheWxvYWQpOw0KPiArCXByZXAtPnBheWxv
YWQuZGF0YVswXSA9IG1rdG1lX3BheWxvYWQ7DQo+ICtvdXQ6DQo+ICsJa3pmcmVlKG9wdGlvbnMp
Ow0KPiArCXJldHVybiByZXQ7DQo+ICt9DQo+ICsNCj4gK3N0cnVjdCBrZXlfdHlwZSBrZXlfdHlw
ZV9ta3RtZSA9IHsNCj4gKwkubmFtZQkJPSAibWt0bWUiLA0KPiArCS5wcmVwYXJzZQk9IG1rdG1l
X3ByZXBhcnNlX2tleSwNCj4gKwkuZnJlZV9wcmVwYXJzZQk9IG1rdG1lX2ZyZWVfcHJlcGFyc2Vk
X2tleSwNCj4gKwkuaW5zdGFudGlhdGUJPSBta3RtZV9pbnN0YW50aWF0ZV9rZXksDQo+ICsJLnVw
ZGF0ZQkJPSBta3RtZV91cGRhdGVfa2V5LA0KPiArCS5kZXNjcmliZQk9IHVzZXJfZGVzY3JpYmUs
DQo+ICsJLmRlc3Ryb3kJPSBta3RtZV9kZXN0cm95X2tleSwNCj4gK307DQo+ICsNCj4gKy8qDQo+
ICsgKiBBbGxvY2F0ZSB0aGUgZ2xvYmFsIG1rdG1lX21hcCBzdHJ1Y3R1cmUgYmFzZWQgb24gdGhl
IGF2YWlsYWJsZSBrZXlpZHMuDQo+ICsgKiBDcmVhdGUgYSBjYWNoZSBmb3IgdGhlIGhhcmR3YXJl
IHN0cnVjdHVyZS4gSW5pdGlhbGl6ZSB0aGUgZW5jcnlwdF9jb3VudA0KPiArICogYXJyYXkgdG8g
dHJhY2sgKiBWTUEncyBwZXIga2V5aWQuIE9uY2UgYWxsIHRoYXQgc3VjY2VlZHMsIHJlZ2lzdGVy
IHRoZQ0KPiArICogJ21rdG1lJyBrZXkgdHlwZS4NCj4gKyAqLw0KPiArc3RhdGljIGludCBfX2lu
aXQgaW5pdF9ta3RtZSh2b2lkKQ0KPiArew0KPiArCWludCByZXQ7DQo+ICsNCj4gKwkvKiBWZXJp
Znkga2V5cyBhcmUgcHJlc2VudCAqLw0KPiArCWlmICghKG1rdG1lX25yX2tleWlkcyA+IDApKQ0K
PiArCQlyZXR1cm4gLUVJTlZBTDsNCj4gKw0KPiArCWlmICghbWt0bWVfbWFwX2FsbG9jKCkpDQo+
ICsJCXJldHVybiAtRU5PTUVNOw0KPiArDQo+ICsJbWt0bWVfcHJvZ19jYWNoZSA9IEtNRU1fQ0FD
SEUobWt0bWVfa2V5X3Byb2dyYW0sIFNMQUJfUEFOSUMpOw0KPiArCWlmICghbWt0bWVfcHJvZ19j
YWNoZSkNCj4gKwkJZ290byBmcmVlX21hcDsNCj4gKw0KPiArCWlmIChta3RtZV9hbGxvY19lbmNy
eXB0X2FycmF5KCkgPCAwKQ0KPiArCQlnb3RvIGZyZWVfY2FjaGU7DQo+ICsNCj4gKwlyZXQgPSBy
ZWdpc3Rlcl9rZXlfdHlwZSgma2V5X3R5cGVfbWt0bWUpOw0KPiArCWlmICghcmV0KQ0KPiArCQly
ZXR1cm4gcmV0OwkJCS8qIFNVQ0NFU1MgKi8NCj4gKw0KPiArCW1rdG1lX2ZyZWVfZW5jcnlwdF9h
cnJheSgpOw0KPiArZnJlZV9jYWNoZToNCj4gKwlrbWVtX2NhY2hlX2Rlc3Ryb3kobWt0bWVfcHJv
Z19jYWNoZSk7DQo+ICtmcmVlX21hcDoNCg0KTWF5YmUgZXJyX2ZyZWVfY2FjaGUgYW5kIGVycl9t
YXAgKG1vcmUgdGhhbiBjb3NtZXRpYyk/DQoNCj4gKwlta3RtZV9tYXBfZnJlZSgpOw0KPiArDQo+
ICsJcmV0dXJuIC1FTk9NRU07DQo+ICt9DQo+ICsNCj4gK2xhdGVfaW5pdGNhbGwoaW5pdF9ta3Rt
ZSk7DQoNCkFzIGZvciBjb2RlIGNoYW5nZS4gT3ZlcmFsbHksIGl0IGxvb2tzIGdvb2QhIEJ1dCBi
ZWZvcmUgdXNpbmcgdGltZQ0KZm9yIGRldGFpbGVkIHJldmlldyBvciB0ZXN0aW5nIChvbmNlIEkg
Z2V0IGEgY2hhbmNlIHRvIGFjcXVpcmUNCnNvbWV0aGluZyB0byB0ZXN0IGl0KSBsZXRzIGdvIHRo
cm91Z2ggdGhlIGRvY3VtZW50YXRpb24gZGlzY3Vzc2lvbi4NCkNsZWFybHkgdGhlcmUgaXMgYnVu
Y2ggb2Ygc3R1ZmYgdGhhdCBjYW4gYmUgY3V0Li4uDQoNCi9KYXJra28NCg=

Re: [RFC v2 10/13] keys/mktme: Add the MKTME Key Service type for memory encryption

[Sakkinen, Jarkko]

On Mon, 2018-12-03 at 23:39 -0800, Alison Schofield wrote:
> MKTME (Multi-Key Total Memory Encryption) is a technology that allows
> transparent memory encryption in upcoming Intel platforms. MKTME will
> support mulitple encryption domains, each having their own key. The main
> use case for the feature is virtual machine isolation. The API needs the
> flexibility to work for a wide range of uses.

Some, maybe brutal, honesty (apologies)...

Have never really got the grip why either SME or TME would make
isolation any better. If you can break into hypervisor, you'll
have these tools availabe:

1. Read page (in encrypted form).
2. Write page (for example replay as pages are not versioned).

with all the side-channel possibilities of course since you can
control the VMs (in which core they execute etc.).

I've seen now SME presentation three times and it always leaves
me an empty feeling. This feels the same same.

> The MKTME key service type manages the addition and removal of the memory
> encryption keys. It maps Userspace Keys to hardware KeyIDs. It programs
> the hardware with the user requested encryption options.
> 
> The only supported encryption algorithm is AES-XTS 128.
> 
> The MKTME key service is half of the MKTME API level solution. It pairs
> with a new memory encryption system call: encrypt_mprotect() that uses
> the keys to encrypt memory.
> 
> See Documentation/x86/mktme/mktme.rst
> 
> Change-Id: Idda4af2beabb739c77719897affff183ee9fa716
> Signed-off-by: Alison Schofield <alison.schofield@intel.com>
> Signed-off-by: Kirill A. Shutemov <kirill.shutemov@linux.intel.com>
> ---
>  arch/x86/Kconfig           |   1 +
>  include/keys/mktme-type.h  |  41 ++++++
>  security/keys/Kconfig      |  11 ++
>  security/keys/Makefile     |   1 +
>  security/keys/mktme_keys.c | 339
> +++++++++++++++++++++++++++++++++++++++++++++
>  5 files changed, 393 insertions(+)
>  create mode 100644 include/keys/mktme-type.h
>  create mode 100644 security/keys/mktme_keys.c
> 
> diff --git a/arch/x86/Kconfig b/arch/x86/Kconfig
> index 7ac78e2856c7..c2e3bb5af077 100644
> --- a/arch/x86/Kconfig
> +++ b/arch/x86/Kconfig
> @@ -1531,6 +1531,7 @@ config X86_INTEL_MKTME
>  	bool "Intel Multi-Key Total Memory Encryption"
>  	select DYNAMIC_PHYSICAL_MASK
>  	select PAGE_EXTENSION
> +	select MKTME_KEYS
>  	depends on X86_64 && CPU_SUP_INTEL
>  	---help---
>  	  Say yes to enable support for Multi-Key Total Memory Encryption.
> diff --git a/include/keys/mktme-type.h b/include/keys/mktme-type.h
> new file mode 100644
> index 000000000000..c63c6568087f
> --- /dev/null
> +++ b/include/keys/mktme-type.h
> @@ -0,0 +1,41 @@
> +/* SPDX-License-Identifier: GPL-2.0 */
> +
> +/* Key service for Multi-KEY Total Memory Encryption */
> +
> +#ifndef _KEYS_MKTME_TYPE_H
> +#define _KEYS_MKTME_TYPE_H
> +
> +#include <linux/key.h>
> +
> +/*
> + * The AES-XTS 128 encryption algorithm requires 128 bits for each
> + * user supplied data key and tweak key.
> + */
> +#define MKTME_AES_XTS_SIZE	16	/* 16 bytes, 128 bits */
> +
> +enum mktme_alg {
> +	MKTME_ALG_AES_XTS_128,
> +};
> +
> +const char *const mktme_alg_names[] = {
> +	[MKTME_ALG_AES_XTS_128]	= "aes-xts-128",
> +};
> +
> +enum mktme_type {
> +	MKTME_TYPE_ERROR = -1,
> +	MKTME_TYPE_USER,
> +	MKTME_TYPE_CPU,
> +	MKTME_TYPE_CLEAR,
> +	MKTME_TYPE_NO_ENCRYPT,
> +};
> +
> +const char *const mktme_type_names[] = {
> +	[MKTME_TYPE_USER]	= "user",
> +	[MKTME_TYPE_CPU]	= "cpu",
> +	[MKTME_TYPE_CLEAR]	= "clear",
> +	[MKTME_TYPE_NO_ENCRYPT]	= "no-encrypt",
> +};
> +
> +extern struct key_type key_type_mktme;
> +
> +#endif /* _KEYS_MKTME_TYPE_H */
> diff --git a/security/keys/Kconfig b/security/keys/Kconfig
> index 6462e6654ccf..c36972113e67 100644
> --- a/security/keys/Kconfig
> +++ b/security/keys/Kconfig
> @@ -101,3 +101,14 @@ config KEY_DH_OPERATIONS
>  	 in the kernel.
>  
>  	 If you are unsure as to whether this is required, answer N.
> +
> +config MKTME_KEYS
> +	bool "Multi-Key Total Memory Encryption Keys"
> +	depends on KEYS && X86_INTEL_MKTME
> +	help
> +	  This option provides support for Multi-Key Total Memory
> +	  Encryption (MKTME) on Intel platforms offering the feature.
> +	  MKTME allows userspace to manage the hardware encryption
> +	  keys through the kernel key services.
> +
> +	  If you are unsure as to whether this is required, answer N.
> diff --git a/security/keys/Makefile b/security/keys/Makefile
> index 9cef54064f60..94c84f10a857 100644
> --- a/security/keys/Makefile
> +++ b/security/keys/Makefile
> @@ -30,3 +30,4 @@ obj-$(CONFIG_ASYMMETRIC_KEY_TYPE) += keyctl_pkey.o
>  obj-$(CONFIG_BIG_KEYS) += big_key.o
>  obj-$(CONFIG_TRUSTED_KEYS) += trusted.o
>  obj-$(CONFIG_ENCRYPTED_KEYS) += encrypted-keys/
> +obj-$(CONFIG_MKTME_KEYS) += mktme_keys.o
> diff --git a/security/keys/mktme_keys.c b/security/keys/mktme_keys.c
> new file mode 100644
> index 000000000000..e615eb58e600
> --- /dev/null
> +++ b/security/keys/mktme_keys.c
> @@ -0,0 +1,339 @@
> +// SPDX-License-Identifier: GPL-3.0
> +
> +/* Documentation/x86/mktme/mktme_keys.rst */
> +
> +#include <linux/cred.h>
> +#include <linux/cpu.h>
> +#include <linux/err.h>
> +#include <linux/init.h>
> +#include <linux/key.h>
> +#include <linux/key-type.h>
> +#include <linux/init.h>
> +#include <linux/parser.h>
> +#include <linux/random.h>
> +#include <linux/slab.h>
> +#include <linux/string.h>
> +#include <asm/intel_pconfig.h>
> +#include <asm/mktme.h>
> +#include <keys/mktme-type.h>
> +#include <keys/user-type.h>
> +
> +#include "internal.h"
> +
> +struct kmem_cache *mktme_prog_cache;	/* Hardware programming cache */
> +
> +static const char * const mktme_program_err[] = {
> +	"KeyID was successfully programmed",	/* 0 */
> +	"Invalid KeyID programming command",	/* 1 */
> +	"Insufficient entropy",			/* 2 */
> +	"KeyID not valid",			/* 3 */
> +	"Invalid encryption algorithm chosen",	/* 4 */
> +	"Failure to access key table",		/* 5 */
> +};
> +
> +enum mktme_opt_id {
> +	OPT_ERROR = -1,
> +	OPT_TYPE,
> +	OPT_KEY,
> +	OPT_TWEAK,
> +	OPT_ALGORITHM,
> +};
> +
> +static const match_table_t mktme_token = {
> +	{OPT_TYPE, "type=%s"},
> +	{OPT_KEY, "key=%s"},
> +	{OPT_TWEAK, "tweak=%s"},
> +	{OPT_ALGORITHM, "algorithm=%s"},
> +	{OPT_ERROR, NULL}
> +};
> +
> +struct mktme_payload {
> +	u32		keyid_ctrl;	/* Command & Encryption Algorithm */
> +	u8		data_key[MKTME_AES_XTS_SIZE];
> +	u8		tweak_key[MKTME_AES_XTS_SIZE];
> +};
> +
> +/* Key Service Method called when Key is garbage collected. */
> +static void mktme_destroy_key(struct key *key)
> +{
> +	key_put_encrypt_ref(mktme_map_keyid_from_key(key));
> +}
> +
> +/* Copy the payload to the HW programming structure and program this KeyID */
> +static int mktme_program_keyid(int keyid, struct mktme_payload *payload)
> +{
> +	struct mktme_key_program *kprog = NULL;
> +	u8 kern_entropy[MKTME_AES_XTS_SIZE];
> +	int i, ret;
> +
> +	kprog = kmem_cache_zalloc(mktme_prog_cache, GFP_KERNEL);
> +	if (!kprog)
> +		return -ENOMEM;
> +
> +	/* Hardware programming requires cached aligned struct */
> +	kprog->keyid = keyid;
> +	kprog->keyid_ctrl = payload->keyid_ctrl;
> +	memcpy(kprog->key_field_1, payload->data_key, MKTME_AES_XTS_SIZE);
> +	memcpy(kprog->key_field_2, payload->tweak_key, MKTME_AES_XTS_SIZE);
> +
> +	/* Strengthen the entropy fields for CPU generated keys */
> +	if ((payload->keyid_ctrl & 0xff) == MKTME_KEYID_SET_KEY_RANDOM) {
> +		get_random_bytes(&kern_entropy, sizeof(kern_entropy));
> +		for (i = 0; i < (MKTME_AES_XTS_SIZE); i++) {
> +			kprog->key_field_1[i] ^= kern_entropy[i];
> +			kprog->key_field_2[i] ^= kern_entropy[i];
> +		}
> +	}
> +	ret = mktme_key_program(kprog);
> +	kmem_cache_free(mktme_prog_cache, kprog);
> +	return ret;
> +}
> +
> +/* Key Service Method to update an existing key. */
> +static int mktme_update_key(struct key *key,
> +			    struct key_preparsed_payload *prep)
> +{
> +	struct mktme_payload *payload = prep->payload.data[0];
> +	int keyid, ref_count;
> +	int ret;
> +
> +	mktme_map_lock();
> +	keyid = mktme_map_keyid_from_key(key);
> +	if (keyid <= 0)
> +		return -EINVAL;
> +	/*
> +	 * ref_count will be at least one when we get here because the
> +	 * key already exists. If ref_count is not > 1, it is safe to
> +	 * update the key while holding the mktme_map_lock.
> +	 */
> +	ref_count = mktme_read_encrypt_ref(keyid);
> +	if (ref_count > 1) {
> +		pr_debug("mktme not updating keyid[%d] encrypt_count[%d]\n",
> +			 keyid, ref_count);
> +		return -EBUSY;
> +	}
> +	ret = mktme_program_keyid(keyid, payload);
> +	if (ret != MKTME_PROG_SUCCESS) {
> +		pr_debug("%s: %s\n", __func__, mktme_program_err[ret]);
> +		ret = -ENOKEY;
> +	}
> +	mktme_map_unlock();
> +	return ret;
> +}
> +
> +/* Key Service Method to create a new key. Payload is preparsed. */
> +int mktme_instantiate_key(struct key *key, struct key_preparsed_payload
> *prep)
> +{
> +	struct mktme_payload *payload = prep->payload.data[0];
> +	int keyid, ret;
> +
> +	mktme_map_lock();
> +	keyid = mktme_map_get_free_keyid();
> +	if (keyid == 0) {
> +		ret = -ENOKEY;
> +		goto out;
> +	}
> +	ret = mktme_program_keyid(keyid, payload);
> +	if (ret != MKTME_PROG_SUCCESS) {
> +		pr_debug("%s: %s\n", __func__, mktme_program_err[ret]);
> +		ret = -ENOKEY;
> +		goto out;
> +	}
> +	mktme_map_set_keyid(keyid, key);
> +	key_get_encrypt_ref(keyid);
> +out:
> +	mktme_map_unlock();
> +	return ret;
> +}
> +
> +/* Verify the user provided the needed arguments for the TYPE of Key */
> +static int mktme_check_options(struct mktme_payload *payload,
> +			       unsigned long token_mask, enum mktme_type type)
> +{
> +	if (!token_mask)
> +		return -EINVAL;
> +
> +	switch (type) {
> +	case MKTME_TYPE_USER:
> +		if (test_bit(OPT_ALGORITHM, &token_mask))
> +			payload->keyid_ctrl |= MKTME_AES_XTS_128;
> +		else
> +			return -EINVAL;
> +
> +		if ((test_bit(OPT_KEY, &token_mask)) &&
> +		    (test_bit(OPT_TWEAK, &token_mask)))
> +			payload->keyid_ctrl |= MKTME_KEYID_SET_KEY_DIRECT;
> +		else
> +			return -EINVAL;
> +		break;
> +
> +	case MKTME_TYPE_CPU:
> +		if (test_bit(OPT_ALGORITHM, &token_mask))
> +			payload->keyid_ctrl |= MKTME_AES_XTS_128;
> +		else
> +			return -EINVAL;
> +
> +		payload->keyid_ctrl |= MKTME_KEYID_SET_KEY_RANDOM;
> +		break;
> +
> +	case MKTME_TYPE_CLEAR:
> +		payload->keyid_ctrl |= MKTME_KEYID_CLEAR_KEY;
> +		break;
> +
> +	case MKTME_TYPE_NO_ENCRYPT:
> +		payload->keyid_ctrl |= MKTME_KEYID_NO_ENCRYPT;
> +		break;
> +
> +	default:
> +		return -EINVAL;
> +	}
> +	return 0;
> +}
> +
> +/* Parse the options and store the key programming data in the payload. */
> +static int mktme_get_options(char *options, struct mktme_payload *payload)
> +{
> +	enum mktme_type type = MKTME_TYPE_ERROR;
> +	substring_t args[MAX_OPT_ARGS];
> +	unsigned long token_mask = 0;
> +	char *p = options;
> +	int ret, token;
> +
> +	while ((p = strsep(&options, " \t"))) {
> +		if (*p == '\0' || *p == ' ' || *p == '\t')
> +			continue;
> +		token = match_token(p, mktme_token, args);
> +		if (test_and_set_bit(token, &token_mask))
> +			return -EINVAL;
> +
> +		switch (token) {
> +		case OPT_KEY:
> +			ret = hex2bin(payload->data_key, args[0].from,
> +				      MKTME_AES_XTS_SIZE);
> +			if (ret < 0)
> +				return -EINVAL;
> +			break;
> +
> +		case OPT_TWEAK:
> +			ret = hex2bin(payload->tweak_key, args[0].from,
> +				      MKTME_AES_XTS_SIZE);
> +			if (ret < 0)
> +				return -EINVAL;
> +			break;
> +
> +		case OPT_TYPE:
> +			type = match_string(mktme_type_names,
> +					    ARRAY_SIZE(mktme_type_names),
> +					    args[0].from);
> +			if (type < 0)
> +				return -EINVAL;
> +			break;
> +
> +		case OPT_ALGORITHM:
> +			ret = match_string(mktme_alg_names,
> +					   ARRAY_SIZE(mktme_alg_names),
> +					   args[0].from);
> +			if (ret < 0)
> +				return -EINVAL;
> +			break;
> +
> +		default:
> +			return -EINVAL;
> +		}
> +	}
> +	return mktme_check_options(payload, token_mask, type);
> +}
> +
> +void mktme_free_preparsed_key(struct key_preparsed_payload *prep)
> +{
> +	kzfree(prep->payload.data[0]);
> +}
> +
> +/*
> + * Key Service Method to preparse a payload before a key is created.
> + * Check permissions and the options. Load the proposed key field
> + * data into the payload for use by instantiate and update methods.
> + */
> +int mktme_preparse_key(struct key_preparsed_payload *prep)
> +{
> +	struct mktme_payload *mktme_payload;
> +	size_t datalen = prep->datalen;
> +	char *options;
> +	int ret;
> +
> +	if (!capable(CAP_SYS_RESOURCE) && !capable(CAP_SYS_ADMIN))
> +		return -EACCES;
> +
> +	if (datalen <= 0 || datalen > 1024 || !prep->data)
> +		return -EINVAL;
> +
> +	options = kmemdup(prep->data, datalen + 1, GFP_KERNEL);
> +	if (!options)
> +		return -ENOMEM;
> +
> +	options[datalen] = '\0';
> +
> +	mktme_payload = kzalloc(sizeof(*mktme_payload), GFP_KERNEL);
> +	if (!mktme_payload) {
> +		ret = -ENOMEM;
> +		goto out;
> +	}
> +	ret = mktme_get_options(options, mktme_payload);
> +	if (ret < 0)
> +		goto out;
> +
> +	prep->quotalen = sizeof(mktme_payload);
> +	prep->payload.data[0] = mktme_payload;
> +out:
> +	kzfree(options);
> +	return ret;
> +}
> +
> +struct key_type key_type_mktme = {
> +	.name		= "mktme",
> +	.preparse	= mktme_preparse_key,
> +	.free_preparse	= mktme_free_preparsed_key,
> +	.instantiate	= mktme_instantiate_key,
> +	.update		= mktme_update_key,
> +	.describe	= user_describe,
> +	.destroy	= mktme_destroy_key,
> +};
> +
> +/*
> + * Allocate the global mktme_map structure based on the available keyids.
> + * Create a cache for the hardware structure. Initialize the encrypt_count
> + * array to track * VMA's per keyid. Once all that succeeds, register the
> + * 'mktme' key type.
> + */
> +static int __init init_mktme(void)
> +{
> +	int ret;
> +
> +	/* Verify keys are present */
> +	if (!(mktme_nr_keyids > 0))
> +		return -EINVAL;
> +
> +	if (!mktme_map_alloc())
> +		return -ENOMEM;
> +
> +	mktme_prog_cache = KMEM_CACHE(mktme_key_program, SLAB_PANIC);
> +	if (!mktme_prog_cache)
> +		goto free_map;
> +
> +	if (mktme_alloc_encrypt_array() < 0)
> +		goto free_cache;
> +
> +	ret = register_key_type(&key_type_mktme);
> +	if (!ret)
> +		return ret;			/* SUCCESS */
> +
> +	mktme_free_encrypt_array();
> +free_cache:
> +	kmem_cache_destroy(mktme_prog_cache);
> +free_map:

Maybe err_free_cache and err_map (more than cosmetic)?

> +	mktme_map_free();
> +
> +	return -ENOMEM;
> +}
> +
> +late_initcall(init_mktme);

As for code change. Overally, it looks good! But before using time
for detailed review or testing (once I get a chance to acquire
something to test it) lets go through the documentation discussion.
Clearly there is bunch of stuff that can be cut...

/Jarkko

Re: [RFC v2 10/13] keys/mktme: Add the MKTME Key Service type for memory encryption

[Sakkinen, Jarkko]

On Mon, 2018-12-03 at 23:39 -0800, Alison Schofield wrote:
> MKTME (Multi-Key Total Memory Encryption) is a technology that allows
> transparent memory encryption in upcoming Intel platforms. MKTME will
> support mulitple encryption domains, each having their own key. The main
> use case for the feature is virtual machine isolation. The API needs the
> flexibility to work for a wide range of uses.

Some, maybe brutal, honesty (apologies)...

Have never really got the grip why either SME or TME would make
isolation any better. If you can break into hypervisor, you'll
have these tools availabe:

1. Read page (in encrypted form).
2. Write page (for example replay as pages are not versioned).

with all the side-channel possibilities of course since you can
control the VMs (in which core they execute etc.).

I've seen now SME presentation three times and it always leaves
me an empty feeling. This feels the same same.

> The MKTME key service type manages the addition and removal of the memory
> encryption keys. It maps Userspace Keys to hardware KeyIDs. It programs
> the hardware with the user requested encryption options.
> 
> The only supported encryption algorithm is AES-XTS 128.
> 
> The MKTME key service is half of the MKTME API level solution. It pairs
> with a new memory encryption system call: encrypt_mprotect() that uses
> the keys to encrypt memory.
> 
> See Documentation/x86/mktme/mktme.rst
> 
> Change-Id: Idda4af2beabb739c77719897affff183ee9fa716
> Signed-off-by: Alison Schofield <alison.schofield@intel.com>
> Signed-off-by: Kirill A. Shutemov <kirill.shutemov@linux.intel.com>
> ---
>  arch/x86/Kconfig           |   1 +
>  include/keys/mktme-type.h  |  41 ++++++
>  security/keys/Kconfig      |  11 ++
>  security/keys/Makefile     |   1 +
>  security/keys/mktme_keys.c | 339
> +++++++++++++++++++++++++++++++++++++++++++++
>  5 files changed, 393 insertions(+)
>  create mode 100644 include/keys/mktme-type.h
>  create mode 100644 security/keys/mktme_keys.c
> 
> diff --git a/arch/x86/Kconfig b/arch/x86/Kconfig
> index 7ac78e2856c7..c2e3bb5af077 100644
> --- a/arch/x86/Kconfig
> +++ b/arch/x86/Kconfig
> @@ -1531,6 +1531,7 @@ config X86_INTEL_MKTME
>  	bool "Intel Multi-Key Total Memory Encryption"
>  	select DYNAMIC_PHYSICAL_MASK
>  	select PAGE_EXTENSION
> +	select MKTME_KEYS
>  	depends on X86_64 && CPU_SUP_INTEL
>  	---help---
>  	  Say yes to enable support for Multi-Key Total Memory Encryption.
> diff --git a/include/keys/mktme-type.h b/include/keys/mktme-type.h
> new file mode 100644
> index 000000000000..c63c6568087f
> --- /dev/null
> +++ b/include/keys/mktme-type.h
> @@ -0,0 +1,41 @@
> +/* SPDX-License-Identifier: GPL-2.0 */
> +
> +/* Key service for Multi-KEY Total Memory Encryption */
> +
> +#ifndef _KEYS_MKTME_TYPE_H
> +#define _KEYS_MKTME_TYPE_H
> +
> +#include <linux/key.h>
> +
> +/*
> + * The AES-XTS 128 encryption algorithm requires 128 bits for each
> + * user supplied data key and tweak key.
> + */
> +#define MKTME_AES_XTS_SIZE	16	/* 16 bytes, 128 bits */
> +
> +enum mktme_alg {
> +	MKTME_ALG_AES_XTS_128,
> +};
> +
> +const char *const mktme_alg_names[] = {
> +	[MKTME_ALG_AES_XTS_128]	= "aes-xts-128",
> +};
> +
> +enum mktme_type {
> +	MKTME_TYPE_ERROR = -1,
> +	MKTME_TYPE_USER,
> +	MKTME_TYPE_CPU,
> +	MKTME_TYPE_CLEAR,
> +	MKTME_TYPE_NO_ENCRYPT,
> +};
> +
> +const char *const mktme_type_names[] = {
> +	[MKTME_TYPE_USER]	= "user",
> +	[MKTME_TYPE_CPU]	= "cpu",
> +	[MKTME_TYPE_CLEAR]	= "clear",
> +	[MKTME_TYPE_NO_ENCRYPT]	= "no-encrypt",
> +};
> +
> +extern struct key_type key_type_mktme;
> +
> +#endif /* _KEYS_MKTME_TYPE_H */
> diff --git a/security/keys/Kconfig b/security/keys/Kconfig
> index 6462e6654ccf..c36972113e67 100644
> --- a/security/keys/Kconfig
> +++ b/security/keys/Kconfig
> @@ -101,3 +101,14 @@ config KEY_DH_OPERATIONS
>  	 in the kernel.
>  
>  	 If you are unsure as to whether this is required, answer N.
> +
> +config MKTME_KEYS
> +	bool "Multi-Key Total Memory Encryption Keys"
> +	depends on KEYS && X86_INTEL_MKTME
> +	help
> +	  This option provides support for Multi-Key Total Memory
> +	  Encryption (MKTME) on Intel platforms offering the feature.
> +	  MKTME allows userspace to manage the hardware encryption
> +	  keys through the kernel key services.
> +
> +	  If you are unsure as to whether this is required, answer N.
> diff --git a/security/keys/Makefile b/security/keys/Makefile
> index 9cef54064f60..94c84f10a857 100644
> --- a/security/keys/Makefile
> +++ b/security/keys/Makefile
> @@ -30,3 +30,4 @@ obj-$(CONFIG_ASYMMETRIC_KEY_TYPE) += keyctl_pkey.o
>  obj-$(CONFIG_BIG_KEYS) += big_key.o
>  obj-$(CONFIG_TRUSTED_KEYS) += trusted.o
>  obj-$(CONFIG_ENCRYPTED_KEYS) += encrypted-keys/
> +obj-$(CONFIG_MKTME_KEYS) += mktme_keys.o
> diff --git a/security/keys/mktme_keys.c b/security/keys/mktme_keys.c
> new file mode 100644
> index 000000000000..e615eb58e600
> --- /dev/null
> +++ b/security/keys/mktme_keys.c
> @@ -0,0 +1,339 @@
> +// SPDX-License-Identifier: GPL-3.0
> +
> +/* Documentation/x86/mktme/mktme_keys.rst */
> +
> +#include <linux/cred.h>
> +#include <linux/cpu.h>
> +#include <linux/err.h>
> +#include <linux/init.h>
> +#include <linux/key.h>
> +#include <linux/key-type.h>
> +#include <linux/init.h>
> +#include <linux/parser.h>
> +#include <linux/random.h>
> +#include <linux/slab.h>
> +#include <linux/string.h>
> +#include <asm/intel_pconfig.h>
> +#include <asm/mktme.h>
> +#include <keys/mktme-type.h>
> +#include <keys/user-type.h>
> +
> +#include "internal.h"
> +
> +struct kmem_cache *mktme_prog_cache;	/* Hardware programming cache */
> +
> +static const char * const mktme_program_err[] = {
> +	"KeyID was successfully programmed",	/* 0 */
> +	"Invalid KeyID programming command",	/* 1 */
> +	"Insufficient entropy",			/* 2 */
> +	"KeyID not valid",			/* 3 */
> +	"Invalid encryption algorithm chosen",	/* 4 */
> +	"Failure to access key table",		/* 5 */
> +};
> +
> +enum mktme_opt_id {
> +	OPT_ERROR = -1,
> +	OPT_TYPE,
> +	OPT_KEY,
> +	OPT_TWEAK,
> +	OPT_ALGORITHM,
> +};
> +
> +static const match_table_t mktme_token = {
> +	{OPT_TYPE, "type=%s"},
> +	{OPT_KEY, "key=%s"},
> +	{OPT_TWEAK, "tweak=%s"},
> +	{OPT_ALGORITHM, "algorithm=%s"},
> +	{OPT_ERROR, NULL}
> +};
> +
> +struct mktme_payload {
> +	u32		keyid_ctrl;	/* Command & Encryption Algorithm */
> +	u8		data_key[MKTME_AES_XTS_SIZE];
> +	u8		tweak_key[MKTME_AES_XTS_SIZE];
> +};
> +
> +/* Key Service Method called when Key is garbage collected. */
> +static void mktme_destroy_key(struct key *key)
> +{
> +	key_put_encrypt_ref(mktme_map_keyid_from_key(key));
> +}
> +
> +/* Copy the payload to the HW programming structure and program this KeyID */
> +static int mktme_program_keyid(int keyid, struct mktme_payload *payload)
> +{
> +	struct mktme_key_program *kprog = NULL;
> +	u8 kern_entropy[MKTME_AES_XTS_SIZE];
> +	int i, ret;
> +
> +	kprog = kmem_cache_zalloc(mktme_prog_cache, GFP_KERNEL);
> +	if (!kprog)
> +		return -ENOMEM;
> +
> +	/* Hardware programming requires cached aligned struct */
> +	kprog->keyid = keyid;
> +	kprog->keyid_ctrl = payload->keyid_ctrl;
> +	memcpy(kprog->key_field_1, payload->data_key, MKTME_AES_XTS_SIZE);
> +	memcpy(kprog->key_field_2, payload->tweak_key, MKTME_AES_XTS_SIZE);
> +
> +	/* Strengthen the entropy fields for CPU generated keys */
> +	if ((payload->keyid_ctrl & 0xff) == MKTME_KEYID_SET_KEY_RANDOM) {
> +		get_random_bytes(&kern_entropy, sizeof(kern_entropy));
> +		for (i = 0; i < (MKTME_AES_XTS_SIZE); i++) {
> +			kprog->key_field_1[i] ^= kern_entropy[i];
> +			kprog->key_field_2[i] ^= kern_entropy[i];
> +		}
> +	}
> +	ret = mktme_key_program(kprog);
> +	kmem_cache_free(mktme_prog_cache, kprog);
> +	return ret;
> +}
> +
> +/* Key Service Method to update an existing key. */
> +static int mktme_update_key(struct key *key,
> +			    struct key_preparsed_payload *prep)
> +{
> +	struct mktme_payload *payload = prep->payload.data[0];
> +	int keyid, ref_count;
> +	int ret;
> +
> +	mktme_map_lock();
> +	keyid = mktme_map_keyid_from_key(key);
> +	if (keyid <= 0)
> +		return -EINVAL;
> +	/*
> +	 * ref_count will be at least one when we get here because the
> +	 * key already exists. If ref_count is not > 1, it is safe to
> +	 * update the key while holding the mktme_map_lock.
> +	 */
> +	ref_count = mktme_read_encrypt_ref(keyid);
> +	if (ref_count > 1) {
> +		pr_debug("mktme not updating keyid[%d] encrypt_count[%d]\n",
> +			 keyid, ref_count);
> +		return -EBUSY;
> +	}
> +	ret = mktme_program_keyid(keyid, payload);
> +	if (ret != MKTME_PROG_SUCCESS) {
> +		pr_debug("%s: %s\n", __func__, mktme_program_err[ret]);
> +		ret = -ENOKEY;
> +	}
> +	mktme_map_unlock();
> +	return ret;
> +}
> +
> +/* Key Service Method to create a new key. Payload is preparsed. */
> +int mktme_instantiate_key(struct key *key, struct key_preparsed_payload
> *prep)
> +{
> +	struct mktme_payload *payload = prep->payload.data[0];
> +	int keyid, ret;
> +
> +	mktme_map_lock();
> +	keyid = mktme_map_get_free_keyid();
> +	if (keyid == 0) {
> +		ret = -ENOKEY;
> +		goto out;
> +	}
> +	ret = mktme_program_keyid(keyid, payload);
> +	if (ret != MKTME_PROG_SUCCESS) {
> +		pr_debug("%s: %s\n", __func__, mktme_program_err[ret]);
> +		ret = -ENOKEY;
> +		goto out;
> +	}
> +	mktme_map_set_keyid(keyid, key);
> +	key_get_encrypt_ref(keyid);
> +out:
> +	mktme_map_unlock();
> +	return ret;
> +}
> +
> +/* Verify the user provided the needed arguments for the TYPE of Key */
> +static int mktme_check_options(struct mktme_payload *payload,
> +			       unsigned long token_mask, enum mktme_type type)
> +{
> +	if (!token_mask)
> +		return -EINVAL;
> +
> +	switch (type) {
> +	case MKTME_TYPE_USER:
> +		if (test_bit(OPT_ALGORITHM, &token_mask))
> +			payload->keyid_ctrl |= MKTME_AES_XTS_128;
> +		else
> +			return -EINVAL;
> +
> +		if ((test_bit(OPT_KEY, &token_mask)) &&
> +		    (test_bit(OPT_TWEAK, &token_mask)))
> +			payload->keyid_ctrl |= MKTME_KEYID_SET_KEY_DIRECT;
> +		else
> +			return -EINVAL;
> +		break;
> +
> +	case MKTME_TYPE_CPU:
> +		if (test_bit(OPT_ALGORITHM, &token_mask))
> +			payload->keyid_ctrl |= MKTME_AES_XTS_128;
> +		else
> +			return -EINVAL;
> +
> +		payload->keyid_ctrl |= MKTME_KEYID_SET_KEY_RANDOM;
> +		break;
> +
> +	case MKTME_TYPE_CLEAR:
> +		payload->keyid_ctrl |= MKTME_KEYID_CLEAR_KEY;
> +		break;
> +
> +	case MKTME_TYPE_NO_ENCRYPT:
> +		payload->keyid_ctrl |= MKTME_KEYID_NO_ENCRYPT;
> +		break;
> +
> +	default:
> +		return -EINVAL;
> +	}
> +	return 0;
> +}
> +
> +/* Parse the options and store the key programming data in the payload. */
> +static int mktme_get_options(char *options, struct mktme_payload *payload)
> +{
> +	enum mktme_type type = MKTME_TYPE_ERROR;
> +	substring_t args[MAX_OPT_ARGS];
> +	unsigned long token_mask = 0;
> +	char *p = options;
> +	int ret, token;
> +
> +	while ((p = strsep(&options, " \t"))) {
> +		if (*p == '\0' || *p == ' ' || *p == '\t')
> +			continue;
> +		token = match_token(p, mktme_token, args);
> +		if (test_and_set_bit(token, &token_mask))
> +			return -EINVAL;
> +
> +		switch (token) {
> +		case OPT_KEY:
> +			ret = hex2bin(payload->data_key, args[0].from,
> +				      MKTME_AES_XTS_SIZE);
> +			if (ret < 0)
> +				return -EINVAL;
> +			break;
> +
> +		case OPT_TWEAK:
> +			ret = hex2bin(payload->tweak_key, args[0].from,
> +				      MKTME_AES_XTS_SIZE);
> +			if (ret < 0)
> +				return -EINVAL;
> +			break;
> +
> +		case OPT_TYPE:
> +			type = match_string(mktme_type_names,
> +					    ARRAY_SIZE(mktme_type_names),
> +					    args[0].from);
> +			if (type < 0)
> +				return -EINVAL;
> +			break;
> +
> +		case OPT_ALGORITHM:
> +			ret = match_string(mktme_alg_names,
> +					   ARRAY_SIZE(mktme_alg_names),
> +					   args[0].from);
> +			if (ret < 0)
> +				return -EINVAL;
> +			break;
> +
> +		default:
> +			return -EINVAL;
> +		}
> +	}
> +	return mktme_check_options(payload, token_mask, type);
> +}
> +
> +void mktme_free_preparsed_key(struct key_preparsed_payload *prep)
> +{
> +	kzfree(prep->payload.data[0]);
> +}
> +
> +/*
> + * Key Service Method to preparse a payload before a key is created.
> + * Check permissions and the options. Load the proposed key field
> + * data into the payload for use by instantiate and update methods.
> + */
> +int mktme_preparse_key(struct key_preparsed_payload *prep)
> +{
> +	struct mktme_payload *mktme_payload;
> +	size_t datalen = prep->datalen;
> +	char *options;
> +	int ret;
> +
> +	if (!capable(CAP_SYS_RESOURCE) && !capable(CAP_SYS_ADMIN))
> +		return -EACCES;
> +
> +	if (datalen <= 0 || datalen > 1024 || !prep->data)
> +		return -EINVAL;
> +
> +	options = kmemdup(prep->data, datalen + 1, GFP_KERNEL);
> +	if (!options)
> +		return -ENOMEM;
> +
> +	options[datalen] = '\0';
> +
> +	mktme_payload = kzalloc(sizeof(*mktme_payload), GFP_KERNEL);
> +	if (!mktme_payload) {
> +		ret = -ENOMEM;
> +		goto out;
> +	}
> +	ret = mktme_get_options(options, mktme_payload);
> +	if (ret < 0)
> +		goto out;
> +
> +	prep->quotalen = sizeof(mktme_payload);
> +	prep->payload.data[0] = mktme_payload;
> +out:
> +	kzfree(options);
> +	return ret;
> +}
> +
> +struct key_type key_type_mktme = {
> +	.name		= "mktme",
> +	.preparse	= mktme_preparse_key,
> +	.free_preparse	= mktme_free_preparsed_key,
> +	.instantiate	= mktme_instantiate_key,
> +	.update		= mktme_update_key,
> +	.describe	= user_describe,
> +	.destroy	= mktme_destroy_key,
> +};
> +
> +/*
> + * Allocate the global mktme_map structure based on the available keyids.
> + * Create a cache for the hardware structure. Initialize the encrypt_count
> + * array to track * VMA's per keyid. Once all that succeeds, register the
> + * 'mktme' key type.
> + */
> +static int __init init_mktme(void)
> +{
> +	int ret;
> +
> +	/* Verify keys are present */
> +	if (!(mktme_nr_keyids > 0))
> +		return -EINVAL;
> +
> +	if (!mktme_map_alloc())
> +		return -ENOMEM;
> +
> +	mktme_prog_cache = KMEM_CACHE(mktme_key_program, SLAB_PANIC);
> +	if (!mktme_prog_cache)
> +		goto free_map;
> +
> +	if (mktme_alloc_encrypt_array() < 0)
> +		goto free_cache;
> +
> +	ret = register_key_type(&key_type_mktme);
> +	if (!ret)
> +		return ret;			/* SUCCESS */
> +
> +	mktme_free_encrypt_array();
> +free_cache:
> +	kmem_cache_destroy(mktme_prog_cache);
> +free_map:

Maybe err_free_cache and err_map (more than cosmetic)?

> +	mktme_map_free();
> +
> +	return -ENOMEM;
> +}
> +
> +late_initcall(init_mktme);

As for code change. Overally, it looks good! But before using time
for detailed review or testing (once I get a chance to acquire
something to test it) lets go through the documentation discussion.
Clearly there is bunch of stuff that can be cut...

/Jarkko

Re: [RFC v2 10/13] keys/mktme: Add the MKTME Key Service type for memory encryption

[Sakkinen, Jarkko]

T24gVGh1LCAyMDE4LTEyLTA2IGF0IDAwOjUxIC0wODAwLCBKYXJra28gU2Fra2luZW4gd3JvdGU6
DQo+IE9uIE1vbiwgMjAxOC0xMi0wMyBhdCAyMzozOSAtMDgwMCwgQWxpc29uIFNjaG9maWVsZCB3
cm90ZToNCj4gPiBNS1RNRSAoTXVsdGktS2V5IFRvdGFsIE1lbW9yeSBFbmNyeXB0aW9uKSBpcyBh
IHRlY2hub2xvZ3kgdGhhdCBhbGxvd3MNCj4gPiB0cmFuc3BhcmVudCBtZW1vcnkgZW5jcnlwdGlv
biBpbiB1cGNvbWluZyBJbnRlbCBwbGF0Zm9ybXMuIE1LVE1FIHdpbGwNCj4gPiBzdXBwb3J0IG11
bGl0cGxlIGVuY3J5cHRpb24gZG9tYWlucywgZWFjaCBoYXZpbmcgdGhlaXIgb3duIGtleS4gVGhl
IG1haW4NCj4gPiB1c2UgY2FzZSBmb3IgdGhlIGZlYXR1cmUgaXMgdmlydHVhbCBtYWNoaW5lIGlz
b2xhdGlvbi4gVGhlIEFQSSBuZWVkcyB0aGUNCj4gPiBmbGV4aWJpbGl0eSB0byB3b3JrIGZvciBh
IHdpZGUgcmFuZ2Ugb2YgdXNlcy4NCj4gDQo+IFNvbWUsIG1heWJlIGJydXRhbCwgaG9uZXN0eSAo
YXBvbG9naWVzKS4uLg0KPiANCj4gSGF2ZSBuZXZlciByZWFsbHkgZ290IHRoZSBncmlwIHdoeSBl
aXRoZXIgU01FIG9yIFRNRSB3b3VsZCBtYWtlDQo+IGlzb2xhdGlvbiBhbnkgYmV0dGVyLiBJZiB5
b3UgY2FuIGJyZWFrIGludG8gaHlwZXJ2aXNvciwgeW91J2xsDQo+IGhhdmUgdGhlc2UgdG9vbHMg
YXZhaWxhYmU6DQo+IA0KPiAxLiBSZWFkIHBhZ2UgKGluIGVuY3J5cHRlZCBmb3JtKS4NCj4gMi4g
V3JpdGUgcGFnZSAoZm9yIGV4YW1wbGUgcmVwbGF5IGFzIHBhZ2VzIGFyZSBub3QgdmVyc2lvbmVk
KS4NCj4gDQo+IHdpdGggYWxsIHRoZSBzaWRlLWNoYW5uZWwgcG9zc2liaWxpdGllcyBvZiBjb3Vy
c2Ugc2luY2UgeW91IGNhbg0KPiBjb250cm9sIHRoZSBWTXMgKGluIHdoaWNoIGNvcmUgdGhleSBl
eGVjdXRlIGV0Yy4pLg0KPiANCj4gSSd2ZSBzZWVuIG5vdyBTTUUgcHJlc2VudGF0aW9uIHRocmVl
IHRpbWVzIGFuZCBpdCBhbHdheXMgbGVhdmVzDQo+IG1lIGFuIGVtcHR5IGZlZWxpbmcuIFRoaXMg
ZmVlbHMgdGhlIHNhbWUgc2FtZS4NCg0KSS5lLiBuZWVkIHRvIHRlbGwgdmVyeSBleHBsaWNpdGx5
IHRoZSBzY2VuYXJpbyB3aGVyZSB0aGlzIHdpbGwNCmhlbHAuIE5vdCBzYXlpbmcgdGhhdCB0aGlz
IHNob3VsZCByZXNvbHZlIGV2ZXJ5dGhpbmcgYnV0IGl0IG11c3QNCnJlc29sdmUgc29tZXRoaW5n
Lg0KDQovSmFya2tvDQo

Re: [RFC v2 10/13] keys/mktme: Add the MKTME Key Service type for memory encryption

[Sakkinen, Jarkko]

On Thu, 2018-12-06 at 00:51 -0800, Jarkko Sakkinen wrote:
> On Mon, 2018-12-03 at 23:39 -0800, Alison Schofield wrote:
> > MKTME (Multi-Key Total Memory Encryption) is a technology that allows
> > transparent memory encryption in upcoming Intel platforms. MKTME will
> > support mulitple encryption domains, each having their own key. The main
> > use case for the feature is virtual machine isolation. The API needs the
> > flexibility to work for a wide range of uses.
> 
> Some, maybe brutal, honesty (apologies)...
> 
> Have never really got the grip why either SME or TME would make
> isolation any better. If you can break into hypervisor, you'll
> have these tools availabe:
> 
> 1. Read page (in encrypted form).
> 2. Write page (for example replay as pages are not versioned).
> 
> with all the side-channel possibilities of course since you can
> control the VMs (in which core they execute etc.).
> 
> I've seen now SME presentation three times and it always leaves
> me an empty feeling. This feels the same same.

I.e. need to tell very explicitly the scenario where this will
help. Not saying that this should resolve everything but it must
resolve something.

/Jarkko

Re: [RFC v2 10/13] keys/mktme: Add the MKTME Key Service type for memory encryption

[Sakkinen, Jarkko]

On Thu, 2018-12-06 at 00:51 -0800, Jarkko Sakkinen wrote:
> On Mon, 2018-12-03 at 23:39 -0800, Alison Schofield wrote:
> > MKTME (Multi-Key Total Memory Encryption) is a technology that allows
> > transparent memory encryption in upcoming Intel platforms. MKTME will
> > support mulitple encryption domains, each having their own key. The main
> > use case for the feature is virtual machine isolation. The API needs the
> > flexibility to work for a wide range of uses.
> 
> Some, maybe brutal, honesty (apologies)...
> 
> Have never really got the grip why either SME or TME would make
> isolation any better. If you can break into hypervisor, you'll
> have these tools availabe:
> 
> 1. Read page (in encrypted form).
> 2. Write page (for example replay as pages are not versioned).
> 
> with all the side-channel possibilities of course since you can
> control the VMs (in which core they execute etc.).
> 
> I've seen now SME presentation three times and it always leaves
> me an empty feeling. This feels the same same.

I.e. need to tell very explicitly the scenario where this will
help. Not saying that this should resolve everything but it must
resolve something.

/Jarkko

Re: [RFC v2 00/13] Multi-Key Total Memory Encryption API (MKTME)

[Kirill A. Shutemov]

On Wed, Dec 05, 2018 at 08:32:52PM +0000, Sakkinen, Jarkko wrote:
> On Tue, 2018-12-04 at 12:46 +0300, Kirill A. Shutemov wrote:
> > On Tue, Dec 04, 2018 at 09:25:50AM +0000, Peter Zijlstra wrote:
> > > On Mon, Dec 03, 2018 at 11:39:47PM -0800, Alison Schofield wrote:
> > > > (Multi-Key Total Memory Encryption)
> > > 
> > > I think that MKTME is a horrible name, and doesn't appear to accurately
> > > describe what it does either. Specifically the 'total' seems out of
> > > place, it doesn't require all memory to be encrypted.
> > 
> > MKTME implies TME. TME is enabled by BIOS and it encrypts all memory with
> > CPU-generated key. MKTME allows to use other keys or disable encryption
> > for a page.
> 
> When you say "disable encryption to a page" does the encryption get
> actually disabled or does the CPU just decrypt it transparently i.e.
> what happens physically?

Yes, it gets disabled. Physically. It overrides TME encryption.

-- 
 Kirill A. Shutemov

Re: [RFC v2 00/13] Multi-Key Total Memory Encryption API (MKTME)

[Kirill A. Shutemov]

On Wed, Dec 05, 2018 at 08:32:52PM +0000, Sakkinen, Jarkko wrote:
> On Tue, 2018-12-04 at 12:46 +0300, Kirill A. Shutemov wrote:
> > On Tue, Dec 04, 2018 at 09:25:50AM +0000, Peter Zijlstra wrote:
> > > On Mon, Dec 03, 2018 at 11:39:47PM -0800, Alison Schofield wrote:
> > > > (Multi-Key Total Memory Encryption)
> > > 
> > > I think that MKTME is a horrible name, and doesn't appear to accurately
> > > describe what it does either. Specifically the 'total' seems out of
> > > place, it doesn't require all memory to be encrypted.
> > 
> > MKTME implies TME. TME is enabled by BIOS and it encrypts all memory with
> > CPU-generated key. MKTME allows to use other keys or disable encryption
> > for a page.
> 
> When you say "disable encryption to a page" does the encryption get
> actually disabled or does the CPU just decrypt it transparently i.e.
> what happens physically?

Yes, it gets disabled. Physically. It overrides TME encryption.

-- 
 Kirill A. Shutemov

Re: [RFC v2 00/13] Multi-Key Total Memory Encryption API (MKTME)

[Kirill A. Shutemov]

On Wed, Dec 05, 2018 at 08:32:52PM +0000, Sakkinen, Jarkko wrote:
> On Tue, 2018-12-04 at 12:46 +0300, Kirill A. Shutemov wrote:
> > On Tue, Dec 04, 2018 at 09:25:50AM +0000, Peter Zijlstra wrote:
> > > On Mon, Dec 03, 2018 at 11:39:47PM -0800, Alison Schofield wrote:
> > > > (Multi-Key Total Memory Encryption)
> > > 
> > > I think that MKTME is a horrible name, and doesn't appear to accurately
> > > describe what it does either. Specifically the 'total' seems out of
> > > place, it doesn't require all memory to be encrypted.
> > 
> > MKTME implies TME. TME is enabled by BIOS and it encrypts all memory with
> > CPU-generated key. MKTME allows to use other keys or disable encryption
> > for a page.
> 
> When you say "disable encryption to a page" does the encryption get
> actually disabled or does the CPU just decrypt it transparently i.e.
> what happens physically?

Yes, it gets disabled. Physically. It overrides TME encryption.

-- 
 Kirill A. Shutemov

Re: [RFC v2 00/13] Multi-Key Total Memory Encryption API (MKTME)

[Dave Hansen]

On 12/6/18 3:22 AM, Kirill A. Shutemov wrote:
>> When you say "disable encryption to a page" does the encryption get
>> actually disabled or does the CPU just decrypt it transparently i.e.
>> what happens physically?
> Yes, it gets disabled. Physically. It overrides TME encryption.

I know MKTME itself has a runtime overhead and we expect it to have a
performance impact in the low single digits.  Does TME have that
overhead?  Presumably MKTME plus no-encryption is not expected to have
the overhead.

We should probably mention that in the changelogs too.

Re: [RFC v2 00/13] Multi-Key Total Memory Encryption API (MKTME)

[Dave Hansen]

On 12/6/18 3:22 AM, Kirill A. Shutemov wrote:
>> When you say "disable encryption to a page" does the encryption get
>> actually disabled or does the CPU just decrypt it transparently i.e.
>> what happens physically?
> Yes, it gets disabled. Physically. It overrides TME encryption.

I know MKTME itself has a runtime overhead and we expect it to have a
performance impact in the low single digits.  Does TME have that
overhead?  Presumably MKTME plus no-encryption is not expected to have
the overhead.

We should probably mention that in the changelogs too.

Re: [RFC v2 00/13] Multi-Key Total Memory Encryption API (MKTME)

[Dave Hansen]

On 12/6/18 3:22 AM, Kirill A. Shutemov wrote:
>> When you say "disable encryption to a page" does the encryption get
>> actually disabled or does the CPU just decrypt it transparently i.e.
>> what happens physically?
> Yes, it gets disabled. Physically. It overrides TME encryption.

I know MKTME itself has a runtime overhead and we expect it to have a
performance impact in the low single digits.  Does TME have that
overhead?  Presumably MKTME plus no-encryption is not expected to have
the overhead.

We should probably mention that in the changelogs too.

Re: [RFC v2 10/13] keys/mktme: Add the MKTME Key Service type for memory encryption

[Dave Hansen]

On 12/6/18 12:51 AM, Sakkinen, Jarkko wrote:
> On Mon, 2018-12-03 at 23:39 -0800, Alison Schofield wrote:
>> MKTME (Multi-Key Total Memory Encryption) is a technology that allows
>> transparent memory encryption in upcoming Intel platforms. MKTME will
>> support mulitple encryption domains, each having their own key. The main
>> use case for the feature is virtual machine isolation. The API needs the
>> flexibility to work for a wide range of uses.
> Some, maybe brutal, honesty (apologies)...
> 
> Have never really got the grip why either SME or TME would make
> isolation any better. If you can break into hypervisor, you'll
> have these tools availabe:

For systems using MKTME, the hypervisor is within the "trust boundary".
 From what I've read, it is a bit _more_ trusted than with AMD's scheme.

But, yes, if you can mount a successful arbitrary code execution attack
against the MKTME hypervisor, you can defeat MKTME's protections.  If
the kernel creates non-encrypted mappings of memory that's being
encrypted with MKTME, an arbitrary read primitive could also be a very
valuable in defeating MKTME's protections.  That's why Andy is proposing
doing something like eXclusive-Page-Frame-Ownership (google XPFO).

Re: [RFC v2 10/13] keys/mktme: Add the MKTME Key Service type for memory encryption

[Dave Hansen]

On 12/6/18 12:51 AM, Sakkinen, Jarkko wrote:
> On Mon, 2018-12-03 at 23:39 -0800, Alison Schofield wrote:
>> MKTME (Multi-Key Total Memory Encryption) is a technology that allows
>> transparent memory encryption in upcoming Intel platforms. MKTME will
>> support mulitple encryption domains, each having their own key. The main
>> use case for the feature is virtual machine isolation. The API needs the
>> flexibility to work for a wide range of uses.
> Some, maybe brutal, honesty (apologies)...
> 
> Have never really got the grip why either SME or TME would make
> isolation any better. If you can break into hypervisor, you'll
> have these tools availabe:

For systems using MKTME, the hypervisor is within the "trust boundary".
 From what I've read, it is a bit _more_ trusted than with AMD's scheme.

But, yes, if you can mount a successful arbitrary code execution attack
against the MKTME hypervisor, you can defeat MKTME's protections.  If
the kernel creates non-encrypted mappings of memory that's being
encrypted with MKTME, an arbitrary read primitive could also be a very
valuable in defeating MKTME's protections.  That's why Andy is proposing
doing something like eXclusive-Page-Frame-Ownership (google XPFO).

Re: [RFC v2 10/13] keys/mktme: Add the MKTME Key Service type for memory encryption

[Dave Hansen]

On 12/6/18 12:51 AM, Sakkinen, Jarkko wrote:
> On Mon, 2018-12-03 at 23:39 -0800, Alison Schofield wrote:
>> MKTME (Multi-Key Total Memory Encryption) is a technology that allows
>> transparent memory encryption in upcoming Intel platforms. MKTME will
>> support mulitple encryption domains, each having their own key. The main
>> use case for the feature is virtual machine isolation. The API needs the
>> flexibility to work for a wide range of uses.
> Some, maybe brutal, honesty (apologies)...
> 
> Have never really got the grip why either SME or TME would make
> isolation any better. If you can break into hypervisor, you'll
> have these tools availabe:

For systems using MKTME, the hypervisor is within the "trust boundary".
 From what I've read, it is a bit _more_ trusted than with AMD's scheme.

But, yes, if you can mount a successful arbitrary code execution attack
against the MKTME hypervisor, you can defeat MKTME's protections.  If
the kernel creates non-encrypted mappings of memory that's being
encrypted with MKTME, an arbitrary read primitive could also be a very
valuable in defeating MKTME's protections.  That's why Andy is proposing
doing something like eXclusive-Page-Frame-Ownership (google XPFO).

Re: [RFC v2 00/13] Multi-Key Total Memory Encryption API (MKTME)

[Dave Hansen]

On 12/5/18 5:09 PM, Andy Lutomirski wrote:
> On Wed, Dec 5, 2018 at 3:49 PM Dave Hansen <dave.hansen@intel.com> wrote:
>> What I was hoping to see was them do this (apologies for the horrible
>> indentation:
>>
>>         ptr = mmap(..., PROT_NONE);
>>         mprotect_pkey(   addr, len, PROT_NONE, pkey);
>>         mprotect_encrypt(addr, len, PROT_NONE, keyid);
>>         mprotect(        addr, len, real_prot);
>>
>> The point is that you *can* stack these things and don't have to have an
>> mprotect_kitchen_sink() if you use PROT_NONE for intermediate
>> permissions during setup.
> 
> Sure, but then why call it mprotect at all?  How about:
> 
> mmap(..., PROT_NONE);
> mencrypt(..., keyid);
> mprotect_pkey(...);

That would totally work too.  I just like the idea of a family of
mprotect() syscalls that do mprotect() plus one other thing.  What
you're proposing is totally equivalent where we have mprotect_pkey()
always being the *last* thing that gets called, plus a family of things
that we expect to get called on something that's probably PROT_NONE.

> But wouldn't this be much nicer:
> 
> int fd = memfd_create(...);
> memfd_set_tme_key(fd, keyid);  /* fails if len != 0 */
> mmap(fd, ...);

No. :)

One really big advantage with protection keys, or this implementation is
that you don't have to implement an allocator.  You can use it with any
old malloc() as long as you own a whole page.

The pages also fundamentally *stay* anonymous in the VM and get all the
goodness that comes with that, like THP.

>>> and it's also functionally just MADV_DONTNEED.  In other words, the
>>> sole user-visible effect appears to be that the existing pages are
>>> blown away.  The fact that it changes the key in use doesn't seem
>>> terribly useful, since it's anonymous memory,
>>
>> It's functionally MADV_DONTNEED, plus a future promise that your writes
>> will never show up as plaintext on the DIMM.
> 
> But that's mostly vacuous.  If I read the docs right, MKTME systems
> also support TME, so you *already* have that promise, unless the
> firmware totally blew it.  If we want a boot option to have the kernel
> use MKTME to forcibly encrypt everything regardless of what the TME
> MSRs say, I'd be entirely on board.  Heck, the implementation would be
> quite simple because we mostly reuse the SME code.

Yeah, that's true.  I seem to always forget about the TME case! :)

"It's functionally MADV_DONTNEED, plus a future promise that your writes
will never be written to the DIMM with the TME key."

But, this gets us back to your very good question about what good this
does in the end.  What value does _that_ scheme provide over TME?  We're
admittedly weak on specific examples there, but I'm working on it.

>>> the direct map as well, probably using the pageattr.c code.
>>
>> The current, public hardware spec has a description of what's required
>> to maintain cache coherency.  Basically, you can keep as many mappings
>> of a physical page as you want, but only write to one mapping at a time,
>> and clflush the old one when you want to write to a new one.
> 
> Surely you at least have to clflush the old mapping and then the new
> mapping, since the new mapping could have been speculatively read.

Nope.  The coherency is "fine" unless you have writeback of an older
cacheline that blows away newer data.  CPUs that support MKTME are
guaranteed to never do writeback of the lines that could be established
speculatively or from prefetching.

>>> Finally, If you're going to teach the kernel how to have some user
>>> pages that aren't in the direct map, you've essentially done XPO,
>>> which is nifty but expensive.  And I think that doing this gets you
>>> essentially all the benefit of MKTME for the non-pmem use case.  Why
>>> exactly would any software want to use anything other than a
>>> CPU-managed key for anything other than pmem?
>>
>> It is handy, for one, to let you "cluster" key usage.  If you have 5
>> Pepsi VMs and 5 Coke VMs, each Pepsi one using the same key and each
>> Coke one using the same key, you can boil it down to only 2 hardware
>> keyid slots that get used, and do this transparently.
> 
> I understand this from a marketing perspective but not a security
> perspective.  Say I'm Coke and you've sold me some VMs that are
> "encrypted with a Coke-specific key and no other VMs get to use that
> key."  I can't think of *any* not-exceedingly-contrived attack in
> which this makes the slightest difference.  If Pepsi tries to attack
> Coke without MKTME, then they'll either need to get the hypervisor to
> leak Coke's data through the direct map or they'll have to find some
> way to corrupt a page table or use something like L1TF to read from a
> physical address Coke owns.  With MKTME, if they can read through the
> host direct map, then they'll get Coke's cleartext, and if they can
> corrupt a page table or use L1TF to read from your memory, they'll get
> Coke's cleartext.

The design definitely has the hypervisor in the trust boundary.  If the
hypervisor is evil, or if someone evil compromises the hypervisor, MKTME
obviously provides less protection.

I guess the question ends up being if this makes its protections weak
enough that we should not bother merging it in its current form.

I still have the homework assignment to go figure out why folks want the
protections as they stand.

Re: [RFC v2 00/13] Multi-Key Total Memory Encryption API (MKTME)

[Dave Hansen]

On 12/5/18 5:09 PM, Andy Lutomirski wrote:
> On Wed, Dec 5, 2018 at 3:49 PM Dave Hansen <dave.hansen@intel.com> wrote:
>> What I was hoping to see was them do this (apologies for the horrible
>> indentation:
>>
>>         ptr = mmap(..., PROT_NONE);
>>         mprotect_pkey(   addr, len, PROT_NONE, pkey);
>>         mprotect_encrypt(addr, len, PROT_NONE, keyid);
>>         mprotect(        addr, len, real_prot);
>>
>> The point is that you *can* stack these things and don't have to have an
>> mprotect_kitchen_sink() if you use PROT_NONE for intermediate
>> permissions during setup.
> 
> Sure, but then why call it mprotect at all?  How about:
> 
> mmap(..., PROT_NONE);
> mencrypt(..., keyid);
> mprotect_pkey(...);

That would totally work too.  I just like the idea of a family of
mprotect() syscalls that do mprotect() plus one other thing.  What
you're proposing is totally equivalent where we have mprotect_pkey()
always being the *last* thing that gets called, plus a family of things
that we expect to get called on something that's probably PROT_NONE.

> But wouldn't this be much nicer:
> 
> int fd = memfd_create(...);
> memfd_set_tme_key(fd, keyid);  /* fails if len != 0 */
> mmap(fd, ...);

No. :)

One really big advantage with protection keys, or this implementation is
that you don't have to implement an allocator.  You can use it with any
old malloc() as long as you own a whole page.

The pages also fundamentally *stay* anonymous in the VM and get all the
goodness that comes with that, like THP.

>>> and it's also functionally just MADV_DONTNEED.  In other words, the
>>> sole user-visible effect appears to be that the existing pages are
>>> blown away.  The fact that it changes the key in use doesn't seem
>>> terribly useful, since it's anonymous memory,
>>
>> It's functionally MADV_DONTNEED, plus a future promise that your writes
>> will never show up as plaintext on the DIMM.
> 
> But that's mostly vacuous.  If I read the docs right, MKTME systems
> also support TME, so you *already* have that promise, unless the
> firmware totally blew it.  If we want a boot option to have the kernel
> use MKTME to forcibly encrypt everything regardless of what the TME
> MSRs say, I'd be entirely on board.  Heck, the implementation would be
> quite simple because we mostly reuse the SME code.

Yeah, that's true.  I seem to always forget about the TME case! :)

"It's functionally MADV_DONTNEED, plus a future promise that your writes
will never be written to the DIMM with the TME key."

But, this gets us back to your very good question about what good this
does in the end.  What value does _that_ scheme provide over TME?  We're
admittedly weak on specific examples there, but I'm working on it.

>>> the direct map as well, probably using the pageattr.c code.
>>
>> The current, public hardware spec has a description of what's required
>> to maintain cache coherency.  Basically, you can keep as many mappings
>> of a physical page as you want, but only write to one mapping at a time,
>> and clflush the old one when you want to write to a new one.
> 
> Surely you at least have to clflush the old mapping and then the new
> mapping, since the new mapping could have been speculatively read.

Nope.  The coherency is "fine" unless you have writeback of an older
cacheline that blows away newer data.  CPUs that support MKTME are
guaranteed to never do writeback of the lines that could be established
speculatively or from prefetching.

>>> Finally, If you're going to teach the kernel how to have some user
>>> pages that aren't in the direct map, you've essentially done XPO,
>>> which is nifty but expensive.  And I think that doing this gets you
>>> essentially all the benefit of MKTME for the non-pmem use case.  Why
>>> exactly would any software want to use anything other than a
>>> CPU-managed key for anything other than pmem?
>>
>> It is handy, for one, to let you "cluster" key usage.  If you have 5
>> Pepsi VMs and 5 Coke VMs, each Pepsi one using the same key and each
>> Coke one using the same key, you can boil it down to only 2 hardware
>> keyid slots that get used, and do this transparently.
> 
> I understand this from a marketing perspective but not a security
> perspective.  Say I'm Coke and you've sold me some VMs that are
> "encrypted with a Coke-specific key and no other VMs get to use that
> key."  I can't think of *any* not-exceedingly-contrived attack in
> which this makes the slightest difference.  If Pepsi tries to attack
> Coke without MKTME, then they'll either need to get the hypervisor to
> leak Coke's data through the direct map or they'll have to find some
> way to corrupt a page table or use something like L1TF to read from a
> physical address Coke owns.  With MKTME, if they can read through the
> host direct map, then they'll get Coke's cleartext, and if they can
> corrupt a page table or use L1TF to read from your memory, they'll get
> Coke's cleartext.

The design definitely has the hypervisor in the trust boundary.  If the
hypervisor is evil, or if someone evil compromises the hypervisor, MKTME
obviously provides less protection.

I guess the question ends up being if this makes its protections weak
enough that we should not bother merging it in its current form.

I still have the homework assignment to go figure out why folks want the
protections as they stand.

Re: [RFC v2 00/13] Multi-Key Total Memory Encryption API (MKTME)

[Andy Lutomirski]

> On Dec 6, 2018, at 7:39 AM, Dave Hansen <dave.hansen@intel.com> wrote:

>>>> the direct map as well, probably using the pageattr.c code.
>>>
>>> The current, public hardware spec has a description of what's required
>>> to maintain cache coherency.  Basically, you can keep as many mappings
>>> of a physical page as you want, but only write to one mapping at a time,
>>> and clflush the old one when you want to write to a new one.
>>
>> Surely you at least have to clflush the old mapping and then the new
>> mapping, since the new mapping could have been speculatively read.
>
> Nope.  The coherency is "fine" unless you have writeback of an older
> cacheline that blows away newer data.  CPUs that support MKTME are
> guaranteed to never do writeback of the lines that could be established
> speculatively or from prefetching.

How is that sufficient?  Suppose I have some physical page mapped with
keys 1 and 2. #1 is logically live and I write to it.  Then I prefetch
or otherwise populate mapping 2 into the cache (in the S state,
presumably).  Now I clflush mapping 1 and read 2.  It contains garbage
in the cache, but the garbage in the cache is inconsistent with the
garbage in memory.  This can’t be a good thing, even if no writeback
occurs.

I suppose the right fix is to clflush the old mapping and then to zero
the new mapping.

>
>>>> Finally, If you're going to teach the kernel how to have some user
>>>> pages that aren't in the direct map, you've essentially done XPO,
>>>> which is nifty but expensive.  And I think that doing this gets you
>>>> essentially all the benefit of MKTME for the non-pmem use case.  Why
>>>> exactly would any software want to use anything other than a
>>>> CPU-managed key for anything other than pmem?
>>>
>>> It is handy, for one, to let you "cluster" key usage.  If you have 5
>>> Pepsi VMs and 5 Coke VMs, each Pepsi one using the same key and each
>>> Coke one using the same key, you can boil it down to only 2 hardware
>>> keyid slots that get used, and do this transparently.
>>
>> I understand this from a marketing perspective but not a security
>> perspective.  Say I'm Coke and you've sold me some VMs that are
>> "encrypted with a Coke-specific key and no other VMs get to use that
>> key."  I can't think of *any* not-exceedingly-contrived attack in
>> which this makes the slightest difference.  If Pepsi tries to attack
>> Coke without MKTME, then they'll either need to get the hypervisor to
>> leak Coke's data through the direct map or they'll have to find some
>> way to corrupt a page table or use something like L1TF to read from a
>> physical address Coke owns.  With MKTME, if they can read through the
>> host direct map, then they'll get Coke's cleartext, and if they can
>> corrupt a page table or use L1TF to read from your memory, they'll get
>> Coke's cleartext.
>
> The design definitely has the hypervisor in the trust boundary.  If the
> hypervisor is evil, or if someone evil compromises the hypervisor, MKTME
> obviously provides less protection.
>
> I guess the question ends up being if this makes its protections weak
> enough that we should not bother merging it in its current form.

Indeed, but I’d ask another question too: I expect that MKTME is weak
enough that it will be improved, and without seeing the improvement,
it seems quite plausible that using the improvement will require
radically reworking the kernel implementation.

As a straw man, suppose we get a way to say “this key may only be
accessed through such-and-such VPID or by using a special new
restricted facility for the hypervisor to request access”.    Now we
have some degree of serious protection, but it doesn’t work, by
design, for anonymous memory.  Similarly, something that looks more
like AMD's SEV would be very very awkward to support with anything
like the current API proposal.

>
> I still have the homework assignment to go figure out why folks want the
> protections as they stand.

Re: [RFC v2 00/13] Multi-Key Total Memory Encryption API (MKTME)

[Andy Lutomirski]

> On Dec 6, 2018, at 7:39 AM, Dave Hansen <dave.hansen@intel.com> wrote:

>>>> the direct map as well, probably using the pageattr.c code.
>>>
>>> The current, public hardware spec has a description of what's required
>>> to maintain cache coherency.  Basically, you can keep as many mappings
>>> of a physical page as you want, but only write to one mapping at a time,
>>> and clflush the old one when you want to write to a new one.
>>
>> Surely you at least have to clflush the old mapping and then the new
>> mapping, since the new mapping could have been speculatively read.
>
> Nope.  The coherency is "fine" unless you have writeback of an older
> cacheline that blows away newer data.  CPUs that support MKTME are
> guaranteed to never do writeback of the lines that could be established
> speculatively or from prefetching.

How is that sufficient?  Suppose I have some physical page mapped with
keys 1 and 2. #1 is logically live and I write to it.  Then I prefetch
or otherwise populate mapping 2 into the cache (in the S state,
presumably).  Now I clflush mapping 1 and read 2.  It contains garbage
in the cache, but the garbage in the cache is inconsistent with the
garbage in memory.  This can’t be a good thing, even if no writeback
occurs.

I suppose the right fix is to clflush the old mapping and then to zero
the new mapping.

>
>>>> Finally, If you're going to teach the kernel how to have some user
>>>> pages that aren't in the direct map, you've essentially done XPO,
>>>> which is nifty but expensive.  And I think that doing this gets you
>>>> essentially all the benefit of MKTME for the non-pmem use case.  Why
>>>> exactly would any software want to use anything other than a
>>>> CPU-managed key for anything other than pmem?
>>>
>>> It is handy, for one, to let you "cluster" key usage.  If you have 5
>>> Pepsi VMs and 5 Coke VMs, each Pepsi one using the same key and each
>>> Coke one using the same key, you can boil it down to only 2 hardware
>>> keyid slots that get used, and do this transparently.
>>
>> I understand this from a marketing perspective but not a security
>> perspective.  Say I'm Coke and you've sold me some VMs that are
>> "encrypted with a Coke-specific key and no other VMs get to use that
>> key."  I can't think of *any* not-exceedingly-contrived attack in
>> which this makes the slightest difference.  If Pepsi tries to attack
>> Coke without MKTME, then they'll either need to get the hypervisor to
>> leak Coke's data through the direct map or they'll have to find some
>> way to corrupt a page table or use something like L1TF to read from a
>> physical address Coke owns.  With MKTME, if they can read through the
>> host direct map, then they'll get Coke's cleartext, and if they can
>> corrupt a page table or use L1TF to read from your memory, they'll get
>> Coke's cleartext.
>
> The design definitely has the hypervisor in the trust boundary.  If the
> hypervisor is evil, or if someone evil compromises the hypervisor, MKTME
> obviously provides less protection.
>
> I guess the question ends up being if this makes its protections weak
> enough that we should not bother merging it in its current form.

Indeed, but I’d ask another question too: I expect that MKTME is weak
enough that it will be improved, and without seeing the improvement,
it seems quite plausible that using the improvement will require
radically reworking the kernel implementation.

As a straw man, suppose we get a way to say “this key may only be
accessed through such-and-such VPID or by using a special new
restricted facility for the hypervisor to request access”.    Now we
have some degree of serious protection, but it doesn’t work, by
design, for anonymous memory.  Similarly, something that looks more
like AMD's SEV would be very very awkward to support with anything
like the current API proposal.

>
> I still have the homework assignment to go figure out why folks want the
> protections as they stand.

Re: [RFC v2 00/13] Multi-Key Total Memory Encryption API (MKTME)

[Dave Hansen]

On 12/6/18 11:10 AM, Andy Lutomirski wrote:
>> On Dec 6, 2018, at 7:39 AM, Dave Hansen <dave.hansen@intel.com> wrote:
>>The coherency is "fine" unless you have writeback of an older
>> cacheline that blows away newer data.  CPUs that support MKTME are
>> guaranteed to never do writeback of the lines that could be established
>> speculatively or from prefetching.
> 
> How is that sufficient?  Suppose I have some physical page mapped with
> keys 1 and 2. #1 is logically live and I write to it.  Then I prefetch
> or otherwise populate mapping 2 into the cache (in the S state,
> presumably).  Now I clflush mapping 1 and read 2.  It contains garbage
> in the cache, but the garbage in the cache is inconsistent with the
> garbage in memory.  This can’t be a good thing, even if no writeback
> occurs.
> 
> I suppose the right fix is to clflush the old mapping and then to zero
> the new mapping.

Yep.  Practically, you need to write to the new mapping to give it any
meaning.  Those writes effectively blow away any previously cached,
garbage contents.

I think you're right, though, that the cached data might not be
_consistent_ with what is in memory.  It feels really dirty, but I can't
think of any problems that it actually causes.

Re: [RFC v2 00/13] Multi-Key Total Memory Encryption API (MKTME)

[Dave Hansen]

On 12/6/18 11:10 AM, Andy Lutomirski wrote:
>> On Dec 6, 2018, at 7:39 AM, Dave Hansen <dave.hansen@intel.com> wrote:
>>The coherency is "fine" unless you have writeback of an older
>> cacheline that blows away newer data.  CPUs that support MKTME are
>> guaranteed to never do writeback of the lines that could be established
>> speculatively or from prefetching.
> 
> How is that sufficient?  Suppose I have some physical page mapped with
> keys 1 and 2. #1 is logically live and I write to it.  Then I prefetch
> or otherwise populate mapping 2 into the cache (in the S state,
> presumably).  Now I clflush mapping 1 and read 2.  It contains garbage
> in the cache, but the garbage in the cache is inconsistent with the
> garbage in memory.  This can’t be a good thing, even if no writeback
> occurs.
> 
> I suppose the right fix is to clflush the old mapping and then to zero
> the new mapping.

Yep.  Practically, you need to write to the new mapping to give it any
meaning.  Those writes effectively blow away any previously cached,
garbage contents.

I think you're right, though, that the cached data might not be
_consistent_ with what is in memory.  It feels really dirty, but I can't
think of any problems that it actually causes.

Re: [RFC v2 00/13] Multi-Key Total Memory Encryption API (MKTME)

[Sakkinen, Jarkko]

T24gVGh1LCAyMDE4LTEyLTA2IGF0IDE0OjIyICswMzAwLCBLaXJpbGwgQS4gU2h1dGVtb3Ygd3Jv
dGU6DQo+IFdoZW4geW91IHNheSAiZGlzYWJsZSBlbmNyeXB0aW9uIHRvIGEgcGFnZSIgZG9lcyB0
aGUgZW5jcnlwdGlvbiBnZXQNCj4gPiBhY3R1YWxseSBkaXNhYmxlZCBvciBkb2VzIHRoZSBDUFUg
anVzdCBkZWNyeXB0IGl0IHRyYW5zcGFyZW50bHkgaS5lLg0KPiA+IHdoYXQgaGFwcGVucyBwaHlz
aWNhbGx5Pw0KPiANCj4gWWVzLCBpdCBnZXRzIGRpc2FibGVkLiBQaHlzaWNhbGx5LiBJdCBvdmVy
cmlkZXMgVE1FIGVuY3J5cHRpb24uDQoNCk9LLCB0aGFua3MgZm9yIGNvbmZpcm1hdGlvbi4gQlRX
LCBob3cgbXVjaCBpcyB0aGUgcGVuYWx0eSB0byBrZWVwIGl0DQphbHdheXMgZW5hYmxlZD8gSXMg
aXQgc29tZXRoaW5nIHRoYXQgd291bGQgbm90IG1ha2Ugc2Vuc2UgZm9yIHNvbWUNCm90aGVyIHJl
YXNvbnM/DQoNCi9KYXJra28NCg=

Re: [RFC v2 00/13] Multi-Key Total Memory Encryption API (MKTME)

[Sakkinen, Jarkko]

On Thu, 2018-12-06 at 14:22 +0300, Kirill A. Shutemov wrote:
> When you say "disable encryption to a page" does the encryption get
> > actually disabled or does the CPU just decrypt it transparently i.e.
> > what happens physically?
> 
> Yes, it gets disabled. Physically. It overrides TME encryption.

OK, thanks for confirmation. BTW, how much is the penalty to keep it
always enabled? Is it something that would not make sense for some
other reasons?

/Jarkko

Re: [RFC v2 00/13] Multi-Key Total Memory Encryption API (MKTME)

[Sakkinen, Jarkko]

On Thu, 2018-12-06 at 14:22 +0300, Kirill A. Shutemov wrote:
> When you say "disable encryption to a page" does the encryption get
> > actually disabled or does the CPU just decrypt it transparently i.e.
> > what happens physically?
> 
> Yes, it gets disabled. Physically. It overrides TME encryption.

OK, thanks for confirmation. BTW, how much is the penalty to keep it
always enabled? Is it something that would not make sense for some
other reasons?

/Jarkko

Re: [RFC v2 10/13] keys/mktme: Add the MKTME Key Service type for memory encryption

[Sakkinen, Jarkko]

T24gVGh1LCAyMDE4LTEyLTA2IGF0IDA3OjExIC0wODAwLCBEYXZlIEhhbnNlbiB3cm90ZToNCj4g
T24gMTIvNi8xOCAxMjo1MSBBTSwgU2Fra2luZW4sIEphcmtrbyB3cm90ZToNCj4gPiBPbiBNb24s
IDIwMTgtMTItMDMgYXQgMjM6MzkgLTA4MDAsIEFsaXNvbiBTY2hvZmllbGQgd3JvdGU6DQo+ID4g
PiBNS1RNRSAoTXVsdGktS2V5IFRvdGFsIE1lbW9yeSBFbmNyeXB0aW9uKSBpcyBhIHRlY2hub2xv
Z3kgdGhhdCBhbGxvd3MNCj4gPiA+IHRyYW5zcGFyZW50IG1lbW9yeSBlbmNyeXB0aW9uIGluIHVw
Y29taW5nIEludGVsIHBsYXRmb3Jtcy4gTUtUTUUgd2lsbA0KPiA+ID4gc3VwcG9ydCBtdWxpdHBs
ZSBlbmNyeXB0aW9uIGRvbWFpbnMsIGVhY2ggaGF2aW5nIHRoZWlyIG93biBrZXkuIFRoZSBtYWlu
DQo+ID4gPiB1c2UgY2FzZSBmb3IgdGhlIGZlYXR1cmUgaXMgdmlydHVhbCBtYWNoaW5lIGlzb2xh
dGlvbi4gVGhlIEFQSSBuZWVkcyB0aGUNCj4gPiA+IGZsZXhpYmlsaXR5IHRvIHdvcmsgZm9yIGEg
d2lkZSByYW5nZSBvZiB1c2VzLg0KPiA+IFNvbWUsIG1heWJlIGJydXRhbCwgaG9uZXN0eSAoYXBv
bG9naWVzKS4uLg0KPiA+IA0KPiA+IEhhdmUgbmV2ZXIgcmVhbGx5IGdvdCB0aGUgZ3JpcCB3aHkg
ZWl0aGVyIFNNRSBvciBUTUUgd291bGQgbWFrZQ0KPiA+IGlzb2xhdGlvbiBhbnkgYmV0dGVyLiBJ
ZiB5b3UgY2FuIGJyZWFrIGludG8gaHlwZXJ2aXNvciwgeW91J2xsDQo+ID4gaGF2ZSB0aGVzZSB0
b29scyBhdmFpbGFiZToNCj4gDQo+IEZvciBzeXN0ZW1zIHVzaW5nIE1LVE1FLCB0aGUgaHlwZXJ2
aXNvciBpcyB3aXRoaW4gdGhlICJ0cnVzdCBib3VuZGFyeSIuDQo+ICBGcm9tIHdoYXQgSSd2ZSBy
ZWFkLCBpdCBpcyBhIGJpdCBfbW9yZV8gdHJ1c3RlZCB0aGFuIHdpdGggQU1EJ3Mgc2NoZW1lLg0K
PiANCj4gQnV0LCB5ZXMsIGlmIHlvdSBjYW4gbW91bnQgYSBzdWNjZXNzZnVsIGFyYml0cmFyeSBj
b2RlIGV4ZWN1dGlvbiBhdHRhY2sNCj4gYWdhaW5zdCB0aGUgTUtUTUUgaHlwZXJ2aXNvciwgeW91
IGNhbiBkZWZlYXQgTUtUTUUncyBwcm90ZWN0aW9ucy4gIElmDQo+IHRoZSBrZXJuZWwgY3JlYXRl
cyBub24tZW5jcnlwdGVkIG1hcHBpbmdzIG9mIG1lbW9yeSB0aGF0J3MgYmVpbmcNCj4gZW5jcnlw
dGVkIHdpdGggTUtUTUUsIGFuIGFyYml0cmFyeSByZWFkIHByaW1pdGl2ZSBjb3VsZCBhbHNvIGJl
IGEgdmVyeQ0KPiB2YWx1YWJsZSBpbiBkZWZlYXRpbmcgTUtUTUUncyBwcm90ZWN0aW9ucy4gIFRo
YXQncyB3aHkgQW5keSBpcyBwcm9wb3NpbmcNCj4gZG9pbmcgc29tZXRoaW5nIGxpa2UgZVhjbHVz
aXZlLVBhZ2UtRnJhbWUtT3duZXJzaGlwIChnb29nbGUgWFBGTykuDQoNClRoYW5rcywgSSB3YXMg
bm90IGF3YXJlIG9mIFhQRk8gYnV0IEkgZm91bmQgYSBuaWNlIH4yIHBhZ2UgYXJ0aWNsZSBhYm91
dCBpdDoNCg0KaHR0cHM6Ly9sd24ubmV0L0FydGljbGVzLzcwMDY0Ny8NCg0KSSB0aGluayB0aGUg
cGVyZm9ybWFuY2UgaGl0IGlzIHRoZSBuZWNlc3NhcnkgcHJpY2UgdG8gcGF5IChpZiB5b3Ugd2Fu
dA0Kc29tZXRoaW5nIG1vcmUgb3BhcXVlIHRoYW4ganVzdCB0aGUgdXN1YWwgIm1pbGl0YXJ5IGdy
YWRlIHNlY3VyaXR5IikuIEF0DQptaW5pbXVtLCBpdCBzaG91bGQgYmUgYW4gb3B0LWluIGZlYXR1
cmUuDQoNCi9KYXJra28NCg=

Re: [RFC v2 10/13] keys/mktme: Add the MKTME Key Service type for memory encryption

[Sakkinen, Jarkko]

On Thu, 2018-12-06 at 07:11 -0800, Dave Hansen wrote:
> On 12/6/18 12:51 AM, Sakkinen, Jarkko wrote:
> > On Mon, 2018-12-03 at 23:39 -0800, Alison Schofield wrote:
> > > MKTME (Multi-Key Total Memory Encryption) is a technology that allows
> > > transparent memory encryption in upcoming Intel platforms. MKTME will
> > > support mulitple encryption domains, each having their own key. The main
> > > use case for the feature is virtual machine isolation. The API needs the
> > > flexibility to work for a wide range of uses.
> > Some, maybe brutal, honesty (apologies)...
> > 
> > Have never really got the grip why either SME or TME would make
> > isolation any better. If you can break into hypervisor, you'll
> > have these tools availabe:
> 
> For systems using MKTME, the hypervisor is within the "trust boundary".
>  From what I've read, it is a bit _more_ trusted than with AMD's scheme.
> 
> But, yes, if you can mount a successful arbitrary code execution attack
> against the MKTME hypervisor, you can defeat MKTME's protections.  If
> the kernel creates non-encrypted mappings of memory that's being
> encrypted with MKTME, an arbitrary read primitive could also be a very
> valuable in defeating MKTME's protections.  That's why Andy is proposing
> doing something like eXclusive-Page-Frame-Ownership (google XPFO).

Thanks, I was not aware of XPFO but I found a nice ~2 page article about it:

https://lwn.net/Articles/700647/

I think the performance hit is the necessary price to pay (if you want
something more opaque than just the usual "military grade security"). At
minimum, it should be an opt-in feature.

/Jarkko

Re: [RFC v2 10/13] keys/mktme: Add the MKTME Key Service type for memory encryption

[Sakkinen, Jarkko]

On Thu, 2018-12-06 at 07:11 -0800, Dave Hansen wrote:
> On 12/6/18 12:51 AM, Sakkinen, Jarkko wrote:
> > On Mon, 2018-12-03 at 23:39 -0800, Alison Schofield wrote:
> > > MKTME (Multi-Key Total Memory Encryption) is a technology that allows
> > > transparent memory encryption in upcoming Intel platforms. MKTME will
> > > support mulitple encryption domains, each having their own key. The main
> > > use case for the feature is virtual machine isolation. The API needs the
> > > flexibility to work for a wide range of uses.
> > Some, maybe brutal, honesty (apologies)...
> > 
> > Have never really got the grip why either SME or TME would make
> > isolation any better. If you can break into hypervisor, you'll
> > have these tools availabe:
> 
> For systems using MKTME, the hypervisor is within the "trust boundary".
>  From what I've read, it is a bit _more_ trusted than with AMD's scheme.
> 
> But, yes, if you can mount a successful arbitrary code execution attack
> against the MKTME hypervisor, you can defeat MKTME's protections.  If
> the kernel creates non-encrypted mappings of memory that's being
> encrypted with MKTME, an arbitrary read primitive could also be a very
> valuable in defeating MKTME's protections.  That's why Andy is proposing
> doing something like eXclusive-Page-Frame-Ownership (google XPFO).

Thanks, I was not aware of XPFO but I found a nice ~2 page article about it:

https://lwn.net/Articles/700647/

I think the performance hit is the necessary price to pay (if you want
something more opaque than just the usual "military grade security"). At
minimum, it should be an opt-in feature.

/Jarkko

Re: [RFC v2 00/13] Multi-Key Total Memory Encryption API (MKTME)

[Huang, Kai]

DQo+IA0KPiBUTUUgaXRzZWxmIHByb3ZpZGVzIGEgdG9uIG9mIHByb3RlY3Rpb24gLS0geW91IGNh
bid0IGp1c3QgYmFyZ2UgaW50bw0KPiB0aGUgZGF0YWNlbnRlciwgcmVmcmlnZXJhdGUgdGhlIERJ
TU1zLCB3YWxrIGF3YXkgd2l0aCB0aGVtLCBhbmQgcmVhZA0KPiBvZmYgZXZlcnlvbmUncyBkYXRh
Lg0KPiANCj4gQW0gSSBtaXNzaW5nIHNvbWV0aGluZz8NCg0KSSB0aGluayB3ZSBjYW4gbWFrZSBz
dWNoIGFzc3VtcHRpb24gaW4gbW9zdCBjYXNlcywgYnV0IEkgdGhpbmsgaXQncyBiZXR0ZXIgdGhh
dCB3ZSBkb24ndCBtYWtlIGFueQ0KYXNzdW1wdGlvbiBhdCBhbGwuIEZvciBleGFtcGxlLCB0aGUg
YWRtaW4gb2YgZGF0YSBjZW50ZXIgKG9yIGFueW9uZSkgd2hvIGhhcyBwaHlzaWNhbCBhY2Nlc3Mg
dG8NCnNlcnZlcnMgbWF5IGRvIHNvbWV0aGluZyBtYWxpY2lvdXMuIEkgYW0gbm90IGV4cGVydCBi
dXQgdGhlcmUgc2hvdWxkIGJlIG90aGVyIHBoeXNpY2FsIGF0dGFjaw0KbWV0aG9kcyBiZXNpZGVz
IGNvbGRib290IGF0dGFjaywgaWYgdGhlIG1hbGljaW91cyBlbXBsb3llZSBjYW4gZ2V0IHBoeXNp
Y2FsIGFjY2VzcyB0byBzZXJ2ZXIgdy9vDQpiZWluZyBkZXRlY3RlZC4NCg0KPiANCj4gPiANCj4g
PiBCdXQsIEkgdGhpbmsgd2hhdCB5b3UncmUgaW1wbHlpbmcgaXMgdGhhdCB0aGUgc2VjdXJpdHkg
cHJvcGVydGllcyBvZg0KPiA+IHVzZXItc3VwcGxpZWQga2V5cyBjYW4gb25seSBiZSAqd29yc2Uq
IHRoYW4gdXNpbmcgQ1BVLWdlbmVyYXRlZCBrZXlzDQo+ID4gKGFzc3VtaW5nIHRoZSBDUFUgZG9l
cyBhIGdvb2Qgam9iIGdlbmVyYXRpbmcgaXQpLiAgU28sIHdoeSBib3RoZXINCj4gPiBhbGxvd2lu
ZyB1c2VyLXNwZWNpZmllZCBrZXlzIGluIHRoZSBmaXJzdCBwbGFjZT8NCj4gDQo+IFRoYXQgdG9v
IDopDQoNCkkgdGhpbmsgb25lIHVzYWdlIG9mIHVzZXItc3BlY2lmaWVkIGtleSBpcyBmb3IgTlZE
SU1NLCBzaW5jZSBDUFUga2V5IHdpbGwgYmUgZ29uZSBhZnRlciBtYWNoaW5lDQpyZWJvb3QsIHRo
ZXJlZm9yZSBpZiBOVkRJTU0gaXMgZW5jcnlwdGVkIGJ5IENQVSBrZXkgd2UgYXJlIG5vdCBhYmxl
IHRvIHJldHJpZXZlIGl0IG9uY2UNCnNodXRkb3duL3JlYm9vdCwgZXRjLg0KDQpUaGVyZSBhcmUg
c29tZSBvdGhlciB1c2UgY2FzZXMgdGhhdCBhbHJlYWR5IHJlcXVpcmUgdGVuYW50IHRvIHNlbmQg
a2V5IHRvIENTUC4gRm9yIGV4YW1wbGUsIHRoZSBWTQ0KaW1hZ2UgY2FuIGJlIHByb3ZpZGVkIGJ5
IHRlbmFudCBhbmQgZW5jcnlwdGVkIGJ5IHRlbmFudCdzIG93biBrZXksIGFuZCB0ZW5hbnQgbmVl
ZHMgdG8gc2VuZCBrZXkgdG8NCkNTUCB3aGVuIGFza2luZyBDU1AgdG8gcnVuIHRoYXQgZW5jcnlw
dGVkIGltYWdlLiBCdXQgdGVuYW50IHdpbGwgbmVlZCB0byB0cnVzdCBDU1AgaW4gc3VjaCBjYXNl
LA0Kd2hpY2ggYnJpbmdzIHVzIHdoeSB0ZW5hbnQgd2FudHMgdG8gdXNlIGhpcyBvd24gaW1hZ2Ug
YXQgZmlyc3QgcGxhY2UgKEkgaGF2ZSB0byBzYXkgSSBteXNlbGYgaXMgbm90DQpjb252aW5jZWQg
dGhlIHZhbHVlIG9mIHN1Y2ggdXNlIGNhc2UpLiBJIHRoaW5rIHRoZXJlIGFyZSB0d28gbGV2ZWxz
IG9mIHRydXN0aW5lc3MgaW52b2x2ZWQgaGVyZTogMSkNCnRlbmFudCBuZWVkcyB0byB0cnVzdCBD
U1AgYW55d2F5OyAyKSBidXQgQ1NQIG5lZWRzIHRvIGNvbnZpbmNlIHRlbmFudCB0aGF0IENTUCBj
YW4gYmUgdHJ1c3RlZCwgaWUsDQpieSBwcm92aW5nIGl0IGNhbiBwcmV2ZW50IHBvdGVudGlhbCBh
dHRhY2sgZnJvbSBtYWxpY2lvdXMgZW1wbG95ZWUgKGllLCBieSByYWlzaW5nIGJhciBieSB1c2lu
Zw0KTUtUTUUpLCBldGMuDQoNClRoYW5rcywNCi1LYWk

Re: [RFC v2 00/13] Multi-Key Total Memory Encryption API (MKTME)

[Huang, Kai]

> 
> TME itself provides a ton of protection -- you can't just barge into
> the datacenter, refrigerate the DIMMs, walk away with them, and read
> off everyone's data.
> 
> Am I missing something?

I think we can make such assumption in most cases, but I think it's better that we don't make any
assumption at all. For example, the admin of data center (or anyone) who has physical access to
servers may do something malicious. I am not expert but there should be other physical attack
methods besides coldboot attack, if the malicious employee can get physical access to server w/o
being detected.

> 
> > 
> > But, I think what you're implying is that the security properties of
> > user-supplied keys can only be *worse* than using CPU-generated keys
> > (assuming the CPU does a good job generating it).  So, why bother
> > allowing user-specified keys in the first place?
> 
> That too :)

I think one usage of user-specified key is for NVDIMM, since CPU key will be gone after machine
reboot, therefore if NVDIMM is encrypted by CPU key we are not able to retrieve it once
shutdown/reboot, etc.

There are some other use cases that already require tenant to send key to CSP. For example, the VM
image can be provided by tenant and encrypted by tenant's own key, and tenant needs to send key to
CSP when asking CSP to run that encrypted image. But tenant will need to trust CSP in such case,
which brings us why tenant wants to use his own image at first place (I have to say I myself is not
convinced the value of such use case). I think there are two levels of trustiness involved here: 1)
tenant needs to trust CSP anyway; 2) but CSP needs to convince tenant that CSP can be trusted, ie,
by proving it can prevent potential attack from malicious employee (ie, by raising bar by using
MKTME), etc.

Thanks,
-Kai

Re: [RFC v2 00/13] Multi-Key Total Memory Encryption API (MKTME)

[Huang, Kai]

> 
> TME itself provides a ton of protection -- you can't just barge into
> the datacenter, refrigerate the DIMMs, walk away with them, and read
> off everyone's data.
> 
> Am I missing something?

I think we can make such assumption in most cases, but I think it's better that we don't make any
assumption at all. For example, the admin of data center (or anyone) who has physical access to
servers may do something malicious. I am not expert but there should be other physical attack
methods besides coldboot attack, if the malicious employee can get physical access to server w/o
being detected.

> 
> > 
> > But, I think what you're implying is that the security properties of
> > user-supplied keys can only be *worse* than using CPU-generated keys
> > (assuming the CPU does a good job generating it).  So, why bother
> > allowing user-specified keys in the first place?
> 
> That too :)

I think one usage of user-specified key is for NVDIMM, since CPU key will be gone after machine
reboot, therefore if NVDIMM is encrypted by CPU key we are not able to retrieve it once
shutdown/reboot, etc.

There are some other use cases that already require tenant to send key to CSP. For example, the VM
image can be provided by tenant and encrypted by tenant's own key, and tenant needs to send key to
CSP when asking CSP to run that encrypted image. But tenant will need to trust CSP in such case,
which brings us why tenant wants to use his own image at first place (I have to say I myself is not
convinced the value of such use case). I think there are two levels of trustiness involved here: 1)
tenant needs to trust CSP anyway; 2) but CSP needs to convince tenant that CSP can be trusted, ie,
by proving it can prevent potential attack from malicious employee (ie, by raising bar by using
MKTME), etc.

Thanks,
-Kai

Re: [RFC v2 00/13] Multi-Key Total Memory Encryption API (MKTME)

[Huang, Kai]

T24gV2VkLCAyMDE4LTEyLTA1IGF0IDIyOjE5ICswMDAwLCBTYWtraW5lbiwgSmFya2tvIHdyb3Rl
Og0KPiBPbiBUdWUsIDIwMTgtMTItMDQgYXQgMTE6MTkgLTA4MDAsIEFuZHkgTHV0b21pcnNraSB3
cm90ZToNCj4gPiBJJ20gbm90IFRob21hcywgYnV0IEkgdGhpbmsgaXQncyB0aGUgd3JvbmcgZGly
ZWN0aW9uLiAgQXMgaXQgc3RhbmRzLA0KPiA+IGVuY3J5cHRfbXByb3RlY3QoKSBpcyBhbiBpbmNv
bXBsZXRlIHZlcnNpb24gb2YgbXByb3RlY3QoKSAoc2luY2UgaXQncw0KPiA+IG1pc3NpbmcgdGhl
IHByb3RlY3Rpb24ga2V5IHN1cHBvcnQpLCBhbmQgaXQncyBhbHNvIGZ1bmN0aW9uYWxseSBqdXN0
DQo+ID4gTUFEVl9ET05UTkVFRC4gIEluIG90aGVyIHdvcmRzLCB0aGUgc29sZSB1c2VyLXZpc2li
bGUgZWZmZWN0IGFwcGVhcnMNCj4gPiB0byBiZSB0aGF0IHRoZSBleGlzdGluZyBwYWdlcyBhcmUg
Ymxvd24gYXdheS4gIFRoZSBmYWN0IHRoYXQgaXQNCj4gPiBjaGFuZ2VzIHRoZSBrZXkgaW4gdXNl
IGRvZXNuJ3Qgc2VlbSB0ZXJyaWJseSB1c2VmdWwsIHNpbmNlIGl0J3MNCj4gPiBhbm9ueW1vdXMg
bWVtb3J5LCBhbmQgdGhlIG1vc3Qgc2VjdXJlIGNob2ljZSBpcyB0byB1c2UgQ1BVLW1hbmFnZWQN
Cj4gPiBrZXlpbmcsIHdoaWNoIGFwcGVhcnMgdG8gYmUgdGhlIGRlZmF1bHQgYW55d2F5IG9uIFRN
RSBzeXN0ZW1zLiAgSXQNCj4gPiBhbHNvIGhhcyB0b3RhbGx5IHVuY2xlYXIgc2VtYW50aWNzIFdS
VCBzd2FwLCBhbmQsIG9mZiB0aGUgdG9wIG9mIG15DQo+ID4gaGVhZCwgaXQgbG9va3MgbGlrZSBp
dCBtYXkgaGF2ZSBzZXJpb3VzIGNhY2hlLWNvaGVyZW5jeSBpc3N1ZXMgYW5kDQo+ID4gbGlrZSBz
d2FwcGluZyB0aGUgcGFnZXMgbWlnaHQgY29ycnVwdCB0aGVtLCBib3RoIGJlY2F1c2UgdGhlcmUg
YXJlIG5vDQo+ID4gZmx1c2hlcyBhbmQgYmVjYXVzZSB0aGUgZGlyZWN0LW1hcCBhbGlhcyBsb29r
cyBsaWtlIGl0IHdpbGwgdXNlIHRoZQ0KPiA+IGRlZmF1bHQga2V5IGFuZCB0aGVyZWZvcmUgYXBw
ZWFyIHRvIGNvbnRhaW4gdGhlIHdyb25nIGRhdGEuDQo+ID4gDQo+ID4gSSB3b3VsZCBwcm9wb3Nl
IGEgdmVyeSBkaWZmZXJlbnQgZGlyZWN0aW9uOiBkb24ndCB0cnkgdG8gc3VwcG9ydCBNS1RNRQ0K
PiA+IGF0IGFsbCBmb3IgYW5vbnltb3VzIG1lbW9yeSwgYW5kIGluc3RlYWQgZmlndXJlIG91dCB0
aGUgaW1wb3J0YW50IHVzZQ0KPiA+IGNhc2VzIGFuZCBzdXBwb3J0IHRoZW0gZGlyZWN0bHkuICBU
aGUgdXNlIGNhc2VzIHRoYXQgSSBjYW4gdGhpbmsgb2YNCj4gPiBvZmYgdGhlIHRvcCBvZiBteSBo
ZWFkIGFyZToNCj4gPiANCj4gPiAxLiBwbWVtLiAgVGhpcyBzaG91bGQgcHJvYmFibHkgdXNlIGEg
dmVyeSBkaWZmZXJlbnQgQVBJLg0KPiA+IA0KPiA+IDIuIFNvbWUga2luZCBvZiBWTSBoYXJkZW5p
bmcsIHdoZXJlIGEgVk0ncyBtZW1vcnkgY2FuIGJlIHByb3RlY3RlZCBhDQo+ID4gbGl0dGxlIHRp
bnkgYml0IGZyb20gdGhlIG1haW4ga2VybmVsLiAgQnV0IEkgZG9uJ3Qgc2VlIHdoeSB0aGlzIGlz
IGFueQ0KPiA+IGJldHRlciB0aGFuIFhQTyAoZVhjbHVzaXZlIFBhZ2UtZnJhbWUgT3duZXJzaGlw
KSwgd2hpY2ggYnJpbmdzIHRvDQo+ID4gbWluZDoNCj4gDQo+IFdoYXQgaXMgdGhlIHRocmVhdCBt
b2RlbCBhbnl3YXkgZm9yIEFNRCBhbmQgSW50ZWwgdGVjaG5vbG9naWVzPw0KPiANCj4gRm9yIG1l
IGl0IGxvb2tzIGxpa2UgdGhhdCB5b3UgY2FuIHJlYWQsIHdyaXRlIGFuZCBldmVuIHJlcGxheSAN
Cj4gZW5jcnlwdGVkIHBhZ2VzIGJvdGggaW4gU01FIGFuZCBUTUUuIA0KDQpSaWdodC4gTmVpdGhl
ciBvZiB0aGVtIChpbmNsdWRpbmcgTUtUTUUpIHByZXZlbnRzIHJlcGxheSBhdHRhY2suIEJ1dCBp
biBteSB1bmRlcnN0YW5kaW5nIFNFViBkb2Vzbid0DQpwcmV2ZW50IHJlcGxheSBhdHRhY2sgZWl0
aGVyIHNpbmNlIGl0IGRvZXNuJ3QgaGF2ZSBpbnRlZ3JpdHkgcHJvdGVjdGlvbi4NCg0KVGhhbmtz
LA0KLUthaQ0K

Re: [RFC v2 00/13] Multi-Key Total Memory Encryption API (MKTME)

[Huang, Kai]

On Wed, 2018-12-05 at 22:19 +0000, Sakkinen, Jarkko wrote:
> On Tue, 2018-12-04 at 11:19 -0800, Andy Lutomirski wrote:
> > I'm not Thomas, but I think it's the wrong direction.  As it stands,
> > encrypt_mprotect() is an incomplete version of mprotect() (since it's
> > missing the protection key support), and it's also functionally just
> > MADV_DONTNEED.  In other words, the sole user-visible effect appears
> > to be that the existing pages are blown away.  The fact that it
> > changes the key in use doesn't seem terribly useful, since it's
> > anonymous memory, and the most secure choice is to use CPU-managed
> > keying, which appears to be the default anyway on TME systems.  It
> > also has totally unclear semantics WRT swap, and, off the top of my
> > head, it looks like it may have serious cache-coherency issues and
> > like swapping the pages might corrupt them, both because there are no
> > flushes and because the direct-map alias looks like it will use the
> > default key and therefore appear to contain the wrong data.
> > 
> > I would propose a very different direction: don't try to support MKTME
> > at all for anonymous memory, and instead figure out the important use
> > cases and support them directly.  The use cases that I can think of
> > off the top of my head are:
> > 
> > 1. pmem.  This should probably use a very different API.
> > 
> > 2. Some kind of VM hardening, where a VM's memory can be protected a
> > little tiny bit from the main kernel.  But I don't see why this is any
> > better than XPO (eXclusive Page-frame Ownership), which brings to
> > mind:
> 
> What is the threat model anyway for AMD and Intel technologies?
> 
> For me it looks like that you can read, write and even replay 
> encrypted pages both in SME and TME. 

Right. Neither of them (including MKTME) prevents replay attack. But in my understanding SEV doesn't
prevent replay attack either since it doesn't have integrity protection.

Thanks,
-Kai

Re: [RFC v2 00/13] Multi-Key Total Memory Encryption API (MKTME)

[Huang, Kai]

On Wed, 2018-12-05 at 22:19 +0000, Sakkinen, Jarkko wrote:
> On Tue, 2018-12-04 at 11:19 -0800, Andy Lutomirski wrote:
> > I'm not Thomas, but I think it's the wrong direction.  As it stands,
> > encrypt_mprotect() is an incomplete version of mprotect() (since it's
> > missing the protection key support), and it's also functionally just
> > MADV_DONTNEED.  In other words, the sole user-visible effect appears
> > to be that the existing pages are blown away.  The fact that it
> > changes the key in use doesn't seem terribly useful, since it's
> > anonymous memory, and the most secure choice is to use CPU-managed
> > keying, which appears to be the default anyway on TME systems.  It
> > also has totally unclear semantics WRT swap, and, off the top of my
> > head, it looks like it may have serious cache-coherency issues and
> > like swapping the pages might corrupt them, both because there are no
> > flushes and because the direct-map alias looks like it will use the
> > default key and therefore appear to contain the wrong data.
> > 
> > I would propose a very different direction: don't try to support MKTME
> > at all for anonymous memory, and instead figure out the important use
> > cases and support them directly.  The use cases that I can think of
> > off the top of my head are:
> > 
> > 1. pmem.  This should probably use a very different API.
> > 
> > 2. Some kind of VM hardening, where a VM's memory can be protected a
> > little tiny bit from the main kernel.  But I don't see why this is any
> > better than XPO (eXclusive Page-frame Ownership), which brings to
> > mind:
> 
> What is the threat model anyway for AMD and Intel technologies?
> 
> For me it looks like that you can read, write and even replay 
> encrypted pages both in SME and TME. 

Right. Neither of them (including MKTME) prevents replay attack. But in my understanding SEV doesn't
prevent replay attack either since it doesn't have integrity protection.

Thanks,
-Kai

Re: [RFC v2 12/13] keys/mktme: Save MKTME data if kernel cmdline parameter allows

[Huang, Kai]

T24gTW9uLCAyMDE4LTEyLTAzIGF0IDIzOjM5IC0wODAwLCBBbGlzb24gU2Nob2ZpZWxkIHdyb3Rl
Og0KPiBNS1RNRSAoTXVsdGktS2V5IFRvdGFsIE1lbW9yeSBFbmNyeXB0aW9uKSBrZXkgcGF5bG9h
ZHMgbWF5IGluY2x1ZGUNCj4gZGF0YSBlbmNyeXB0aW9uIGtleXMsIHR3ZWFrIGtleXMsIGFuZCBh
ZGRpdGlvbmFsIGVudHJvcHkgYml0cy4gVGhlc2UNCj4gYXJlIHVzZWQgdG8gcHJvZ3JhbSB0aGUg
TUtUTUUgZW5jcnlwdGlvbiBoYXJkd2FyZS4gQnkgZGVmYXVsdCwgdGhlDQo+IGtlcm5lbCBkZXN0
cm95cyB0aGlzIHBheWxvYWQgZGF0YSBvbmNlIHRoZSBoYXJkd2FyZSBpcyBwcm9ncmFtbWVkLg0K
PiANCj4gSG93ZXZlciwgaW4gb3JkZXIgdG8gZnVsbHkgc3VwcG9ydCBDUFUgSG90cGx1Zywgc2F2
aW5nIHRoZSBrZXkgZGF0YQ0KPiBiZWNvbWVzIGltcG9ydGFudC4gVGhlIE1LVE1FIEtleSBTZXJ2
aWNlIGNhbm5vdCBhbGxvdyBhIG5ldyBwaHlzaWNhbA0KPiBwYWNrYWdlIHRvIGNvbWUgb25saW5l
IHVubGVzcyBpdCBjYW4gcHJvZ3JhbSB0aGUgbmV3IHBhY2thZ2VzIEtleSBUYWJsZQ0KPiB0byBt
YXRjaCB0aGUgS2V5IFRhYmxlcyBvZiBhbGwgZXhpc3RpbmcgcGh5c2ljYWwgcGFja2FnZXMuDQo+
IA0KPiBXaXRoIENQVSBnZW5lcmF0ZWQga2V5cyAoYS5rLmEuIHJhbmRvbSBrZXlzIG9yIGVwaGVt
ZXJhbCBrZXlzKSB0aGUNCj4gc2F2aW5nIG9mIHVzZXIga2V5IGRhdGEgaXMgbm90IGFuIGlzc3Vl
LiBUaGUga2VybmVsIGFuZCBNS1RNRSBoYXJkd2FyZQ0KPiBjYW4gZ2VuZXJhdGUgc3Ryb25nIGVu
Y3J5cHRpb24ga2V5cyB3aXRob3V0IHJlY2FsbGluZyBhbnkgdXNlciBzdXBwbGllZA0KPiBkYXRh
Lg0KPiANCj4gV2l0aCBVU0VSIGRpcmVjdGVkIGtleXMgKGEuay5hLiB1c2VyIHR5cGUpIHNhdmlu
ZyB0aGUga2V5IHByb2dyYW1taW5nDQo+IGRhdGEgKGRhdGEgYW5kIHR3ZWFrIGtleSkgYmVjb21l
cyBhbiBpc3N1ZS4gVGhlIGRhdGEgYW5kIHR3ZWFrIGtleXMNCj4gYXJlIHJlcXVpcmVkIHRvIHBy
b2dyYW0gdGhvc2Uga2V5cyBvbiBhIG5ldyBwaHlzaWNhbCBwYWNrYWdlLg0KPiANCj4gSW4gcHJl
cGFyYXRpb24gZm9yIGFkZGluZyBDUFUgaG90cGx1ZyBzdXBwb3J0Og0KPiANCj4gICAgQWRkIGFu
ICdta3RtZV92YXVsdCcgd2hlcmUga2V5IGRhdGEgaXMgc3RvcmVkLg0KPiANCj4gICAgQWRkICdt
a3RtZV9zYXZla2V5cycga2VybmVsIGNvbW1hbmQgbGluZSBwYXJhbWV0ZXIgdGhhdCBkaXJlY3Rz
DQo+ICAgIHdoYXQga2V5IGRhdGEgY2FuIGJlIHN0b3JlZC4gSWYgaXQgaXMgbm90IHNldCwga2Vy
bmVsIGRvZXMgbm90DQo+ICAgIHN0b3JlIHVzZXJzIGRhdGEga2V5IG9yIHR3ZWFrIGtleS4NCj4g
DQo+ICAgIEFkZCAnbWt0bWVfYml0bWFwX3VzZXJfdHlwZScgdG8gdHJhY2sgd2hlbiBVU0VSIHR5
cGUga2V5cyBhcmUgaW4NCj4gICAgdXNlLiBJZiBubyBVU0VSIHR5cGUga2V5cyBhcmUgY3VycmVu
dGx5IGluIHVzZSwgYSBwaHlzaWNhbCBwYWNrYWdlDQo+ICAgIG1heSBiZSBicm91Z2h0IG9ubGlu
ZSwgZGVzcGl0ZSB0aGUgYWJzZW5jZSBvZiAnbWt0bWVfc2F2ZWtleXMnLg0KDQpPdmVyYWxsLCBJ
IGFtIG5vdCBzdXJlIHdoZXRoZXIgc2F2aW5nIGtleSBpcyBnb29kIGlkZWEsIHNpbmNlIGl0IGJy
ZWFrcyBjb2xkYm9vdCBhdHRhY2sgSU1ITy4gV2UNCm5lZWQgdG8gdHJhZGVvZmYgYmV0d2VlbiBz
dXBwb3J0aW5nIENQVSBob3RwbHVnIGFuZCBzZWN1cml0eS4gSSBhbSBub3Qgc3VyZSB3aGV0aGVy
IHN1cHBvcnRpbmcgQ1BVDQpob3RwbHVnIGlzIHRoYXQgaW1wb3J0YW50LCBzaW5jZSBmb3Igc29t
ZSBvdGhlciBmZWF0dXJlcyBzdWNoIGFzIFNHWCwgd2UgZG9uJ3Qgc3VwcG9ydCBDUFUgaG90cGx1
Zw0KYW55d2F5Lg0KDQpBbHRlcm5hdGl2ZWx5LCB3ZSBjYW4gY2hvb3NlIHRvIHVzZSBwZXItc29j
a2V0IGtleUlELCBidXQgbm90IHRvIHByb2dyYW0ga2V5SUQgZ2xvYmFsbHkgYWNyb3NzIGFsbA0K
c29ja2V0cywgc28geW91IGRvbid0IGhhdmUgdG8gc2F2ZSBrZXkgd2hpbGUgc3RpbGwgc3VwcG9y
dGluZyBDUFUgaG90cGx1Zy4NCg0KVGhhbmtzLA0KLUthaQ=

Re: [RFC v2 12/13] keys/mktme: Save MKTME data if kernel cmdline parameter allows

[Huang, Kai]

On Mon, 2018-12-03 at 23:39 -0800, Alison Schofield wrote:
> MKTME (Multi-Key Total Memory Encryption) key payloads may include
> data encryption keys, tweak keys, and additional entropy bits. These
> are used to program the MKTME encryption hardware. By default, the
> kernel destroys this payload data once the hardware is programmed.
> 
> However, in order to fully support CPU Hotplug, saving the key data
> becomes important. The MKTME Key Service cannot allow a new physical
> package to come online unless it can program the new packages Key Table
> to match the Key Tables of all existing physical packages.
> 
> With CPU generated keys (a.k.a. random keys or ephemeral keys) the
> saving of user key data is not an issue. The kernel and MKTME hardware
> can generate strong encryption keys without recalling any user supplied
> data.
> 
> With USER directed keys (a.k.a. user type) saving the key programming
> data (data and tweak key) becomes an issue. The data and tweak keys
> are required to program those keys on a new physical package.
> 
> In preparation for adding CPU hotplug support:
> 
>    Add an 'mktme_vault' where key data is stored.
> 
>    Add 'mktme_savekeys' kernel command line parameter that directs
>    what key data can be stored. If it is not set, kernel does not
>    store users data key or tweak key.
> 
>    Add 'mktme_bitmap_user_type' to track when USER type keys are in
>    use. If no USER type keys are currently in use, a physical package
>    may be brought online, despite the absence of 'mktme_savekeys'.

Overall, I am not sure whether saving key is good idea, since it breaks coldboot attack IMHO. We
need to tradeoff between supporting CPU hotplug and security. I am not sure whether supporting CPU
hotplug is that important, since for some other features such as SGX, we don't support CPU hotplug
anyway.

Alternatively, we can choose to use per-socket keyID, but not to program keyID globally across all
sockets, so you don't have to save key while still supporting CPU hotplug.

Thanks,
-Kai

Re: [RFC v2 12/13] keys/mktme: Save MKTME data if kernel cmdline parameter allows

[Huang, Kai]

On Mon, 2018-12-03 at 23:39 -0800, Alison Schofield wrote:
> MKTME (Multi-Key Total Memory Encryption) key payloads may include
> data encryption keys, tweak keys, and additional entropy bits. These
> are used to program the MKTME encryption hardware. By default, the
> kernel destroys this payload data once the hardware is programmed.
> 
> However, in order to fully support CPU Hotplug, saving the key data
> becomes important. The MKTME Key Service cannot allow a new physical
> package to come online unless it can program the new packages Key Table
> to match the Key Tables of all existing physical packages.
> 
> With CPU generated keys (a.k.a. random keys or ephemeral keys) the
> saving of user key data is not an issue. The kernel and MKTME hardware
> can generate strong encryption keys without recalling any user supplied
> data.
> 
> With USER directed keys (a.k.a. user type) saving the key programming
> data (data and tweak key) becomes an issue. The data and tweak keys
> are required to program those keys on a new physical package.
> 
> In preparation for adding CPU hotplug support:
> 
>    Add an 'mktme_vault' where key data is stored.
> 
>    Add 'mktme_savekeys' kernel command line parameter that directs
>    what key data can be stored. If it is not set, kernel does not
>    store users data key or tweak key.
> 
>    Add 'mktme_bitmap_user_type' to track when USER type keys are in
>    use. If no USER type keys are currently in use, a physical package
>    may be brought online, despite the absence of 'mktme_savekeys'.

Overall, I am not sure whether saving key is good idea, since it breaks coldboot attack IMHO. We
need to tradeoff between supporting CPU hotplug and security. I am not sure whether supporting CPU
hotplug is that important, since for some other features such as SGX, we don't support CPU hotplug
anyway.

Alternatively, we can choose to use per-socket keyID, but not to program keyID globally across all
sockets, so you don't have to save key while still supporting CPU hotplug.

Thanks,
-Kai

Re: [RFC v2 12/13] keys/mktme: Save MKTME data if kernel cmdline parameter allows

[Alison Schofield]

On Thu, Dec 06, 2018 at 06:14:03PM -0800, Huang, Kai wrote:
> On Mon, 2018-12-03 at 23:39 -0800, Alison Schofield wrote:

8< ------------

> >    Add an 'mktme_vault' where key data is stored.
> > 
> >    Add 'mktme_savekeys' kernel command line parameter that directs
> >    what key data can be stored. If it is not set, kernel does not
> >    store users data key or tweak key.
> > 
> >    Add 'mktme_bitmap_user_type' to track when USER type keys are in
> >    use. If no USER type keys are currently in use, a physical package
> >    may be brought online, despite the absence of 'mktme_savekeys'.
> 
> Overall, I am not sure whether saving key is good idea, since it breaks coldboot attack IMHO. We
> need to tradeoff between supporting CPU hotplug and security. I am not sure whether supporting CPU
> hotplug is that important, since for some other features such as SGX, we don't support CPU hotplug
> anyway.

Yes, saving the key data exposes it in a cold boot attack.

Here we have 2 conflicting requirements. Do not save the data and
support CPU hotplug. I don't think CPU hotplug support is budging!
If the risk of offering the mktme_savekeys option is too dangerous,
then we can't have user type keys.
Is mktme_savekeys options too risky to offer?
(That's not just a question for you Kai ;). I'll pursue too.)
> 
> Alternatively, we can choose to use per-socket keyID, but not to program keyID globally across all
> sockets, so you don't have to save key while still supporting CPU hotplug.

An alternative, with a lot of impact to the core linux support for
MKTME.  I don't think we need to go there. I'll leave this thought for
a Kirill or Dave to perhaps elaborate on. 

Alison 
> 
> Thanks,
> -Kai

Re: [RFC v2 12/13] keys/mktme: Save MKTME data if kernel cmdline parameter allows

[Alison Schofield]

On Thu, Dec 06, 2018 at 06:14:03PM -0800, Huang, Kai wrote:
> On Mon, 2018-12-03 at 23:39 -0800, Alison Schofield wrote:

8< ------------

> >    Add an 'mktme_vault' where key data is stored.
> > 
> >    Add 'mktme_savekeys' kernel command line parameter that directs
> >    what key data can be stored. If it is not set, kernel does not
> >    store users data key or tweak key.
> > 
> >    Add 'mktme_bitmap_user_type' to track when USER type keys are in
> >    use. If no USER type keys are currently in use, a physical package
> >    may be brought online, despite the absence of 'mktme_savekeys'.
> 
> Overall, I am not sure whether saving key is good idea, since it breaks coldboot attack IMHO. We
> need to tradeoff between supporting CPU hotplug and security. I am not sure whether supporting CPU
> hotplug is that important, since for some other features such as SGX, we don't support CPU hotplug
> anyway.

Yes, saving the key data exposes it in a cold boot attack.

Here we have 2 conflicting requirements. Do not save the data and
support CPU hotplug. I don't think CPU hotplug support is budging!
If the risk of offering the mktme_savekeys option is too dangerous,
then we can't have user type keys.
Is mktme_savekeys options too risky to offer?
(That's not just a question for you Kai ;). I'll pursue too.)
> 
> Alternatively, we can choose to use per-socket keyID, but not to program keyID globally across all
> sockets, so you don't have to save key while still supporting CPU hotplug.

An alternative, with a lot of impact to the core linux support for
MKTME.  I don't think we need to go there. I'll leave this thought for
a Kirill or Dave to perhaps elaborate on. 

Alison 
> 
> Thanks,
> -Kai

Re: [RFC v2 12/13] keys/mktme: Save MKTME data if kernel cmdline parameter allows

[Alison Schofield]

On Thu, Dec 06, 2018 at 06:14:03PM -0800, Huang, Kai wrote:
> On Mon, 2018-12-03 at 23:39 -0800, Alison Schofield wrote:

8< ------------

> >    Add an 'mktme_vault' where key data is stored.
> > 
> >    Add 'mktme_savekeys' kernel command line parameter that directs
> >    what key data can be stored. If it is not set, kernel does not
> >    store users data key or tweak key.
> > 
> >    Add 'mktme_bitmap_user_type' to track when USER type keys are in
> >    use. If no USER type keys are currently in use, a physical package
> >    may be brought online, despite the absence of 'mktme_savekeys'.
> 
> Overall, I am not sure whether saving key is good idea, since it breaks coldboot attack IMHO. We
> need to tradeoff between supporting CPU hotplug and security. I am not sure whether supporting CPU
> hotplug is that important, since for some other features such as SGX, we don't support CPU hotplug
> anyway.

Yes, saving the key data exposes it in a cold boot attack.

Here we have 2 conflicting requirements. Do not save the data and
support CPU hotplug. I don't think CPU hotplug support is budging!
If the risk of offering the mktme_savekeys option is too dangerous,
then we can't have user type keys.
Is mktme_savekeys options too risky to offer?
(That's not just a question for you Kai ;). I'll pursue too.)
> 
> Alternatively, we can choose to use per-socket keyID, but not to program keyID globally across all
> sockets, so you don't have to save key while still supporting CPU hotplug.

An alternative, with a lot of impact to the core linux support for
MKTME.  I don't think we need to go there. I'll leave this thought for
a Kirill or Dave to perhaps elaborate on. 

Alison 
> 
> Thanks,
> -Kai

Re: [RFC v2 00/13] Multi-Key Total Memory Encryption API (MKTME)

[Dave Hansen]

On 12/6/18 5:55 PM, Huang, Kai wrote:
> I think one usage of user-specified key is for NVDIMM, since CPU key
> will be gone after machine reboot, therefore if NVDIMM is encrypted
> by CPU key we are not able to retrieve it once shutdown/reboot, etc.

I think we all agree that the NVDIMM uses are really useful.

But, these patches don't implement that.  So, if NVDIMMs are the only
reasonable use case, we shouldn't merge these patches until we add
NVDIMM support.

Re: [RFC v2 00/13] Multi-Key Total Memory Encryption API (MKTME)

[Dave Hansen]

On 12/6/18 5:55 PM, Huang, Kai wrote:
> I think one usage of user-specified key is for NVDIMM, since CPU key
> will be gone after machine reboot, therefore if NVDIMM is encrypted
> by CPU key we are not able to retrieve it once shutdown/reboot, etc.

I think we all agree that the NVDIMM uses are really useful.

But, these patches don't implement that.  So, if NVDIMMs are the only
reasonable use case, we shouldn't merge these patches until we add
NVDIMM support.

Re: [RFC v2 00/13] Multi-Key Total Memory Encryption API (MKTME)

[Dave Hansen]

On 12/6/18 5:55 PM, Huang, Kai wrote:
> I think one usage of user-specified key is for NVDIMM, since CPU key
> will be gone after machine reboot, therefore if NVDIMM is encrypted
> by CPU key we are not able to retrieve it once shutdown/reboot, etc.

I think we all agree that the NVDIMM uses are really useful.

But, these patches don't implement that.  So, if NVDIMMs are the only
reasonable use case, we shouldn't merge these patches until we add
NVDIMM support.

Re: [RFC v2 12/13] keys/mktme: Save MKTME data if kernel cmdline parameter allows

[Jarkko Sakkinen]

On Thu, Dec 06, 2018 at 06:14:03PM -0800, Huang, Kai wrote:
> On Mon, 2018-12-03 at 23:39 -0800, Alison Schofield wrote:
> > MKTME (Multi-Key Total Memory Encryption) key payloads may include
> > data encryption keys, tweak keys, and additional entropy bits. These
> > are used to program the MKTME encryption hardware. By default, the
> > kernel destroys this payload data once the hardware is programmed.
> > 
> > However, in order to fully support CPU Hotplug, saving the key data
> > becomes important. The MKTME Key Service cannot allow a new physical
> > package to come online unless it can program the new packages Key Table
> > to match the Key Tables of all existing physical packages.
> > 
> > With CPU generated keys (a.k.a. random keys or ephemeral keys) the
> > saving of user key data is not an issue. The kernel and MKTME hardware
> > can generate strong encryption keys without recalling any user supplied
> > data.
> > 
> > With USER directed keys (a.k.a. user type) saving the key programming
> > data (data and tweak key) becomes an issue. The data and tweak keys
> > are required to program those keys on a new physical package.
> > 
> > In preparation for adding CPU hotplug support:
> > 
> >    Add an 'mktme_vault' where key data is stored.
> > 
> >    Add 'mktme_savekeys' kernel command line parameter that directs
> >    what key data can be stored. If it is not set, kernel does not
> >    store users data key or tweak key.
> > 
> >    Add 'mktme_bitmap_user_type' to track when USER type keys are in
> >    use. If no USER type keys are currently in use, a physical package
> >    may be brought online, despite the absence of 'mktme_savekeys'.
> 
> Overall, I am not sure whether saving key is good idea, since it
> breaks coldboot attack IMHO. We need to tradeoff between supporting
> CPU hotplug and security. I am not sure whether supporting CPU hotplug
> is that important, since for some other features such as SGX, we don't
> support CPU hotplug anyway.

What is the application for saving the key anyway?

With my current knowledge, I'm not even sure what is the application
for user provided keys.

/Jarkko

Re: [RFC v2 12/13] keys/mktme: Save MKTME data if kernel cmdline parameter allows

[Jarkko Sakkinen]

On Thu, Dec 06, 2018 at 06:14:03PM -0800, Huang, Kai wrote:
> On Mon, 2018-12-03 at 23:39 -0800, Alison Schofield wrote:
> > MKTME (Multi-Key Total Memory Encryption) key payloads may include
> > data encryption keys, tweak keys, and additional entropy bits. These
> > are used to program the MKTME encryption hardware. By default, the
> > kernel destroys this payload data once the hardware is programmed.
> > 
> > However, in order to fully support CPU Hotplug, saving the key data
> > becomes important. The MKTME Key Service cannot allow a new physical
> > package to come online unless it can program the new packages Key Table
> > to match the Key Tables of all existing physical packages.
> > 
> > With CPU generated keys (a.k.a. random keys or ephemeral keys) the
> > saving of user key data is not an issue. The kernel and MKTME hardware
> > can generate strong encryption keys without recalling any user supplied
> > data.
> > 
> > With USER directed keys (a.k.a. user type) saving the key programming
> > data (data and tweak key) becomes an issue. The data and tweak keys
> > are required to program those keys on a new physical package.
> > 
> > In preparation for adding CPU hotplug support:
> > 
> >    Add an 'mktme_vault' where key data is stored.
> > 
> >    Add 'mktme_savekeys' kernel command line parameter that directs
> >    what key data can be stored. If it is not set, kernel does not
> >    store users data key or tweak key.
> > 
> >    Add 'mktme_bitmap_user_type' to track when USER type keys are in
> >    use. If no USER type keys are currently in use, a physical package
> >    may be brought online, despite the absence of 'mktme_savekeys'.
> 
> Overall, I am not sure whether saving key is good idea, since it
> breaks coldboot attack IMHO. We need to tradeoff between supporting
> CPU hotplug and security. I am not sure whether supporting CPU hotplug
> is that important, since for some other features such as SGX, we don't
> support CPU hotplug anyway.

What is the application for saving the key anyway?

With my current knowledge, I'm not even sure what is the application
for user provided keys.

/Jarkko

Re: [RFC v2 12/13] keys/mktme: Save MKTME data if kernel cmdline parameter allows

[Jarkko Sakkinen]

On Thu, Dec 06, 2018 at 06:14:03PM -0800, Huang, Kai wrote:
> On Mon, 2018-12-03 at 23:39 -0800, Alison Schofield wrote:
> > MKTME (Multi-Key Total Memory Encryption) key payloads may include
> > data encryption keys, tweak keys, and additional entropy bits. These
> > are used to program the MKTME encryption hardware. By default, the
> > kernel destroys this payload data once the hardware is programmed.
> > 
> > However, in order to fully support CPU Hotplug, saving the key data
> > becomes important. The MKTME Key Service cannot allow a new physical
> > package to come online unless it can program the new packages Key Table
> > to match the Key Tables of all existing physical packages.
> > 
> > With CPU generated keys (a.k.a. random keys or ephemeral keys) the
> > saving of user key data is not an issue. The kernel and MKTME hardware
> > can generate strong encryption keys without recalling any user supplied
> > data.
> > 
> > With USER directed keys (a.k.a. user type) saving the key programming
> > data (data and tweak key) becomes an issue. The data and tweak keys
> > are required to program those keys on a new physical package.
> > 
> > In preparation for adding CPU hotplug support:
> > 
> >    Add an 'mktme_vault' where key data is stored.
> > 
> >    Add 'mktme_savekeys' kernel command line parameter that directs
> >    what key data can be stored. If it is not set, kernel does not
> >    store users data key or tweak key.
> > 
> >    Add 'mktme_bitmap_user_type' to track when USER type keys are in
> >    use. If no USER type keys are currently in use, a physical package
> >    may be brought online, despite the absence of 'mktme_savekeys'.
> 
> Overall, I am not sure whether saving key is good idea, since it
> breaks coldboot attack IMHO. We need to tradeoff between supporting
> CPU hotplug and security. I am not sure whether supporting CPU hotplug
> is that important, since for some other features such as SGX, we don't
> support CPU hotplug anyway.

What is the application for saving the key anyway?

With my current knowledge, I'm not even sure what is the application
for user provided keys.

/Jarkko

Re: [RFC v2 12/13] keys/mktme: Save MKTME data if kernel cmdline parameter allows

[Jarkko Sakkinen]

On Thu, Dec 06, 2018 at 10:39:18PM -0800, Jarkko Sakkinen wrote:
> On Thu, Dec 06, 2018 at 06:14:03PM -0800, Huang, Kai wrote:
> > On Mon, 2018-12-03 at 23:39 -0800, Alison Schofield wrote:
> > > MKTME (Multi-Key Total Memory Encryption) key payloads may include
> > > data encryption keys, tweak keys, and additional entropy bits. These
> > > are used to program the MKTME encryption hardware. By default, the
> > > kernel destroys this payload data once the hardware is programmed.
> > > 
> > > However, in order to fully support CPU Hotplug, saving the key data
> > > becomes important. The MKTME Key Service cannot allow a new physical
> > > package to come online unless it can program the new packages Key Table
> > > to match the Key Tables of all existing physical packages.
> > > 
> > > With CPU generated keys (a.k.a. random keys or ephemeral keys) the
> > > saving of user key data is not an issue. The kernel and MKTME hardware
> > > can generate strong encryption keys without recalling any user supplied
> > > data.
> > > 
> > > With USER directed keys (a.k.a. user type) saving the key programming
> > > data (data and tweak key) becomes an issue. The data and tweak keys
> > > are required to program those keys on a new physical package.
> > > 
> > > In preparation for adding CPU hotplug support:
> > > 
> > >    Add an 'mktme_vault' where key data is stored.
> > > 
> > >    Add 'mktme_savekeys' kernel command line parameter that directs
> > >    what key data can be stored. If it is not set, kernel does not
> > >    store users data key or tweak key.
> > > 
> > >    Add 'mktme_bitmap_user_type' to track when USER type keys are in
> > >    use. If no USER type keys are currently in use, a physical package
> > >    may be brought online, despite the absence of 'mktme_savekeys'.
> > 
> > Overall, I am not sure whether saving key is good idea, since it
> > breaks coldboot attack IMHO. We need to tradeoff between supporting
> > CPU hotplug and security. I am not sure whether supporting CPU hotplug
> > is that important, since for some other features such as SGX, we don't
> > support CPU hotplug anyway.
> 
> What is the application for saving the key anyway?
> 
> With my current knowledge, I'm not even sure what is the application
> for user provided keys.

Ugh, right of course, you need to save the key in order to support
hotplug.

Cold boot is like the main security use case for this (probably would
be worth to mention this in the documentation).

/Jarkko

Re: [RFC v2 12/13] keys/mktme: Save MKTME data if kernel cmdline parameter allows

[Jarkko Sakkinen]

On Thu, Dec 06, 2018 at 10:39:18PM -0800, Jarkko Sakkinen wrote:
> On Thu, Dec 06, 2018 at 06:14:03PM -0800, Huang, Kai wrote:
> > On Mon, 2018-12-03 at 23:39 -0800, Alison Schofield wrote:
> > > MKTME (Multi-Key Total Memory Encryption) key payloads may include
> > > data encryption keys, tweak keys, and additional entropy bits. These
> > > are used to program the MKTME encryption hardware. By default, the
> > > kernel destroys this payload data once the hardware is programmed.
> > > 
> > > However, in order to fully support CPU Hotplug, saving the key data
> > > becomes important. The MKTME Key Service cannot allow a new physical
> > > package to come online unless it can program the new packages Key Table
> > > to match the Key Tables of all existing physical packages.
> > > 
> > > With CPU generated keys (a.k.a. random keys or ephemeral keys) the
> > > saving of user key data is not an issue. The kernel and MKTME hardware
> > > can generate strong encryption keys without recalling any user supplied
> > > data.
> > > 
> > > With USER directed keys (a.k.a. user type) saving the key programming
> > > data (data and tweak key) becomes an issue. The data and tweak keys
> > > are required to program those keys on a new physical package.
> > > 
> > > In preparation for adding CPU hotplug support:
> > > 
> > >    Add an 'mktme_vault' where key data is stored.
> > > 
> > >    Add 'mktme_savekeys' kernel command line parameter that directs
> > >    what key data can be stored. If it is not set, kernel does not
> > >    store users data key or tweak key.
> > > 
> > >    Add 'mktme_bitmap_user_type' to track when USER type keys are in
> > >    use. If no USER type keys are currently in use, a physical package
> > >    may be brought online, despite the absence of 'mktme_savekeys'.
> > 
> > Overall, I am not sure whether saving key is good idea, since it
> > breaks coldboot attack IMHO. We need to tradeoff between supporting
> > CPU hotplug and security. I am not sure whether supporting CPU hotplug
> > is that important, since for some other features such as SGX, we don't
> > support CPU hotplug anyway.
> 
> What is the application for saving the key anyway?
> 
> With my current knowledge, I'm not even sure what is the application
> for user provided keys.

Ugh, right of course, you need to save the key in order to support
hotplug.

Cold boot is like the main security use case for this (probably would
be worth to mention this in the documentation).

/Jarkko

Re: [RFC v2 12/13] keys/mktme: Save MKTME data if kernel cmdline parameter allows

[Jarkko Sakkinen]

On Thu, Dec 06, 2018 at 10:39:18PM -0800, Jarkko Sakkinen wrote:
> On Thu, Dec 06, 2018 at 06:14:03PM -0800, Huang, Kai wrote:
> > On Mon, 2018-12-03 at 23:39 -0800, Alison Schofield wrote:
> > > MKTME (Multi-Key Total Memory Encryption) key payloads may include
> > > data encryption keys, tweak keys, and additional entropy bits. These
> > > are used to program the MKTME encryption hardware. By default, the
> > > kernel destroys this payload data once the hardware is programmed.
> > > 
> > > However, in order to fully support CPU Hotplug, saving the key data
> > > becomes important. The MKTME Key Service cannot allow a new physical
> > > package to come online unless it can program the new packages Key Table
> > > to match the Key Tables of all existing physical packages.
> > > 
> > > With CPU generated keys (a.k.a. random keys or ephemeral keys) the
> > > saving of user key data is not an issue. The kernel and MKTME hardware
> > > can generate strong encryption keys without recalling any user supplied
> > > data.
> > > 
> > > With USER directed keys (a.k.a. user type) saving the key programming
> > > data (data and tweak key) becomes an issue. The data and tweak keys
> > > are required to program those keys on a new physical package.
> > > 
> > > In preparation for adding CPU hotplug support:
> > > 
> > >    Add an 'mktme_vault' where key data is stored.
> > > 
> > >    Add 'mktme_savekeys' kernel command line parameter that directs
> > >    what key data can be stored. If it is not set, kernel does not
> > >    store users data key or tweak key.
> > > 
> > >    Add 'mktme_bitmap_user_type' to track when USER type keys are in
> > >    use. If no USER type keys are currently in use, a physical package
> > >    may be brought online, despite the absence of 'mktme_savekeys'.
> > 
> > Overall, I am not sure whether saving key is good idea, since it
> > breaks coldboot attack IMHO. We need to tradeoff between supporting
> > CPU hotplug and security. I am not sure whether supporting CPU hotplug
> > is that important, since for some other features such as SGX, we don't
> > support CPU hotplug anyway.
> 
> What is the application for saving the key anyway?
> 
> With my current knowledge, I'm not even sure what is the application
> for user provided keys.

Ugh, right of course, you need to save the key in order to support
hotplug.

Cold boot is like the main security use case for this (probably would
be worth to mention this in the documentation).

/Jarkko

Re: [RFC v2 00/13] Multi-Key Total Memory Encryption API (MKTME)

[Jarkko Sakkinen]

On Thu, Dec 06, 2018 at 06:05:50PM -0800, Huang, Kai wrote:
> On Wed, 2018-12-05 at 22:19 +0000, Sakkinen, Jarkko wrote:
> > On Tue, 2018-12-04 at 11:19 -0800, Andy Lutomirski wrote:
> > > I'm not Thomas, but I think it's the wrong direction.  As it stands,
> > > encrypt_mprotect() is an incomplete version of mprotect() (since it's
> > > missing the protection key support), and it's also functionally just
> > > MADV_DONTNEED.  In other words, the sole user-visible effect appears
> > > to be that the existing pages are blown away.  The fact that it
> > > changes the key in use doesn't seem terribly useful, since it's
> > > anonymous memory, and the most secure choice is to use CPU-managed
> > > keying, which appears to be the default anyway on TME systems.  It
> > > also has totally unclear semantics WRT swap, and, off the top of my
> > > head, it looks like it may have serious cache-coherency issues and
> > > like swapping the pages might corrupt them, both because there are no
> > > flushes and because the direct-map alias looks like it will use the
> > > default key and therefore appear to contain the wrong data.
> > > 
> > > I would propose a very different direction: don't try to support MKTME
> > > at all for anonymous memory, and instead figure out the important use
> > > cases and support them directly.  The use cases that I can think of
> > > off the top of my head are:
> > > 
> > > 1. pmem.  This should probably use a very different API.
> > > 
> > > 2. Some kind of VM hardening, where a VM's memory can be protected a
> > > little tiny bit from the main kernel.  But I don't see why this is any
> > > better than XPO (eXclusive Page-frame Ownership), which brings to
> > > mind:
> > 
> > What is the threat model anyway for AMD and Intel technologies?
> > 
> > For me it looks like that you can read, write and even replay 
> > encrypted pages both in SME and TME. 
> 
> Right. Neither of them (including MKTME) prevents replay attack. But
> in my understanding SEV doesn't prevent replay attack either since it
> doesn't have integrity protection.

Yep, it doesn't :-) That's why I've been wondering after seeing
presentations concerning SME and SVE what they are good for.

Cold boot attacks are definitely at least something where these
techs can help...

/Jarkko

Re: [RFC v2 00/13] Multi-Key Total Memory Encryption API (MKTME)

[Jarkko Sakkinen]

On Thu, Dec 06, 2018 at 06:05:50PM -0800, Huang, Kai wrote:
> On Wed, 2018-12-05 at 22:19 +0000, Sakkinen, Jarkko wrote:
> > On Tue, 2018-12-04 at 11:19 -0800, Andy Lutomirski wrote:
> > > I'm not Thomas, but I think it's the wrong direction.  As it stands,
> > > encrypt_mprotect() is an incomplete version of mprotect() (since it's
> > > missing the protection key support), and it's also functionally just
> > > MADV_DONTNEED.  In other words, the sole user-visible effect appears
> > > to be that the existing pages are blown away.  The fact that it
> > > changes the key in use doesn't seem terribly useful, since it's
> > > anonymous memory, and the most secure choice is to use CPU-managed
> > > keying, which appears to be the default anyway on TME systems.  It
> > > also has totally unclear semantics WRT swap, and, off the top of my
> > > head, it looks like it may have serious cache-coherency issues and
> > > like swapping the pages might corrupt them, both because there are no
> > > flushes and because the direct-map alias looks like it will use the
> > > default key and therefore appear to contain the wrong data.
> > > 
> > > I would propose a very different direction: don't try to support MKTME
> > > at all for anonymous memory, and instead figure out the important use
> > > cases and support them directly.  The use cases that I can think of
> > > off the top of my head are:
> > > 
> > > 1. pmem.  This should probably use a very different API.
> > > 
> > > 2. Some kind of VM hardening, where a VM's memory can be protected a
> > > little tiny bit from the main kernel.  But I don't see why this is any
> > > better than XPO (eXclusive Page-frame Ownership), which brings to
> > > mind:
> > 
> > What is the threat model anyway for AMD and Intel technologies?
> > 
> > For me it looks like that you can read, write and even replay 
> > encrypted pages both in SME and TME. 
> 
> Right. Neither of them (including MKTME) prevents replay attack. But
> in my understanding SEV doesn't prevent replay attack either since it
> doesn't have integrity protection.

Yep, it doesn't :-) That's why I've been wondering after seeing
presentations concerning SME and SVE what they are good for.

Cold boot attacks are definitely at least something where these
techs can help...

/Jarkko

Re: [RFC v2 00/13] Multi-Key Total Memory Encryption API (MKTME)

[Jarkko Sakkinen]

On Thu, Dec 06, 2018 at 06:05:50PM -0800, Huang, Kai wrote:
> On Wed, 2018-12-05 at 22:19 +0000, Sakkinen, Jarkko wrote:
> > On Tue, 2018-12-04 at 11:19 -0800, Andy Lutomirski wrote:
> > > I'm not Thomas, but I think it's the wrong direction.  As it stands,
> > > encrypt_mprotect() is an incomplete version of mprotect() (since it's
> > > missing the protection key support), and it's also functionally just
> > > MADV_DONTNEED.  In other words, the sole user-visible effect appears
> > > to be that the existing pages are blown away.  The fact that it
> > > changes the key in use doesn't seem terribly useful, since it's
> > > anonymous memory, and the most secure choice is to use CPU-managed
> > > keying, which appears to be the default anyway on TME systems.  It
> > > also has totally unclear semantics WRT swap, and, off the top of my
> > > head, it looks like it may have serious cache-coherency issues and
> > > like swapping the pages might corrupt them, both because there are no
> > > flushes and because the direct-map alias looks like it will use the
> > > default key and therefore appear to contain the wrong data.
> > > 
> > > I would propose a very different direction: don't try to support MKTME
> > > at all for anonymous memory, and instead figure out the important use
> > > cases and support them directly.  The use cases that I can think of
> > > off the top of my head are:
> > > 
> > > 1. pmem.  This should probably use a very different API.
> > > 
> > > 2. Some kind of VM hardening, where a VM's memory can be protected a
> > > little tiny bit from the main kernel.  But I don't see why this is any
> > > better than XPO (eXclusive Page-frame Ownership), which brings to
> > > mind:
> > 
> > What is the threat model anyway for AMD and Intel technologies?
> > 
> > For me it looks like that you can read, write and even replay 
> > encrypted pages both in SME and TME. 
> 
> Right. Neither of them (including MKTME) prevents replay attack. But
> in my understanding SEV doesn't prevent replay attack either since it
> doesn't have integrity protection.

Yep, it doesn't :-) That's why I've been wondering after seeing
presentations concerning SME and SVE what they are good for.

Cold boot attacks are definitely at least something where these
techs can help...

/Jarkko

Re: [RFC v2 00/13] Multi-Key Total Memory Encryption API (MKTME)

[Huang, Kai]

T24gVGh1LCAyMDE4LTEyLTA2IGF0IDA2OjU5IC0wODAwLCBEYXZlIEhhbnNlbiB3cm90ZToNCj4g
T24gMTIvNi8xOCAzOjIyIEFNLCBLaXJpbGwgQS4gU2h1dGVtb3Ygd3JvdGU6DQo+ID4gPiBXaGVu
IHlvdSBzYXkgImRpc2FibGUgZW5jcnlwdGlvbiB0byBhIHBhZ2UiIGRvZXMgdGhlIGVuY3J5cHRp
b24gZ2V0DQo+ID4gPiBhY3R1YWxseSBkaXNhYmxlZCBvciBkb2VzIHRoZSBDUFUganVzdCBkZWNy
eXB0IGl0IHRyYW5zcGFyZW50bHkgaS5lLg0KPiA+ID4gd2hhdCBoYXBwZW5zIHBoeXNpY2FsbHk/
DQo+ID4gDQo+ID4gWWVzLCBpdCBnZXRzIGRpc2FibGVkLiBQaHlzaWNhbGx5LiBJdCBvdmVycmlk
ZXMgVE1FIGVuY3J5cHRpb24uDQo+IA0KPiBJIGtub3cgTUtUTUUgaXRzZWxmIGhhcyBhIHJ1bnRp
bWUgb3ZlcmhlYWQgYW5kIHdlIGV4cGVjdCBpdCB0byBoYXZlIGENCj4gcGVyZm9ybWFuY2UgaW1w
YWN0IGluIHRoZSBsb3cgc2luZ2xlIGRpZ2l0cy4gIERvZXMgVE1FIGhhdmUgdGhhdA0KPiBvdmVy
aGVhZD8gIFByZXN1bWFibHkgTUtUTUUgcGx1cyBuby1lbmNyeXB0aW9uIGlzIG5vdCBleHBlY3Rl
ZCB0byBoYXZlDQo+IHRoZSBvdmVyaGVhZC4NCj4gDQo+IFdlIHNob3VsZCBwcm9iYWJseSBtZW50
aW9uIHRoYXQgaW4gdGhlIGNoYW5nZWxvZ3MgdG9vLg0KPiANCg0KSSBiZWxpZXZlIGluIHRlcm1z
IG9mIGhhcmR3YXJlIGNyeXB0byBvdmVyaGVhZCBNS1RNRSBhbmQgVE1FIHNob3VsZCBoYXZlIHRo
ZSBzYW1lIChleGNlcHQgTUtUTUUgbm8tDQplbmNyeXB0IGNhc2U/KS4gQnV0IE1LVE1FIG1pZ2h0
IGhhdmUgYWRkaXRpb25hbCBvdmVyaGVhZCBmcm9tIHNvZnR3YXJlIGltcGxlbWVudGF0aW9uIGlu
IGtlcm5lbD8NCg0KVGhhbmtzLA0KLUthaQ=

Re: [RFC v2 00/13] Multi-Key Total Memory Encryption API (MKTME)

[Huang, Kai]

On Thu, 2018-12-06 at 06:59 -0800, Dave Hansen wrote:
> On 12/6/18 3:22 AM, Kirill A. Shutemov wrote:
> > > When you say "disable encryption to a page" does the encryption get
> > > actually disabled or does the CPU just decrypt it transparently i.e.
> > > what happens physically?
> > 
> > Yes, it gets disabled. Physically. It overrides TME encryption.
> 
> I know MKTME itself has a runtime overhead and we expect it to have a
> performance impact in the low single digits.  Does TME have that
> overhead?  Presumably MKTME plus no-encryption is not expected to have
> the overhead.
> 
> We should probably mention that in the changelogs too.
> 

I believe in terms of hardware crypto overhead MKTME and TME should have the same (except MKTME no-
encrypt case?). But MKTME might have additional overhead from software implementation in kernel?

Thanks,
-Kai

Re: [RFC v2 00/13] Multi-Key Total Memory Encryption API (MKTME)

[Huang, Kai]

On Thu, 2018-12-06 at 06:59 -0800, Dave Hansen wrote:
> On 12/6/18 3:22 AM, Kirill A. Shutemov wrote:
> > > When you say "disable encryption to a page" does the encryption get
> > > actually disabled or does the CPU just decrypt it transparently i.e.
> > > what happens physically?
> > 
> > Yes, it gets disabled. Physically. It overrides TME encryption.
> 
> I know MKTME itself has a runtime overhead and we expect it to have a
> performance impact in the low single digits.  Does TME have that
> overhead?  Presumably MKTME plus no-encryption is not expected to have
> the overhead.
> 
> We should probably mention that in the changelogs too.
> 

I believe in terms of hardware crypto overhead MKTME and TME should have the same (except MKTME no-
encrypt case?). But MKTME might have additional overhead from software implementation in kernel?

Thanks,
-Kai

Re: [RFC v2 12/13] keys/mktme: Save MKTME data if kernel cmdline parameter allows

[Kirill A. Shutemov]

On Fri, Dec 07, 2018 at 02:14:03AM +0000, Huang, Kai wrote:
> Alternatively, we can choose to use per-socket keyID, but not to program
> keyID globally across all sockets, so you don't have to save key while
> still supporting CPU hotplug.

Per-socket KeyID approach would make things more complex. For instance
KeyID on its own will not be enough to refer a key. You will need a node
too. It will also require a way to track whether theirs an KeyID on other
node for the key.

It also makes memory management less flexible: runtime migration of the
memory between nodes will be limited and it can hurt memory availablity
for non-encrypted tasks too.

In general, I don't see per-socket KeyID handling very attractive. It
creates more problems than solves.

-- 
 Kirill A. Shutemov

Re: [RFC v2 12/13] keys/mktme: Save MKTME data if kernel cmdline parameter allows

[Kirill A. Shutemov]

On Fri, Dec 07, 2018 at 02:14:03AM +0000, Huang, Kai wrote:
> Alternatively, we can choose to use per-socket keyID, but not to program
> keyID globally across all sockets, so you don't have to save key while
> still supporting CPU hotplug.

Per-socket KeyID approach would make things more complex. For instance
KeyID on its own will not be enough to refer a key. You will need a node
too. It will also require a way to track whether theirs an KeyID on other
node for the key.

It also makes memory management less flexible: runtime migration of the
memory between nodes will be limited and it can hurt memory availablity
for non-encrypted tasks too.

In general, I don't see per-socket KeyID handling very attractive. It
creates more problems than solves.

-- 
 Kirill A. Shutemov

Re: [RFC v2 12/13] keys/mktme: Save MKTME data if kernel cmdline parameter allows

[Kirill A. Shutemov]

On Fri, Dec 07, 2018 at 02:14:03AM +0000, Huang, Kai wrote:
> Alternatively, we can choose to use per-socket keyID, but not to program
> keyID globally across all sockets, so you don't have to save key while
> still supporting CPU hotplug.

Per-socket KeyID approach would make things more complex. For instance
KeyID on its own will not be enough to refer a key. You will need a node
too. It will also require a way to track whether theirs an KeyID on other
node for the key.

It also makes memory management less flexible: runtime migration of the
memory between nodes will be limited and it can hurt memory availablity
for non-encrypted tasks too.

In general, I don't see per-socket KeyID handling very attractive. It
creates more problems than solves.

-- 
 Kirill A. Shutemov

Re: [RFC v2 00/13] Multi-Key Total Memory Encryption API (MKTME)

[Kirill A. Shutemov]

On Thu, Dec 06, 2018 at 09:23:20PM +0000, Sakkinen, Jarkko wrote:
> On Thu, 2018-12-06 at 14:22 +0300, Kirill A. Shutemov wrote:
> > When you say "disable encryption to a page" does the encryption get
> > > actually disabled or does the CPU just decrypt it transparently i.e.
> > > what happens physically?
> > 
> > Yes, it gets disabled. Physically. It overrides TME encryption.
> 
> OK, thanks for confirmation. BTW, how much is the penalty to keep it
> always enabled? Is it something that would not make sense for some
> other reasons?

We don't have any numbers to share at this point.

-- 
 Kirill A. Shutemov

Re: [RFC v2 00/13] Multi-Key Total Memory Encryption API (MKTME)

[Kirill A. Shutemov]

On Thu, Dec 06, 2018 at 09:23:20PM +0000, Sakkinen, Jarkko wrote:
> On Thu, 2018-12-06 at 14:22 +0300, Kirill A. Shutemov wrote:
> > When you say "disable encryption to a page" does the encryption get
> > > actually disabled or does the CPU just decrypt it transparently i.e.
> > > what happens physically?
> > 
> > Yes, it gets disabled. Physically. It overrides TME encryption.
> 
> OK, thanks for confirmation. BTW, how much is the penalty to keep it
> always enabled? Is it something that would not make sense for some
> other reasons?

We don't have any numbers to share at this point.

-- 
 Kirill A. Shutemov

Re: [RFC v2 00/13] Multi-Key Total Memory Encryption API (MKTME)

[Kirill A. Shutemov]

On Thu, Dec 06, 2018 at 09:23:20PM +0000, Sakkinen, Jarkko wrote:
> On Thu, 2018-12-06 at 14:22 +0300, Kirill A. Shutemov wrote:
> > When you say "disable encryption to a page" does the encryption get
> > > actually disabled or does the CPU just decrypt it transparently i.e.
> > > what happens physically?
> > 
> > Yes, it gets disabled. Physically. It overrides TME encryption.
> 
> OK, thanks for confirmation. BTW, how much is the penalty to keep it
> always enabled? Is it something that would not make sense for some
> other reasons?

We don't have any numbers to share at this point.

-- 
 Kirill A. Shutemov

Re: [RFC v2 00/13] Multi-Key Total Memory Encryption API (MKTME)

[Kirill A. Shutemov]

On Wed, Dec 05, 2018 at 10:19:20PM +0000, Sakkinen, Jarkko wrote:
> On Tue, 2018-12-04 at 11:19 -0800, Andy Lutomirski wrote:
> > I'm not Thomas, but I think it's the wrong direction.  As it stands,
> > encrypt_mprotect() is an incomplete version of mprotect() (since it's
> > missing the protection key support), and it's also functionally just
> > MADV_DONTNEED.  In other words, the sole user-visible effect appears
> > to be that the existing pages are blown away.  The fact that it
> > changes the key in use doesn't seem terribly useful, since it's
> > anonymous memory, and the most secure choice is to use CPU-managed
> > keying, which appears to be the default anyway on TME systems.  It
> > also has totally unclear semantics WRT swap, and, off the top of my
> > head, it looks like it may have serious cache-coherency issues and
> > like swapping the pages might corrupt them, both because there are no
> > flushes and because the direct-map alias looks like it will use the
> > default key and therefore appear to contain the wrong data.
> > 
> > I would propose a very different direction: don't try to support MKTME
> > at all for anonymous memory, and instead figure out the important use
> > cases and support them directly.  The use cases that I can think of
> > off the top of my head are:
> > 
> > 1. pmem.  This should probably use a very different API.
> > 
> > 2. Some kind of VM hardening, where a VM's memory can be protected a
> > little tiny bit from the main kernel.  But I don't see why this is any
> > better than XPO (eXclusive Page-frame Ownership), which brings to
> > mind:
> 
> What is the threat model anyway for AMD and Intel technologies?
> 
> For me it looks like that you can read, write and even replay 
> encrypted pages both in SME and TME. 

What replay attack are you talking about? MKTME uses AES-XTS with physical
address tweak. So the data is tied to the place in physical address space
and replacing one encrypted page with another encrypted page from
different address will produce garbage on decryption.

-- 
 Kirill A. Shutemov

Re: [RFC v2 00/13] Multi-Key Total Memory Encryption API (MKTME)

[Kirill A. Shutemov]

On Wed, Dec 05, 2018 at 10:19:20PM +0000, Sakkinen, Jarkko wrote:
> On Tue, 2018-12-04 at 11:19 -0800, Andy Lutomirski wrote:
> > I'm not Thomas, but I think it's the wrong direction.  As it stands,
> > encrypt_mprotect() is an incomplete version of mprotect() (since it's
> > missing the protection key support), and it's also functionally just
> > MADV_DONTNEED.  In other words, the sole user-visible effect appears
> > to be that the existing pages are blown away.  The fact that it
> > changes the key in use doesn't seem terribly useful, since it's
> > anonymous memory, and the most secure choice is to use CPU-managed
> > keying, which appears to be the default anyway on TME systems.  It
> > also has totally unclear semantics WRT swap, and, off the top of my
> > head, it looks like it may have serious cache-coherency issues and
> > like swapping the pages might corrupt them, both because there are no
> > flushes and because the direct-map alias looks like it will use the
> > default key and therefore appear to contain the wrong data.
> > 
> > I would propose a very different direction: don't try to support MKTME
> > at all for anonymous memory, and instead figure out the important use
> > cases and support them directly.  The use cases that I can think of
> > off the top of my head are:
> > 
> > 1. pmem.  This should probably use a very different API.
> > 
> > 2. Some kind of VM hardening, where a VM's memory can be protected a
> > little tiny bit from the main kernel.  But I don't see why this is any
> > better than XPO (eXclusive Page-frame Ownership), which brings to
> > mind:
> 
> What is the threat model anyway for AMD and Intel technologies?
> 
> For me it looks like that you can read, write and even replay 
> encrypted pages both in SME and TME. 

What replay attack are you talking about? MKTME uses AES-XTS with physical
address tweak. So the data is tied to the place in physical address space
and replacing one encrypted page with another encrypted page from
different address will produce garbage on decryption.

-- 
 Kirill A. Shutemov

Re: [RFC v2 00/13] Multi-Key Total Memory Encryption API (MKTME)

[Kirill A. Shutemov]

On Wed, Dec 05, 2018 at 10:19:20PM +0000, Sakkinen, Jarkko wrote:
> On Tue, 2018-12-04 at 11:19 -0800, Andy Lutomirski wrote:
> > I'm not Thomas, but I think it's the wrong direction.  As it stands,
> > encrypt_mprotect() is an incomplete version of mprotect() (since it's
> > missing the protection key support), and it's also functionally just
> > MADV_DONTNEED.  In other words, the sole user-visible effect appears
> > to be that the existing pages are blown away.  The fact that it
> > changes the key in use doesn't seem terribly useful, since it's
> > anonymous memory, and the most secure choice is to use CPU-managed
> > keying, which appears to be the default anyway on TME systems.  It
> > also has totally unclear semantics WRT swap, and, off the top of my
> > head, it looks like it may have serious cache-coherency issues and
> > like swapping the pages might corrupt them, both because there are no
> > flushes and because the direct-map alias looks like it will use the
> > default key and therefore appear to contain the wrong data.
> > 
> > I would propose a very different direction: don't try to support MKTME
> > at all for anonymous memory, and instead figure out the important use
> > cases and support them directly.  The use cases that I can think of
> > off the top of my head are:
> > 
> > 1. pmem.  This should probably use a very different API.
> > 
> > 2. Some kind of VM hardening, where a VM's memory can be protected a
> > little tiny bit from the main kernel.  But I don't see why this is any
> > better than XPO (eXclusive Page-frame Ownership), which brings to
> > mind:
> 
> What is the threat model anyway for AMD and Intel technologies?
> 
> For me it looks like that you can read, write and even replay 
> encrypted pages both in SME and TME. 

What replay attack are you talking about? MKTME uses AES-XTS with physical
address tweak. So the data is tied to the place in physical address space
and replacing one encrypted page with another encrypted page from
different address will produce garbage on decryption.

-- 
 Kirill A. Shutemov

Re: [RFC v2 00/13] Multi-Key Total Memory Encryption API (MKTME)

[Sakkinen, Jarkko]

T24gRnJpLCAyMDE4LTEyLTA3IGF0IDE0OjU3ICswMzAwLCBLaXJpbGwgQS4gU2h1dGVtb3Ygd3Jv
dGU6DQo+ID4gV2hhdCBpcyB0aGUgdGhyZWF0IG1vZGVsIGFueXdheSBmb3IgQU1EIGFuZCBJbnRl
bCB0ZWNobm9sb2dpZXM/DQo+ID4gDQo+ID4gRm9yIG1lIGl0IGxvb2tzIGxpa2UgdGhhdCB5b3Ug
Y2FuIHJlYWQsIHdyaXRlIGFuZCBldmVuIHJlcGxheSANCj4gPiBlbmNyeXB0ZWQgcGFnZXMgYm90
aCBpbiBTTUUgYW5kIFRNRS4gDQo+IA0KPiBXaGF0IHJlcGxheSBhdHRhY2sgYXJlIHlvdSB0YWxr
aW5nIGFib3V0PyBNS1RNRSB1c2VzIEFFUy1YVFMgd2l0aCBwaHlzaWNhbA0KPiBhZGRyZXNzIHR3
ZWFrLiBTbyB0aGUgZGF0YSBpcyB0aWVkIHRvIHRoZSBwbGFjZSBpbiBwaHlzaWNhbCBhZGRyZXNz
IHNwYWNlIGFuZA0KPiByZXBsYWNpbmcgb25lIGVuY3J5cHRlZCBwYWdlIHdpdGggYW5vdGhlciBl
bmNyeXB0ZWQgcGFnZSBmcm9tIGRpZmZlcmVudA0KPiBhZGRyZXNzIHdpbGwgcHJvZHVjZSBnYXJi
YWdlIG9uIGRlY3J5cHRpb24uDQoNCkp1c3QgdHJ5aW5nIHRvIHVuZGVyc3RhbmQgaG93IHRoaXMg
d29ya3MuDQoNClNvIHlvdSB1c2UgcGh5c2ljYWwgYWRkcmVzcyBsaWtlIGEgbm9uY2UvdmVyc2lv
biBmb3IgdGhlIHBhZ2UgYW5kDQp0aHVzIHByZXZlbnQgcmVwbGF5PyBXYXMgbm90IGF3YXJlIG9m
IHRoaXMuDQoNCi9KYXJra28NCg=

Re: [RFC v2 00/13] Multi-Key Total Memory Encryption API (MKTME)

[Sakkinen, Jarkko]

On Fri, 2018-12-07 at 14:57 +0300, Kirill A. Shutemov wrote:
> > What is the threat model anyway for AMD and Intel technologies?
> > 
> > For me it looks like that you can read, write and even replay 
> > encrypted pages both in SME and TME. 
> 
> What replay attack are you talking about? MKTME uses AES-XTS with physical
> address tweak. So the data is tied to the place in physical address space and
> replacing one encrypted page with another encrypted page from different
> address will produce garbage on decryption.

Just trying to understand how this works.

So you use physical address like a nonce/version for the page and
thus prevent replay? Was not aware of this.

/Jarkko

Re: [RFC v2 00/13] Multi-Key Total Memory Encryption API (MKTME)

[Sakkinen, Jarkko]

On Fri, 2018-12-07 at 14:57 +0300, Kirill A. Shutemov wrote:
> > What is the threat model anyway for AMD and Intel technologies?
> > 
> > For me it looks like that you can read, write and even replay 
> > encrypted pages both in SME and TME. 
> 
> What replay attack are you talking about? MKTME uses AES-XTS with physical
> address tweak. So the data is tied to the place in physical address space and
> replacing one encrypted page with another encrypted page from different
> address will produce garbage on decryption.

Just trying to understand how this works.

So you use physical address like a nonce/version for the page and
thus prevent replay? Was not aware of this.

/Jarkko

Re: [RFC v2 00/13] Multi-Key Total Memory Encryption API (MKTME)

[Eric Rannaud]

On Fri, Dec 7, 2018 at 3:57 AM Kirill A. Shutemov <kirill@shutemov.name> wrote:
> > What is the threat model anyway for AMD and Intel technologies?
> >
> > For me it looks like that you can read, write and even replay
> > encrypted pages both in SME and TME.
>
> What replay attack are you talking about? MKTME uses AES-XTS with physical
> address tweak. So the data is tied to the place in physical address space
> and replacing one encrypted page with another encrypted page from
> different address will produce garbage on decryption.

What if you have some control over the physical addresses you write
the stolen encrypted page to? For instance, VM_Eve might manage to use
physical address space previously used by VM_Alice by getting the
hypervisor to move memory around (memory pressure, force other VMs out
via some type of DOS attack, etc.).

Say:
    C is VM_Alice's clear text at hwaddr
    E = mktme_encrypt(VM_Allice_key, hwaddr, C)
    Eve somehow stole the encrypted bits E

Eve would need to write the page E without further encryption to make
sure that the DRAM contains the original stolen bits E, not encrypted
again with VM_Eve's key or mktme_encrypt(VM_Eve_key, hwaddr, E) would
be present in the DRAM which is not helpful. But with MKTME under the
current proposal VM_Eve can disable encryption for a given mapping,
right? (See also Note 1)

Eve gets the HV to move VM_Alice back over the same physical address,
Eve "somehow" gets VM_Alice to read that page and use its content
(which would likely be a use of uninitialized memory bug, from
VM_Alice's perspective) and you have a replay attack?

For TME, this doesn't work as you cannot partially disable encryption,
so if Eve tries to write the stolen encrypted bits E, even in the
"right place", they get encrypted again to tme_encrypt(hwaddr, E).
Upon decryption, VM_Alice will get E, not C.

Note 1: Actually, even if with MKTME you cannot disable encryption but
*if* Eve knows its own key, Eve can always write a preimage P that the
CPU encrypts to E for VM_Alice to read back and decrypt:
    P = mktme_decrypt(VM_Eve_key, hwaddr, E)

This is not possible with TME as Eve doesn't know the key used by the
CPU and cannot compute P.

Re: [RFC v2 00/13] Multi-Key Total Memory Encryption API (MKTME)

[Eric Rannaud]

On Fri, Dec 7, 2018 at 3:57 AM Kirill A. Shutemov <kirill@shutemov.name> wrote:
> > What is the threat model anyway for AMD and Intel technologies?
> >
> > For me it looks like that you can read, write and even replay
> > encrypted pages both in SME and TME.
>
> What replay attack are you talking about? MKTME uses AES-XTS with physical
> address tweak. So the data is tied to the place in physical address space
> and replacing one encrypted page with another encrypted page from
> different address will produce garbage on decryption.

What if you have some control over the physical addresses you write
the stolen encrypted page to? For instance, VM_Eve might manage to use
physical address space previously used by VM_Alice by getting the
hypervisor to move memory around (memory pressure, force other VMs out
via some type of DOS attack, etc.).

Say:
    C is VM_Alice's clear text at hwaddr
    E = mktme_encrypt(VM_Allice_key, hwaddr, C)
    Eve somehow stole the encrypted bits E

Eve would need to write the page E without further encryption to make
sure that the DRAM contains the original stolen bits E, not encrypted
again with VM_Eve's key or mktme_encrypt(VM_Eve_key, hwaddr, E) would
be present in the DRAM which is not helpful. But with MKTME under the
current proposal VM_Eve can disable encryption for a given mapping,
right? (See also Note 1)

Eve gets the HV to move VM_Alice back over the same physical address,
Eve "somehow" gets VM_Alice to read that page and use its content
(which would likely be a use of uninitialized memory bug, from
VM_Alice's perspective) and you have a replay attack?

For TME, this doesn't work as you cannot partially disable encryption,
so if Eve tries to write the stolen encrypted bits E, even in the
"right place", they get encrypted again to tme_encrypt(hwaddr, E).
Upon decryption, VM_Alice will get E, not C.

Note 1: Actually, even if with MKTME you cannot disable encryption but
*if* Eve knows its own key, Eve can always write a preimage P that the
CPU encrypts to E for VM_Alice to read back and decrypt:
    P = mktme_decrypt(VM_Eve_key, hwaddr, E)

This is not possible with TME as Eve doesn't know the key used by the
CPU and cannot compute P.

Re: [RFC v2 00/13] Multi-Key Total Memory Encryption API (MKTME)

[Sakkinen, Jarkko]

T24gRnJpLCAyMDE4LTEyLTA3IGF0IDEzOjU5IC0wODAwLCBKYXJra28gU2Fra2luZW4gd3JvdGU6
DQo+IE9uIEZyaSwgMjAxOC0xMi0wNyBhdCAxNDo1NyArMDMwMCwgS2lyaWxsIEEuIFNodXRlbW92
IHdyb3RlOg0KPiA+ID4gV2hhdCBpcyB0aGUgdGhyZWF0IG1vZGVsIGFueXdheSBmb3IgQU1EIGFu
ZCBJbnRlbCB0ZWNobm9sb2dpZXM/DQo+ID4gPiANCj4gPiA+IEZvciBtZSBpdCBsb29rcyBsaWtl
IHRoYXQgeW91IGNhbiByZWFkLCB3cml0ZSBhbmQgZXZlbiByZXBsYXkgDQo+ID4gPiBlbmNyeXB0
ZWQgcGFnZXMgYm90aCBpbiBTTUUgYW5kIFRNRS4gDQo+ID4gDQo+ID4gV2hhdCByZXBsYXkgYXR0
YWNrIGFyZSB5b3UgdGFsa2luZyBhYm91dD8gTUtUTUUgdXNlcyBBRVMtWFRTIHdpdGggcGh5c2lj
YWwNCj4gPiBhZGRyZXNzIHR3ZWFrLiBTbyB0aGUgZGF0YSBpcyB0aWVkIHRvIHRoZSBwbGFjZSBp
biBwaHlzaWNhbCBhZGRyZXNzIHNwYWNlDQo+ID4gYW5kDQo+ID4gcmVwbGFjaW5nIG9uZSBlbmNy
eXB0ZWQgcGFnZSB3aXRoIGFub3RoZXIgZW5jcnlwdGVkIHBhZ2UgZnJvbSBkaWZmZXJlbnQNCj4g
PiBhZGRyZXNzIHdpbGwgcHJvZHVjZSBnYXJiYWdlIG9uIGRlY3J5cHRpb24uDQo+IA0KPiBKdXN0
IHRyeWluZyB0byB1bmRlcnN0YW5kIGhvdyB0aGlzIHdvcmtzLg0KPiANCj4gU28geW91IHVzZSBw
aHlzaWNhbCBhZGRyZXNzIGxpa2UgYSBub25jZS92ZXJzaW9uIGZvciB0aGUgcGFnZSBhbmQNCj4g
dGh1cyBwcmV2ZW50IHJlcGxheT8gV2FzIG5vdCBhd2FyZSBvZiB0aGlzLg0KDQpUaGUgYnJ1dGFs
IGZhY3QgaXMgdGhhdCBhIHBoeXNpY2FsIGFkZHJlc3MgaXMgYW4gYXN0cm9ub21pY2FsIHN0cmV0
Y2gNCmZyb20gYSByYW5kb20gdmFsdWUgb3IgaW5jcmVhc2luZyBjb3VudGVyLiBUaHVzLCBpdCBp
cyBmYWlyIHRvIHNheSB0aGF0DQpNS1RNRSBwcm92aWRlcyBvbmx5IG5haXZlIG1lYXN1cmVzIGFn
YWluc3QgcmVwbGF5IGF0dGFja3MuLi4NCg0KL0phcmtrbw0K

Re: [RFC v2 00/13] Multi-Key Total Memory Encryption API (MKTME)

[Sakkinen, Jarkko]

On Fri, 2018-12-07 at 13:59 -0800, Jarkko Sakkinen wrote:
> On Fri, 2018-12-07 at 14:57 +0300, Kirill A. Shutemov wrote:
> > > What is the threat model anyway for AMD and Intel technologies?
> > > 
> > > For me it looks like that you can read, write and even replay 
> > > encrypted pages both in SME and TME. 
> > 
> > What replay attack are you talking about? MKTME uses AES-XTS with physical
> > address tweak. So the data is tied to the place in physical address space
> > and
> > replacing one encrypted page with another encrypted page from different
> > address will produce garbage on decryption.
> 
> Just trying to understand how this works.
> 
> So you use physical address like a nonce/version for the page and
> thus prevent replay? Was not aware of this.

The brutal fact is that a physical address is an astronomical stretch
from a random value or increasing counter. Thus, it is fair to say that
MKTME provides only naive measures against replay attacks...

/Jarkko

Re: [RFC v2 00/13] Multi-Key Total Memory Encryption API (MKTME)

[Sakkinen, Jarkko]

On Fri, 2018-12-07 at 13:59 -0800, Jarkko Sakkinen wrote:
> On Fri, 2018-12-07 at 14:57 +0300, Kirill A. Shutemov wrote:
> > > What is the threat model anyway for AMD and Intel technologies?
> > > 
> > > For me it looks like that you can read, write and even replay 
> > > encrypted pages both in SME and TME. 
> > 
> > What replay attack are you talking about? MKTME uses AES-XTS with physical
> > address tweak. So the data is tied to the place in physical address space
> > and
> > replacing one encrypted page with another encrypted page from different
> > address will produce garbage on decryption.
> 
> Just trying to understand how this works.
> 
> So you use physical address like a nonce/version for the page and
> thus prevent replay? Was not aware of this.

The brutal fact is that a physical address is an astronomical stretch
from a random value or increasing counter. Thus, it is fair to say that
MKTME provides only naive measures against replay attacks...

/Jarkko

Re: [RFC v2 00/13] Multi-Key Total Memory Encryption API (MKTME)

[Andy Lutomirski]

On Fri, Dec 7, 2018 at 3:45 PM Sakkinen, Jarkko
<jarkko.sakkinen@intel.com> wrote:
>
> On Fri, 2018-12-07 at 13:59 -0800, Jarkko Sakkinen wrote:
> > On Fri, 2018-12-07 at 14:57 +0300, Kirill A. Shutemov wrote:
> > > > What is the threat model anyway for AMD and Intel technologies?
> > > >
> > > > For me it looks like that you can read, write and even replay
> > > > encrypted pages both in SME and TME.
> > >
> > > What replay attack are you talking about? MKTME uses AES-XTS with physical
> > > address tweak. So the data is tied to the place in physical address space
> > > and
> > > replacing one encrypted page with another encrypted page from different
> > > address will produce garbage on decryption.
> >
> > Just trying to understand how this works.
> >
> > So you use physical address like a nonce/version for the page and
> > thus prevent replay? Was not aware of this.
>
> The brutal fact is that a physical address is an astronomical stretch
> from a random value or increasing counter. Thus, it is fair to say that
> MKTME provides only naive measures against replay attacks...
>

And this is potentially a big deal, since there are much simpler
replay attacks that can compromise the system.  For example, if I can
replay the contents of a page table, I can write to freed memory.

--Andy

Re: [RFC v2 00/13] Multi-Key Total Memory Encryption API (MKTME)

[Andy Lutomirski]

On Fri, Dec 7, 2018 at 3:45 PM Sakkinen, Jarkko
<jarkko.sakkinen@intel.com> wrote:
>
> On Fri, 2018-12-07 at 13:59 -0800, Jarkko Sakkinen wrote:
> > On Fri, 2018-12-07 at 14:57 +0300, Kirill A. Shutemov wrote:
> > > > What is the threat model anyway for AMD and Intel technologies?
> > > >
> > > > For me it looks like that you can read, write and even replay
> > > > encrypted pages both in SME and TME.
> > >
> > > What replay attack are you talking about? MKTME uses AES-XTS with physical
> > > address tweak. So the data is tied to the place in physical address space
> > > and
> > > replacing one encrypted page with another encrypted page from different
> > > address will produce garbage on decryption.
> >
> > Just trying to understand how this works.
> >
> > So you use physical address like a nonce/version for the page and
> > thus prevent replay? Was not aware of this.
>
> The brutal fact is that a physical address is an astronomical stretch
> from a random value or increasing counter. Thus, it is fair to say that
> MKTME provides only naive measures against replay attacks...
>

And this is potentially a big deal, since there are much simpler
replay attacks that can compromise the system.  For example, if I can
replay the contents of a page table, I can write to freed memory.

--Andy

Re: [RFC v2 00/13] Multi-Key Total Memory Encryption API (MKTME)

[Andy Lutomirski]

> On Dec 6, 2018, at 5:55 PM, Huang, Kai <kai.huang@intel.com> wrote:
>
>
>>
>> TME itself provides a ton of protection -- you can't just barge into
>> the datacenter, refrigerate the DIMMs, walk away with them, and read
>> off everyone's data.
>>
>> Am I missing something?
>
> I think we can make such assumption in most cases, but I think it's better that we don't make any
> assumption at all. For example, the admin of data center (or anyone) who has physical access to
> servers may do something malicious. I am not expert but there should be other physical attack
> methods besides coldboot attack, if the malicious employee can get physical access to server w/o
> being detected.
>
>>
>>>
>>> But, I think what you're implying is that the security properties of
>>> user-supplied keys can only be *worse* than using CPU-generated keys
>>> (assuming the CPU does a good job generating it).  So, why bother
>>> allowing user-specified keys in the first place?
>>
>> That too :)
>
> I think one usage of user-specified key is for NVDIMM, since CPU key will be gone after machine
> reboot, therefore if NVDIMM is encrypted by CPU key we are not able to retrieve it once
> shutdown/reboot, etc.
>
> There are some other use cases that already require tenant to send key to CSP. For example, the VM
> image can be provided by tenant and encrypted by tenant's own key, and tenant needs to send key to
> CSP when asking CSP to run that encrypted image.


I can imagine a few reasons why one would want to encrypt one’s image.
For example, the CSP could issue a public key and state, or even
attest, that the key is wrapped and locked to particular PCRs of their
TPM or otherwise protected by an enclave that verifies that the key is
only used to decrypt the image for the benefit of a hypervisor.

I don’t see what MKTME has to do with this.  The only remotely
plausible way I can see to use MKTME for this is to have the
hypervisor load a TPM (or other enclave) protected key into an MKTME
user key slot and to load customer-provided ciphertext into the
corresponding physical memory (using an MKTME no-encrypt slot).  But
this has three major problems.  First, it's effectively just a fancy
way to avoid one AES pass over the data.  Second, sensible scheme for
this type of VM image protection would use *authenticated* encryption
or at least verify a signature, which MKTME can't do.  The third
problem is the real show-stopper, though: this scheme requires that
the ciphertext go into predetermined physical addresses, which would
be a giant mess.

Re: [RFC v2 00/13] Multi-Key Total Memory Encryption API (MKTME)

[Andy Lutomirski]

> On Dec 6, 2018, at 5:55 PM, Huang, Kai <kai.huang@intel.com> wrote:
>
>
>>
>> TME itself provides a ton of protection -- you can't just barge into
>> the datacenter, refrigerate the DIMMs, walk away with them, and read
>> off everyone's data.
>>
>> Am I missing something?
>
> I think we can make such assumption in most cases, but I think it's better that we don't make any
> assumption at all. For example, the admin of data center (or anyone) who has physical access to
> servers may do something malicious. I am not expert but there should be other physical attack
> methods besides coldboot attack, if the malicious employee can get physical access to server w/o
> being detected.
>
>>
>>>
>>> But, I think what you're implying is that the security properties of
>>> user-supplied keys can only be *worse* than using CPU-generated keys
>>> (assuming the CPU does a good job generating it).  So, why bother
>>> allowing user-specified keys in the first place?
>>
>> That too :)
>
> I think one usage of user-specified key is for NVDIMM, since CPU key will be gone after machine
> reboot, therefore if NVDIMM is encrypted by CPU key we are not able to retrieve it once
> shutdown/reboot, etc.
>
> There are some other use cases that already require tenant to send key to CSP. For example, the VM
> image can be provided by tenant and encrypted by tenant's own key, and tenant needs to send key to
> CSP when asking CSP to run that encrypted image.


I can imagine a few reasons why one would want to encrypt one’s image.
For example, the CSP could issue a public key and state, or even
attest, that the key is wrapped and locked to particular PCRs of their
TPM or otherwise protected by an enclave that verifies that the key is
only used to decrypt the image for the benefit of a hypervisor.

I don’t see what MKTME has to do with this.  The only remotely
plausible way I can see to use MKTME for this is to have the
hypervisor load a TPM (or other enclave) protected key into an MKTME
user key slot and to load customer-provided ciphertext into the
corresponding physical memory (using an MKTME no-encrypt slot).  But
this has three major problems.  First, it's effectively just a fancy
way to avoid one AES pass over the data.  Second, sensible scheme for
this type of VM image protection would use *authenticated* encryption
or at least verify a signature, which MKTME can't do.  The third
problem is the real show-stopper, though: this scheme requires that
the ciphertext go into predetermined physical addresses, which would
be a giant mess.

Re: [RFC v2 00/13] Multi-Key Total Memory Encryption API (MKTME)

[Dave Hansen]

On 12/7/18 3:53 PM, Andy Lutomirski wrote:
> The third problem is the real show-stopper, though: this scheme
> requires that the ciphertext go into predetermined physical
> addresses, which would be a giant mess.

There's a more fundamental problem than that.  The tweak fed into the
actual AES-XTS operation is determined by the firmware, programmed into
the memory controller, and is not visible to software.  So, not only
would you need to put stuff at a fixed physical address, the tweaks can
change from boot-to-boot, so whatever you did would only be good for one
boot.

Re: [RFC v2 00/13] Multi-Key Total Memory Encryption API (MKTME)

[Dave Hansen]

On 12/7/18 3:53 PM, Andy Lutomirski wrote:
> The third problem is the real show-stopper, though: this scheme
> requires that the ciphertext go into predetermined physical
> addresses, which would be a giant mess.

There's a more fundamental problem than that.  The tweak fed into the
actual AES-XTS operation is determined by the firmware, programmed into
the memory controller, and is not visible to software.  So, not only
would you need to put stuff at a fixed physical address, the tweaks can
change from boot-to-boot, so whatever you did would only be good for one
boot.

Re: [RFC v2 00/13] Multi-Key Total Memory Encryption API (MKTME)

[Huang, Kai]

T24gRnJpLCAyMDE4LTEyLTA3IGF0IDIzOjQ1ICswMDAwLCBTYWtraW5lbiwgSmFya2tvIHdyb3Rl
Og0KPiBPbiBGcmksIDIwMTgtMTItMDcgYXQgMTM6NTkgLTA4MDAsIEphcmtrbyBTYWtraW5lbiB3
cm90ZToNCj4gPiBPbiBGcmksIDIwMTgtMTItMDcgYXQgMTQ6NTcgKzAzMDAsIEtpcmlsbCBBLiBT
aHV0ZW1vdiB3cm90ZToNCj4gPiA+ID4gV2hhdCBpcyB0aGUgdGhyZWF0IG1vZGVsIGFueXdheSBm
b3IgQU1EIGFuZCBJbnRlbCB0ZWNobm9sb2dpZXM/DQo+ID4gPiA+IA0KPiA+ID4gPiBGb3IgbWUg
aXQgbG9va3MgbGlrZSB0aGF0IHlvdSBjYW4gcmVhZCwgd3JpdGUgYW5kIGV2ZW4gcmVwbGF5IA0K
PiA+ID4gPiBlbmNyeXB0ZWQgcGFnZXMgYm90aCBpbiBTTUUgYW5kIFRNRS4gDQo+ID4gPiANCj4g
PiA+IFdoYXQgcmVwbGF5IGF0dGFjayBhcmUgeW91IHRhbGtpbmcgYWJvdXQ/IE1LVE1FIHVzZXMg
QUVTLVhUUyB3aXRoIHBoeXNpY2FsDQo+ID4gPiBhZGRyZXNzIHR3ZWFrLiBTbyB0aGUgZGF0YSBp
cyB0aWVkIHRvIHRoZSBwbGFjZSBpbiBwaHlzaWNhbCBhZGRyZXNzIHNwYWNlDQo+ID4gPiBhbmQN
Cj4gPiA+IHJlcGxhY2luZyBvbmUgZW5jcnlwdGVkIHBhZ2Ugd2l0aCBhbm90aGVyIGVuY3J5cHRl
ZCBwYWdlIGZyb20gZGlmZmVyZW50DQo+ID4gPiBhZGRyZXNzIHdpbGwgcHJvZHVjZSBnYXJiYWdl
IG9uIGRlY3J5cHRpb24uDQo+ID4gDQo+ID4gSnVzdCB0cnlpbmcgdG8gdW5kZXJzdGFuZCBob3cg
dGhpcyB3b3Jrcy4NCj4gPiANCj4gPiBTbyB5b3UgdXNlIHBoeXNpY2FsIGFkZHJlc3MgbGlrZSBh
IG5vbmNlL3ZlcnNpb24gZm9yIHRoZSBwYWdlIGFuZA0KPiA+IHRodXMgcHJldmVudCByZXBsYXk/
IFdhcyBub3QgYXdhcmUgb2YgdGhpcy4NCj4gDQo+IFRoZSBicnV0YWwgZmFjdCBpcyB0aGF0IGEg
cGh5c2ljYWwgYWRkcmVzcyBpcyBhbiBhc3Ryb25vbWljYWwgc3RyZXRjaA0KPiBmcm9tIGEgcmFu
ZG9tIHZhbHVlIG9yIGluY3JlYXNpbmcgY291bnRlci4gVGh1cywgaXQgaXMgZmFpciB0byBzYXkg
dGhhdA0KPiBNS1RNRSBwcm92aWRlcyBvbmx5IG5haXZlIG1lYXN1cmVzIGFnYWluc3QgcmVwbGF5
IGF0dGFja3MuLi4NCj4gDQo+IC9KYXJra28NCg0KQ3VycmVudGx5IHRoZXJlJ3Mgbm8gbm9uY2Ug
dG8gcHJvdGVjdCBjYWNoZSBsaW5lIHNvIFRNRS9NS1RNRSBpcyBub3QgYWJsZSB0byBwcmV2ZW50
IHJlcGxheSBhdHRhY2sNCnlvdSBtZW50aW9uZWQuIEN1cnJlbnRseSBNS1RNRSBvbmx5IGludm9s
dmVzIEFFUy1YVFMtMTI4IGVuY3J5cHRpb24gYnV0IG5vdGhpbmcgZWxzZS4gQnV0IGxpa2UgSQ0K
c2FpZCBpZiBJIHVuZGVyc3RhbmQgY29ycmVjdGx5IGV2ZW4gU0VWIGRvZXNuJ3QgaGF2ZSBpbnRl
Z3JpdHkgcHJvdGVjdGlvbiBzbyBub3QgYWJsZSB0byBwcmV2ZW50DQpyZXBseSBhdHRhY2sgYXMg
d2VsbC4NCg0KVGhhbmtzLA0KLUthaQ=

Re: [RFC v2 00/13] Multi-Key Total Memory Encryption API (MKTME)

[Huang, Kai]

On Fri, 2018-12-07 at 23:45 +0000, Sakkinen, Jarkko wrote:
> On Fri, 2018-12-07 at 13:59 -0800, Jarkko Sakkinen wrote:
> > On Fri, 2018-12-07 at 14:57 +0300, Kirill A. Shutemov wrote:
> > > > What is the threat model anyway for AMD and Intel technologies?
> > > > 
> > > > For me it looks like that you can read, write and even replay 
> > > > encrypted pages both in SME and TME. 
> > > 
> > > What replay attack are you talking about? MKTME uses AES-XTS with physical
> > > address tweak. So the data is tied to the place in physical address space
> > > and
> > > replacing one encrypted page with another encrypted page from different
> > > address will produce garbage on decryption.
> > 
> > Just trying to understand how this works.
> > 
> > So you use physical address like a nonce/version for the page and
> > thus prevent replay? Was not aware of this.
> 
> The brutal fact is that a physical address is an astronomical stretch
> from a random value or increasing counter. Thus, it is fair to say that
> MKTME provides only naive measures against replay attacks...
> 
> /Jarkko

Currently there's no nonce to protect cache line so TME/MKTME is not able to prevent replay attack
you mentioned. Currently MKTME only involves AES-XTS-128 encryption but nothing else. But like I
said if I understand correctly even SEV doesn't have integrity protection so not able to prevent
reply attack as well.

Thanks,
-Kai

Re: [RFC v2 00/13] Multi-Key Total Memory Encryption API (MKTME)

[Huang, Kai]

On Fri, 2018-12-07 at 23:45 +0000, Sakkinen, Jarkko wrote:
> On Fri, 2018-12-07 at 13:59 -0800, Jarkko Sakkinen wrote:
> > On Fri, 2018-12-07 at 14:57 +0300, Kirill A. Shutemov wrote:
> > > > What is the threat model anyway for AMD and Intel technologies?
> > > > 
> > > > For me it looks like that you can read, write and even replay 
> > > > encrypted pages both in SME and TME. 
> > > 
> > > What replay attack are you talking about? MKTME uses AES-XTS with physical
> > > address tweak. So the data is tied to the place in physical address space
> > > and
> > > replacing one encrypted page with another encrypted page from different
> > > address will produce garbage on decryption.
> > 
> > Just trying to understand how this works.
> > 
> > So you use physical address like a nonce/version for the page and
> > thus prevent replay? Was not aware of this.
> 
> The brutal fact is that a physical address is an astronomical stretch
> from a random value or increasing counter. Thus, it is fair to say that
> MKTME provides only naive measures against replay attacks...
> 
> /Jarkko

Currently there's no nonce to protect cache line so TME/MKTME is not able to prevent replay attack
you mentioned. Currently MKTME only involves AES-XTS-128 encryption but nothing else. But like I
said if I understand correctly even SEV doesn't have integrity protection so not able to prevent
reply attack as well.

Thanks,
-Kai

Re: [RFC v2 00/13] Multi-Key Total Memory Encryption API (MKTME)

[Huang, Kai]

DQo+ID4gVGhlcmUgYXJlIHNvbWUgb3RoZXIgdXNlIGNhc2VzIHRoYXQgYWxyZWFkeSByZXF1aXJl
IHRlbmFudCB0byBzZW5kIGtleSB0byBDU1AuIEZvciBleGFtcGxlLCB0aGUNCj4gPiBWTQ0KPiA+
IGltYWdlIGNhbiBiZSBwcm92aWRlZCBieSB0ZW5hbnQgYW5kIGVuY3J5cHRlZCBieSB0ZW5hbnQn
cyBvd24ga2V5LCBhbmQgdGVuYW50IG5lZWRzIHRvIHNlbmQga2V5DQo+ID4gdG8NCj4gPiBDU1Ag
d2hlbiBhc2tpbmcgQ1NQIHRvIHJ1biB0aGF0IGVuY3J5cHRlZCBpbWFnZS4NCj4gDQo+IA0KPiBJ
IGNhbiBpbWFnaW5lIGEgZmV3IHJlYXNvbnMgd2h5IG9uZSB3b3VsZCB3YW50IHRvIGVuY3J5cHQg
b25l4oCZcyBpbWFnZS4NCj4gRm9yIGV4YW1wbGUsIHRoZSBDU1AgY291bGQgaXNzdWUgYSBwdWJs
aWMga2V5IGFuZCBzdGF0ZSwgb3IgZXZlbg0KPiBhdHRlc3QsIHRoYXQgdGhlIGtleSBpcyB3cmFw
cGVkIGFuZCBsb2NrZWQgdG8gcGFydGljdWxhciBQQ1JzIG9mIHRoZWlyDQo+IFRQTSBvciBvdGhl
cndpc2UgcHJvdGVjdGVkIGJ5IGFuIGVuY2xhdmUgdGhhdCB2ZXJpZmllcyB0aGF0IHRoZSBrZXkg
aXMNCj4gb25seSB1c2VkIHRvIGRlY3J5cHQgdGhlIGltYWdlIGZvciB0aGUgYmVuZWZpdCBvZiBh
IGh5cGVydmlzb3IuDQoNClJpZ2h0LiBJIHRoaW5rIGJlZm9yZSB0ZW5hbnQgcmVsZWFzZXMga2V5
IHRvIENTUCBpdCBzaG91bGQgYWx3YXlzIHVzZSBhdHRlc3RhdGlvbiBhdXRob3JpdHkgdG8NCnZl
cmlmeSB0aGUgdHJ1c3RpbmVzcyBvZiBjb21wdXRlciBub2RlLiBJIGNhbiB1bmRlcnN0YW5kIHRo
YXQgdGhlIGtleSBjYW4gYmUgd3JhcHBlZCBieSBUUE0gYmVmb3JlDQpzZW5kaW5nIHRvIENTUCBi
dXQgbmVlZCBzb21lIGNhdGNoIHVwIGFib3V0IHVzaW5nIGVuY2xhdmUgcGFydC4gDQoNClRoZSB0
aGluZyBpcyBjb21wdXRlciBub2RlIGNhbiBiZSB0cnVzdGVkIGRvZXNuJ3QgbWVhbiBpdCBjYW5u
b3QgYmUgYXR0YWNrZWQsIG9yIGV2ZW4gaXQgZG9lc24ndA0KbWVhbiBpdCBjYW4gcHJldmVudCwg
aWUgc29tZSBtYWxpY2lvdXMgYWRtaW4sIHRvIGdldCB0ZW5hbnQga2V5IGV2ZW4gYnkgdXNpbmcg
bGVnaXRpbWF0ZSB3YXkuIFRoZXJlDQphcmUgbWFueSBTVyBjb21wb25lbnRzIGludm9sdmVkIGhl
cmUuIEFueXdheSB0aGlzIGlzIG5vdCByZWxhdGVkIHRvIE1LVE1FIGl0c2VsZiBsaWtlIHlvdSBt
ZW50aW9uZWQNCmJlbG93LCB0aGVyZWZvcmUgdGhlIHBvaW50IGlzLCBhcyB3ZSBhbHJlYWR5IHNl
ZSBNS1RNRSBpdHNlbGYgcHJvdmlkZXMgdmVyeSB3ZWFrIHNlY3VyaXR5DQpwcm90ZWN0aW9uLCB3
ZSBuZWVkIHRvIHNlZSB3aGV0aGVyIE1LVE1FIGhhcyB2YWx1ZSBmcm9tIHRoZSB3aG9sZSB1c2Ug
Y2FzZSdzIHBvaW50IG9mIHZpZXcNCihpbmNsdWRpbmcgYWxsIHRoZSB0aGluZ3MgeW91IG1lbnRp
b25lZCBhYm92ZSkgLS0gd2UgZGVmaW5lIHRoZSB3aG9sZSB1c2UgY2FzZSwgd2UgY2xlYXJseSBz
dGF0ZQ0Kd2hvL3doYXQgc2hvdWxkIGJlIGluIHRydXN0IGJvdW5kYXJ5LCBhbmQgd2hhdCB3ZSBj
YW4gcHJldmVudCwgZXRjLg0KDQo+IA0KPiBJIGRvbuKAmXQgc2VlIHdoYXQgTUtUTUUgaGFzIHRv
IGRvIHdpdGggdGhpcy4gVGhlIG9ubHkgcmVtb3RlbHkNCj4gcGxhdXNpYmxlIHdheSBJIGNhbiBz
ZWUgdG8gdXNlIE1LVE1FIGZvciB0aGlzIGlzIHRvIGhhdmUgdGhlDQo+IGh5cGVydmlzb3IgbG9h
ZCBhIFRQTSAob3Igb3RoZXIgZW5jbGF2ZSkgcHJvdGVjdGVkIGtleSBpbnRvIGFuIE1LVE1FDQo+
IHVzZXIga2V5IHNsb3QgYW5kIHRvIGxvYWQgY3VzdG9tZXItcHJvdmlkZWQgY2lwaGVydGV4dCBp
bnRvIHRoZQ0KPiBjb3JyZXNwb25kaW5nIHBoeXNpY2FsIG1lbW9yeSAodXNpbmcgYW4gTUtUTUUg
bm8tZW5jcnlwdCBzbG90KS4gIEJ1dA0KPiB0aGlzIGhhcyB0aHJlZSBtYWpvciBwcm9ibGVtcy4g
IEZpcnN0LCBpdCdzIGVmZmVjdGl2ZWx5IGp1c3QgYSBmYW5jeQ0KPiB3YXkgdG8gYXZvaWQgb25l
IEFFUyBwYXNzIG92ZXIgdGhlIGRhdGEuICBTZWNvbmQsIHNlbnNpYmxlIHNjaGVtZSBmb3INCj4g
dGhpcyB0eXBlIG9mIFZNIGltYWdlIHByb3RlY3Rpb24gd291bGQgdXNlICphdXRoZW50aWNhdGVk
KiBlbmNyeXB0aW9uDQo+IG9yIGF0IGxlYXN0IHZlcmlmeSBhIHNpZ25hdHVyZSwgd2hpY2ggTUtU
TUUgY2FuJ3QgZG8uICBUaGUgdGhpcmQNCj4gcHJvYmxlbSBpcyB0aGUgcmVhbCBzaG93LXN0b3Bw
ZXIsIHRob3VnaDogdGhpcyBzY2hlbWUgcmVxdWlyZXMgdGhhdA0KPiB0aGUgY2lwaGVydGV4dCBn
byBpbnRvIHByZWRldGVybWluZWQgcGh5c2ljYWwgYWRkcmVzc2VzLCB3aGljaCB3b3VsZA0KPiBi
ZSBhIGdpYW50IG1lc3MuDQoNCk15IGludGVudGlvbiB3YXMgdG8gc2F5IGlmIHdlIGFyZSBhbHJl
YWR5IHNlbmRpbmcga2V5IHRvIENTUCwgdGhlbiB3ZSBtYXkgcHJlZmVyIHRvIHVzZSB0aGUga2V5
IGZvcg0KTUtUTUUgVk0gcnVudGltZSBwcm90ZWN0aW9uIGFzIHdlbGwsIGJ1dCBsaWtlIHlvdSBz
YWlkIHdlIG1heSBub3QgaGF2ZSByZWFsIHNlY3VyaXR5IGdhaW4gaGVyZQ0KY29tcGFyaW5nIHRv
IFRNRSwgc28gSSBhZ3JlZSB3ZSBuZWVkIHRvIGZpbmQgb3V0IG9uZSBzcGVjaWZpYyBjYXNlIHRv
IHByb3ZlIHRoYXQuDQoNClRoYW5rcywNCi1LYWk

Re: [RFC v2 00/13] Multi-Key Total Memory Encryption API (MKTME)

[Huang, Kai]

> > There are some other use cases that already require tenant to send key to CSP. For example, the
> > VM
> > image can be provided by tenant and encrypted by tenant's own key, and tenant needs to send key
> > to
> > CSP when asking CSP to run that encrypted image.
> 
> 
> I can imagine a few reasons why one would want to encrypt one’s image.
> For example, the CSP could issue a public key and state, or even
> attest, that the key is wrapped and locked to particular PCRs of their
> TPM or otherwise protected by an enclave that verifies that the key is
> only used to decrypt the image for the benefit of a hypervisor.

Right. I think before tenant releases key to CSP it should always use attestation authority to
verify the trustiness of computer node. I can understand that the key can be wrapped by TPM before
sending to CSP but need some catch up about using enclave part. 

The thing is computer node can be trusted doesn't mean it cannot be attacked, or even it doesn't
mean it can prevent, ie some malicious admin, to get tenant key even by using legitimate way. There
are many SW components involved here. Anyway this is not related to MKTME itself like you mentioned
below, therefore the point is, as we already see MKTME itself provides very weak security
protection, we need to see whether MKTME has value from the whole use case's point of view
(including all the things you mentioned above) -- we define the whole use case, we clearly state
who/what should be in trust boundary, and what we can prevent, etc.

> 
> I don’t see what MKTME has to do with this. The only remotely
> plausible way I can see to use MKTME for this is to have the
> hypervisor load a TPM (or other enclave) protected key into an MKTME
> user key slot and to load customer-provided ciphertext into the
> corresponding physical memory (using an MKTME no-encrypt slot).  But
> this has three major problems.  First, it's effectively just a fancy
> way to avoid one AES pass over the data.  Second, sensible scheme for
> this type of VM image protection would use *authenticated* encryption
> or at least verify a signature, which MKTME can't do.  The third
> problem is the real show-stopper, though: this scheme requires that
> the ciphertext go into predetermined physical addresses, which would
> be a giant mess.

My intention was to say if we are already sending key to CSP, then we may prefer to use the key for
MKTME VM runtime protection as well, but like you said we may not have real security gain here
comparing to TME, so I agree we need to find out one specific case to prove that.

Thanks,
-Kai

Re: [RFC v2 00/13] Multi-Key Total Memory Encryption API (MKTME)

[Huang, Kai]

> > There are some other use cases that already require tenant to send key to CSP. For example, the
> > VM
> > image can be provided by tenant and encrypted by tenant's own key, and tenant needs to send key
> > to
> > CSP when asking CSP to run that encrypted image.
> 
> 
> I can imagine a few reasons why one would want to encrypt one’s image.
> For example, the CSP could issue a public key and state, or even
> attest, that the key is wrapped and locked to particular PCRs of their
> TPM or otherwise protected by an enclave that verifies that the key is
> only used to decrypt the image for the benefit of a hypervisor.

Right. I think before tenant releases key to CSP it should always use attestation authority to
verify the trustiness of computer node. I can understand that the key can be wrapped by TPM before
sending to CSP but need some catch up about using enclave part. 

The thing is computer node can be trusted doesn't mean it cannot be attacked, or even it doesn't
mean it can prevent, ie some malicious admin, to get tenant key even by using legitimate way. There
are many SW components involved here. Anyway this is not related to MKTME itself like you mentioned
below, therefore the point is, as we already see MKTME itself provides very weak security
protection, we need to see whether MKTME has value from the whole use case's point of view
(including all the things you mentioned above) -- we define the whole use case, we clearly state
who/what should be in trust boundary, and what we can prevent, etc.

> 
> I don’t see what MKTME has to do with this. The only remotely
> plausible way I can see to use MKTME for this is to have the
> hypervisor load a TPM (or other enclave) protected key into an MKTME
> user key slot and to load customer-provided ciphertext into the
> corresponding physical memory (using an MKTME no-encrypt slot).  But
> this has three major problems.  First, it's effectively just a fancy
> way to avoid one AES pass over the data.  Second, sensible scheme for
> this type of VM image protection would use *authenticated* encryption
> or at least verify a signature, which MKTME can't do.  The third
> problem is the real show-stopper, though: this scheme requires that
> the ciphertext go into predetermined physical addresses, which would
> be a giant mess.

My intention was to say if we are already sending key to CSP, then we may prefer to use the key for
MKTME VM runtime protection as well, but like you said we may not have real security gain here
comparing to TME, so I agree we need to find out one specific case to prove that.

Thanks,
-Kai

Re: [RFC v2 00/13] Multi-Key Total Memory Encryption API (MKTME)

[Sakkinen, Jarkko]

T24gU2F0LCAyMDE4LTEyLTA4IGF0IDA5OjMzICswODAwLCBIdWFuZywgS2FpIHdyb3RlOg0KPiBD
dXJyZW50bHkgdGhlcmUncyBubyBub25jZSB0byBwcm90ZWN0IGNhY2hlIGxpbmUgc28gVE1FL01L
VE1FIGlzIG5vdCBhYmxlIHRvDQo+IHByZXZlbnQgcmVwbGF5IGF0dGFjaw0KPiB5b3UgbWVudGlv
bmVkLiBDdXJyZW50bHkgTUtUTUUgb25seSBpbnZvbHZlcyBBRVMtWFRTLTEyOCBlbmNyeXB0aW9u
IGJ1dA0KPiBub3RoaW5nIGVsc2UuIEJ1dCBsaWtlIEkNCj4gc2FpZCBpZiBJIHVuZGVyc3RhbmQg
Y29ycmVjdGx5IGV2ZW4gU0VWIGRvZXNuJ3QgaGF2ZSBpbnRlZ3JpdHkgcHJvdGVjdGlvbiBzbw0K
PiBub3QgYWJsZSB0byBwcmV2ZW50DQo+IHJlcGx5IGF0dGFjayBhcyB3ZWxsLg0KDQpZb3UncmUg
YWJzb2x1dGVseSBjb3JyZWN0Lg0KDQpUaGVyZSdzIGEgYWxzbyBnb29kIHBhcGVyIG9uIFNFViBz
dWJ2ZXJ0aW9uOg0KDQpodHRwczovL2FyeGl2Lm9yZy9wZGYvMTgwNS4wOTYwNC5wZGYNCg0KSSBk
b24ndCB0aGluayB0aGlzIG1ha2VzIE1LVE1FIG9yIFNFViB1c2Vsc3MsIGJ1dCB5ZWFoLCBpdCBp
cyBhDQpjb25zdHJhaW50IHRoYXQgbmVlZHMgdG8gYmUgdGFrZW4gaW50byBjb25zaWRlcmF0aW9u
IHdoZW4gZmluZGluZyB0aGUNCmJlc3Qgd2F5IHRvIHVzZSB0aGVzZSB0ZWNobm9sb2dpZXMgaW4g
TGludXguDQoNCi9KYXJra28NCg=

Re: [RFC v2 00/13] Multi-Key Total Memory Encryption API (MKTME)

[Sakkinen, Jarkko]

On Sat, 2018-12-08 at 09:33 +0800, Huang, Kai wrote:
> Currently there's no nonce to protect cache line so TME/MKTME is not able to
> prevent replay attack
> you mentioned. Currently MKTME only involves AES-XTS-128 encryption but
> nothing else. But like I
> said if I understand correctly even SEV doesn't have integrity protection so
> not able to prevent
> reply attack as well.

You're absolutely correct.

There's a also good paper on SEV subvertion:

https://arxiv.org/pdf/1805.09604.pdf

I don't think this makes MKTME or SEV uselss, but yeah, it is a
constraint that needs to be taken into consideration when finding the
best way to use these technologies in Linux.

/Jarkko

Re: [RFC v2 00/13] Multi-Key Total Memory Encryption API (MKTME)

[Sakkinen, Jarkko]

On Sat, 2018-12-08 at 09:33 +0800, Huang, Kai wrote:
> Currently there's no nonce to protect cache line so TME/MKTME is not able to
> prevent replay attack
> you mentioned. Currently MKTME only involves AES-XTS-128 encryption but
> nothing else. But like I
> said if I understand correctly even SEV doesn't have integrity protection so
> not able to prevent
> reply attack as well.

You're absolutely correct.

There's a also good paper on SEV subvertion:

https://arxiv.org/pdf/1805.09604.pdf

I don't think this makes MKTME or SEV uselss, but yeah, it is a
constraint that needs to be taken into consideration when finding the
best way to use these technologies in Linux.

/Jarkko

Re: [RFC v2 00/13] Multi-Key Total Memory Encryption API (MKTME)

[Sakkinen, Jarkko]

T24gRnJpLCAyMDE4LTEyLTA3IGF0IDE1OjQ1IC0wODAwLCBKYXJra28gU2Fra2luZW4gd3JvdGU6
DQo+IFRoZSBicnV0YWwgZmFjdCBpcyB0aGF0IGEgcGh5c2ljYWwgYWRkcmVzcyBpcyBhbiBhc3Ry
b25vbWljYWwgc3RyZXRjaA0KPiBmcm9tIGEgcmFuZG9tIHZhbHVlIG9yIGluY3JlYXNpbmcgY291
bnRlci4gVGh1cywgaXQgaXMgZmFpciB0byBzYXkgdGhhdA0KPiBNS1RNRSBwcm92aWRlcyBvbmx5
IG5haXZlIG1lYXN1cmVzIGFnYWluc3QgcmVwbGF5IGF0dGFja3MuLi4NCg0KSSdsbCB0cnkgdG8g
c3VtbWFyaXplIGhvdyBJIHVuZGVyc3RhbmQgdGhlIGhpZ2ggbGV2ZWwgc2VjdXJpdHkNCm1vZGVs
IG9mIE1LVE1FIGJlY2F1c2UgKHdvdWxkIGJlIGdvb2QgaWRlYSB0byBkb2N1bWVudCBpdCkuDQoN
CkFzc3VtcHRpb25zOg0KDQoxLiBUaGUgaHlwZXJ2aXNvciBoYXMgbm90IGJlZW4gaW5maWx0cmF0
ZWQuDQoyLiBUaGUgaHlwZXJ2aXNvciBkb2VzIG5vdCBsZWFrIHNlY3JldHMuDQoNCldoZW4gKDEp
IGFuZCAoMikgaG9sZCBbMV0sIHdlIGhhcmRlbiBWTXMgaW4gdHdvIGRpZmZlcmVudCB3YXlzOg0K
DQpBLiBWTXMgY2Fubm90IGxlYWsgZGF0YSB0byBlYWNoIG90aGVyIG9yIGNhbiB0aGV5IHdpdGgg
TDFURiB3aGVuIEhUDQogICBpcyBlbmFibGVkPw0KQi4gUHJvdGVjdHMgYWdhaW5zdCBjb2xkIGJv
b3QgYXR0YWNrcy4NCg0KSXNuJ3QgdGhpcyB3aGF0IHRoaXMgYWJvdXQgaW4gdGhlIG51dHNoZWxs
IHJvdWdobHk/DQoNClsxXSBYUEZPIGNvdWxkIHBvdGVudGlhbGx5IGJlIGFuIG9wdC1pbiBmZWF0
dXJlIHRoYXQgcmVkdWNlcyB0aGUNCiAgICBkYW1hZ2Ugd2hlbiBlaXRoZXIgb2YgdGhlc2UgYXNz
dW1wdGlvbnMgaGFzIGJlZW4gYnJva2VuLg0KDQovSmFya2tvDQo

Re: [RFC v2 00/13] Multi-Key Total Memory Encryption API (MKTME)

[Sakkinen, Jarkko]

On Fri, 2018-12-07 at 15:45 -0800, Jarkko Sakkinen wrote:
> The brutal fact is that a physical address is an astronomical stretch
> from a random value or increasing counter. Thus, it is fair to say that
> MKTME provides only naive measures against replay attacks...

I'll try to summarize how I understand the high level security
model of MKTME because (would be good idea to document it).

Assumptions:

1. The hypervisor has not been infiltrated.
2. The hypervisor does not leak secrets.

When (1) and (2) hold [1], we harden VMs in two different ways:

A. VMs cannot leak data to each other or can they with L1TF when HT
   is enabled?
B. Protects against cold boot attacks.

Isn't this what this about in the nutshell roughly?

[1] XPFO could potentially be an opt-in feature that reduces the
    damage when either of these assumptions has been broken.

/Jarkko

Re: [RFC v2 00/13] Multi-Key Total Memory Encryption API (MKTME)

[Sakkinen, Jarkko]

On Fri, 2018-12-07 at 15:45 -0800, Jarkko Sakkinen wrote:
> The brutal fact is that a physical address is an astronomical stretch
> from a random value or increasing counter. Thus, it is fair to say that
> MKTME provides only naive measures against replay attacks...

I'll try to summarize how I understand the high level security
model of MKTME because (would be good idea to document it).

Assumptions:

1. The hypervisor has not been infiltrated.
2. The hypervisor does not leak secrets.

When (1) and (2) hold [1], we harden VMs in two different ways:

A. VMs cannot leak data to each other or can they with L1TF when HT
   is enabled?
B. Protects against cold boot attacks.

Isn't this what this about in the nutshell roughly?

[1] XPFO could potentially be an opt-in feature that reduces the
    damage when either of these assumptions has been broken.

/Jarkko

Re: [RFC v2 00/13] Multi-Key Total Memory Encryption API (MKTME)

[Andy Lutomirski]

On Wed, Dec 12, 2018 at 7:31 AM Sakkinen, Jarkko
<jarkko.sakkinen@intel.com> wrote:
>
> On Fri, 2018-12-07 at 15:45 -0800, Jarkko Sakkinen wrote:
> > The brutal fact is that a physical address is an astronomical stretch
> > from a random value or increasing counter. Thus, it is fair to say that
> > MKTME provides only naive measures against replay attacks...
>
> I'll try to summarize how I understand the high level security
> model of MKTME because (would be good idea to document it).
>
> Assumptions:
>
> 1. The hypervisor has not been infiltrated.
> 2. The hypervisor does not leak secrets.
>
> When (1) and (2) hold [1], we harden VMs in two different ways:
>
> A. VMs cannot leak data to each other or can they with L1TF when HT
>    is enabled?

I strongly suspect that, on L1TF-vulnerable CPUs, MKTME provides no
protection whatsoever.  It sounds like MKTME is implemented in the
memory controller -- as far as the rest of the CPU and the cache
hierarchy are concerned, the MKTME key selction bits are just part of
the physical address.  So an attack like L1TF that leaks a cacheline
that's selected by physical address will leak the cleartext if the key
selection bits are set correctly.

(I suppose that, if the attacker needs to brute-force the physical
address, then MKTME makes it a bit harder because the effective
physical address space is larger.)

> B. Protects against cold boot attacks.

TME does this, AFAIK.  MKTME does, too, unless the "user" mode is
used, in which case the protection is weaker.

>
> Isn't this what this about in the nutshell roughly?
>
> [1] XPFO could potentially be an opt-in feature that reduces the
>     damage when either of these assumptions has been broken.
>
> /Jarkko

Re: [RFC v2 00/13] Multi-Key Total Memory Encryption API (MKTME)

[Andy Lutomirski]

On Wed, Dec 12, 2018 at 7:31 AM Sakkinen, Jarkko
<jarkko.sakkinen@intel.com> wrote:
>
> On Fri, 2018-12-07 at 15:45 -0800, Jarkko Sakkinen wrote:
> > The brutal fact is that a physical address is an astronomical stretch
> > from a random value or increasing counter. Thus, it is fair to say that
> > MKTME provides only naive measures against replay attacks...
>
> I'll try to summarize how I understand the high level security
> model of MKTME because (would be good idea to document it).
>
> Assumptions:
>
> 1. The hypervisor has not been infiltrated.
> 2. The hypervisor does not leak secrets.
>
> When (1) and (2) hold [1], we harden VMs in two different ways:
>
> A. VMs cannot leak data to each other or can they with L1TF when HT
>    is enabled?

I strongly suspect that, on L1TF-vulnerable CPUs, MKTME provides no
protection whatsoever.  It sounds like MKTME is implemented in the
memory controller -- as far as the rest of the CPU and the cache
hierarchy are concerned, the MKTME key selction bits are just part of
the physical address.  So an attack like L1TF that leaks a cacheline
that's selected by physical address will leak the cleartext if the key
selection bits are set correctly.

(I suppose that, if the attacker needs to brute-force the physical
address, then MKTME makes it a bit harder because the effective
physical address space is larger.)

> B. Protects against cold boot attacks.

TME does this, AFAIK.  MKTME does, too, unless the "user" mode is
used, in which case the protection is weaker.

>
> Isn't this what this about in the nutshell roughly?
>
> [1] XPFO could potentially be an opt-in feature that reduces the
>     damage when either of these assumptions has been broken.
>
> /Jarkko

Re: [RFC v2 00/13] Multi-Key Total Memory Encryption API (MKTME)

[Sakkinen, Jarkko]

T24gV2VkLCAyMDE4LTEyLTEyIGF0IDA4OjI5IC0wODAwLCBBbmR5IEx1dG9taXJza2kgd3JvdGU6
DQo+IE9uIFdlZCwgRGVjIDEyLCAyMDE4IGF0IDc6MzEgQU0gU2Fra2luZW4sIEphcmtrbw0KPiA8
amFya2tvLnNha2tpbmVuQGludGVsLmNvbT4gd3JvdGU6DQo+ID4gT24gRnJpLCAyMDE4LTEyLTA3
IGF0IDE1OjQ1IC0wODAwLCBKYXJra28gU2Fra2luZW4gd3JvdGU6DQo+ID4gPiBUaGUgYnJ1dGFs
IGZhY3QgaXMgdGhhdCBhIHBoeXNpY2FsIGFkZHJlc3MgaXMgYW4gYXN0cm9ub21pY2FsIHN0cmV0
Y2gNCj4gPiA+IGZyb20gYSByYW5kb20gdmFsdWUgb3IgaW5jcmVhc2luZyBjb3VudGVyLiBUaHVz
LCBpdCBpcyBmYWlyIHRvIHNheSB0aGF0DQo+ID4gPiBNS1RNRSBwcm92aWRlcyBvbmx5IG5haXZl
IG1lYXN1cmVzIGFnYWluc3QgcmVwbGF5IGF0dGFja3MuLi4NCj4gPiANCj4gPiBJJ2xsIHRyeSB0
byBzdW1tYXJpemUgaG93IEkgdW5kZXJzdGFuZCB0aGUgaGlnaCBsZXZlbCBzZWN1cml0eQ0KPiA+
IG1vZGVsIG9mIE1LVE1FIGJlY2F1c2UgKHdvdWxkIGJlIGdvb2QgaWRlYSB0byBkb2N1bWVudCBp
dCkuDQo+ID4gDQo+ID4gQXNzdW1wdGlvbnM6DQo+ID4gDQo+ID4gMS4gVGhlIGh5cGVydmlzb3Ig
aGFzIG5vdCBiZWVuIGluZmlsdHJhdGVkLg0KPiA+IDIuIFRoZSBoeXBlcnZpc29yIGRvZXMgbm90
IGxlYWsgc2VjcmV0cy4NCj4gPiANCj4gPiBXaGVuICgxKSBhbmQgKDIpIGhvbGQgWzFdLCB3ZSBo
YXJkZW4gVk1zIGluIHR3byBkaWZmZXJlbnQgd2F5czoNCj4gPiANCj4gPiBBLiBWTXMgY2Fubm90
IGxlYWsgZGF0YSB0byBlYWNoIG90aGVyIG9yIGNhbiB0aGV5IHdpdGggTDFURiB3aGVuIEhUDQo+
ID4gICAgaXMgZW5hYmxlZD8NCj4gDQo+IEkgc3Ryb25nbHkgc3VzcGVjdCB0aGF0LCBvbiBMMVRG
LXZ1bG5lcmFibGUgQ1BVcywgTUtUTUUgcHJvdmlkZXMgbm8NCj4gcHJvdGVjdGlvbiB3aGF0c29l
dmVyLiAgSXQgc291bmRzIGxpa2UgTUtUTUUgaXMgaW1wbGVtZW50ZWQgaW4gdGhlDQo+IG1lbW9y
eSBjb250cm9sbGVyIC0tIGFzIGZhciBhcyB0aGUgcmVzdCBvZiB0aGUgQ1BVIGFuZCB0aGUgY2Fj
aGUNCj4gaGllcmFyY2h5IGFyZSBjb25jZXJuZWQsIHRoZSBNS1RNRSBrZXkgc2VsY3Rpb24gYml0
cyBhcmUganVzdCBwYXJ0IG9mDQo+IHRoZSBwaHlzaWNhbCBhZGRyZXNzLiAgU28gYW4gYXR0YWNr
IGxpa2UgTDFURiB0aGF0IGxlYWtzIGEgY2FjaGVsaW5lDQo+IHRoYXQncyBzZWxlY3RlZCBieSBw
aHlzaWNhbCBhZGRyZXNzIHdpbGwgbGVhayB0aGUgY2xlYXJ0ZXh0IGlmIHRoZSBrZXkNCj4gc2Vs
ZWN0aW9uIGJpdHMgYXJlIHNldCBjb3JyZWN0bHkuDQo+IA0KPiAoSSBzdXBwb3NlIHRoYXQsIGlm
IHRoZSBhdHRhY2tlciBuZWVkcyB0byBicnV0ZS1mb3JjZSB0aGUgcGh5c2ljYWwNCj4gYWRkcmVz
cywgdGhlbiBNS1RNRSBtYWtlcyBpdCBhIGJpdCBoYXJkZXIgYmVjYXVzZSB0aGUgZWZmZWN0aXZl
DQo+IHBoeXNpY2FsIGFkZHJlc3Mgc3BhY2UgaXMgbGFyZ2VyLikNCj4gDQo+ID4gQi4gUHJvdGVj
dHMgYWdhaW5zdCBjb2xkIGJvb3QgYXR0YWNrcy4NCj4gDQo+IFRNRSBkb2VzIHRoaXMsIEFGQUlL
LiAgTUtUTUUgZG9lcywgdG9vLCB1bmxlc3MgdGhlICJ1c2VyIiBtb2RlIGlzDQo+IHVzZWQsIGlu
IHdoaWNoIGNhc2UgdGhlIHByb3RlY3Rpb24gaXMgd2Vha2VyLg0KPiANCj4gPiBJc24ndCB0aGlz
IHdoYXQgdGhpcyBhYm91dCBpbiB0aGUgbnV0c2hlbGwgcm91Z2hseT8NCj4gPiANCj4gPiBbMV0g
WFBGTyBjb3VsZCBwb3RlbnRpYWxseSBiZSBhbiBvcHQtaW4gZmVhdHVyZSB0aGF0IHJlZHVjZXMg
dGhlDQo+ID4gICAgIGRhbWFnZSB3aGVuIGVpdGhlciBvZiB0aGVzZSBhc3N1bXB0aW9ucyBoYXMg
YmVlbiBicm9rZW4uDQoNClRoaXMgYWxsIHNob3VsZCBiZSBzdW1tYXJpemVkIGluIHRoZSBkb2N1
bWVudGF0aW9uIChoaWdoLWxldmVsIG1vZGVsDQphbmQgY29ybmVyIGNhc2VzKS4NCg0KL0phcmtr
bw0K

Re: [RFC v2 00/13] Multi-Key Total Memory Encryption API (MKTME)

[Sakkinen, Jarkko]

On Wed, 2018-12-12 at 08:29 -0800, Andy Lutomirski wrote:
> On Wed, Dec 12, 2018 at 7:31 AM Sakkinen, Jarkko
> <jarkko.sakkinen@intel.com> wrote:
> > On Fri, 2018-12-07 at 15:45 -0800, Jarkko Sakkinen wrote:
> > > The brutal fact is that a physical address is an astronomical stretch
> > > from a random value or increasing counter. Thus, it is fair to say that
> > > MKTME provides only naive measures against replay attacks...
> > 
> > I'll try to summarize how I understand the high level security
> > model of MKTME because (would be good idea to document it).
> > 
> > Assumptions:
> > 
> > 1. The hypervisor has not been infiltrated.
> > 2. The hypervisor does not leak secrets.
> > 
> > When (1) and (2) hold [1], we harden VMs in two different ways:
> > 
> > A. VMs cannot leak data to each other or can they with L1TF when HT
> >    is enabled?
> 
> I strongly suspect that, on L1TF-vulnerable CPUs, MKTME provides no
> protection whatsoever.  It sounds like MKTME is implemented in the
> memory controller -- as far as the rest of the CPU and the cache
> hierarchy are concerned, the MKTME key selction bits are just part of
> the physical address.  So an attack like L1TF that leaks a cacheline
> that's selected by physical address will leak the cleartext if the key
> selection bits are set correctly.
> 
> (I suppose that, if the attacker needs to brute-force the physical
> address, then MKTME makes it a bit harder because the effective
> physical address space is larger.)
> 
> > B. Protects against cold boot attacks.
> 
> TME does this, AFAIK.  MKTME does, too, unless the "user" mode is
> used, in which case the protection is weaker.
> 
> > Isn't this what this about in the nutshell roughly?
> > 
> > [1] XPFO could potentially be an opt-in feature that reduces the
> >     damage when either of these assumptions has been broken.

This all should be summarized in the documentation (high-level model
and corner cases).

/Jarkko

Re: [RFC v2 00/13] Multi-Key Total Memory Encryption API (MKTME)

[Sakkinen, Jarkko]

On Wed, 2018-12-12 at 08:29 -0800, Andy Lutomirski wrote:
> On Wed, Dec 12, 2018 at 7:31 AM Sakkinen, Jarkko
> <jarkko.sakkinen@intel.com> wrote:
> > On Fri, 2018-12-07 at 15:45 -0800, Jarkko Sakkinen wrote:
> > > The brutal fact is that a physical address is an astronomical stretch
> > > from a random value or increasing counter. Thus, it is fair to say that
> > > MKTME provides only naive measures against replay attacks...
> > 
> > I'll try to summarize how I understand the high level security
> > model of MKTME because (would be good idea to document it).
> > 
> > Assumptions:
> > 
> > 1. The hypervisor has not been infiltrated.
> > 2. The hypervisor does not leak secrets.
> > 
> > When (1) and (2) hold [1], we harden VMs in two different ways:
> > 
> > A. VMs cannot leak data to each other or can they with L1TF when HT
> >    is enabled?
> 
> I strongly suspect that, on L1TF-vulnerable CPUs, MKTME provides no
> protection whatsoever.  It sounds like MKTME is implemented in the
> memory controller -- as far as the rest of the CPU and the cache
> hierarchy are concerned, the MKTME key selction bits are just part of
> the physical address.  So an attack like L1TF that leaks a cacheline
> that's selected by physical address will leak the cleartext if the key
> selection bits are set correctly.
> 
> (I suppose that, if the attacker needs to brute-force the physical
> address, then MKTME makes it a bit harder because the effective
> physical address space is larger.)
> 
> > B. Protects against cold boot attacks.
> 
> TME does this, AFAIK.  MKTME does, too, unless the "user" mode is
> used, in which case the protection is weaker.
> 
> > Isn't this what this about in the nutshell roughly?
> > 
> > [1] XPFO could potentially be an opt-in feature that reduces the
> >     damage when either of these assumptions has been broken.

This all should be summarized in the documentation (high-level model
and corner cases).

/Jarkko

RE: [RFC v2 00/13] Multi-Key Total Memory Encryption API (MKTME)

[Huang, Kai]

PiBJIHN0cm9uZ2x5IHN1c3BlY3QgdGhhdCwgb24gTDFURi12dWxuZXJhYmxlIENQVXMsIE1LVE1F
IHByb3ZpZGVzIG5vDQo+IHByb3RlY3Rpb24gd2hhdHNvZXZlci4gIEl0IHNvdW5kcyBsaWtlIE1L
VE1FIGlzIGltcGxlbWVudGVkIGluIHRoZQ0KPiBtZW1vcnkgY29udHJvbGxlciAtLSBhcyBmYXIg
YXMgdGhlIHJlc3Qgb2YgdGhlIENQVSBhbmQgdGhlIGNhY2hlIGhpZXJhcmNoeQ0KPiBhcmUgY29u
Y2VybmVkLCB0aGUgTUtUTUUga2V5IHNlbGN0aW9uIGJpdHMgYXJlIGp1c3QgcGFydCBvZiB0aGUg
cGh5c2ljYWwNCj4gYWRkcmVzcy4gIFNvIGFuIGF0dGFjayBsaWtlIEwxVEYgdGhhdCBsZWFrcyBh
IGNhY2hlbGluZSB0aGF0J3Mgc2VsZWN0ZWQgYnkNCj4gcGh5c2ljYWwgYWRkcmVzcyB3aWxsIGxl
YWsgdGhlIGNsZWFydGV4dCBpZiB0aGUga2V5IHNlbGVjdGlvbiBiaXRzIGFyZSBzZXQNCj4gY29y
cmVjdGx5Lg0KDQpSaWdodC4gTUtUTUUgZG9lc24ndCBwcmV2ZW50IGNhY2hlIGJhc2VkIGF0dGFj
ay4gRGF0YSBpbiBjYWNoZSBpcyBpbiBjbGVhci4NCg0KVGhhbmtzLA0KLUthaQ0KDQo

RE: [RFC v2 00/13] Multi-Key Total Memory Encryption API (MKTME)

[Huang, Kai]

> I strongly suspect that, on L1TF-vulnerable CPUs, MKTME provides no
> protection whatsoever.  It sounds like MKTME is implemented in the
> memory controller -- as far as the rest of the CPU and the cache hierarchy
> are concerned, the MKTME key selction bits are just part of the physical
> address.  So an attack like L1TF that leaks a cacheline that's selected by
> physical address will leak the cleartext if the key selection bits are set
> correctly.

Right. MKTME doesn't prevent cache based attack. Data in cache is in clear.

Thanks,
-Kai

RE: [RFC v2 00/13] Multi-Key Total Memory Encryption API (MKTME)

[Huang, Kai]

> I strongly suspect that, on L1TF-vulnerable CPUs, MKTME provides no
> protection whatsoever.  It sounds like MKTME is implemented in the
> memory controller -- as far as the rest of the CPU and the cache hierarchy
> are concerned, the MKTME key selction bits are just part of the physical
> address.  So an attack like L1TF that leaks a cacheline that's selected by
> physical address will leak the cleartext if the key selection bits are set
> correctly.

Right. MKTME doesn't prevent cache based attack. Data in cache is in clear.

Thanks,
-Kai

RE: [RFC v2 00/13] Multi-Key Total Memory Encryption API (MKTME)

[Huang, Kai]

PiBUaGlzIGFsbCBzaG91bGQgYmUgc3VtbWFyaXplZCBpbiB0aGUgZG9jdW1lbnRhdGlvbiAoaGln
aC1sZXZlbCBtb2RlbCBhbmQNCj4gY29ybmVyIGNhc2VzKS4NCg0KSSBhbSBub3Qgc3VyZSB3aGV0
aGVyIGl0IGlzIG5lY2Vzc2FyeSB0byBkb2N1bWVudCBMMVRGIGV4cGxpY2l0bHksIHNpbmNlIGl0
IGlzIHF1aXRlIG9idmlvdXMgdGhhdCBNS1RNRSBkb2Vzbid0IHByZXZlbnQgdGhhdC4gSU1ITyBp
ZiBuZWVkZWQgd2Ugb25seSBuZWVkIHRvIG1lbnRpb24gTUtUTUUgZG9lc24ndCBwcmV2ZW50IGFu
eSBzb3J0IG9mIGNhY2hlIGJhc2VkIGF0dGFjaywgc2luY2UgZGF0YSBpbiBjYWNoZSBpcyBpbiBj
bGVhci4NCg0KSW4gZmFjdCBTR1ggZG9lc24ndCBwcmV2ZW50IHRoaXMgZWl0aGVyLi4NCg0KVGhh
bmtzLA0KLUthaQ0K

RE: [RFC v2 00/13] Multi-Key Total Memory Encryption API (MKTME)

[Huang, Kai]

> This all should be summarized in the documentation (high-level model and
> corner cases).

I am not sure whether it is necessary to document L1TF explicitly, since it is quite obvious that MKTME doesn't prevent that. IMHO if needed we only need to mention MKTME doesn't prevent any sort of cache based attack, since data in cache is in clear.

In fact SGX doesn't prevent this either..

Thanks,
-Kai

RE: [RFC v2 00/13] Multi-Key Total Memory Encryption API (MKTME)

[Huang, Kai]

> This all should be summarized in the documentation (high-level model and
> corner cases).

I am not sure whether it is necessary to document L1TF explicitly, since it is quite obvious that MKTME doesn't prevent that. IMHO if needed we only need to mention MKTME doesn't prevent any sort of cache based attack, since data in cache is in clear.

In fact SGX doesn't prevent this either..

Thanks,
-Kai

Re: [RFC v2 00/13] Multi-Key Total Memory Encryption API (MKTME)

[Sakkinen, Jarkko]

T24gVGh1LCAyMDE4LTEyLTEzIGF0IDA3OjI3ICswODAwLCBIdWFuZywgS2FpIHdyb3RlOg0KPiA+
IFRoaXMgYWxsIHNob3VsZCBiZSBzdW1tYXJpemVkIGluIHRoZSBkb2N1bWVudGF0aW9uIChoaWdo
LWxldmVsIG1vZGVsIGFuZA0KPiA+IGNvcm5lciBjYXNlcykuDQo+IA0KPiBJIGFtIG5vdCBzdXJl
IHdoZXRoZXIgaXQgaXMgbmVjZXNzYXJ5IHRvIGRvY3VtZW50IEwxVEYgZXhwbGljaXRseSwgc2lu
Y2UgaXQgaXMNCj4gcXVpdGUgb2J2aW91cyB0aGF0IE1LVE1FIGRvZXNuJ3QgcHJldmVudCB0aGF0
LiBJTUhPIGlmIG5lZWRlZCB3ZSBvbmx5IG5lZWQgdG8NCj4gbWVudGlvbiBNS1RNRSBkb2Vzbid0
IHByZXZlbnQgYW55IHNvcnQgb2YgY2FjaGUgYmFzZWQgYXR0YWNrLCBzaW5jZSBkYXRhIGluDQo+
IGNhY2hlIGlzIGluIGNsZWFyLg0KPiANCj4gSW4gZmFjdCBTR1ggZG9lc24ndCBwcmV2ZW50IHRo
aXMgZWl0aGVyLi4NCg0KU29ycnksIHdhcyBhIGJpdCB1bmNsZWFyLiBJIG1lYW50IHRoZSBhc3N1
bXB0aW9ucyBhbmQgZ29hbHMuDQoNCi9KYXJra28NCg=

Re: [RFC v2 00/13] Multi-Key Total Memory Encryption API (MKTME)

[Sakkinen, Jarkko]

On Thu, 2018-12-13 at 07:27 +0800, Huang, Kai wrote:
> > This all should be summarized in the documentation (high-level model and
> > corner cases).
> 
> I am not sure whether it is necessary to document L1TF explicitly, since it is
> quite obvious that MKTME doesn't prevent that. IMHO if needed we only need to
> mention MKTME doesn't prevent any sort of cache based attack, since data in
> cache is in clear.
> 
> In fact SGX doesn't prevent this either..

Sorry, was a bit unclear. I meant the assumptions and goals.

/Jarkko

Re: [RFC v2 00/13] Multi-Key Total Memory Encryption API (MKTME)

[Sakkinen, Jarkko]

On Thu, 2018-12-13 at 07:27 +0800, Huang, Kai wrote:
> > This all should be summarized in the documentation (high-level model and
> > corner cases).
> 
> I am not sure whether it is necessary to document L1TF explicitly, since it is
> quite obvious that MKTME doesn't prevent that. IMHO if needed we only need to
> mention MKTME doesn't prevent any sort of cache based attack, since data in
> cache is in clear.
> 
> In fact SGX doesn't prevent this either..

Sorry, was a bit unclear. I meant the assumptions and goals.

/Jarkko

Re: [RFC v2 00/13] Multi-Key Total Memory Encryption API (MKTME)

[Sakkinen, Jarkko]

T24gVGh1LCAyMDE4LTEyLTEzIGF0IDA3OjQ5ICswMjAwLCBKYXJra28gU2Fra2luZW4gd3JvdGU6
DQo+IE9uIFRodSwgMjAxOC0xMi0xMyBhdCAwNzoyNyArMDgwMCwgSHVhbmcsIEthaSB3cm90ZToN
Cj4gPiA+IFRoaXMgYWxsIHNob3VsZCBiZSBzdW1tYXJpemVkIGluIHRoZSBkb2N1bWVudGF0aW9u
IChoaWdoLWxldmVsIG1vZGVsIGFuZA0KPiA+ID4gY29ybmVyIGNhc2VzKS4NCj4gPiANCj4gPiBJ
IGFtIG5vdCBzdXJlIHdoZXRoZXIgaXQgaXMgbmVjZXNzYXJ5IHRvIGRvY3VtZW50IEwxVEYgZXhw
bGljaXRseSwgc2luY2UgaXQNCj4gPiBpcw0KPiA+IHF1aXRlIG9idmlvdXMgdGhhdCBNS1RNRSBk
b2Vzbid0IHByZXZlbnQgdGhhdC4gSU1ITyBpZiBuZWVkZWQgd2Ugb25seSBuZWVkDQo+ID4gdG8N
Cj4gPiBtZW50aW9uIE1LVE1FIGRvZXNuJ3QgcHJldmVudCBhbnkgc29ydCBvZiBjYWNoZSBiYXNl
ZCBhdHRhY2ssIHNpbmNlIGRhdGEgaW4NCj4gPiBjYWNoZSBpcyBpbiBjbGVhci4NCj4gPiANCj4g
PiBJbiBmYWN0IFNHWCBkb2Vzbid0IHByZXZlbnQgdGhpcyBlaXRoZXIuLg0KPiANCj4gU29ycnks
IHdhcyBhIGJpdCB1bmNsZWFyLiBJIG1lYW50IHRoZSBhc3N1bXB0aW9ucyBhbmQgZ29hbHMuDQoN
CkkuZS4gd2hhdCBJIHB1dCBpbiBteSBlYXJsaWVyIHJlc3BvbnNlLCB3aGF0IGJlbG9uZ3MgdG8g
VENCIGFuZCB3aGF0DQp0eXBlcyBhZHZlcnNhcmllcyBpcyBwdXJzdWVkIHRvIGJlIHByb3RlY3Rl
ZC4NCg0KL0phcmtrbw0K

Re: [RFC v2 00/13] Multi-Key Total Memory Encryption API (MKTME)

[Sakkinen, Jarkko]

On Thu, 2018-12-13 at 07:49 +0200, Jarkko Sakkinen wrote:
> On Thu, 2018-12-13 at 07:27 +0800, Huang, Kai wrote:
> > > This all should be summarized in the documentation (high-level model and
> > > corner cases).
> > 
> > I am not sure whether it is necessary to document L1TF explicitly, since it
> > is
> > quite obvious that MKTME doesn't prevent that. IMHO if needed we only need
> > to
> > mention MKTME doesn't prevent any sort of cache based attack, since data in
> > cache is in clear.
> > 
> > In fact SGX doesn't prevent this either..
> 
> Sorry, was a bit unclear. I meant the assumptions and goals.

I.e. what I put in my earlier response, what belongs to TCB and what
types adversaries is pursued to be protected.

/Jarkko

Re: [RFC v2 00/13] Multi-Key Total Memory Encryption API (MKTME)

[Sakkinen, Jarkko]

On Thu, 2018-12-13 at 07:49 +0200, Jarkko Sakkinen wrote:
> On Thu, 2018-12-13 at 07:27 +0800, Huang, Kai wrote:
> > > This all should be summarized in the documentation (high-level model and
> > > corner cases).
> > 
> > I am not sure whether it is necessary to document L1TF explicitly, since it
> > is
> > quite obvious that MKTME doesn't prevent that. IMHO if needed we only need
> > to
> > mention MKTME doesn't prevent any sort of cache based attack, since data in
> > cache is in clear.
> > 
> > In fact SGX doesn't prevent this either..
> 
> Sorry, was a bit unclear. I meant the assumptions and goals.

I.e. what I put in my earlier response, what belongs to TCB and what
types adversaries is pursued to be protected.

/Jarkko