[PATCH v11 00/13] Intel SGX1 support

[PATCH v11 00/13] Intel SGX1 support

[Jarkko Sakkinen]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain; charset=y, Size: 13643 bytes --]

Intel(R) SGX is a set of CPU instructions that can be used by applications
to set aside private regions of code and data. The code outside the enclave
is disallowed to access the memory inside the enclave by the CPU access
control.  In a way you can think that SGX provides inverted sandbox. It
protects the application from a malicious host.

There is a new hardware unit in the processor called Memory Encryption
Engine (MEE) starting from the Skylake microacrhitecture. BIOS can define
one or many MEE regions that can hold enclave data by configuring them with
PRMRR registers.

The MEE automatically encrypts the data leaving the processor package to
the MEE regions. The data is encrypted using a random key whose life-time
is exactly one power cycle.

You can tell if your CPU supports SGX by looking into /proc/cpuinfo:

	cat /proc/cpuinfo  | grep sgx

v11:
* Polished ENCLS wrappers with refined exception handling.
* ksgxswapd was not stopped (regression in v5) in
  sgx_page_cache_teardown(), which causes a leaked kthread after driver
  deinitialization.
* Shutdown sgx_le_proxy when going to suspend because its EPC pages will be
  invalidated when resuming, which will cause it not function properly
  anymore.
* Set EINITTOKEN.VALID to zero for a token that is passed when
  SGXLEPUBKEYHASH matches MRSIGNER as alloc_page() does not give a zero
  page.
* Fixed the check in sgx_edbgrd() for a TCS page. Allowed to read offsets
  around the flags field, which causes a #GP. Only flags read is readable.
* On read access memcpy() call inside sgx_vma_access() had src and dest
  parameters in wrong order.
* The build issue with CONFIG_KASAN is now fixed. Added undefined symbols
  to LE even if “KASAN_SANITIZE := false” was set in the makefile. 
* Fixed a regression in the #PF handler. If a page has
  SGX_ENCL_PAGE_RESERVED flag the #PF handler should unconditionally fail.
  It did not, which caused weird races when trying to change other parts of
  swapping code.
* EPC management has been refactored to a flat LRU cache and moved to
  arch/x86. The swapper thread reads a cluster of EPC pages and swaps all
  of them. It can now swap from multiple enclaves in the same round.
* For the sake of consistency with SGX_IOC_ENCLAVE_ADD_PAGE, return -EINVAL
  when an enclave is already initialized or dead instead of zero.

v10:
* Cleaned up anon inode based IPC between the ring-0 and ring-3 parts
  of the driver.
* Unset the reserved flag from an enclave page if EDBGRD/WR fails
  (regression in v6).
* Close the anon inode when LE is stopped (regression in v9).
* Update the documentation with a more detailed description of SGX.

v9:
* Replaced kernel-LE IPC based on pipes with an anonymous inode.
  The driver does not require anymore new exports.

v8:
* Check that public key MSRs match the LE public key hash in the
  driver initialization when the MSRs are read-only.
* Fix the race in VA slot allocation by checking the fullness
  immediately after succeesful allocation.
* Fix the race in hash mrsigner calculation between the launch
  enclave and user enclaves by having a separate lock for hash
  calculation.

v7:
* Fixed offset calculation in sgx_edbgr/wr(). Address was masked with PAGE_MASK
  when it should have been masked with ~PAGE_MASK.
* Fixed a memory leak in sgx_ioc_enclave_create().
* Simplified swapping code by using a pointer array for a cluster
  instead of a linked list.
* Squeezed struct sgx_encl_page to 32 bytes.
* Fixed deferencing of an RSA key on OpenSSL 1.1.0.
* Modified TC's CMAC to use kernel AES-NI. Restructured the code
  a bit in order to better align with kernel conventions.

v6:
* Fixed semaphore underrun when accessing /dev/sgx from the launch enclave.
* In sgx_encl_create() s/IS_ERR(secs)/IS_ERR(encl)/.
* Removed virtualization chapter from the documentation.
* Changed the default filename for the signing key as signing_key.pem.
* Reworked EPC management in a way that instead of a linked list of
  struct sgx_epc_page instances there is an array of integers that
  encodes address and bank of an EPC page (the same data as 'pa' field
  earlier). The locking has been moved to the EPC bank level instead
  of a global lock.
* Relaxed locking requirements for EPC management. EPC pages can be
  released back to the EPC bank concurrently.
* Cleaned up ptrace() code.
* Refined commit messages for new architectural constants.
* Sorted includes in every source file.
* Sorted local variable declarations according to the line length in
  every function.
* Style fixes based on Darren's comments to sgx_le.c.

v5:
* Described IPC between the Launch Enclave and kernel in the commit messages.
* Fixed all relevant checkpatch.pl issues that I have forgot fix in earlier
  versions except those that exist in the imported TinyCrypt code.
* Fixed spelling mistakes in the documentation.
* Forgot to check the return value of sgx_drv_subsys_init().
* Encapsulated properly page cache init and teardown.
* Collect epc pages to a temp list in sgx_add_epc_bank
* Removed SGX_ENCLAVE_INIT_ARCH constant.

v4:
* Tied life-cycle of the sgx_le_proxy process to /dev/sgx.
* Removed __exit annotation from sgx_drv_subsys_exit().
* Fixed a leak of a backing page in sgx_process_add_page_req() in the
  case when vm_insert_pfn() fails.
* Removed unused symbol exports for sgx_page_cache.c.
* Updated sgx_alloc_page() to require encl parameter and documented the
  behavior (Sean Christopherson).
* Refactored a more lean API for sgx_encl_find() and documented the behavior.
* Moved #PF handler to sgx_fault.c.
* Replaced subsys_system_register() with plain bus_register().
* Retry EINIT 2nd time only if MSRs are not locked.

v3:
* Check that FEATURE_CONTROL_LOCKED and FEATURE_CONTROL_SGX_ENABLE are set.
* Return -ERESTARTSYS in __sgx_encl_add_page() when sgx_alloc_page() fails.
* Use unused bits in epc_page->pa to store the bank number.
* Removed #ifdef for WQ_NONREENTRANT.
* If mmu_notifier_register() fails with -EINTR, return -ERESTARTSYS.
* Added --remove-section=.got.plt to objcopy flags in order to prevent a
  dummy .got.plt, which will cause an inconsistent size for the LE.
* Documented sgx_encl_* functions.
* Added remark about AES implementation used inside the LE.
* Removed redundant sgx_sys_exit() from le/main.c.
* Fixed struct sgx_secinfo alignment from 128 to 64 bytes.
* Validate miscselect in sgx_encl_create().
* Fixed SSA frame size calculation to take the misc region into account.
* Implemented consistent exception handling to __encls() and __encls_ret().
* Implemented a proper device model in order to allow sysfs attributes
  and in-kernel API.
* Cleaned up various "find enclave" implementations to the unified
  sgx_encl_find().
* Validate that vm_pgoff is zero.
* Discard backing pages with shmem_truncate_range() after EADD.
* Added missing EEXTEND operations to LE signing and launch.
* Fixed SSA size for GPRS region from 168 to 184 bytes.
* Fixed the checks for TCS flags. Now DBGOPTIN is allowed.
* Check that TCS addresses are in ELRANGE and not just page aligned.
* Require kernel to be compiled with X64_64 and CPU_SUP_INTEL.
* Fixed an incorrect value for SGX_ATTR_DEBUG from 0x01 to 0x02.

v2:
* get_rand_uint32() changed the value of the pointer instead of value
  where it is pointing at.
* Launch enclave incorrectly used sigstruct attributes-field instead of
  enclave attributes-field.
* Removed unused struct sgx_add_page_req from sgx_ioctl.c
* Removed unused sgx_has_sgx2.
* Updated arch/x86/include/asm/sgx.h so that it provides stub
  implementations when sgx in not enabled.
* Removed cruft rdmsr-calls from sgx_set_pubkeyhash_msrs().
* return -ENOMEM in sgx_alloc_page() when VA pages consume too much space
* removed unused global sgx_nr_pids
* moved sgx_encl_release to sgx_encl.c
* return -ERESTARTSYS instead of -EINTR in sgx_encl_init()


Jarkko Sakkinen (7):
  x86, sgx: updated MAINTAINERS
  x86, sgx: added ENCLS wrappers
  x86, sgx: basic routines for enclave page cache
  intel_sgx: driver for Intel Software Guard Extensions
  intel_sgx: ptrace() support
  intel_sgx: driver documentation
  intel_sgx: in-kernel launch enclave

Kai Huang (1):
  x86, sgx: add SGX definitions to cpufeature

Sean Christopherson (5):
  compiler.h, kasan: add __SANITIZE_ADDRESS__ check for
    __no_kasan_or_inline
  x86, sgx: add SGX definitions to msr-index.h
  x86, cpufeatures: add Intel-defined SGX leaf CPUID_12_EAX
  crypto: aesni: add minimal build option for SGX LE
  x86, sgx: detect Intel SGX

 Documentation/index.rst                       |   1 +
 Documentation/x86/intel_sgx.rst               | 195 ++++
 MAINTAINERS                                   |   7 +
 arch/x86/Kconfig                              |  19 +
 arch/x86/crypto/aesni-intel_asm.S             |  11 +
 arch/x86/include/asm/cpufeature.h             |   7 +-
 arch/x86/include/asm/cpufeatures.h            |  10 +-
 arch/x86/include/asm/disabled-features.h      |   3 +-
 arch/x86/include/asm/msr-index.h              |   8 +
 arch/x86/include/asm/required-features.h      |   3 +-
 arch/x86/include/asm/sgx.h                    | 289 +++++
 arch/x86/include/asm/sgx_arch.h               | 224 ++++
 arch/x86/include/asm/sgx_le.h                 |  17 +
 arch/x86/include/asm/sgx_pr.h                 |  36 +
 arch/x86/include/uapi/asm/sgx.h               | 138 +++
 arch/x86/kernel/cpu/Makefile                  |   1 +
 arch/x86/kernel/cpu/common.c                  |   7 +
 arch/x86/kernel/cpu/intel_sgx.c               | 492 +++++++++
 arch/x86/kvm/cpuid.h                          |   1 +
 drivers/platform/x86/Kconfig                  |   2 +
 drivers/platform/x86/Makefile                 |   1 +
 drivers/platform/x86/intel_sgx/Kconfig        |  34 +
 drivers/platform/x86/intel_sgx/Makefile       |  32 +
 drivers/platform/x86/intel_sgx/le/Makefile    |  34 +
 .../x86/intel_sgx/le/enclave/Makefile         |  54 +
 .../intel_sgx/le/enclave/aesni-intel_asm.S    |   1 +
 .../x86/intel_sgx/le/enclave/cmac_mode.c      | 209 ++++
 .../x86/intel_sgx/le/enclave/cmac_mode.h      |  54 +
 .../x86/intel_sgx/le/enclave/encl_bootstrap.S | 116 ++
 .../platform/x86/intel_sgx/le/enclave/main.c  | 146 +++
 .../platform/x86/intel_sgx/le/enclave/main.h  |  19 +
 .../x86/intel_sgx/le/enclave/sgx_le.lds       |  33 +
 .../x86/intel_sgx/le/enclave/sgxsign.c        | 551 ++++++++++
 .../x86/intel_sgx/le/enclave/string.c         |   1 +
 drivers/platform/x86/intel_sgx/le/entry.S     |  70 ++
 .../x86/intel_sgx/le/include/sgx_asm.h        |  15 +
 drivers/platform/x86/intel_sgx/le/main.c      | 140 +++
 drivers/platform/x86/intel_sgx/le/main.h      |  30 +
 .../platform/x86/intel_sgx/le/sgx_le_piggy.S  |  22 +
 drivers/platform/x86/intel_sgx/le/string.c    |  39 +
 drivers/platform/x86/intel_sgx/sgx.h          | 181 ++++
 drivers/platform/x86/intel_sgx/sgx_encl.c     | 988 ++++++++++++++++++
 .../platform/x86/intel_sgx/sgx_encl_page.c    | 294 ++++++
 drivers/platform/x86/intel_sgx/sgx_fault.c    | 159 +++
 drivers/platform/x86/intel_sgx/sgx_ioctl.c    | 235 +++++
 drivers/platform/x86/intel_sgx/sgx_le.c       | 303 ++++++
 .../x86/intel_sgx/sgx_le_proxy_piggy.S        |  22 +
 drivers/platform/x86/intel_sgx/sgx_main.c     | 373 +++++++
 drivers/platform/x86/intel_sgx/sgx_vma.c      | 182 ++++
 include/linux/compiler.h                      |   2 +-
 50 files changed, 5805 insertions(+), 6 deletions(-)
 create mode 100644 Documentation/x86/intel_sgx.rst
 create mode 100644 arch/x86/include/asm/sgx.h
 create mode 100644 arch/x86/include/asm/sgx_arch.h
 create mode 100644 arch/x86/include/asm/sgx_le.h
 create mode 100644 arch/x86/include/asm/sgx_pr.h
 create mode 100644 arch/x86/include/uapi/asm/sgx.h
 create mode 100644 arch/x86/kernel/cpu/intel_sgx.c
 create mode 100644 drivers/platform/x86/intel_sgx/Kconfig
 create mode 100644 drivers/platform/x86/intel_sgx/Makefile
 create mode 100644 drivers/platform/x86/intel_sgx/le/Makefile
 create mode 100644 drivers/platform/x86/intel_sgx/le/enclave/Makefile
 create mode 120000 drivers/platform/x86/intel_sgx/le/enclave/aesni-intel_asm.S
 create mode 100644 drivers/platform/x86/intel_sgx/le/enclave/cmac_mode.c
 create mode 100644 drivers/platform/x86/intel_sgx/le/enclave/cmac_mode.h
 create mode 100644 drivers/platform/x86/intel_sgx/le/enclave/encl_bootstrap.S
 create mode 100644 drivers/platform/x86/intel_sgx/le/enclave/main.c
 create mode 100644 drivers/platform/x86/intel_sgx/le/enclave/main.h
 create mode 100644 drivers/platform/x86/intel_sgx/le/enclave/sgx_le.lds
 create mode 100644 drivers/platform/x86/intel_sgx/le/enclave/sgxsign.c
 create mode 120000 drivers/platform/x86/intel_sgx/le/enclave/string.c
 create mode 100644 drivers/platform/x86/intel_sgx/le/entry.S
 create mode 100644 drivers/platform/x86/intel_sgx/le/include/sgx_asm.h
 create mode 100644 drivers/platform/x86/intel_sgx/le/main.c
 create mode 100644 drivers/platform/x86/intel_sgx/le/main.h
 create mode 100644 drivers/platform/x86/intel_sgx/le/sgx_le_piggy.S
 create mode 100644 drivers/platform/x86/intel_sgx/le/string.c
 create mode 100644 drivers/platform/x86/intel_sgx/sgx.h
 create mode 100644 drivers/platform/x86/intel_sgx/sgx_encl.c
 create mode 100644 drivers/platform/x86/intel_sgx/sgx_encl_page.c
 create mode 100644 drivers/platform/x86/intel_sgx/sgx_fault.c
 create mode 100644 drivers/platform/x86/intel_sgx/sgx_ioctl.c
 create mode 100644 drivers/platform/x86/intel_sgx/sgx_le.c
 create mode 100644 drivers/platform/x86/intel_sgx/sgx_le_proxy_piggy.S
 create mode 100644 drivers/platform/x86/intel_sgx/sgx_main.c
 create mode 100644 drivers/platform/x86/intel_sgx/sgx_vma.c

-- 
2.17.0

[PATCH v11 00/13] Intel SGX1 support

[Jarkko Sakkinen]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain; charset=y, Size: 13643 bytes --]

Intel(R) SGX is a set of CPU instructions that can be used by applications
to set aside private regions of code and data. The code outside the enclave
is disallowed to access the memory inside the enclave by the CPU access
control.  In a way you can think that SGX provides inverted sandbox. It
protects the application from a malicious host.

There is a new hardware unit in the processor called Memory Encryption
Engine (MEE) starting from the Skylake microacrhitecture. BIOS can define
one or many MEE regions that can hold enclave data by configuring them with
PRMRR registers.

The MEE automatically encrypts the data leaving the processor package to
the MEE regions. The data is encrypted using a random key whose life-time
is exactly one power cycle.

You can tell if your CPU supports SGX by looking into /proc/cpuinfo:

	cat /proc/cpuinfo  | grep sgx

v11:
* Polished ENCLS wrappers with refined exception handling.
* ksgxswapd was not stopped (regression in v5) in
  sgx_page_cache_teardown(), which causes a leaked kthread after driver
  deinitialization.
* Shutdown sgx_le_proxy when going to suspend because its EPC pages will be
  invalidated when resuming, which will cause it not function properly
  anymore.
* Set EINITTOKEN.VALID to zero for a token that is passed when
  SGXLEPUBKEYHASH matches MRSIGNER as alloc_page() does not give a zero
  page.
* Fixed the check in sgx_edbgrd() for a TCS page. Allowed to read offsets
  around the flags field, which causes a #GP. Only flags read is readable.
* On read access memcpy() call inside sgx_vma_access() had src and dest
  parameters in wrong order.
* The build issue with CONFIG_KASAN is now fixed. Added undefined symbols
  to LE even if “KASAN_SANITIZE := false” was set in the makefile. 
* Fixed a regression in the #PF handler. If a page has
  SGX_ENCL_PAGE_RESERVED flag the #PF handler should unconditionally fail.
  It did not, which caused weird races when trying to change other parts of
  swapping code.
* EPC management has been refactored to a flat LRU cache and moved to
  arch/x86. The swapper thread reads a cluster of EPC pages and swaps all
  of them. It can now swap from multiple enclaves in the same round.
* For the sake of consistency with SGX_IOC_ENCLAVE_ADD_PAGE, return -EINVAL
  when an enclave is already initialized or dead instead of zero.

v10:
* Cleaned up anon inode based IPC between the ring-0 and ring-3 parts
  of the driver.
* Unset the reserved flag from an enclave page if EDBGRD/WR fails
  (regression in v6).
* Close the anon inode when LE is stopped (regression in v9).
* Update the documentation with a more detailed description of SGX.

v9:
* Replaced kernel-LE IPC based on pipes with an anonymous inode.
  The driver does not require anymore new exports.

v8:
* Check that public key MSRs match the LE public key hash in the
  driver initialization when the MSRs are read-only.
* Fix the race in VA slot allocation by checking the fullness
  immediately after succeesful allocation.
* Fix the race in hash mrsigner calculation between the launch
  enclave and user enclaves by having a separate lock for hash
  calculation.

v7:
* Fixed offset calculation in sgx_edbgr/wr(). Address was masked with PAGE_MASK
  when it should have been masked with ~PAGE_MASK.
* Fixed a memory leak in sgx_ioc_enclave_create().
* Simplified swapping code by using a pointer array for a cluster
  instead of a linked list.
* Squeezed struct sgx_encl_page to 32 bytes.
* Fixed deferencing of an RSA key on OpenSSL 1.1.0.
* Modified TC's CMAC to use kernel AES-NI. Restructured the code
  a bit in order to better align with kernel conventions.

v6:
* Fixed semaphore underrun when accessing /dev/sgx from the launch enclave.
* In sgx_encl_create() s/IS_ERR(secs)/IS_ERR(encl)/.
* Removed virtualization chapter from the documentation.
* Changed the default filename for the signing key as signing_key.pem.
* Reworked EPC management in a way that instead of a linked list of
  struct sgx_epc_page instances there is an array of integers that
  encodes address and bank of an EPC page (the same data as 'pa' field
  earlier). The locking has been moved to the EPC bank level instead
  of a global lock.
* Relaxed locking requirements for EPC management. EPC pages can be
  released back to the EPC bank concurrently.
* Cleaned up ptrace() code.
* Refined commit messages for new architectural constants.
* Sorted includes in every source file.
* Sorted local variable declarations according to the line length in
  every function.
* Style fixes based on Darren's comments to sgx_le.c.

v5:
* Described IPC between the Launch Enclave and kernel in the commit messages.
* Fixed all relevant checkpatch.pl issues that I have forgot fix in earlier
  versions except those that exist in the imported TinyCrypt code.
* Fixed spelling mistakes in the documentation.
* Forgot to check the return value of sgx_drv_subsys_init().
* Encapsulated properly page cache init and teardown.
* Collect epc pages to a temp list in sgx_add_epc_bank
* Removed SGX_ENCLAVE_INIT_ARCH constant.

v4:
* Tied life-cycle of the sgx_le_proxy process to /dev/sgx.
* Removed __exit annotation from sgx_drv_subsys_exit().
* Fixed a leak of a backing page in sgx_process_add_page_req() in the
  case when vm_insert_pfn() fails.
* Removed unused symbol exports for sgx_page_cache.c.
* Updated sgx_alloc_page() to require encl parameter and documented the
  behavior (Sean Christopherson).
* Refactored a more lean API for sgx_encl_find() and documented the behavior.
* Moved #PF handler to sgx_fault.c.
* Replaced subsys_system_register() with plain bus_register().
* Retry EINIT 2nd time only if MSRs are not locked.

v3:
* Check that FEATURE_CONTROL_LOCKED and FEATURE_CONTROL_SGX_ENABLE are set.
* Return -ERESTARTSYS in __sgx_encl_add_page() when sgx_alloc_page() fails.
* Use unused bits in epc_page->pa to store the bank number.
* Removed #ifdef for WQ_NONREENTRANT.
* If mmu_notifier_register() fails with -EINTR, return -ERESTARTSYS.
* Added --remove-section=.got.plt to objcopy flags in order to prevent a
  dummy .got.plt, which will cause an inconsistent size for the LE.
* Documented sgx_encl_* functions.
* Added remark about AES implementation used inside the LE.
* Removed redundant sgx_sys_exit() from le/main.c.
* Fixed struct sgx_secinfo alignment from 128 to 64 bytes.
* Validate miscselect in sgx_encl_create().
* Fixed SSA frame size calculation to take the misc region into account.
* Implemented consistent exception handling to __encls() and __encls_ret().
* Implemented a proper device model in order to allow sysfs attributes
  and in-kernel API.
* Cleaned up various "find enclave" implementations to the unified
  sgx_encl_find().
* Validate that vm_pgoff is zero.
* Discard backing pages with shmem_truncate_range() after EADD.
* Added missing EEXTEND operations to LE signing and launch.
* Fixed SSA size for GPRS region from 168 to 184 bytes.
* Fixed the checks for TCS flags. Now DBGOPTIN is allowed.
* Check that TCS addresses are in ELRANGE and not just page aligned.
* Require kernel to be compiled with X64_64 and CPU_SUP_INTEL.
* Fixed an incorrect value for SGX_ATTR_DEBUG from 0x01 to 0x02.

v2:
* get_rand_uint32() changed the value of the pointer instead of value
  where it is pointing at.
* Launch enclave incorrectly used sigstruct attributes-field instead of
  enclave attributes-field.
* Removed unused struct sgx_add_page_req from sgx_ioctl.c
* Removed unused sgx_has_sgx2.
* Updated arch/x86/include/asm/sgx.h so that it provides stub
  implementations when sgx in not enabled.
* Removed cruft rdmsr-calls from sgx_set_pubkeyhash_msrs().
* return -ENOMEM in sgx_alloc_page() when VA pages consume too much space
* removed unused global sgx_nr_pids
* moved sgx_encl_release to sgx_encl.c
* return -ERESTARTSYS instead of -EINTR in sgx_encl_init()


Jarkko Sakkinen (7):
  x86, sgx: updated MAINTAINERS
  x86, sgx: added ENCLS wrappers
  x86, sgx: basic routines for enclave page cache
  intel_sgx: driver for Intel Software Guard Extensions
  intel_sgx: ptrace() support
  intel_sgx: driver documentation
  intel_sgx: in-kernel launch enclave

Kai Huang (1):
  x86, sgx: add SGX definitions to cpufeature

Sean Christopherson (5):
  compiler.h, kasan: add __SANITIZE_ADDRESS__ check for
    __no_kasan_or_inline
  x86, sgx: add SGX definitions to msr-index.h
  x86, cpufeatures: add Intel-defined SGX leaf CPUID_12_EAX
  crypto: aesni: add minimal build option for SGX LE
  x86, sgx: detect Intel SGX

 Documentation/index.rst                       |   1 +
 Documentation/x86/intel_sgx.rst               | 195 ++++
 MAINTAINERS                                   |   7 +
 arch/x86/Kconfig                              |  19 +
 arch/x86/crypto/aesni-intel_asm.S             |  11 +
 arch/x86/include/asm/cpufeature.h             |   7 +-
 arch/x86/include/asm/cpufeatures.h            |  10 +-
 arch/x86/include/asm/disabled-features.h      |   3 +-
 arch/x86/include/asm/msr-index.h              |   8 +
 arch/x86/include/asm/required-features.h      |   3 +-
 arch/x86/include/asm/sgx.h                    | 289 +++++
 arch/x86/include/asm/sgx_arch.h               | 224 ++++
 arch/x86/include/asm/sgx_le.h                 |  17 +
 arch/x86/include/asm/sgx_pr.h                 |  36 +
 arch/x86/include/uapi/asm/sgx.h               | 138 +++
 arch/x86/kernel/cpu/Makefile                  |   1 +
 arch/x86/kernel/cpu/common.c                  |   7 +
 arch/x86/kernel/cpu/intel_sgx.c               | 492 +++++++++
 arch/x86/kvm/cpuid.h                          |   1 +
 drivers/platform/x86/Kconfig                  |   2 +
 drivers/platform/x86/Makefile                 |   1 +
 drivers/platform/x86/intel_sgx/Kconfig        |  34 +
 drivers/platform/x86/intel_sgx/Makefile       |  32 +
 drivers/platform/x86/intel_sgx/le/Makefile    |  34 +
 .../x86/intel_sgx/le/enclave/Makefile         |  54 +
 .../intel_sgx/le/enclave/aesni-intel_asm.S    |   1 +
 .../x86/intel_sgx/le/enclave/cmac_mode.c      | 209 ++++
 .../x86/intel_sgx/le/enclave/cmac_mode.h      |  54 +
 .../x86/intel_sgx/le/enclave/encl_bootstrap.S | 116 ++
 .../platform/x86/intel_sgx/le/enclave/main.c  | 146 +++
 .../platform/x86/intel_sgx/le/enclave/main.h  |  19 +
 .../x86/intel_sgx/le/enclave/sgx_le.lds       |  33 +
 .../x86/intel_sgx/le/enclave/sgxsign.c        | 551 ++++++++++
 .../x86/intel_sgx/le/enclave/string.c         |   1 +
 drivers/platform/x86/intel_sgx/le/entry.S     |  70 ++
 .../x86/intel_sgx/le/include/sgx_asm.h        |  15 +
 drivers/platform/x86/intel_sgx/le/main.c      | 140 +++
 drivers/platform/x86/intel_sgx/le/main.h      |  30 +
 .../platform/x86/intel_sgx/le/sgx_le_piggy.S  |  22 +
 drivers/platform/x86/intel_sgx/le/string.c    |  39 +
 drivers/platform/x86/intel_sgx/sgx.h          | 181 ++++
 drivers/platform/x86/intel_sgx/sgx_encl.c     | 988 ++++++++++++++++++
 .../platform/x86/intel_sgx/sgx_encl_page.c    | 294 ++++++
 drivers/platform/x86/intel_sgx/sgx_fault.c    | 159 +++
 drivers/platform/x86/intel_sgx/sgx_ioctl.c    | 235 +++++
 drivers/platform/x86/intel_sgx/sgx_le.c       | 303 ++++++
 .../x86/intel_sgx/sgx_le_proxy_piggy.S        |  22 +
 drivers/platform/x86/intel_sgx/sgx_main.c     | 373 +++++++
 drivers/platform/x86/intel_sgx/sgx_vma.c      | 182 ++++
 include/linux/compiler.h                      |   2 +-
 50 files changed, 5805 insertions(+), 6 deletions(-)
 create mode 100644 Documentation/x86/intel_sgx.rst
 create mode 100644 arch/x86/include/asm/sgx.h
 create mode 100644 arch/x86/include/asm/sgx_arch.h
 create mode 100644 arch/x86/include/asm/sgx_le.h
 create mode 100644 arch/x86/include/asm/sgx_pr.h
 create mode 100644 arch/x86/include/uapi/asm/sgx.h
 create mode 100644 arch/x86/kernel/cpu/intel_sgx.c
 create mode 100644 drivers/platform/x86/intel_sgx/Kconfig
 create mode 100644 drivers/platform/x86/intel_sgx/Makefile
 create mode 100644 drivers/platform/x86/intel_sgx/le/Makefile
 create mode 100644 drivers/platform/x86/intel_sgx/le/enclave/Makefile
 create mode 120000 drivers/platform/x86/intel_sgx/le/enclave/aesni-intel_asm.S
 create mode 100644 drivers/platform/x86/intel_sgx/le/enclave/cmac_mode.c
 create mode 100644 drivers/platform/x86/intel_sgx/le/enclave/cmac_mode.h
 create mode 100644 drivers/platform/x86/intel_sgx/le/enclave/encl_bootstrap.S
 create mode 100644 drivers/platform/x86/intel_sgx/le/enclave/main.c
 create mode 100644 drivers/platform/x86/intel_sgx/le/enclave/main.h
 create mode 100644 drivers/platform/x86/intel_sgx/le/enclave/sgx_le.lds
 create mode 100644 drivers/platform/x86/intel_sgx/le/enclave/sgxsign.c
 create mode 120000 drivers/platform/x86/intel_sgx/le/enclave/string.c
 create mode 100644 drivers/platform/x86/intel_sgx/le/entry.S
 create mode 100644 drivers/platform/x86/intel_sgx/le/include/sgx_asm.h
 create mode 100644 drivers/platform/x86/intel_sgx/le/main.c
 create mode 100644 drivers/platform/x86/intel_sgx/le/main.h
 create mode 100644 drivers/platform/x86/intel_sgx/le/sgx_le_piggy.S
 create mode 100644 drivers/platform/x86/intel_sgx/le/string.c
 create mode 100644 drivers/platform/x86/intel_sgx/sgx.h
 create mode 100644 drivers/platform/x86/intel_sgx/sgx_encl.c
 create mode 100644 drivers/platform/x86/intel_sgx/sgx_encl_page.c
 create mode 100644 drivers/platform/x86/intel_sgx/sgx_fault.c
 create mode 100644 drivers/platform/x86/intel_sgx/sgx_ioctl.c
 create mode 100644 drivers/platform/x86/intel_sgx/sgx_le.c
 create mode 100644 drivers/platform/x86/intel_sgx/sgx_le_proxy_piggy.S
 create mode 100644 drivers/platform/x86/intel_sgx/sgx_main.c
 create mode 100644 drivers/platform/x86/intel_sgx/sgx_vma.c

-- 
2.17.0

[PATCH v11 00/13] Intel SGX1 support

[Jarkko Sakkinen]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain; charset=y, Size: 13643 bytes --]

Intel(R) SGX is a set of CPU instructions that can be used by applications
to set aside private regions of code and data. The code outside the enclave
is disallowed to access the memory inside the enclave by the CPU access
control.  In a way you can think that SGX provides inverted sandbox. It
protects the application from a malicious host.

There is a new hardware unit in the processor called Memory Encryption
Engine (MEE) starting from the Skylake microacrhitecture. BIOS can define
one or many MEE regions that can hold enclave data by configuring them with
PRMRR registers.

The MEE automatically encrypts the data leaving the processor package to
the MEE regions. The data is encrypted using a random key whose life-time
is exactly one power cycle.

You can tell if your CPU supports SGX by looking into /proc/cpuinfo:

	cat /proc/cpuinfo  | grep sgx

v11:
* Polished ENCLS wrappers with refined exception handling.
* ksgxswapd was not stopped (regression in v5) in
  sgx_page_cache_teardown(), which causes a leaked kthread after driver
  deinitialization.
* Shutdown sgx_le_proxy when going to suspend because its EPC pages will be
  invalidated when resuming, which will cause it not function properly
  anymore.
* Set EINITTOKEN.VALID to zero for a token that is passed when
  SGXLEPUBKEYHASH matches MRSIGNER as alloc_page() does not give a zero
  page.
* Fixed the check in sgx_edbgrd() for a TCS page. Allowed to read offsets
  around the flags field, which causes a #GP. Only flags read is readable.
* On read access memcpy() call inside sgx_vma_access() had src and dest
  parameters in wrong order.
* The build issue with CONFIG_KASAN is now fixed. Added undefined symbols
  to LE even if “KASAN_SANITIZE := false” was set in the makefile. 
* Fixed a regression in the #PF handler. If a page has
  SGX_ENCL_PAGE_RESERVED flag the #PF handler should unconditionally fail.
  It did not, which caused weird races when trying to change other parts of
  swapping code.
* EPC management has been refactored to a flat LRU cache and moved to
  arch/x86. The swapper thread reads a cluster of EPC pages and swaps all
  of them. It can now swap from multiple enclaves in the same round.
* For the sake of consistency with SGX_IOC_ENCLAVE_ADD_PAGE, return -EINVAL
  when an enclave is already initialized or dead instead of zero.

v10:
* Cleaned up anon inode based IPC between the ring-0 and ring-3 parts
  of the driver.
* Unset the reserved flag from an enclave page if EDBGRD/WR fails
  (regression in v6).
* Close the anon inode when LE is stopped (regression in v9).
* Update the documentation with a more detailed description of SGX.

v9:
* Replaced kernel-LE IPC based on pipes with an anonymous inode.
  The driver does not require anymore new exports.

v8:
* Check that public key MSRs match the LE public key hash in the
  driver initialization when the MSRs are read-only.
* Fix the race in VA slot allocation by checking the fullness
  immediately after succeesful allocation.
* Fix the race in hash mrsigner calculation between the launch
  enclave and user enclaves by having a separate lock for hash
  calculation.

v7:
* Fixed offset calculation in sgx_edbgr/wr(). Address was masked with PAGE_MASK
  when it should have been masked with ~PAGE_MASK.
* Fixed a memory leak in sgx_ioc_enclave_create().
* Simplified swapping code by using a pointer array for a cluster
  instead of a linked list.
* Squeezed struct sgx_encl_page to 32 bytes.
* Fixed deferencing of an RSA key on OpenSSL 1.1.0.
* Modified TC's CMAC to use kernel AES-NI. Restructured the code
  a bit in order to better align with kernel conventions.

v6:
* Fixed semaphore underrun when accessing /dev/sgx from the launch enclave.
* In sgx_encl_create() s/IS_ERR(secs)/IS_ERR(encl)/.
* Removed virtualization chapter from the documentation.
* Changed the default filename for the signing key as signing_key.pem.
* Reworked EPC management in a way that instead of a linked list of
  struct sgx_epc_page instances there is an array of integers that
  encodes address and bank of an EPC page (the same data as 'pa' field
  earlier). The locking has been moved to the EPC bank level instead
  of a global lock.
* Relaxed locking requirements for EPC management. EPC pages can be
  released back to the EPC bank concurrently.
* Cleaned up ptrace() code.
* Refined commit messages for new architectural constants.
* Sorted includes in every source file.
* Sorted local variable declarations according to the line length in
  every function.
* Style fixes based on Darren's comments to sgx_le.c.

v5:
* Described IPC between the Launch Enclave and kernel in the commit messages.
* Fixed all relevant checkpatch.pl issues that I have forgot fix in earlier
  versions except those that exist in the imported TinyCrypt code.
* Fixed spelling mistakes in the documentation.
* Forgot to check the return value of sgx_drv_subsys_init().
* Encapsulated properly page cache init and teardown.
* Collect epc pages to a temp list in sgx_add_epc_bank
* Removed SGX_ENCLAVE_INIT_ARCH constant.

v4:
* Tied life-cycle of the sgx_le_proxy process to /dev/sgx.
* Removed __exit annotation from sgx_drv_subsys_exit().
* Fixed a leak of a backing page in sgx_process_add_page_req() in the
  case when vm_insert_pfn() fails.
* Removed unused symbol exports for sgx_page_cache.c.
* Updated sgx_alloc_page() to require encl parameter and documented the
  behavior (Sean Christopherson).
* Refactored a more lean API for sgx_encl_find() and documented the behavior.
* Moved #PF handler to sgx_fault.c.
* Replaced subsys_system_register() with plain bus_register().
* Retry EINIT 2nd time only if MSRs are not locked.

v3:
* Check that FEATURE_CONTROL_LOCKED and FEATURE_CONTROL_SGX_ENABLE are set.
* Return -ERESTARTSYS in __sgx_encl_add_page() when sgx_alloc_page() fails.
* Use unused bits in epc_page->pa to store the bank number.
* Removed #ifdef for WQ_NONREENTRANT.
* If mmu_notifier_register() fails with -EINTR, return -ERESTARTSYS.
* Added --remove-section=.got.plt to objcopy flags in order to prevent a
  dummy .got.plt, which will cause an inconsistent size for the LE.
* Documented sgx_encl_* functions.
* Added remark about AES implementation used inside the LE.
* Removed redundant sgx_sys_exit() from le/main.c.
* Fixed struct sgx_secinfo alignment from 128 to 64 bytes.
* Validate miscselect in sgx_encl_create().
* Fixed SSA frame size calculation to take the misc region into account.
* Implemented consistent exception handling to __encls() and __encls_ret().
* Implemented a proper device model in order to allow sysfs attributes
  and in-kernel API.
* Cleaned up various "find enclave" implementations to the unified
  sgx_encl_find().
* Validate that vm_pgoff is zero.
* Discard backing pages with shmem_truncate_range() after EADD.
* Added missing EEXTEND operations to LE signing and launch.
* Fixed SSA size for GPRS region from 168 to 184 bytes.
* Fixed the checks for TCS flags. Now DBGOPTIN is allowed.
* Check that TCS addresses are in ELRANGE and not just page aligned.
* Require kernel to be compiled with X64_64 and CPU_SUP_INTEL.
* Fixed an incorrect value for SGX_ATTR_DEBUG from 0x01 to 0x02.

v2:
* get_rand_uint32() changed the value of the pointer instead of value
  where it is pointing at.
* Launch enclave incorrectly used sigstruct attributes-field instead of
  enclave attributes-field.
* Removed unused struct sgx_add_page_req from sgx_ioctl.c
* Removed unused sgx_has_sgx2.
* Updated arch/x86/include/asm/sgx.h so that it provides stub
  implementations when sgx in not enabled.
* Removed cruft rdmsr-calls from sgx_set_pubkeyhash_msrs().
* return -ENOMEM in sgx_alloc_page() when VA pages consume too much space
* removed unused global sgx_nr_pids
* moved sgx_encl_release to sgx_encl.c
* return -ERESTARTSYS instead of -EINTR in sgx_encl_init()


Jarkko Sakkinen (7):
  x86, sgx: updated MAINTAINERS
  x86, sgx: added ENCLS wrappers
  x86, sgx: basic routines for enclave page cache
  intel_sgx: driver for Intel Software Guard Extensions
  intel_sgx: ptrace() support
  intel_sgx: driver documentation
  intel_sgx: in-kernel launch enclave

Kai Huang (1):
  x86, sgx: add SGX definitions to cpufeature

Sean Christopherson (5):
  compiler.h, kasan: add __SANITIZE_ADDRESS__ check for
    __no_kasan_or_inline
  x86, sgx: add SGX definitions to msr-index.h
  x86, cpufeatures: add Intel-defined SGX leaf CPUID_12_EAX
  crypto: aesni: add minimal build option for SGX LE
  x86, sgx: detect Intel SGX

 Documentation/index.rst                       |   1 +
 Documentation/x86/intel_sgx.rst               | 195 ++++
 MAINTAINERS                                   |   7 +
 arch/x86/Kconfig                              |  19 +
 arch/x86/crypto/aesni-intel_asm.S             |  11 +
 arch/x86/include/asm/cpufeature.h             |   7 +-
 arch/x86/include/asm/cpufeatures.h            |  10 +-
 arch/x86/include/asm/disabled-features.h      |   3 +-
 arch/x86/include/asm/msr-index.h              |   8 +
 arch/x86/include/asm/required-features.h      |   3 +-
 arch/x86/include/asm/sgx.h                    | 289 +++++
 arch/x86/include/asm/sgx_arch.h               | 224 ++++
 arch/x86/include/asm/sgx_le.h                 |  17 +
 arch/x86/include/asm/sgx_pr.h                 |  36 +
 arch/x86/include/uapi/asm/sgx.h               | 138 +++
 arch/x86/kernel/cpu/Makefile                  |   1 +
 arch/x86/kernel/cpu/common.c                  |   7 +
 arch/x86/kernel/cpu/intel_sgx.c               | 492 +++++++++
 arch/x86/kvm/cpuid.h                          |   1 +
 drivers/platform/x86/Kconfig                  |   2 +
 drivers/platform/x86/Makefile                 |   1 +
 drivers/platform/x86/intel_sgx/Kconfig        |  34 +
 drivers/platform/x86/intel_sgx/Makefile       |  32 +
 drivers/platform/x86/intel_sgx/le/Makefile    |  34 +
 .../x86/intel_sgx/le/enclave/Makefile         |  54 +
 .../intel_sgx/le/enclave/aesni-intel_asm.S    |   1 +
 .../x86/intel_sgx/le/enclave/cmac_mode.c      | 209 ++++
 .../x86/intel_sgx/le/enclave/cmac_mode.h      |  54 +
 .../x86/intel_sgx/le/enclave/encl_bootstrap.S | 116 ++
 .../platform/x86/intel_sgx/le/enclave/main.c  | 146 +++
 .../platform/x86/intel_sgx/le/enclave/main.h  |  19 +
 .../x86/intel_sgx/le/enclave/sgx_le.lds       |  33 +
 .../x86/intel_sgx/le/enclave/sgxsign.c        | 551 ++++++++++
 .../x86/intel_sgx/le/enclave/string.c         |   1 +
 drivers/platform/x86/intel_sgx/le/entry.S     |  70 ++
 .../x86/intel_sgx/le/include/sgx_asm.h        |  15 +
 drivers/platform/x86/intel_sgx/le/main.c      | 140 +++
 drivers/platform/x86/intel_sgx/le/main.h      |  30 +
 .../platform/x86/intel_sgx/le/sgx_le_piggy.S  |  22 +
 drivers/platform/x86/intel_sgx/le/string.c    |  39 +
 drivers/platform/x86/intel_sgx/sgx.h          | 181 ++++
 drivers/platform/x86/intel_sgx/sgx_encl.c     | 988 ++++++++++++++++++
 .../platform/x86/intel_sgx/sgx_encl_page.c    | 294 ++++++
 drivers/platform/x86/intel_sgx/sgx_fault.c    | 159 +++
 drivers/platform/x86/intel_sgx/sgx_ioctl.c    | 235 +++++
 drivers/platform/x86/intel_sgx/sgx_le.c       | 303 ++++++
 .../x86/intel_sgx/sgx_le_proxy_piggy.S        |  22 +
 drivers/platform/x86/intel_sgx/sgx_main.c     | 373 +++++++
 drivers/platform/x86/intel_sgx/sgx_vma.c      | 182 ++++
 include/linux/compiler.h                      |   2 +-
 50 files changed, 5805 insertions(+), 6 deletions(-)
 create mode 100644 Documentation/x86/intel_sgx.rst
 create mode 100644 arch/x86/include/asm/sgx.h
 create mode 100644 arch/x86/include/asm/sgx_arch.h
 create mode 100644 arch/x86/include/asm/sgx_le.h
 create mode 100644 arch/x86/include/asm/sgx_pr.h
 create mode 100644 arch/x86/include/uapi/asm/sgx.h
 create mode 100644 arch/x86/kernel/cpu/intel_sgx.c
 create mode 100644 drivers/platform/x86/intel_sgx/Kconfig
 create mode 100644 drivers/platform/x86/intel_sgx/Makefile
 create mode 100644 drivers/platform/x86/intel_sgx/le/Makefile
 create mode 100644 drivers/platform/x86/intel_sgx/le/enclave/Makefile
 create mode 120000 drivers/platform/x86/intel_sgx/le/enclave/aesni-intel_asm.S
 create mode 100644 drivers/platform/x86/intel_sgx/le/enclave/cmac_mode.c
 create mode 100644 drivers/platform/x86/intel_sgx/le/enclave/cmac_mode.h
 create mode 100644 drivers/platform/x86/intel_sgx/le/enclave/encl_bootstrap.S
 create mode 100644 drivers/platform/x86/intel_sgx/le/enclave/main.c
 create mode 100644 drivers/platform/x86/intel_sgx/le/enclave/main.h
 create mode 100644 drivers/platform/x86/intel_sgx/le/enclave/sgx_le.lds
 create mode 100644 drivers/platform/x86/intel_sgx/le/enclave/sgxsign.c
 create mode 120000 drivers/platform/x86/intel_sgx/le/enclave/string.c
 create mode 100644 drivers/platform/x86/intel_sgx/le/entry.S
 create mode 100644 drivers/platform/x86/intel_sgx/le/include/sgx_asm.h
 create mode 100644 drivers/platform/x86/intel_sgx/le/main.c
 create mode 100644 drivers/platform/x86/intel_sgx/le/main.h
 create mode 100644 drivers/platform/x86/intel_sgx/le/sgx_le_piggy.S
 create mode 100644 drivers/platform/x86/intel_sgx/le/string.c
 create mode 100644 drivers/platform/x86/intel_sgx/sgx.h
 create mode 100644 drivers/platform/x86/intel_sgx/sgx_encl.c
 create mode 100644 drivers/platform/x86/intel_sgx/sgx_encl_page.c
 create mode 100644 drivers/platform/x86/intel_sgx/sgx_fault.c
 create mode 100644 drivers/platform/x86/intel_sgx/sgx_ioctl.c
 create mode 100644 drivers/platform/x86/intel_sgx/sgx_le.c
 create mode 100644 drivers/platform/x86/intel_sgx/sgx_le_proxy_piggy.S
 create mode 100644 drivers/platform/x86/intel_sgx/sgx_main.c
 create mode 100644 drivers/platform/x86/intel_sgx/sgx_vma.c

-- 
2.17.0

[PATCH v11 00/13] Intel SGX1 support

[Jarkko Sakkinen]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain; charset=y, Size: 13837 bytes --]

Intel(R) SGX is a set of CPU instructions that can be used by applications
to set aside private regions of code and data. The code outside the enclave
is disallowed to access the memory inside the enclave by the CPU access
control.  In a way you can think that SGX provides inverted sandbox. It
protects the application from a malicious host.

There is a new hardware unit in the processor called Memory Encryption
Engine (MEE) starting from the Skylake microacrhitecture. BIOS can define
one or many MEE regions that can hold enclave data by configuring them with
PRMRR registers.

The MEE automatically encrypts the data leaving the processor package to
the MEE regions. The data is encrypted using a random key whose life-time
is exactly one power cycle.

You can tell if your CPU supports SGX by looking into /proc/cpuinfo:

	cat /proc/cpuinfo  | grep sgx

v11:
* Polished ENCLS wrappers with refined exception handling.
* ksgxswapd was not stopped (regression in v5) in
  sgx_page_cache_teardown(), which causes a leaked kthread after driver
  deinitialization.
* Shutdown sgx_le_proxy when going to suspend because its EPC pages will be
  invalidated when resuming, which will cause it not function properly
  anymore.
* Set EINITTOKEN.VALID to zero for a token that is passed when
  SGXLEPUBKEYHASH matches MRSIGNER as alloc_page() does not give a zero
  page.
* Fixed the check in sgx_edbgrd() for a TCS page. Allowed to read offsets
  around the flags field, which causes a #GP. Only flags read is readable.
* On read access memcpy() call inside sgx_vma_access() had src and dest
  parameters in wrong order.
* The build issue with CONFIG_KASAN is now fixed. Added undefined symbols
  to LE even if “KASAN_SANITIZE := false” was set in the makefile. 
* Fixed a regression in the #PF handler. If a page has
  SGX_ENCL_PAGE_RESERVED flag the #PF handler should unconditionally fail.
  It did not, which caused weird races when trying to change other parts of
  swapping code.
* EPC management has been refactored to a flat LRU cache and moved to
  arch/x86. The swapper thread reads a cluster of EPC pages and swaps all
  of them. It can now swap from multiple enclaves in the same round.
* For the sake of consistency with SGX_IOC_ENCLAVE_ADD_PAGE, return -EINVAL
  when an enclave is already initialized or dead instead of zero.

v10:
* Cleaned up anon inode based IPC between the ring-0 and ring-3 parts
  of the driver.
* Unset the reserved flag from an enclave page if EDBGRD/WR fails
  (regression in v6).
* Close the anon inode when LE is stopped (regression in v9).
* Update the documentation with a more detailed description of SGX.

v9:
* Replaced kernel-LE IPC based on pipes with an anonymous inode.
  The driver does not require anymore new exports.

v8:
* Check that public key MSRs match the LE public key hash in the
  driver initialization when the MSRs are read-only.
* Fix the race in VA slot allocation by checking the fullness
  immediately after succeesful allocation.
* Fix the race in hash mrsigner calculation between the launch
  enclave and user enclaves by having a separate lock for hash
  calculation.

v7:
* Fixed offset calculation in sgx_edbgr/wr(). Address was masked with PAGE_MASK
  when it should have been masked with ~PAGE_MASK.
* Fixed a memory leak in sgx_ioc_enclave_create().
* Simplified swapping code by using a pointer array for a cluster
  instead of a linked list.
* Squeezed struct sgx_encl_page to 32 bytes.
* Fixed deferencing of an RSA key on OpenSSL 1.1.0.
* Modified TC's CMAC to use kernel AES-NI. Restructured the code
  a bit in order to better align with kernel conventions.

v6:
* Fixed semaphore underrun when accessing /dev/sgx from the launch enclave.
* In sgx_encl_create() s/IS_ERR(secs)/IS_ERR(encl)/.
* Removed virtualization chapter from the documentation.
* Changed the default filename for the signing key as signing_key.pem.
* Reworked EPC management in a way that instead of a linked list of
  struct sgx_epc_page instances there is an array of integers that
  encodes address and bank of an EPC page (the same data as 'pa' field
  earlier). The locking has been moved to the EPC bank level instead
  of a global lock.
* Relaxed locking requirements for EPC management. EPC pages can be
  released back to the EPC bank concurrently.
* Cleaned up ptrace() code.
* Refined commit messages for new architectural constants.
* Sorted includes in every source file.
* Sorted local variable declarations according to the line length in
  every function.
* Style fixes based on Darren's comments to sgx_le.c.

v5:
* Described IPC between the Launch Enclave and kernel in the commit messages.
* Fixed all relevant checkpatch.pl issues that I have forgot fix in earlier
  versions except those that exist in the imported TinyCrypt code.
* Fixed spelling mistakes in the documentation.
* Forgot to check the return value of sgx_drv_subsys_init().
* Encapsulated properly page cache init and teardown.
* Collect epc pages to a temp list in sgx_add_epc_bank
* Removed SGX_ENCLAVE_INIT_ARCH constant.

v4:
* Tied life-cycle of the sgx_le_proxy process to /dev/sgx.
* Removed __exit annotation from sgx_drv_subsys_exit().
* Fixed a leak of a backing page in sgx_process_add_page_req() in the
  case when vm_insert_pfn() fails.
* Removed unused symbol exports for sgx_page_cache.c.
* Updated sgx_alloc_page() to require encl parameter and documented the
  behavior (Sean Christopherson).
* Refactored a more lean API for sgx_encl_find() and documented the behavior.
* Moved #PF handler to sgx_fault.c.
* Replaced subsys_system_register() with plain bus_register().
* Retry EINIT 2nd time only if MSRs are not locked.

v3:
* Check that FEATURE_CONTROL_LOCKED and FEATURE_CONTROL_SGX_ENABLE are set.
* Return -ERESTARTSYS in __sgx_encl_add_page() when sgx_alloc_page() fails.
* Use unused bits in epc_page->pa to store the bank number.
* Removed #ifdef for WQ_NONREENTRANT.
* If mmu_notifier_register() fails with -EINTR, return -ERESTARTSYS.
* Added --remove-section=.got.plt to objcopy flags in order to prevent a
  dummy .got.plt, which will cause an inconsistent size for the LE.
* Documented sgx_encl_* functions.
* Added remark about AES implementation used inside the LE.
* Removed redundant sgx_sys_exit() from le/main.c.
* Fixed struct sgx_secinfo alignment from 128 to 64 bytes.
* Validate miscselect in sgx_encl_create().
* Fixed SSA frame size calculation to take the misc region into account.
* Implemented consistent exception handling to __encls() and __encls_ret().
* Implemented a proper device model in order to allow sysfs attributes
  and in-kernel API.
* Cleaned up various "find enclave" implementations to the unified
  sgx_encl_find().
* Validate that vm_pgoff is zero.
* Discard backing pages with shmem_truncate_range() after EADD.
* Added missing EEXTEND operations to LE signing and launch.
* Fixed SSA size for GPRS region from 168 to 184 bytes.
* Fixed the checks for TCS flags. Now DBGOPTIN is allowed.
* Check that TCS addresses are in ELRANGE and not just page aligned.
* Require kernel to be compiled with X64_64 and CPU_SUP_INTEL.
* Fixed an incorrect value for SGX_ATTR_DEBUG from 0x01 to 0x02.

v2:
* get_rand_uint32() changed the value of the pointer instead of value
  where it is pointing at.
* Launch enclave incorrectly used sigstruct attributes-field instead of
  enclave attributes-field.
* Removed unused struct sgx_add_page_req from sgx_ioctl.c
* Removed unused sgx_has_sgx2.
* Updated arch/x86/include/asm/sgx.h so that it provides stub
  implementations when sgx in not enabled.
* Removed cruft rdmsr-calls from sgx_set_pubkeyhash_msrs().
* return -ENOMEM in sgx_alloc_page() when VA pages consume too much space
* removed unused global sgx_nr_pids
* moved sgx_encl_release to sgx_encl.c
* return -ERESTARTSYS instead of -EINTR in sgx_encl_init()


Jarkko Sakkinen (7):
  x86, sgx: updated MAINTAINERS
  x86, sgx: added ENCLS wrappers
  x86, sgx: basic routines for enclave page cache
  intel_sgx: driver for Intel Software Guard Extensions
  intel_sgx: ptrace() support
  intel_sgx: driver documentation
  intel_sgx: in-kernel launch enclave

Kai Huang (1):
  x86, sgx: add SGX definitions to cpufeature

Sean Christopherson (5):
  compiler.h, kasan: add __SANITIZE_ADDRESS__ check for
    __no_kasan_or_inline
  x86, sgx: add SGX definitions to msr-index.h
  x86, cpufeatures: add Intel-defined SGX leaf CPUID_12_EAX
  crypto: aesni: add minimal build option for SGX LE
  x86, sgx: detect Intel SGX

 Documentation/index.rst                       |   1 +
 Documentation/x86/intel_sgx.rst               | 195 ++++
 MAINTAINERS                                   |   7 +
 arch/x86/Kconfig                              |  19 +
 arch/x86/crypto/aesni-intel_asm.S             |  11 +
 arch/x86/include/asm/cpufeature.h             |   7 +-
 arch/x86/include/asm/cpufeatures.h            |  10 +-
 arch/x86/include/asm/disabled-features.h      |   3 +-
 arch/x86/include/asm/msr-index.h              |   8 +
 arch/x86/include/asm/required-features.h      |   3 +-
 arch/x86/include/asm/sgx.h                    | 289 +++++
 arch/x86/include/asm/sgx_arch.h               | 224 ++++
 arch/x86/include/asm/sgx_le.h                 |  17 +
 arch/x86/include/asm/sgx_pr.h                 |  36 +
 arch/x86/include/uapi/asm/sgx.h               | 138 +++
 arch/x86/kernel/cpu/Makefile                  |   1 +
 arch/x86/kernel/cpu/common.c                  |   7 +
 arch/x86/kernel/cpu/intel_sgx.c               | 492 +++++++++
 arch/x86/kvm/cpuid.h                          |   1 +
 drivers/platform/x86/Kconfig                  |   2 +
 drivers/platform/x86/Makefile                 |   1 +
 drivers/platform/x86/intel_sgx/Kconfig        |  34 +
 drivers/platform/x86/intel_sgx/Makefile       |  32 +
 drivers/platform/x86/intel_sgx/le/Makefile    |  34 +
 .../x86/intel_sgx/le/enclave/Makefile         |  54 +
 .../intel_sgx/le/enclave/aesni-intel_asm.S    |   1 +
 .../x86/intel_sgx/le/enclave/cmac_mode.c      | 209 ++++
 .../x86/intel_sgx/le/enclave/cmac_mode.h      |  54 +
 .../x86/intel_sgx/le/enclave/encl_bootstrap.S | 116 ++
 .../platform/x86/intel_sgx/le/enclave/main.c  | 146 +++
 .../platform/x86/intel_sgx/le/enclave/main.h  |  19 +
 .../x86/intel_sgx/le/enclave/sgx_le.lds       |  33 +
 .../x86/intel_sgx/le/enclave/sgxsign.c        | 551 ++++++++++
 .../x86/intel_sgx/le/enclave/string.c         |   1 +
 drivers/platform/x86/intel_sgx/le/entry.S     |  70 ++
 .../x86/intel_sgx/le/include/sgx_asm.h        |  15 +
 drivers/platform/x86/intel_sgx/le/main.c      | 140 +++
 drivers/platform/x86/intel_sgx/le/main.h      |  30 +
 .../platform/x86/intel_sgx/le/sgx_le_piggy.S  |  22 +
 drivers/platform/x86/intel_sgx/le/string.c    |  39 +
 drivers/platform/x86/intel_sgx/sgx.h          | 181 ++++
 drivers/platform/x86/intel_sgx/sgx_encl.c     | 988 ++++++++++++++++++
 .../platform/x86/intel_sgx/sgx_encl_page.c    | 294 ++++++
 drivers/platform/x86/intel_sgx/sgx_fault.c    | 159 +++
 drivers/platform/x86/intel_sgx/sgx_ioctl.c    | 235 +++++
 drivers/platform/x86/intel_sgx/sgx_le.c       | 303 ++++++
 .../x86/intel_sgx/sgx_le_proxy_piggy.S        |  22 +
 drivers/platform/x86/intel_sgx/sgx_main.c     | 373 +++++++
 drivers/platform/x86/intel_sgx/sgx_vma.c      | 182 ++++
 include/linux/compiler.h                      |   2 +-
 50 files changed, 5805 insertions(+), 6 deletions(-)
 create mode 100644 Documentation/x86/intel_sgx.rst
 create mode 100644 arch/x86/include/asm/sgx.h
 create mode 100644 arch/x86/include/asm/sgx_arch.h
 create mode 100644 arch/x86/include/asm/sgx_le.h
 create mode 100644 arch/x86/include/asm/sgx_pr.h
 create mode 100644 arch/x86/include/uapi/asm/sgx.h
 create mode 100644 arch/x86/kernel/cpu/intel_sgx.c
 create mode 100644 drivers/platform/x86/intel_sgx/Kconfig
 create mode 100644 drivers/platform/x86/intel_sgx/Makefile
 create mode 100644 drivers/platform/x86/intel_sgx/le/Makefile
 create mode 100644 drivers/platform/x86/intel_sgx/le/enclave/Makefile
 create mode 120000 drivers/platform/x86/intel_sgx/le/enclave/aesni-intel_asm.S
 create mode 100644 drivers/platform/x86/intel_sgx/le/enclave/cmac_mode.c
 create mode 100644 drivers/platform/x86/intel_sgx/le/enclave/cmac_mode.h
 create mode 100644 drivers/platform/x86/intel_sgx/le/enclave/encl_bootstrap.S
 create mode 100644 drivers/platform/x86/intel_sgx/le/enclave/main.c
 create mode 100644 drivers/platform/x86/intel_sgx/le/enclave/main.h
 create mode 100644 drivers/platform/x86/intel_sgx/le/enclave/sgx_le.lds
 create mode 100644 drivers/platform/x86/intel_sgx/le/enclave/sgxsign.c
 create mode 120000 drivers/platform/x86/intel_sgx/le/enclave/string.c
 create mode 100644 drivers/platform/x86/intel_sgx/le/entry.S
 create mode 100644 drivers/platform/x86/intel_sgx/le/include/sgx_asm.h
 create mode 100644 drivers/platform/x86/intel_sgx/le/main.c
 create mode 100644 drivers/platform/x86/intel_sgx/le/main.h
 create mode 100644 drivers/platform/x86/intel_sgx/le/sgx_le_piggy.S
 create mode 100644 drivers/platform/x86/intel_sgx/le/string.c
 create mode 100644 drivers/platform/x86/intel_sgx/sgx.h
 create mode 100644 drivers/platform/x86/intel_sgx/sgx_encl.c
 create mode 100644 drivers/platform/x86/intel_sgx/sgx_encl_page.c
 create mode 100644 drivers/platform/x86/intel_sgx/sgx_fault.c
 create mode 100644 drivers/platform/x86/intel_sgx/sgx_ioctl.c
 create mode 100644 drivers/platform/x86/intel_sgx/sgx_le.c
 create mode 100644 drivers/platform/x86/intel_sgx/sgx_le_proxy_piggy.S
 create mode 100644 drivers/platform/x86/intel_sgx/sgx_main.c
 create mode 100644 drivers/platform/x86/intel_sgx/sgx_vma.c

-- 
2.17.0

--
To unsubscribe from this list: send the line "unsubscribe linux-doc" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[PATCH v11 01/13] compiler.h, kasan: add __SANITIZE_ADDRESS__ check for __no_kasan_or_inline

[Jarkko Sakkinen]

From: Sean Christopherson <sean.j.christopherson@intel.com>

Add 'defined(__SANITIZE_ADDRESS__)' to check to determine whether
__no_kasan_or_inline should be 'no_kasan' or 'inline'.  Files that
compile with KASAN disabled may not link to KASAN, in which case
the __maybe_unused attribute added in the 'no_kasan' option can
cause linker errors due to unused functions that manually invoke
KASAN functions, e.g. read_word_at_a_time(), not being discarded.

Signed-off-by: Sean Christopherson <sean.j.christopherson@intel.com>
---
 include/linux/compiler.h | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/include/linux/compiler.h b/include/linux/compiler.h
index ab4711c63601..e7a863a3ac8c 100644
--- a/include/linux/compiler.h
+++ b/include/linux/compiler.h
@@ -188,7 +188,7 @@ void __read_once_size(const volatile void *p, void *res, int size)
 	__READ_ONCE_SIZE;
 }
 
-#ifdef CONFIG_KASAN
+#if defined(CONFIG_KASAN) && defined(__SANITIZE_ADDRESS__)
 /*
  * We can't declare function 'inline' because __no_sanitize_address confilcts
  * with inlining. Attempt to inline it may cause a build failure.
-- 
2.17.0

[PATCH v11 01/13] compiler.h, kasan: add __SANITIZE_ADDRESS__ check for __no_kasan_or_inline

[Jarkko Sakkinen]

From: Sean Christopherson <sean.j.christopherson@intel.com>

Add 'defined(__SANITIZE_ADDRESS__)' to check to determine whether
__no_kasan_or_inline should be 'no_kasan' or 'inline'.  Files that
compile with KASAN disabled may not link to KASAN, in which case
the __maybe_unused attribute added in the 'no_kasan' option can
cause linker errors due to unused functions that manually invoke
KASAN functions, e.g. read_word_at_a_time(), not being discarded.

Signed-off-by: Sean Christopherson <sean.j.christopherson@intel.com>
---
 include/linux/compiler.h | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/include/linux/compiler.h b/include/linux/compiler.h
index ab4711c63601..e7a863a3ac8c 100644
--- a/include/linux/compiler.h
+++ b/include/linux/compiler.h
@@ -188,7 +188,7 @@ void __read_once_size(const volatile void *p, void *res, int size)
 	__READ_ONCE_SIZE;
 }
 
-#ifdef CONFIG_KASAN
+#if defined(CONFIG_KASAN) && defined(__SANITIZE_ADDRESS__)
 /*
  * We can't declare function 'inline' because __no_sanitize_address confilcts
  * with inlining. Attempt to inline it may cause a build failure.
-- 
2.17.0

[PATCH v11 02/13] x86, sgx: updated MAINTAINERS

[Jarkko Sakkinen]

Signed-off-by: Jarkko Sakkinen <jarkko.sakkinen@linux.intel.com>
---
 MAINTAINERS | 7 +++++++
 1 file changed, 7 insertions(+)

diff --git a/MAINTAINERS b/MAINTAINERS
index ca4afd68530c..f32374728571 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -7329,6 +7329,13 @@ L:	linux-gpio@vger.kernel.org
 S:	Maintained
 F:	drivers/gpio/gpio-intel-mid.c
 
+INTEL SGX
+M:	Jarkko Sakkinen <jarkko.sakkinen@linux.intel.com>
+L:	intel-sgx-kernel-dev@lists.01.org
+Q:	https://patchwork.kernel.org/project/intel-sgx/list/
+F:	drivers/platform/x86/intel_sgx/
+K:	\bSGX_
+
 INVENSENSE MPU-3050 GYROSCOPE DRIVER
 M:	Linus Walleij <linus.walleij@linaro.org>
 L:	linux-iio@vger.kernel.org
-- 
2.17.0

[PATCH v11 02/13] x86, sgx: updated MAINTAINERS

[Jarkko Sakkinen]

Signed-off-by: Jarkko Sakkinen <jarkko.sakkinen@linux.intel.com>
---
 MAINTAINERS | 7 +++++++
 1 file changed, 7 insertions(+)

diff --git a/MAINTAINERS b/MAINTAINERS
index ca4afd68530c..f32374728571 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -7329,6 +7329,13 @@ L:	linux-gpio@vger.kernel.org
 S:	Maintained
 F:	drivers/gpio/gpio-intel-mid.c
 
+INTEL SGX
+M:	Jarkko Sakkinen <jarkko.sakkinen@linux.intel.com>
+L:	intel-sgx-kernel-dev@lists.01.org
+Q:	https://patchwork.kernel.org/project/intel-sgx/list/
+F:	drivers/platform/x86/intel_sgx/
+K:	\bSGX_
+
 INVENSENSE MPU-3050 GYROSCOPE DRIVER
 M:	Linus Walleij <linus.walleij@linaro.org>
 L:	linux-iio@vger.kernel.org
-- 
2.17.0

[PATCH v11 03/13] x86, sgx: add SGX definitions to cpufeature

[Jarkko Sakkinen]

From: Kai Huang <kai.huang@linux.intel.com>

Added X86_FEATURE_SGX and X86_FEATURE_SGX_LC definitions that define the
bits CPUID level 7 bits for determining whether the CPU supports SGX and
launch configuration other than the Intel proprietary key. If this the
case, IA32_SGXLEPUBKEYHASHn MSRs (0 < n < 4) are available for defining
the root key for enclaves.

Signed-off-by: Kai Huang <kai.huang@linux.intel.com>
Tested-by: Serge Ayoun <serge.ayoun@intel.com>
Reviewed-by: Jarkko Sakkinen <jarkko.sakkinen@linux.intel.com>
Tested-by: Jarkko Sakkinen <jarkko.sakkinen@linux.intel.com>
Signed-off-by: Jarkko Sakkinen <jarkko.sakkinen@linux.intel.com>
---
 arch/x86/include/asm/cpufeatures.h | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/arch/x86/include/asm/cpufeatures.h b/arch/x86/include/asm/cpufeatures.h
index fb00a2fca990..54d5269e1b86 100644
--- a/arch/x86/include/asm/cpufeatures.h
+++ b/arch/x86/include/asm/cpufeatures.h
@@ -234,6 +234,7 @@
 /* Intel-defined CPU features, CPUID level 0x00000007:0 (EBX), word 9 */
 #define X86_FEATURE_FSGSBASE		( 9*32+ 0) /* RDFSBASE, WRFSBASE, RDGSBASE, WRGSBASE instructions*/
 #define X86_FEATURE_TSC_ADJUST		( 9*32+ 1) /* TSC adjustment MSR 0x3B */
+#define X86_FEATURE_SGX			( 9*32+ 2) /* Software Guard Extensions */
 #define X86_FEATURE_BMI1		( 9*32+ 3) /* 1st group bit manipulation extensions */
 #define X86_FEATURE_HLE			( 9*32+ 4) /* Hardware Lock Elision */
 #define X86_FEATURE_AVX2		( 9*32+ 5) /* AVX2 instructions */
@@ -327,6 +328,7 @@
 #define X86_FEATURE_LA57		(16*32+16) /* 5-level page tables */
 #define X86_FEATURE_RDPID		(16*32+22) /* RDPID instruction */
 #define X86_FEATURE_CLDEMOTE		(16*32+25) /* CLDEMOTE instruction */
+#define X86_FEATURE_SGX_LC		(16*32+30) /* supports SGX launch configuration */
 
 /* AMD-defined CPU features, CPUID level 0x80000007 (EBX), word 17 */
 #define X86_FEATURE_OVERFLOW_RECOV	(17*32+ 0) /* MCA overflow recovery support */
-- 
2.17.0

[PATCH v11 03/13] x86, sgx: add SGX definitions to cpufeature

[Jarkko Sakkinen]

From: Kai Huang <kai.huang@linux.intel.com>

Added X86_FEATURE_SGX and X86_FEATURE_SGX_LC definitions that define the
bits CPUID level 7 bits for determining whether the CPU supports SGX and
launch configuration other than the Intel proprietary key. If this the
case, IA32_SGXLEPUBKEYHASHn MSRs (0 < n < 4) are available for defining
the root key for enclaves.

Signed-off-by: Kai Huang <kai.huang@linux.intel.com>
Tested-by: Serge Ayoun <serge.ayoun@intel.com>
Reviewed-by: Jarkko Sakkinen <jarkko.sakkinen@linux.intel.com>
Tested-by: Jarkko Sakkinen <jarkko.sakkinen@linux.intel.com>
Signed-off-by: Jarkko Sakkinen <jarkko.sakkinen@linux.intel.com>
---
 arch/x86/include/asm/cpufeatures.h | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/arch/x86/include/asm/cpufeatures.h b/arch/x86/include/asm/cpufeatures.h
index fb00a2fca990..54d5269e1b86 100644
--- a/arch/x86/include/asm/cpufeatures.h
+++ b/arch/x86/include/asm/cpufeatures.h
@@ -234,6 +234,7 @@
 /* Intel-defined CPU features, CPUID level 0x00000007:0 (EBX), word 9 */
 #define X86_FEATURE_FSGSBASE		( 9*32+ 0) /* RDFSBASE, WRFSBASE, RDGSBASE, WRGSBASE instructions*/
 #define X86_FEATURE_TSC_ADJUST		( 9*32+ 1) /* TSC adjustment MSR 0x3B */
+#define X86_FEATURE_SGX			( 9*32+ 2) /* Software Guard Extensions */
 #define X86_FEATURE_BMI1		( 9*32+ 3) /* 1st group bit manipulation extensions */
 #define X86_FEATURE_HLE			( 9*32+ 4) /* Hardware Lock Elision */
 #define X86_FEATURE_AVX2		( 9*32+ 5) /* AVX2 instructions */
@@ -327,6 +328,7 @@
 #define X86_FEATURE_LA57		(16*32+16) /* 5-level page tables */
 #define X86_FEATURE_RDPID		(16*32+22) /* RDPID instruction */
 #define X86_FEATURE_CLDEMOTE		(16*32+25) /* CLDEMOTE instruction */
+#define X86_FEATURE_SGX_LC		(16*32+30) /* supports SGX launch configuration */
 
 /* AMD-defined CPU features, CPUID level 0x80000007 (EBX), word 17 */
 #define X86_FEATURE_OVERFLOW_RECOV	(17*32+ 0) /* MCA overflow recovery support */
-- 
2.17.0

[PATCH v11 04/13] x86, sgx: add SGX definitions to msr-index.h

[Jarkko Sakkinen]

From: Sean Christopherson <sean.j.christopherson@intel.com>

ENCLS and ENCLU are usable if and only if SGX_ENABLE is set and After
SGX is activated the IA32_SGXLEPUBKEYHASHn MSRs are writable if
SGX_LC_WR is set and the feature control is locked.

SGX related bits in IA32_FEATURE_CONTROL cannot be set before SGX is
activated by the pre-boot firmware. SGX activation is triggered by
setting bit 0 in the MSR 0x7a. Until SGX is activated, the LE hash MSRs
are writable to allow pre-boot firmware to lock down the LE root key
with a non-Intel value.

Signed-off-by: Sean Christopherson <sean.j.christopherson@intel.com>
Signed-off-by: Haim Cohen <haim.cohen@intel.com>
Tested-by: Serge Ayoun <serge.ayoun@intel.com>
Reviewed-by: Jarkko Sakkinen <jarkko.sakkinen@linux.intel.com>
Tested-by: Jarkko Sakkinen <jarkko.sakkinen@linux.intel.com>
Signed-off-by: Jarkko Sakkinen <jarkko.sakkinen@linux.intel.com>
---
 arch/x86/include/asm/msr-index.h | 8 ++++++++
 1 file changed, 8 insertions(+)

diff --git a/arch/x86/include/asm/msr-index.h b/arch/x86/include/asm/msr-index.h
index fda2114197b3..a7355fb7344f 100644
--- a/arch/x86/include/asm/msr-index.h
+++ b/arch/x86/include/asm/msr-index.h
@@ -479,6 +479,8 @@
 #define FEATURE_CONTROL_LOCKED				(1<<0)
 #define FEATURE_CONTROL_VMXON_ENABLED_INSIDE_SMX	(1<<1)
 #define FEATURE_CONTROL_VMXON_ENABLED_OUTSIDE_SMX	(1<<2)
+#define FEATURE_CONTROL_SGX_ENABLE                      (1<<18)
+#define FEATURE_CONTROL_SGX_LE_WR			(1<<17)
 #define FEATURE_CONTROL_LMCE				(1<<20)
 
 #define MSR_IA32_APICBASE		0x0000001b
@@ -545,6 +547,12 @@
 #define PACKAGE_THERM_INT_LOW_ENABLE		(1 << 1)
 #define PACKAGE_THERM_INT_PLN_ENABLE		(1 << 24)
 
+/* Intel SGX MSRs */
+#define MSR_IA32_SGXLEPUBKEYHASH0	0x0000008C
+#define MSR_IA32_SGXLEPUBKEYHASH1	0x0000008D
+#define MSR_IA32_SGXLEPUBKEYHASH2	0x0000008E
+#define MSR_IA32_SGXLEPUBKEYHASH3	0x0000008F
+
 /* Thermal Thresholds Support */
 #define THERM_INT_THRESHOLD0_ENABLE    (1 << 15)
 #define THERM_SHIFT_THRESHOLD0        8
-- 
2.17.0

[PATCH v11 04/13] x86, sgx: add SGX definitions to msr-index.h

[Jarkko Sakkinen]

From: Sean Christopherson <sean.j.christopherson@intel.com>

ENCLS and ENCLU are usable if and only if SGX_ENABLE is set and After
SGX is activated the IA32_SGXLEPUBKEYHASHn MSRs are writable if
SGX_LC_WR is set and the feature control is locked.

SGX related bits in IA32_FEATURE_CONTROL cannot be set before SGX is
activated by the pre-boot firmware. SGX activation is triggered by
setting bit 0 in the MSR 0x7a. Until SGX is activated, the LE hash MSRs
are writable to allow pre-boot firmware to lock down the LE root key
with a non-Intel value.

Signed-off-by: Sean Christopherson <sean.j.christopherson@intel.com>
Signed-off-by: Haim Cohen <haim.cohen@intel.com>
Tested-by: Serge Ayoun <serge.ayoun@intel.com>
Reviewed-by: Jarkko Sakkinen <jarkko.sakkinen@linux.intel.com>
Tested-by: Jarkko Sakkinen <jarkko.sakkinen@linux.intel.com>
Signed-off-by: Jarkko Sakkinen <jarkko.sakkinen@linux.intel.com>
---
 arch/x86/include/asm/msr-index.h | 8 ++++++++
 1 file changed, 8 insertions(+)

diff --git a/arch/x86/include/asm/msr-index.h b/arch/x86/include/asm/msr-index.h
index fda2114197b3..a7355fb7344f 100644
--- a/arch/x86/include/asm/msr-index.h
+++ b/arch/x86/include/asm/msr-index.h
@@ -479,6 +479,8 @@
 #define FEATURE_CONTROL_LOCKED				(1<<0)
 #define FEATURE_CONTROL_VMXON_ENABLED_INSIDE_SMX	(1<<1)
 #define FEATURE_CONTROL_VMXON_ENABLED_OUTSIDE_SMX	(1<<2)
+#define FEATURE_CONTROL_SGX_ENABLE                      (1<<18)
+#define FEATURE_CONTROL_SGX_LE_WR			(1<<17)
 #define FEATURE_CONTROL_LMCE				(1<<20)
 
 #define MSR_IA32_APICBASE		0x0000001b
@@ -545,6 +547,12 @@
 #define PACKAGE_THERM_INT_LOW_ENABLE		(1 << 1)
 #define PACKAGE_THERM_INT_PLN_ENABLE		(1 << 24)
 
+/* Intel SGX MSRs */
+#define MSR_IA32_SGXLEPUBKEYHASH0	0x0000008C
+#define MSR_IA32_SGXLEPUBKEYHASH1	0x0000008D
+#define MSR_IA32_SGXLEPUBKEYHASH2	0x0000008E
+#define MSR_IA32_SGXLEPUBKEYHASH3	0x0000008F
+
 /* Thermal Thresholds Support */
 #define THERM_INT_THRESHOLD0_ENABLE    (1 << 15)
 #define THERM_SHIFT_THRESHOLD0        8
-- 
2.17.0

[PATCH v11 05/13] x86, cpufeatures: add Intel-defined SGX leaf CPUID_12_EAX

[Jarkko Sakkinen]

From: Sean Christopherson <sean.j.christopherson@intel.com>

CPUID_12_EAX is an Intel-defined feature bits leaf dedicated for SGX.
There are currently four documented feature bits, with more expected
in the not-too-distant future.

Signed-off-by: Sean Christopherson <sean.j.christopherson@intel.com>
---
 arch/x86/include/asm/cpufeature.h        | 7 +++++--
 arch/x86/include/asm/cpufeatures.h       | 8 +++++++-
 arch/x86/include/asm/disabled-features.h | 3 ++-
 arch/x86/include/asm/required-features.h | 3 ++-
 arch/x86/kernel/cpu/common.c             | 7 +++++++
 arch/x86/kvm/cpuid.h                     | 1 +
 6 files changed, 24 insertions(+), 5 deletions(-)

diff --git a/arch/x86/include/asm/cpufeature.h b/arch/x86/include/asm/cpufeature.h
index aced6c9290d6..940f0c01d5f8 100644
--- a/arch/x86/include/asm/cpufeature.h
+++ b/arch/x86/include/asm/cpufeature.h
@@ -30,6 +30,7 @@ enum cpuid_leafs
 	CPUID_7_ECX,
 	CPUID_8000_0007_EBX,
 	CPUID_7_EDX,
+	CPUID_12_EAX,
 };
 
 #ifdef CONFIG_X86_FEATURE_NAMES
@@ -81,8 +82,9 @@ extern const char * const x86_bug_flags[NBUGINTS*32];
 	   CHECK_BIT_IN_MASK_WORD(REQUIRED_MASK, 16, feature_bit) ||	\
 	   CHECK_BIT_IN_MASK_WORD(REQUIRED_MASK, 17, feature_bit) ||	\
 	   CHECK_BIT_IN_MASK_WORD(REQUIRED_MASK, 18, feature_bit) ||	\
+	   CHECK_BIT_IN_MASK_WORD(REQUIRED_MASK, 19, feature_bit) ||	\
 	   REQUIRED_MASK_CHECK					  ||	\
-	   BUILD_BUG_ON_ZERO(NCAPINTS != 19))
+	   BUILD_BUG_ON_ZERO(NCAPINTS != 20))
 
 #define DISABLED_MASK_BIT_SET(feature_bit)				\
 	 ( CHECK_BIT_IN_MASK_WORD(DISABLED_MASK,  0, feature_bit) ||	\
@@ -104,8 +106,9 @@ extern const char * const x86_bug_flags[NBUGINTS*32];
 	   CHECK_BIT_IN_MASK_WORD(DISABLED_MASK, 16, feature_bit) ||	\
 	   CHECK_BIT_IN_MASK_WORD(DISABLED_MASK, 17, feature_bit) ||	\
 	   CHECK_BIT_IN_MASK_WORD(DISABLED_MASK, 18, feature_bit) ||	\
+	   CHECK_BIT_IN_MASK_WORD(DISABLED_MASK, 19, feature_bit) ||	\
 	   DISABLED_MASK_CHECK					  ||	\
-	   BUILD_BUG_ON_ZERO(NCAPINTS != 19))
+	   BUILD_BUG_ON_ZERO(NCAPINTS != 20))
 
 #define cpu_has(c, bit)							\
 	(__builtin_constant_p(bit) && REQUIRED_MASK_BIT_SET(bit) ? 1 :	\
diff --git a/arch/x86/include/asm/cpufeatures.h b/arch/x86/include/asm/cpufeatures.h
index 54d5269e1b86..4f730957b4ae 100644
--- a/arch/x86/include/asm/cpufeatures.h
+++ b/arch/x86/include/asm/cpufeatures.h
@@ -13,7 +13,7 @@
 /*
  * Defines x86 CPU feature bits
  */
-#define NCAPINTS			19	   /* N 32-bit words worth of info */
+#define NCAPINTS			20	   /* N 32-bit words worth of info */
 #define NBUGINTS			1	   /* N 32-bit bug flags */
 
 /*
@@ -344,6 +344,12 @@
 #define X86_FEATURE_ARCH_CAPABILITIES	(18*32+29) /* IA32_ARCH_CAPABILITIES MSR (Intel) */
 #define X86_FEATURE_SPEC_CTRL_SSBD	(18*32+31) /* "" Speculative Store Bypass Disable */
 
+/* Intel SGX CPU features, CPUID level 0x000000012:0 (EAX), word 19 */
+#define X86_FEATURE_SGX1		(19*32+ 0) /* SGX1 leaf functions */
+#define X86_FEATURE_SGX2		(19*32+ 2) /* SGX2 leaf functions */
+#define X86_FEATURE_SGX_ENCLV		(19*32+ 5) /* SGX ENCLV instruction, leafs E[INC|DEC]VIRTCHILD, ESETCONTEXT */
+#define X86_FEATURE_SGX_ENCLS_C		(19*32+ 6) /* SGX ENCLS leafs ERDINFO, ETRACK, ELDBC and ELDUC */
+
 /*
  * BUG word(s)
  */
diff --git a/arch/x86/include/asm/disabled-features.h b/arch/x86/include/asm/disabled-features.h
index 33833d1909af..c81b5d446a3e 100644
--- a/arch/x86/include/asm/disabled-features.h
+++ b/arch/x86/include/asm/disabled-features.h
@@ -78,6 +78,7 @@
 #define DISABLED_MASK16	(DISABLE_PKU|DISABLE_OSPKE|DISABLE_LA57|DISABLE_UMIP)
 #define DISABLED_MASK17	0
 #define DISABLED_MASK18	0
-#define DISABLED_MASK_CHECK BUILD_BUG_ON_ZERO(NCAPINTS != 19)
+#define DISABLED_MASK19	0
+#define DISABLED_MASK_CHECK BUILD_BUG_ON_ZERO(NCAPINTS != 20)
 
 #endif /* _ASM_X86_DISABLED_FEATURES_H */
diff --git a/arch/x86/include/asm/required-features.h b/arch/x86/include/asm/required-features.h
index 6847d85400a8..fa5700097f64 100644
--- a/arch/x86/include/asm/required-features.h
+++ b/arch/x86/include/asm/required-features.h
@@ -101,6 +101,7 @@
 #define REQUIRED_MASK16	0
 #define REQUIRED_MASK17	0
 #define REQUIRED_MASK18	0
-#define REQUIRED_MASK_CHECK BUILD_BUG_ON_ZERO(NCAPINTS != 19)
+#define REQUIRED_MASK19	0
+#define REQUIRED_MASK_CHECK BUILD_BUG_ON_ZERO(NCAPINTS != 20)
 
 #endif /* _ASM_X86_REQUIRED_FEATURES_H */
diff --git a/arch/x86/kernel/cpu/common.c b/arch/x86/kernel/cpu/common.c
index 38276f58d3bf..5e712fa6bf4f 100644
--- a/arch/x86/kernel/cpu/common.c
+++ b/arch/x86/kernel/cpu/common.c
@@ -843,6 +843,13 @@ void get_cpu_cap(struct cpuinfo_x86 *c)
 		}
 	}
 
+	/* Intel SGX features: level 0x00000012 */
+	if (c->cpuid_level >= 0x00000012) {
+		cpuid(0x00000012, &eax, &ebx, &ecx, &edx);
+
+		c->x86_capability[CPUID_12_EAX] = eax;
+	}
+
 	/* AMD-defined flags: level 0x80000001 */
 	eax = cpuid_eax(0x80000000);
 	c->extended_cpuid_level = eax;
diff --git a/arch/x86/kvm/cpuid.h b/arch/x86/kvm/cpuid.h
index 9a327d5b6d1f..669c1774afdb 100644
--- a/arch/x86/kvm/cpuid.h
+++ b/arch/x86/kvm/cpuid.h
@@ -55,6 +55,7 @@ static const struct cpuid_reg reverse_cpuid[] = {
 	[CPUID_7_ECX]         = {         7, 0, CPUID_ECX},
 	[CPUID_8000_0007_EBX] = {0x80000007, 0, CPUID_EBX},
 	[CPUID_7_EDX]         = {         7, 0, CPUID_EDX},
+	[CPUID_12_EAX]        = {        12, 0, CPUID_EAX},
 };
 
 static __always_inline struct cpuid_reg x86_feature_cpuid(unsigned x86_feature)
-- 
2.17.0

[PATCH v11 05/13] x86, cpufeatures: add Intel-defined SGX leaf CPUID_12_EAX

[Jarkko Sakkinen]

From: Sean Christopherson <sean.j.christopherson@intel.com>

CPUID_12_EAX is an Intel-defined feature bits leaf dedicated for SGX.
There are currently four documented feature bits, with more expected
in the not-too-distant future.

Signed-off-by: Sean Christopherson <sean.j.christopherson@intel.com>
---
 arch/x86/include/asm/cpufeature.h        | 7 +++++--
 arch/x86/include/asm/cpufeatures.h       | 8 +++++++-
 arch/x86/include/asm/disabled-features.h | 3 ++-
 arch/x86/include/asm/required-features.h | 3 ++-
 arch/x86/kernel/cpu/common.c             | 7 +++++++
 arch/x86/kvm/cpuid.h                     | 1 +
 6 files changed, 24 insertions(+), 5 deletions(-)

diff --git a/arch/x86/include/asm/cpufeature.h b/arch/x86/include/asm/cpufeature.h
index aced6c9290d6..940f0c01d5f8 100644
--- a/arch/x86/include/asm/cpufeature.h
+++ b/arch/x86/include/asm/cpufeature.h
@@ -30,6 +30,7 @@ enum cpuid_leafs
 	CPUID_7_ECX,
 	CPUID_8000_0007_EBX,
 	CPUID_7_EDX,
+	CPUID_12_EAX,
 };
 
 #ifdef CONFIG_X86_FEATURE_NAMES
@@ -81,8 +82,9 @@ extern const char * const x86_bug_flags[NBUGINTS*32];
 	   CHECK_BIT_IN_MASK_WORD(REQUIRED_MASK, 16, feature_bit) ||	\
 	   CHECK_BIT_IN_MASK_WORD(REQUIRED_MASK, 17, feature_bit) ||	\
 	   CHECK_BIT_IN_MASK_WORD(REQUIRED_MASK, 18, feature_bit) ||	\
+	   CHECK_BIT_IN_MASK_WORD(REQUIRED_MASK, 19, feature_bit) ||	\
 	   REQUIRED_MASK_CHECK					  ||	\
-	   BUILD_BUG_ON_ZERO(NCAPINTS != 19))
+	   BUILD_BUG_ON_ZERO(NCAPINTS != 20))
 
 #define DISABLED_MASK_BIT_SET(feature_bit)				\
 	 ( CHECK_BIT_IN_MASK_WORD(DISABLED_MASK,  0, feature_bit) ||	\
@@ -104,8 +106,9 @@ extern const char * const x86_bug_flags[NBUGINTS*32];
 	   CHECK_BIT_IN_MASK_WORD(DISABLED_MASK, 16, feature_bit) ||	\
 	   CHECK_BIT_IN_MASK_WORD(DISABLED_MASK, 17, feature_bit) ||	\
 	   CHECK_BIT_IN_MASK_WORD(DISABLED_MASK, 18, feature_bit) ||	\
+	   CHECK_BIT_IN_MASK_WORD(DISABLED_MASK, 19, feature_bit) ||	\
 	   DISABLED_MASK_CHECK					  ||	\
-	   BUILD_BUG_ON_ZERO(NCAPINTS != 19))
+	   BUILD_BUG_ON_ZERO(NCAPINTS != 20))
 
 #define cpu_has(c, bit)							\
 	(__builtin_constant_p(bit) && REQUIRED_MASK_BIT_SET(bit) ? 1 :	\
diff --git a/arch/x86/include/asm/cpufeatures.h b/arch/x86/include/asm/cpufeatures.h
index 54d5269e1b86..4f730957b4ae 100644
--- a/arch/x86/include/asm/cpufeatures.h
+++ b/arch/x86/include/asm/cpufeatures.h
@@ -13,7 +13,7 @@
 /*
  * Defines x86 CPU feature bits
  */
-#define NCAPINTS			19	   /* N 32-bit words worth of info */
+#define NCAPINTS			20	   /* N 32-bit words worth of info */
 #define NBUGINTS			1	   /* N 32-bit bug flags */
 
 /*
@@ -344,6 +344,12 @@
 #define X86_FEATURE_ARCH_CAPABILITIES	(18*32+29) /* IA32_ARCH_CAPABILITIES MSR (Intel) */
 #define X86_FEATURE_SPEC_CTRL_SSBD	(18*32+31) /* "" Speculative Store Bypass Disable */
 
+/* Intel SGX CPU features, CPUID level 0x000000012:0 (EAX), word 19 */
+#define X86_FEATURE_SGX1		(19*32+ 0) /* SGX1 leaf functions */
+#define X86_FEATURE_SGX2		(19*32+ 2) /* SGX2 leaf functions */
+#define X86_FEATURE_SGX_ENCLV		(19*32+ 5) /* SGX ENCLV instruction, leafs E[INC|DEC]VIRTCHILD, ESETCONTEXT */
+#define X86_FEATURE_SGX_ENCLS_C		(19*32+ 6) /* SGX ENCLS leafs ERDINFO, ETRACK, ELDBC and ELDUC */
+
 /*
  * BUG word(s)
  */
diff --git a/arch/x86/include/asm/disabled-features.h b/arch/x86/include/asm/disabled-features.h
index 33833d1909af..c81b5d446a3e 100644
--- a/arch/x86/include/asm/disabled-features.h
+++ b/arch/x86/include/asm/disabled-features.h
@@ -78,6 +78,7 @@
 #define DISABLED_MASK16	(DISABLE_PKU|DISABLE_OSPKE|DISABLE_LA57|DISABLE_UMIP)
 #define DISABLED_MASK17	0
 #define DISABLED_MASK18	0
-#define DISABLED_MASK_CHECK BUILD_BUG_ON_ZERO(NCAPINTS != 19)
+#define DISABLED_MASK19	0
+#define DISABLED_MASK_CHECK BUILD_BUG_ON_ZERO(NCAPINTS != 20)
 
 #endif /* _ASM_X86_DISABLED_FEATURES_H */
diff --git a/arch/x86/include/asm/required-features.h b/arch/x86/include/asm/required-features.h
index 6847d85400a8..fa5700097f64 100644
--- a/arch/x86/include/asm/required-features.h
+++ b/arch/x86/include/asm/required-features.h
@@ -101,6 +101,7 @@
 #define REQUIRED_MASK16	0
 #define REQUIRED_MASK17	0
 #define REQUIRED_MASK18	0
-#define REQUIRED_MASK_CHECK BUILD_BUG_ON_ZERO(NCAPINTS != 19)
+#define REQUIRED_MASK19	0
+#define REQUIRED_MASK_CHECK BUILD_BUG_ON_ZERO(NCAPINTS != 20)
 
 #endif /* _ASM_X86_REQUIRED_FEATURES_H */
diff --git a/arch/x86/kernel/cpu/common.c b/arch/x86/kernel/cpu/common.c
index 38276f58d3bf..5e712fa6bf4f 100644
--- a/arch/x86/kernel/cpu/common.c
+++ b/arch/x86/kernel/cpu/common.c
@@ -843,6 +843,13 @@ void get_cpu_cap(struct cpuinfo_x86 *c)
 		}
 	}
 
+	/* Intel SGX features: level 0x00000012 */
+	if (c->cpuid_level >= 0x00000012) {
+		cpuid(0x00000012, &eax, &ebx, &ecx, &edx);
+
+		c->x86_capability[CPUID_12_EAX] = eax;
+	}
+
 	/* AMD-defined flags: level 0x80000001 */
 	eax = cpuid_eax(0x80000000);
 	c->extended_cpuid_level = eax;
diff --git a/arch/x86/kvm/cpuid.h b/arch/x86/kvm/cpuid.h
index 9a327d5b6d1f..669c1774afdb 100644
--- a/arch/x86/kvm/cpuid.h
+++ b/arch/x86/kvm/cpuid.h
@@ -55,6 +55,7 @@ static const struct cpuid_reg reverse_cpuid[] = {
 	[CPUID_7_ECX]         = {         7, 0, CPUID_ECX},
 	[CPUID_8000_0007_EBX] = {0x80000007, 0, CPUID_EBX},
 	[CPUID_7_EDX]         = {         7, 0, CPUID_EDX},
+	[CPUID_12_EAX]        = {        12, 0, CPUID_EAX},
 };
 
 static __always_inline struct cpuid_reg x86_feature_cpuid(unsigned x86_feature)
-- 
2.17.0

[PATCH v11 06/13] crypto: aesni: add minimal build option for SGX LE

[Jarkko Sakkinen]

From: Sean Christopherson <sean.j.christopherson@intel.com>

Allow building a minimal subset of the low-level AESNI functions by
defining AESNI_INTEL_MINIMAL.  The SGX Launch Enclave will utilize
a small number of AESNI functions for creating CMACs when generating
tokens for userspace enclaves.

Reducing the size of the LE is high priority as EPC space is at a
premium and initializing/measuring EPC pages is extremely slow, and
defining only the minimal set of AESNI functions reduces the size of
the in-kernel LE by over 50%.  Because the LE is a (very) non-standard
build environment, using linker tricks e.g. --gc-sections to remove
the unused functions is not an option.

Eliminating the unused AESNI functions also eliminates all usage of
the retpoline macros, e.g. CALL_NOSPEC, which allows the LE linker
script to assert that the alternatives and retpoline sections don't
exist in the final binary.  Because the LE's code cannot be patched,
i.e. retpoline can't be enabled via alternatives, we want to assert
that we're not expecting a security feature that can't be enabled.

Signed-off-by: Sean Christopherson <sean.j.christopherson@intel.com>
---
 arch/x86/crypto/aesni-intel_asm.S | 11 +++++++++++
 1 file changed, 11 insertions(+)

diff --git a/arch/x86/crypto/aesni-intel_asm.S b/arch/x86/crypto/aesni-intel_asm.S
index e762ef417562..5a0a487466d5 100644
--- a/arch/x86/crypto/aesni-intel_asm.S
+++ b/arch/x86/crypto/aesni-intel_asm.S
@@ -45,6 +45,8 @@
 #define MOVADQ	movaps
 #define MOVUDQ	movups
 
+#ifndef AESNI_INTEL_MINIMAL
+
 #ifdef __x86_64__
 
 # constants in mergeable sections, linker can reorder and merge
@@ -133,6 +135,8 @@ ALL_F:      .octa 0xffffffffffffffffffffffffffffffff
 #define keysize 2*15*16(%arg1)
 #endif
 
+#endif /* AESNI_INTEL_MINIMAL */
+
 
 #define STATE1	%xmm0
 #define STATE2	%xmm4
@@ -506,6 +510,8 @@ _T_16_\@:
 _return_T_done_\@:
 .endm
 
+#ifndef AESNI_INTEL_MINIMAL
+
 #ifdef __x86_64__
 /* GHASH_MUL MACRO to implement: Data*HashKey mod (128,127,126,121,0)
 *
@@ -1760,6 +1766,7 @@ ENDPROC(aesni_gcm_finalize)
 
 #endif
 
+#endif /* AESNI_INTEL_MINIMAL */
 
 .align 4
 _key_expansion_128:
@@ -2031,6 +2038,8 @@ _aesni_enc1:
 	ret
 ENDPROC(_aesni_enc1)
 
+#ifndef AESNI_INTEL_MINIMAL
+
 /*
  * _aesni_enc4:	internal ABI
  * input:
@@ -2840,3 +2849,5 @@ ENTRY(aesni_xts_crypt8)
 ENDPROC(aesni_xts_crypt8)
 
 #endif
+
+#endif /* AESNI_INTEL_MINIMAL */
-- 
2.17.0

[PATCH v11 06/13] crypto: aesni: add minimal build option for SGX LE

[Jarkko Sakkinen]

From: Sean Christopherson <sean.j.christopherson@intel.com>

Allow building a minimal subset of the low-level AESNI functions by
defining AESNI_INTEL_MINIMAL.  The SGX Launch Enclave will utilize
a small number of AESNI functions for creating CMACs when generating
tokens for userspace enclaves.

Reducing the size of the LE is high priority as EPC space is at a
premium and initializing/measuring EPC pages is extremely slow, and
defining only the minimal set of AESNI functions reduces the size of
the in-kernel LE by over 50%.  Because the LE is a (very) non-standard
build environment, using linker tricks e.g. --gc-sections to remove
the unused functions is not an option.

Eliminating the unused AESNI functions also eliminates all usage of
the retpoline macros, e.g. CALL_NOSPEC, which allows the LE linker
script to assert that the alternatives and retpoline sections don't
exist in the final binary.  Because the LE's code cannot be patched,
i.e. retpoline can't be enabled via alternatives, we want to assert
that we're not expecting a security feature that can't be enabled.

Signed-off-by: Sean Christopherson <sean.j.christopherson@intel.com>
---
 arch/x86/crypto/aesni-intel_asm.S | 11 +++++++++++
 1 file changed, 11 insertions(+)

diff --git a/arch/x86/crypto/aesni-intel_asm.S b/arch/x86/crypto/aesni-intel_asm.S
index e762ef417562..5a0a487466d5 100644
--- a/arch/x86/crypto/aesni-intel_asm.S
+++ b/arch/x86/crypto/aesni-intel_asm.S
@@ -45,6 +45,8 @@
 #define MOVADQ	movaps
 #define MOVUDQ	movups
 
+#ifndef AESNI_INTEL_MINIMAL
+
 #ifdef __x86_64__
 
 # constants in mergeable sections, linker can reorder and merge
@@ -133,6 +135,8 @@ ALL_F:      .octa 0xffffffffffffffffffffffffffffffff
 #define keysize 2*15*16(%arg1)
 #endif
 
+#endif /* AESNI_INTEL_MINIMAL */
+
 
 #define STATE1	%xmm0
 #define STATE2	%xmm4
@@ -506,6 +510,8 @@ _T_16_\@:
 _return_T_done_\@:
 .endm
 
+#ifndef AESNI_INTEL_MINIMAL
+
 #ifdef __x86_64__
 /* GHASH_MUL MACRO to implement: Data*HashKey mod (128,127,126,121,0)
 *
@@ -1760,6 +1766,7 @@ ENDPROC(aesni_gcm_finalize)
 
 #endif
 
+#endif /* AESNI_INTEL_MINIMAL */
 
 .align 4
 _key_expansion_128:
@@ -2031,6 +2038,8 @@ _aesni_enc1:
 	ret
 ENDPROC(_aesni_enc1)
 
+#ifndef AESNI_INTEL_MINIMAL
+
 /*
  * _aesni_enc4:	internal ABI
  * input:
@@ -2840,3 +2849,5 @@ ENTRY(aesni_xts_crypt8)
 ENDPROC(aesni_xts_crypt8)
 
 #endif
+
+#endif /* AESNI_INTEL_MINIMAL */
-- 
2.17.0

[PATCH v11 07/13] x86, sgx: detect Intel SGX

[Jarkko Sakkinen]

From: Sean Christopherson <sean.j.christopherson@intel.com>

Intel(R) SGX is a set of CPU instructions that can be used by applications
to set aside private regions of code and data. The code outside the enclave
is disallowed to access the memory inside the enclave by the CPU access
control.

This commit adds the check for SGX to arch/x86 and a new config option,
INTEL_SGX_CORE. Exposes a boolean variable 'sgx_enabled' to query whether
or not the SGX support is available.

Signed-off-by: Sean Christopherson <sean.j.christopherson@intel.com>
Reviewed-by: Jarkko Sakkinen <jarkko.sakkinen@linux.intel.com>
Tested-by: Jarkko Sakkinen <jarkko.sakkinen@linux.intel.com>
Signed-off-by: Jarkko Sakkinen <jarkko.sakkinen@linux.intel.com>
---
 arch/x86/Kconfig                | 19 ++++++++++++
 arch/x86/include/asm/sgx.h      | 25 ++++++++++++++++
 arch/x86/include/asm/sgx_pr.h   | 20 +++++++++++++
 arch/x86/kernel/cpu/Makefile    |  1 +
 arch/x86/kernel/cpu/intel_sgx.c | 53 +++++++++++++++++++++++++++++++++
 5 files changed, 118 insertions(+)
 create mode 100644 arch/x86/include/asm/sgx.h
 create mode 100644 arch/x86/include/asm/sgx_pr.h
 create mode 100644 arch/x86/kernel/cpu/intel_sgx.c

diff --git a/arch/x86/Kconfig b/arch/x86/Kconfig
index c07f492b871a..42015d5366ef 100644
--- a/arch/x86/Kconfig
+++ b/arch/x86/Kconfig
@@ -1925,6 +1925,25 @@ config X86_INTEL_MEMORY_PROTECTION_KEYS
 
 	  If unsure, say y.
 
+config INTEL_SGX_CORE
+	prompt "Intel SGX core functionality"
+	def_bool n
+	depends on X86_64 && CPU_SUP_INTEL
+	help
+	Intel Software Guard eXtensions (SGX) is a set of CPU instructions
+	that allows ring 3 applications to create enclaves; private regions
+	of memory that are protected, by hardware, from unauthorized access
+	and/or modification.
+
+	This option enables kernel recognition of SGX, high-level management
+	of the Enclave Page Cache (EPC), tracking and writing of SGX Launch
+	Enclave Hash MSRs, and allows for virtualization of SGX via KVM. By
+	iteslf, this option does not provide SGX support to userspace.
+
+	For details, see Documentation/x86/intel_sgx.rst
+
+	If unsure, say N.
+
 config EFI
 	bool "EFI runtime service support"
 	depends on ACPI
diff --git a/arch/x86/include/asm/sgx.h b/arch/x86/include/asm/sgx.h
new file mode 100644
index 000000000000..fa3e6e0eb8af
--- /dev/null
+++ b/arch/x86/include/asm/sgx.h
@@ -0,0 +1,25 @@
+// SPDX-License-Identifier: (GPL-2.0 OR BSD-3-Clause)
+// Copyright(c) 2016-18 Intel Corporation.
+//
+// Authors:
+//
+// Jarkko Sakkinen <jarkko.sakkinen@linux.intel.com>
+// Suresh Siddha <suresh.b.siddha@intel.com>
+// Sean Christopherson <sean.j.christopherson@intel.com>
+
+#ifndef _ASM_X86_SGX_H
+#define _ASM_X86_SGX_H
+
+#include <linux/types.h>
+
+#define SGX_CPUID 0x12
+
+enum sgx_cpuid {
+	SGX_CPUID_CAPABILITIES	= 0,
+	SGX_CPUID_ATTRIBUTES	= 1,
+	SGX_CPUID_EPC_BANKS	= 2,
+};
+
+extern bool sgx_enabled;
+
+#endif /* _ASM_X86_SGX_H */
diff --git a/arch/x86/include/asm/sgx_pr.h b/arch/x86/include/asm/sgx_pr.h
new file mode 100644
index 000000000000..876dc44c2ccc
--- /dev/null
+++ b/arch/x86/include/asm/sgx_pr.h
@@ -0,0 +1,20 @@
+// SPDX-License-Identifier: (GPL-2.0 OR BSD-3-Clause)
+// Copyright(c) 2016-17 Intel Corporation.
+//
+// Authors:
+//
+// Jarkko Sakkinen <jarkko.sakkinen@linux.intel.com>
+// Suresh Siddha <suresh.b.siddha@intel.com>
+// Serge Ayoun <serge.ayoun@intel.com>
+// Shay Katz-zamir <shay.katz-zamir@intel.com>
+
+#ifndef _ASM_X86_SGX_PR_H
+#define _ASM_X86_SGX_PR_H
+
+#include <linux/printk.h>
+#include <linux/ratelimit.h>
+
+#undef pr_fmt
+#define pr_fmt(fmt) "intel_sgx: " fmt
+
+#endif /* _ASM_X86_SGX_PR_H */
diff --git a/arch/x86/kernel/cpu/Makefile b/arch/x86/kernel/cpu/Makefile
index a66229f51b12..9552ff5b4ec3 100644
--- a/arch/x86/kernel/cpu/Makefile
+++ b/arch/x86/kernel/cpu/Makefile
@@ -36,6 +36,7 @@ obj-$(CONFIG_CPU_SUP_TRANSMETA_32)	+= transmeta.o
 obj-$(CONFIG_CPU_SUP_UMC_32)		+= umc.o
 
 obj-$(CONFIG_INTEL_RDT)	+= intel_rdt.o intel_rdt_rdtgroup.o intel_rdt_monitor.o intel_rdt_ctrlmondata.o
+obj-$(CONFIG_INTEL_SGX_CORE)		+= intel_sgx.o
 
 obj-$(CONFIG_X86_MCE)			+= mcheck/
 obj-$(CONFIG_MTRR)			+= mtrr/
diff --git a/arch/x86/kernel/cpu/intel_sgx.c b/arch/x86/kernel/cpu/intel_sgx.c
new file mode 100644
index 000000000000..db6b315334f4
--- /dev/null
+++ b/arch/x86/kernel/cpu/intel_sgx.c
@@ -0,0 +1,53 @@
+// SPDX-License-Identifier: (GPL-2.0 OR BSD-3-Clause)
+// Copyright(c) 2016-17 Intel Corporation.
+//
+// Authors:
+//
+// Jarkko Sakkinen <jarkko.sakkinen@linux.intel.com>
+// Suresh Siddha <suresh.b.siddha@intel.com>
+// Serge Ayoun <serge.ayoun@intel.com>
+// Shay Katz-zamir <shay.katz-zamir@intel.com>
+// Sean Christopherson <sean.j.christopherson@intel.com>
+
+#include <asm/sgx.h>
+#include <asm/sgx_pr.h>
+#include <linux/freezer.h>
+#include <linux/highmem.h>
+#include <linux/kthread.h>
+#include <linux/ratelimit.h>
+#include <linux/sched/signal.h>
+#include <linux/slab.h>
+
+bool sgx_enabled __ro_after_init = false;
+EXPORT_SYMBOL(sgx_enabled);
+
+static __init bool sgx_is_enabled(void)
+{
+	unsigned long fc;
+
+	if (boot_cpu_data.x86_vendor != X86_VENDOR_INTEL)
+		return false;
+
+	if (!boot_cpu_has(X86_FEATURE_SGX))
+		return false;
+
+	if (!boot_cpu_has(X86_FEATURE_SGX1))
+		return false;
+
+	rdmsrl(MSR_IA32_FEATURE_CONTROL, fc);
+	if (!(fc & FEATURE_CONTROL_LOCKED))
+		return false;
+
+	if (!(fc & FEATURE_CONTROL_SGX_ENABLE))
+		return false;
+
+	return true;
+}
+
+static __init int sgx_init(void)
+{
+	sgx_enabled = sgx_is_enabled();
+	return 0;
+}
+
+arch_initcall(sgx_init);
-- 
2.17.0

[PATCH v11 07/13] x86, sgx: detect Intel SGX

[Jarkko Sakkinen]

From: Sean Christopherson <sean.j.christopherson@intel.com>

Intel(R) SGX is a set of CPU instructions that can be used by applications
to set aside private regions of code and data. The code outside the enclave
is disallowed to access the memory inside the enclave by the CPU access
control.

This commit adds the check for SGX to arch/x86 and a new config option,
INTEL_SGX_CORE. Exposes a boolean variable 'sgx_enabled' to query whether
or not the SGX support is available.

Signed-off-by: Sean Christopherson <sean.j.christopherson@intel.com>
Reviewed-by: Jarkko Sakkinen <jarkko.sakkinen@linux.intel.com>
Tested-by: Jarkko Sakkinen <jarkko.sakkinen@linux.intel.com>
Signed-off-by: Jarkko Sakkinen <jarkko.sakkinen@linux.intel.com>
---
 arch/x86/Kconfig                | 19 ++++++++++++
 arch/x86/include/asm/sgx.h      | 25 ++++++++++++++++
 arch/x86/include/asm/sgx_pr.h   | 20 +++++++++++++
 arch/x86/kernel/cpu/Makefile    |  1 +
 arch/x86/kernel/cpu/intel_sgx.c | 53 +++++++++++++++++++++++++++++++++
 5 files changed, 118 insertions(+)
 create mode 100644 arch/x86/include/asm/sgx.h
 create mode 100644 arch/x86/include/asm/sgx_pr.h
 create mode 100644 arch/x86/kernel/cpu/intel_sgx.c

diff --git a/arch/x86/Kconfig b/arch/x86/Kconfig
index c07f492b871a..42015d5366ef 100644
--- a/arch/x86/Kconfig
+++ b/arch/x86/Kconfig
@@ -1925,6 +1925,25 @@ config X86_INTEL_MEMORY_PROTECTION_KEYS
 
 	  If unsure, say y.
 
+config INTEL_SGX_CORE
+	prompt "Intel SGX core functionality"
+	def_bool n
+	depends on X86_64 && CPU_SUP_INTEL
+	help
+	Intel Software Guard eXtensions (SGX) is a set of CPU instructions
+	that allows ring 3 applications to create enclaves; private regions
+	of memory that are protected, by hardware, from unauthorized access
+	and/or modification.
+
+	This option enables kernel recognition of SGX, high-level management
+	of the Enclave Page Cache (EPC), tracking and writing of SGX Launch
+	Enclave Hash MSRs, and allows for virtualization of SGX via KVM. By
+	iteslf, this option does not provide SGX support to userspace.
+
+	For details, see Documentation/x86/intel_sgx.rst
+
+	If unsure, say N.
+
 config EFI
 	bool "EFI runtime service support"
 	depends on ACPI
diff --git a/arch/x86/include/asm/sgx.h b/arch/x86/include/asm/sgx.h
new file mode 100644
index 000000000000..fa3e6e0eb8af
--- /dev/null
+++ b/arch/x86/include/asm/sgx.h
@@ -0,0 +1,25 @@
+// SPDX-License-Identifier: (GPL-2.0 OR BSD-3-Clause)
+// Copyright(c) 2016-18 Intel Corporation.
+//
+// Authors:
+//
+// Jarkko Sakkinen <jarkko.sakkinen@linux.intel.com>
+// Suresh Siddha <suresh.b.siddha@intel.com>
+// Sean Christopherson <sean.j.christopherson@intel.com>
+
+#ifndef _ASM_X86_SGX_H
+#define _ASM_X86_SGX_H
+
+#include <linux/types.h>
+
+#define SGX_CPUID 0x12
+
+enum sgx_cpuid {
+	SGX_CPUID_CAPABILITIES	= 0,
+	SGX_CPUID_ATTRIBUTES	= 1,
+	SGX_CPUID_EPC_BANKS	= 2,
+};
+
+extern bool sgx_enabled;
+
+#endif /* _ASM_X86_SGX_H */
diff --git a/arch/x86/include/asm/sgx_pr.h b/arch/x86/include/asm/sgx_pr.h
new file mode 100644
index 000000000000..876dc44c2ccc
--- /dev/null
+++ b/arch/x86/include/asm/sgx_pr.h
@@ -0,0 +1,20 @@
+// SPDX-License-Identifier: (GPL-2.0 OR BSD-3-Clause)
+// Copyright(c) 2016-17 Intel Corporation.
+//
+// Authors:
+//
+// Jarkko Sakkinen <jarkko.sakkinen@linux.intel.com>
+// Suresh Siddha <suresh.b.siddha@intel.com>
+// Serge Ayoun <serge.ayoun@intel.com>
+// Shay Katz-zamir <shay.katz-zamir@intel.com>
+
+#ifndef _ASM_X86_SGX_PR_H
+#define _ASM_X86_SGX_PR_H
+
+#include <linux/printk.h>
+#include <linux/ratelimit.h>
+
+#undef pr_fmt
+#define pr_fmt(fmt) "intel_sgx: " fmt
+
+#endif /* _ASM_X86_SGX_PR_H */
diff --git a/arch/x86/kernel/cpu/Makefile b/arch/x86/kernel/cpu/Makefile
index a66229f51b12..9552ff5b4ec3 100644
--- a/arch/x86/kernel/cpu/Makefile
+++ b/arch/x86/kernel/cpu/Makefile
@@ -36,6 +36,7 @@ obj-$(CONFIG_CPU_SUP_TRANSMETA_32)	+= transmeta.o
 obj-$(CONFIG_CPU_SUP_UMC_32)		+= umc.o
 
 obj-$(CONFIG_INTEL_RDT)	+= intel_rdt.o intel_rdt_rdtgroup.o intel_rdt_monitor.o intel_rdt_ctrlmondata.o
+obj-$(CONFIG_INTEL_SGX_CORE)		+= intel_sgx.o
 
 obj-$(CONFIG_X86_MCE)			+= mcheck/
 obj-$(CONFIG_MTRR)			+= mtrr/
diff --git a/arch/x86/kernel/cpu/intel_sgx.c b/arch/x86/kernel/cpu/intel_sgx.c
new file mode 100644
index 000000000000..db6b315334f4
--- /dev/null
+++ b/arch/x86/kernel/cpu/intel_sgx.c
@@ -0,0 +1,53 @@
+// SPDX-License-Identifier: (GPL-2.0 OR BSD-3-Clause)
+// Copyright(c) 2016-17 Intel Corporation.
+//
+// Authors:
+//
+// Jarkko Sakkinen <jarkko.sakkinen@linux.intel.com>
+// Suresh Siddha <suresh.b.siddha@intel.com>
+// Serge Ayoun <serge.ayoun@intel.com>
+// Shay Katz-zamir <shay.katz-zamir@intel.com>
+// Sean Christopherson <sean.j.christopherson@intel.com>
+
+#include <asm/sgx.h>
+#include <asm/sgx_pr.h>
+#include <linux/freezer.h>
+#include <linux/highmem.h>
+#include <linux/kthread.h>
+#include <linux/ratelimit.h>
+#include <linux/sched/signal.h>
+#include <linux/slab.h>
+
+bool sgx_enabled __ro_after_init = false;
+EXPORT_SYMBOL(sgx_enabled);
+
+static __init bool sgx_is_enabled(void)
+{
+	unsigned long fc;
+
+	if (boot_cpu_data.x86_vendor != X86_VENDOR_INTEL)
+		return false;
+
+	if (!boot_cpu_has(X86_FEATURE_SGX))
+		return false;
+
+	if (!boot_cpu_has(X86_FEATURE_SGX1))
+		return false;
+
+	rdmsrl(MSR_IA32_FEATURE_CONTROL, fc);
+	if (!(fc & FEATURE_CONTROL_LOCKED))
+		return false;
+
+	if (!(fc & FEATURE_CONTROL_SGX_ENABLE))
+		return false;
+
+	return true;
+}
+
+static __init int sgx_init(void)
+{
+	sgx_enabled = sgx_is_enabled();
+	return 0;
+}
+
+arch_initcall(sgx_init);
-- 
2.17.0

[PATCH v11 08/13] x86, sgx: added ENCLS wrappers

[Jarkko Sakkinen]

This commit adds wrappers for Intel(R) SGX ENCLS opcode functionality.

Signed-off-by: Jarkko Sakkinen <jarkko.sakkinen@linux.intel.com>
---
 arch/x86/include/asm/sgx.h | 198 +++++++++++++++++++++++++++++++++++++
 1 file changed, 198 insertions(+)

diff --git a/arch/x86/include/asm/sgx.h b/arch/x86/include/asm/sgx.h
index fa3e6e0eb8af..a2f727f85b91 100644
--- a/arch/x86/include/asm/sgx.h
+++ b/arch/x86/include/asm/sgx.h
@@ -10,6 +10,10 @@
 #ifndef _ASM_X86_SGX_H
 #define _ASM_X86_SGX_H
 
+#include <asm/sgx_arch.h>
+#include <asm/asm.h>
+#include <linux/bitops.h>
+#include <linux/err.h>
 #include <linux/types.h>
 
 #define SGX_CPUID 0x12
@@ -20,6 +24,200 @@ enum sgx_cpuid {
 	SGX_CPUID_EPC_BANKS	= 2,
 };
 
+enum sgx_commands {
+	ECREATE	= 0x0,
+	EADD	= 0x1,
+	EINIT	= 0x2,
+	EREMOVE	= 0x3,
+	EDGBRD	= 0x4,
+	EDGBWR	= 0x5,
+	EEXTEND	= 0x6,
+	ELDU	= 0x8,
+	EBLOCK	= 0x9,
+	EPA	= 0xA,
+	EWB	= 0xB,
+	ETRACK	= 0xC,
+	EAUG	= 0xD,
+	EMODPR	= 0xE,
+	EMODT	= 0xF,
+};
+
+#define IS_ENCLS_FAULT(r) ((r) & 0xffff0000)
+#define ENCLS_FAULT_VECTOR(r) ((r) >> 16)
+
+#define ENCLS_TO_ERR(r) (IS_ENCLS_FAULT(r) ? -EFAULT :		\
+			(r) == SGX_UNMASKED_EVENT ? -EINTR :	\
+			(r) == SGX_MAC_COMPARE_FAIL ? -EIO :	\
+			(r) == SGX_ENTRYEPOCH_LOCKED ? -EBUSY : -EPERM)
+
+#define __encls_ret_N(rax, inputs...)			\
+	({						\
+	int ret;					\
+	asm volatile(					\
+	"1: .byte 0x0f, 0x01, 0xcf;\n\t"		\
+	"2:\n"						\
+	".section .fixup,\"ax\"\n"			\
+	"3: shll $16,%%eax\n"				\
+	"   jmp 2b\n"					\
+	".previous\n"					\
+	_ASM_EXTABLE_FAULT(1b, 3b)			\
+	: "=a"(ret)					\
+	: "a"(rax), inputs				\
+	: "memory");					\
+	ret;						\
+	})
+
+#define __encls_ret_1(rax, rcx)				\
+	({						\
+	__encls_ret_N(rax, "c"(rcx));			\
+	})
+
+#define __encls_ret_2(rax, rbx, rcx)			\
+	({						\
+	__encls_ret_N(rax, "b"(rbx), "c"(rcx));		\
+	})
+
+#define __encls_ret_3(rax, rbx, rcx, rdx)			\
+	({							\
+	__encls_ret_N(rax, "b"(rbx), "c"(rcx), "d"(rdx));	\
+	})
+
+#define __encls_N(rax, rbx_out, inputs...)		\
+	({						\
+	int ret;					\
+	asm volatile(					\
+	"1: .byte 0x0f, 0x01, 0xcf;\n\t"		\
+	"   xor %%eax,%%eax;\n"				\
+	"2:\n"						\
+	".section .fixup,\"ax\"\n"			\
+	"3: shll $16,%%eax\n"				\
+	"   jmp 2b\n"					\
+	".previous\n"					\
+	_ASM_EXTABLE_FAULT(1b, 3b)				\
+	: "=a"(ret), "=b"(rbx_out)			\
+	: "a"(rax), inputs				\
+	: "memory");					\
+	ret;						\
+	})
+
+#define __encls_2(rax, rbx, rcx)				\
+	({							\
+	unsigned long ign_rbx_out;				\
+	__encls_N(rax, ign_rbx_out, "b"(rbx), "c"(rcx));	\
+	})
+
+#define __encls_1_1(rax, data, rcx)			\
+	({						\
+	unsigned long rbx_out;				\
+	int ret = __encls_N(rax, rbx_out, "c"(rcx));	\
+	if (!ret)					\
+		data = rbx_out;				\
+	ret;						\
+	})
+
+static inline int __ecreate(struct sgx_pageinfo *pginfo, void *secs)
+{
+	return __encls_2(ECREATE, pginfo, secs);
+}
+
+static inline int __eextend(void *secs, void *epc)
+{
+	return __encls_2(EEXTEND, secs, epc);
+}
+
+static inline int __eadd(struct sgx_pageinfo *pginfo, void *epc)
+{
+	return __encls_2(EADD, pginfo, epc);
+}
+
+static inline int __einit(void *sigstruct, struct sgx_einittoken *einittoken,
+			  void *secs)
+{
+	return __encls_ret_3(EINIT, sigstruct, secs, einittoken);
+}
+
+static inline int __eremove(void *epc)
+{
+	return __encls_ret_1(EREMOVE, epc);
+}
+
+static inline int __edbgwr(unsigned long addr, unsigned long *data)
+{
+	return __encls_2(EDGBWR, *data, addr);
+}
+
+static inline int __edbgrd(unsigned long addr, unsigned long *data)
+{
+	return __encls_1_1(EDGBRD, *data, addr);
+}
+
+static inline int __etrack(void *epc)
+{
+	return __encls_ret_1(ETRACK, epc);
+}
+
+static inline int __eldu(struct sgx_pageinfo *pginfo, void *epc, void *va)
+{
+	return __encls_ret_3(ELDU, pginfo, epc, va);
+}
+
+static inline int __eblock(void *epc)
+{
+	return __encls_ret_1(EBLOCK, epc);
+}
+
+static inline int __epa(void *epc)
+{
+	unsigned long rbx = SGX_PAGE_TYPE_VA;
+
+	return __encls_2(EPA, rbx, epc);
+}
+
+static inline int __ewb(struct sgx_pageinfo *pginfo, void *epc, void *va)
+{
+	return __encls_ret_3(EWB, pginfo, epc, va);
+}
+
+static inline int __eaug(struct sgx_pageinfo *pginfo, void *epc)
+{
+	return __encls_2(EAUG, pginfo, epc);
+}
+
+static inline int __emodpr(struct sgx_secinfo *secinfo, void *epc)
+{
+	return __encls_ret_2(EMODPR, secinfo, epc);
+}
+
+static inline int __emodt(struct sgx_secinfo *secinfo, void *epc)
+{
+	return __encls_ret_2(EMODT, secinfo, epc);
+}
+
 extern bool sgx_enabled;
 
+#define SGX_FN(name, params...)		\
+{					\
+	void *epc;			\
+	int ret;			\
+	epc = sgx_get_page(epc_page);	\
+	ret = __##name(params);		\
+	sgx_put_page(epc);		\
+	return ret;			\
+}
+
+#define BUILD_SGX_FN(fn, name)				\
+static inline int fn(struct sgx_epc_page *epc_page)	\
+	SGX_FN(name, epc)
+BUILD_SGX_FN(sgx_eremove, eremove)
+BUILD_SGX_FN(sgx_eblock, eblock)
+BUILD_SGX_FN(sgx_etrack, etrack)
+BUILD_SGX_FN(sgx_epa, epa)
+
+static inline int sgx_emodpr(struct sgx_secinfo *secinfo,
+			     struct sgx_epc_page *epc_page)
+	SGX_FN(emodpr, secinfo, epc)
+static inline int sgx_emodt(struct sgx_secinfo *secinfo,
+			    struct sgx_epc_page *epc_page)
+	SGX_FN(emodt, secinfo, epc)
+
 #endif /* _ASM_X86_SGX_H */
-- 
2.17.0

[PATCH v11 08/13] x86, sgx: added ENCLS wrappers

[Jarkko Sakkinen]

This commit adds wrappers for Intel(R) SGX ENCLS opcode functionality.

Signed-off-by: Jarkko Sakkinen <jarkko.sakkinen@linux.intel.com>
---
 arch/x86/include/asm/sgx.h | 198 +++++++++++++++++++++++++++++++++++++
 1 file changed, 198 insertions(+)

diff --git a/arch/x86/include/asm/sgx.h b/arch/x86/include/asm/sgx.h
index fa3e6e0eb8af..a2f727f85b91 100644
--- a/arch/x86/include/asm/sgx.h
+++ b/arch/x86/include/asm/sgx.h
@@ -10,6 +10,10 @@
 #ifndef _ASM_X86_SGX_H
 #define _ASM_X86_SGX_H
 
+#include <asm/sgx_arch.h>
+#include <asm/asm.h>
+#include <linux/bitops.h>
+#include <linux/err.h>
 #include <linux/types.h>
 
 #define SGX_CPUID 0x12
@@ -20,6 +24,200 @@ enum sgx_cpuid {
 	SGX_CPUID_EPC_BANKS	= 2,
 };
 
+enum sgx_commands {
+	ECREATE	= 0x0,
+	EADD	= 0x1,
+	EINIT	= 0x2,
+	EREMOVE	= 0x3,
+	EDGBRD	= 0x4,
+	EDGBWR	= 0x5,
+	EEXTEND	= 0x6,
+	ELDU	= 0x8,
+	EBLOCK	= 0x9,
+	EPA	= 0xA,
+	EWB	= 0xB,
+	ETRACK	= 0xC,
+	EAUG	= 0xD,
+	EMODPR	= 0xE,
+	EMODT	= 0xF,
+};
+
+#define IS_ENCLS_FAULT(r) ((r) & 0xffff0000)
+#define ENCLS_FAULT_VECTOR(r) ((r) >> 16)
+
+#define ENCLS_TO_ERR(r) (IS_ENCLS_FAULT(r) ? -EFAULT :		\
+			(r) == SGX_UNMASKED_EVENT ? -EINTR :	\
+			(r) == SGX_MAC_COMPARE_FAIL ? -EIO :	\
+			(r) == SGX_ENTRYEPOCH_LOCKED ? -EBUSY : -EPERM)
+
+#define __encls_ret_N(rax, inputs...)			\
+	({						\
+	int ret;					\
+	asm volatile(					\
+	"1: .byte 0x0f, 0x01, 0xcf;\n\t"		\
+	"2:\n"						\
+	".section .fixup,\"ax\"\n"			\
+	"3: shll $16,%%eax\n"				\
+	"   jmp 2b\n"					\
+	".previous\n"					\
+	_ASM_EXTABLE_FAULT(1b, 3b)			\
+	: "=a"(ret)					\
+	: "a"(rax), inputs				\
+	: "memory");					\
+	ret;						\
+	})
+
+#define __encls_ret_1(rax, rcx)				\
+	({						\
+	__encls_ret_N(rax, "c"(rcx));			\
+	})
+
+#define __encls_ret_2(rax, rbx, rcx)			\
+	({						\
+	__encls_ret_N(rax, "b"(rbx), "c"(rcx));		\
+	})
+
+#define __encls_ret_3(rax, rbx, rcx, rdx)			\
+	({							\
+	__encls_ret_N(rax, "b"(rbx), "c"(rcx), "d"(rdx));	\
+	})
+
+#define __encls_N(rax, rbx_out, inputs...)		\
+	({						\
+	int ret;					\
+	asm volatile(					\
+	"1: .byte 0x0f, 0x01, 0xcf;\n\t"		\
+	"   xor %%eax,%%eax;\n"				\
+	"2:\n"						\
+	".section .fixup,\"ax\"\n"			\
+	"3: shll $16,%%eax\n"				\
+	"   jmp 2b\n"					\
+	".previous\n"					\
+	_ASM_EXTABLE_FAULT(1b, 3b)				\
+	: "=a"(ret), "=b"(rbx_out)			\
+	: "a"(rax), inputs				\
+	: "memory");					\
+	ret;						\
+	})
+
+#define __encls_2(rax, rbx, rcx)				\
+	({							\
+	unsigned long ign_rbx_out;				\
+	__encls_N(rax, ign_rbx_out, "b"(rbx), "c"(rcx));	\
+	})
+
+#define __encls_1_1(rax, data, rcx)			\
+	({						\
+	unsigned long rbx_out;				\
+	int ret = __encls_N(rax, rbx_out, "c"(rcx));	\
+	if (!ret)					\
+		data = rbx_out;				\
+	ret;						\
+	})
+
+static inline int __ecreate(struct sgx_pageinfo *pginfo, void *secs)
+{
+	return __encls_2(ECREATE, pginfo, secs);
+}
+
+static inline int __eextend(void *secs, void *epc)
+{
+	return __encls_2(EEXTEND, secs, epc);
+}
+
+static inline int __eadd(struct sgx_pageinfo *pginfo, void *epc)
+{
+	return __encls_2(EADD, pginfo, epc);
+}
+
+static inline int __einit(void *sigstruct, struct sgx_einittoken *einittoken,
+			  void *secs)
+{
+	return __encls_ret_3(EINIT, sigstruct, secs, einittoken);
+}
+
+static inline int __eremove(void *epc)
+{
+	return __encls_ret_1(EREMOVE, epc);
+}
+
+static inline int __edbgwr(unsigned long addr, unsigned long *data)
+{
+	return __encls_2(EDGBWR, *data, addr);
+}
+
+static inline int __edbgrd(unsigned long addr, unsigned long *data)
+{
+	return __encls_1_1(EDGBRD, *data, addr);
+}
+
+static inline int __etrack(void *epc)
+{
+	return __encls_ret_1(ETRACK, epc);
+}
+
+static inline int __eldu(struct sgx_pageinfo *pginfo, void *epc, void *va)
+{
+	return __encls_ret_3(ELDU, pginfo, epc, va);
+}
+
+static inline int __eblock(void *epc)
+{
+	return __encls_ret_1(EBLOCK, epc);
+}
+
+static inline int __epa(void *epc)
+{
+	unsigned long rbx = SGX_PAGE_TYPE_VA;
+
+	return __encls_2(EPA, rbx, epc);
+}
+
+static inline int __ewb(struct sgx_pageinfo *pginfo, void *epc, void *va)
+{
+	return __encls_ret_3(EWB, pginfo, epc, va);
+}
+
+static inline int __eaug(struct sgx_pageinfo *pginfo, void *epc)
+{
+	return __encls_2(EAUG, pginfo, epc);
+}
+
+static inline int __emodpr(struct sgx_secinfo *secinfo, void *epc)
+{
+	return __encls_ret_2(EMODPR, secinfo, epc);
+}
+
+static inline int __emodt(struct sgx_secinfo *secinfo, void *epc)
+{
+	return __encls_ret_2(EMODT, secinfo, epc);
+}
+
 extern bool sgx_enabled;
 
+#define SGX_FN(name, params...)		\
+{					\
+	void *epc;			\
+	int ret;			\
+	epc = sgx_get_page(epc_page);	\
+	ret = __##name(params);		\
+	sgx_put_page(epc);		\
+	return ret;			\
+}
+
+#define BUILD_SGX_FN(fn, name)				\
+static inline int fn(struct sgx_epc_page *epc_page)	\
+	SGX_FN(name, epc)
+BUILD_SGX_FN(sgx_eremove, eremove)
+BUILD_SGX_FN(sgx_eblock, eblock)
+BUILD_SGX_FN(sgx_etrack, etrack)
+BUILD_SGX_FN(sgx_epa, epa)
+
+static inline int sgx_emodpr(struct sgx_secinfo *secinfo,
+			     struct sgx_epc_page *epc_page)
+	SGX_FN(emodpr, secinfo, epc)
+static inline int sgx_emodt(struct sgx_secinfo *secinfo,
+			    struct sgx_epc_page *epc_page)
+	SGX_FN(emodt, secinfo, epc)
+
 #endif /* _ASM_X86_SGX_H */
-- 
2.17.0

[PATCH v11 09/13] x86, sgx: basic routines for enclave page cache

[Jarkko Sakkinen]

SGX has a set of data structures to maintain information about the enclaves
and their security properties. BIOS reserves a fixed size region of
physical memory for these structures by setting Processor Reserved Memory
Range Registers (PRMRR). This memory area is called Enclave Page Cache
(EPC).

This commit implements the basic routines to allocate and free pages from
different EPC banks. There is also a swapper thread ksgxswapd for EPC pages
that gets woken up by sgx_alloc_page() when we run below the low watermark.
The swapper thread continues swapping pages up until it reaches the high
watermark.

Each subsystem that uses SGX must provide a set of callbacks for EPC
pages that are used to reclaim, block and write an EPC page. Kernel
takes the responsibility of maintaining LRU cache for them.

Signed-off-by: Jarkko Sakkinen <jarkko.sakkinen@linux.intel.com>
---
 arch/x86/include/asm/sgx.h      |  67 +++++
 arch/x86/include/asm/sgx_arch.h | 224 ++++++++++++++++
 arch/x86/kernel/cpu/intel_sgx.c | 443 +++++++++++++++++++++++++++++++-
 3 files changed, 732 insertions(+), 2 deletions(-)
 create mode 100644 arch/x86/include/asm/sgx_arch.h

diff --git a/arch/x86/include/asm/sgx.h b/arch/x86/include/asm/sgx.h
index a2f727f85b91..ae738e16ba6c 100644
--- a/arch/x86/include/asm/sgx.h
+++ b/arch/x86/include/asm/sgx.h
@@ -14,6 +14,7 @@
 #include <asm/asm.h>
 #include <linux/bitops.h>
 #include <linux/err.h>
+#include <linux/rwsem.h>
 #include <linux/types.h>
 
 #define SGX_CPUID 0x12
@@ -193,7 +194,73 @@ static inline int __emodt(struct sgx_secinfo *secinfo, void *epc)
 	return __encls_ret_2(EMODT, secinfo, epc);
 }
 
+#define SGX_MAX_EPC_BANKS 8
+
+#define SGX_EPC_BANK(epc_page) \
+	(&sgx_epc_banks[(unsigned long)(epc_page->desc) & ~PAGE_MASK])
+#define SGX_EPC_PFN(epc_page) PFN_DOWN((unsigned long)(epc_page->desc))
+#define SGX_EPC_ADDR(epc_page) ((unsigned long)(epc_page->desc) & PAGE_MASK)
+
+struct sgx_epc_page;
+
+struct sgx_epc_page_ops {
+	bool (*get)(struct sgx_epc_page *epc_page);
+	void (*put)(struct sgx_epc_page *epc_page);
+	bool (*reclaim)(struct sgx_epc_page *epc_page);
+	void (*block)(struct sgx_epc_page *epc_page);
+	void (*write)(struct sgx_epc_page *epc_page);
+};
+
+struct sgx_epc_page_impl {
+	const struct sgx_epc_page_ops *ops;
+};
+
+struct sgx_epc_page {
+	unsigned long desc;
+	struct sgx_epc_page_impl *impl;
+	struct list_head list;
+};
+
+struct sgx_epc_bank {
+	unsigned long pa;
+	unsigned long va;
+	unsigned long size;
+	struct sgx_epc_page *pages_data;
+	struct sgx_epc_page **pages;
+	atomic_t free_cnt;
+	struct rw_semaphore lock;
+};
+
 extern bool sgx_enabled;
+extern bool sgx_lc_enabled;
+extern atomic_t sgx_nr_free_pages;
+extern struct sgx_epc_bank sgx_epc_banks[];
+extern int sgx_nr_epc_banks;
+extern struct list_head sgx_active_page_list;
+extern struct spinlock sgx_active_page_list_lock;
+
+enum sgx_alloc_flags {
+	SGX_ALLOC_ATOMIC	= BIT(0),
+};
+
+struct sgx_epc_page *sgx_try_alloc_page(struct sgx_epc_page_impl *impl);
+struct sgx_epc_page *sgx_alloc_page(struct sgx_epc_page_impl *impl,
+				    unsigned int flags);
+int sgx_free_page(struct sgx_epc_page *page);
+void *sgx_get_page(struct sgx_epc_page *ptr);
+void sgx_put_page(void *epc_page_ptr);
+struct page *sgx_get_backing(struct file *file, pgoff_t index);
+void sgx_put_backing(struct page *backing_page, bool write);
+int sgx_einit(struct sgx_sigstruct *sigstruct, struct sgx_einittoken *token,
+	      struct sgx_epc_page *secs_page, u64 le_pubkey_hash[4]);
+
+struct sgx_launch_request {
+	u8 mrenclave[32];
+	u8 mrsigner[32];
+	uint64_t attributes;
+	uint64_t xfrm;
+	struct sgx_einittoken token;
+};
 
 #define SGX_FN(name, params...)		\
 {					\
diff --git a/arch/x86/include/asm/sgx_arch.h b/arch/x86/include/asm/sgx_arch.h
new file mode 100644
index 000000000000..eb64bd92f84e
--- /dev/null
+++ b/arch/x86/include/asm/sgx_arch.h
@@ -0,0 +1,224 @@
+// SPDX-License-Identifier: (GPL-2.0 OR BSD-3-Clause)
+// Copyright(c) 2016-17 Intel Corporation.
+//
+// Authors:
+//
+// Jarkko Sakkinen <jarkko.sakkinen@linux.intel.com>
+// Suresh Siddha <suresh.b.siddha@intel.com>
+
+#ifndef _ASM_X86_SGX_ARCH_H
+#define _ASM_X86_SGX_ARCH_H
+
+#include <linux/types.h>
+
+#define SGX_SSA_GPRS_SIZE		182
+#define SGX_SSA_MISC_EXINFO_SIZE	16
+
+enum sgx_misc {
+	SGX_MISC_EXINFO		= 0x01,
+};
+
+#define SGX_MISC_RESERVED_MASK 0xFFFFFFFFFFFFFFFEL
+
+enum sgx_attribute {
+	SGX_ATTR_DEBUG		= 0x02,
+	SGX_ATTR_MODE64BIT	= 0x04,
+	SGX_ATTR_PROVISIONKEY	= 0x10,
+	SGX_ATTR_EINITTOKENKEY	= 0x20,
+};
+
+#define SGX_ATTR_RESERVED_MASK 0xFFFFFFFFFFFFFFC9L
+
+#define SGX_SECS_RESERVED1_SIZE 24
+#define SGX_SECS_RESERVED2_SIZE 32
+#define SGX_SECS_RESERVED3_SIZE 96
+#define SGX_SECS_RESERVED4_SIZE 3836
+
+struct sgx_secs {
+	uint64_t size;
+	uint64_t base;
+	uint32_t ssaframesize;
+	uint32_t miscselect;
+	uint8_t reserved1[SGX_SECS_RESERVED1_SIZE];
+	uint64_t attributes;
+	uint64_t xfrm;
+	uint32_t mrenclave[8];
+	uint8_t reserved2[SGX_SECS_RESERVED2_SIZE];
+	uint32_t mrsigner[8];
+	uint8_t	reserved3[SGX_SECS_RESERVED3_SIZE];
+	uint16_t isvvprodid;
+	uint16_t isvsvn;
+	uint8_t reserved4[SGX_SECS_RESERVED4_SIZE];
+};
+
+enum sgx_tcs_flags {
+	SGX_TCS_DBGOPTIN	= 0x01, /* cleared on EADD */
+};
+
+#define SGX_TCS_RESERVED_MASK 0xFFFFFFFFFFFFFFFEL
+
+struct sgx_tcs {
+	uint64_t state;
+	uint64_t flags;
+	uint64_t ossa;
+	uint32_t cssa;
+	uint32_t nssa;
+	uint64_t oentry;
+	uint64_t aep;
+	uint64_t ofsbase;
+	uint64_t ogsbase;
+	uint32_t fslimit;
+	uint32_t gslimit;
+	uint64_t reserved[503];
+};
+
+struct sgx_pageinfo {
+	uint64_t linaddr;
+	uint64_t srcpge;
+	union {
+		uint64_t secinfo;
+		uint64_t pcmd;
+	};
+	uint64_t secs;
+} __attribute__((aligned(32)));
+
+
+#define SGX_SECINFO_PERMISSION_MASK	0x0000000000000007L
+#define SGX_SECINFO_PAGE_TYPE_MASK	0x000000000000FF00L
+#define SGX_SECINFO_RESERVED_MASK	0xFFFFFFFFFFFF00F8L
+
+enum sgx_page_type {
+	SGX_PAGE_TYPE_SECS	= 0x00,
+	SGX_PAGE_TYPE_TCS	= 0x01,
+	SGX_PAGE_TYPE_REG	= 0x02,
+	SGX_PAGE_TYPE_VA	= 0x03,
+	SGX_PAGE_TYPE_TRIM	= 0x04,
+};
+
+enum sgx_secinfo_flags {
+	SGX_SECINFO_R		= 0x01,
+	SGX_SECINFO_W		= 0x02,
+	SGX_SECINFO_X		= 0x04,
+	SGX_SECINFO_SECS	= (SGX_PAGE_TYPE_SECS << 8),
+	SGX_SECINFO_TCS		= (SGX_PAGE_TYPE_TCS << 8),
+	SGX_SECINFO_REG		= (SGX_PAGE_TYPE_REG << 8),
+	SGX_SECINFO_TRIM	= (SGX_PAGE_TYPE_TRIM << 8),
+};
+
+struct sgx_secinfo {
+	uint64_t flags;
+	uint64_t reserved[7];
+} __attribute__((aligned(64)));
+
+struct sgx_pcmd {
+	struct sgx_secinfo secinfo;
+	uint64_t enclave_id;
+	uint8_t reserved[40];
+	uint8_t mac[16];
+};
+
+#define SGX_MODULUS_SIZE 384
+
+struct sgx_sigstruct_header {
+	uint64_t header1[2];
+	uint32_t vendor;
+	uint32_t date;
+	uint64_t header2[2];
+	uint32_t swdefined;
+	uint8_t reserved1[84];
+};
+
+struct sgx_sigstruct_body {
+	uint32_t miscselect;
+	uint32_t miscmask;
+	uint8_t reserved2[20];
+	uint64_t attributes;
+	uint64_t xfrm;
+	uint8_t attributemask[16];
+	uint8_t mrenclave[32];
+	uint8_t reserved3[32];
+	uint16_t isvprodid;
+	uint16_t isvsvn;
+} __attribute__((__packed__));
+
+struct sgx_sigstruct {
+	struct sgx_sigstruct_header header;
+	uint8_t modulus[SGX_MODULUS_SIZE];
+	uint32_t exponent;
+	uint8_t signature[SGX_MODULUS_SIZE];
+	struct sgx_sigstruct_body body;
+	uint8_t reserved4[12];
+	uint8_t q1[SGX_MODULUS_SIZE];
+	uint8_t q2[SGX_MODULUS_SIZE];
+};
+
+struct sgx_sigstruct_payload {
+	struct sgx_sigstruct_header header;
+	struct sgx_sigstruct_body body;
+};
+
+struct sgx_einittoken_payload {
+	uint32_t valid;
+	uint32_t reserved1[11];
+	uint64_t attributes;
+	uint64_t xfrm;
+	uint8_t mrenclave[32];
+	uint8_t reserved2[32];
+	uint8_t mrsigner[32];
+	uint8_t reserved3[32];
+};
+
+struct sgx_einittoken {
+	struct sgx_einittoken_payload payload;
+	uint8_t cpusvnle[16];
+	uint16_t isvprodidle;
+	uint16_t isvsvnle;
+	uint8_t reserved2[24];
+	uint32_t maskedmiscselectle;
+	uint64_t maskedattributesle;
+	uint64_t maskedxfrmle;
+	uint8_t keyid[32];
+	uint8_t mac[16];
+};
+
+struct sgx_report {
+	uint8_t cpusvn[16];
+	uint32_t miscselect;
+	uint8_t reserved1[28];
+	uint64_t attributes;
+	uint64_t xfrm;
+	uint8_t mrenclave[32];
+	uint8_t reserved2[32];
+	uint8_t mrsigner[32];
+	uint8_t reserved3[96];
+	uint16_t isvprodid;
+	uint16_t isvsvn;
+	uint8_t reserved4[60];
+	uint8_t reportdata[64];
+	uint8_t keyid[32];
+	uint8_t mac[16];
+};
+
+struct sgx_targetinfo {
+	uint8_t mrenclave[32];
+	uint64_t attributes;
+	uint64_t xfrm;
+	uint8_t reserved1[4];
+	uint32_t miscselect;
+	uint8_t reserved2[456];
+};
+
+struct sgx_keyrequest {
+	uint16_t keyname;
+	uint16_t keypolicy;
+	uint16_t isvsvn;
+	uint16_t reserved1;
+	uint8_t cpusvn[16];
+	uint64_t attributemask;
+	uint64_t xfrmmask;
+	uint8_t keyid[32];
+	uint32_t miscmask;
+	uint8_t reserved2[436];
+};
+
+#endif /* _ASM_X86_SGX_ARCH_H */
diff --git a/arch/x86/kernel/cpu/intel_sgx.c b/arch/x86/kernel/cpu/intel_sgx.c
index db6b315334f4..ae2b5c5b455f 100644
--- a/arch/x86/kernel/cpu/intel_sgx.c
+++ b/arch/x86/kernel/cpu/intel_sgx.c
@@ -14,14 +14,439 @@
 #include <linux/freezer.h>
 #include <linux/highmem.h>
 #include <linux/kthread.h>
+#include <linux/pagemap.h>
 #include <linux/ratelimit.h>
 #include <linux/sched/signal.h>
+#include <linux/shmem_fs.h>
 #include <linux/slab.h>
 
+#define SGX_NR_TO_SCAN	16
+#define SGX_NR_LOW_PAGES 32
+#define SGX_NR_HIGH_PAGES 64
+
 bool sgx_enabled __ro_after_init = false;
 EXPORT_SYMBOL(sgx_enabled);
+bool sgx_lc_enabled __ro_after_init;
+EXPORT_SYMBOL(sgx_lc_enabled);
+atomic_t sgx_nr_free_pages = ATOMIC_INIT(0);
+EXPORT_SYMBOL(sgx_nr_free_pages);
+struct sgx_epc_bank sgx_epc_banks[SGX_MAX_EPC_BANKS];
+EXPORT_SYMBOL(sgx_epc_banks);
+int sgx_nr_epc_banks;
+EXPORT_SYMBOL(sgx_nr_epc_banks);
+LIST_HEAD(sgx_active_page_list);
+EXPORT_SYMBOL(sgx_active_page_list);
+DEFINE_SPINLOCK(sgx_active_page_list_lock);
+EXPORT_SYMBOL(sgx_active_page_list_lock);
+
+static struct task_struct *ksgxswapd_tsk;
+static DECLARE_WAIT_QUEUE_HEAD(ksgxswapd_waitq);
+
+/*
+ * Writing the LE hash MSRs is extraordinarily expensive, e.g.
+ * 3-4x slower than normal MSRs, so we use a per-cpu cache to
+ * track the last known value of the MSRs to avoid unnecessarily
+ * writing the MSRs with the current value.  Because most Linux
+ * kernels will use an LE that is signed with a non-Intel key,
+ * i.e. the first EINIT will need to write the MSRs regardless
+ * of the cache, the cache is intentionally left uninitialized
+ * during boot as initializing the cache would be pure overhead
+ * for the majority of systems.  Furthermore, the MSRs are per-cpu
+ * and the boot-time values aren't guaranteed to be identical
+ * across cpus, so we'd have to run code all all cpus to properly
+ * init the cache.  All in all, the complexity and overhead of
+ * initializing the cache is not justified.
+ */
+static DEFINE_PER_CPU(u64 [4], sgx_le_pubkey_hash_cache);
+
+static void sgx_swap_cluster(void)
+{
+	struct sgx_epc_page *cluster[SGX_NR_TO_SCAN + 1];
+	struct sgx_epc_page *epc_page;
+	int i;
+	int j;
+
+	memset(cluster, 0, sizeof(cluster));
+
+	for (i = 0, j = 0; i < SGX_NR_TO_SCAN; i++) {
+		spin_lock(&sgx_active_page_list_lock);
+		if (list_empty(&sgx_active_page_list)) {
+			spin_unlock(&sgx_active_page_list_lock);
+			break;
+		}
+		epc_page = list_first_entry(&sgx_active_page_list,
+					    struct sgx_epc_page, list);
+		if (!epc_page->impl->ops->get(epc_page)) {
+			list_move_tail(&epc_page->list, &sgx_active_page_list);
+			spin_unlock(&sgx_active_page_list_lock);
+			continue;
+		}
+		list_del(&epc_page->list);
+		spin_unlock(&sgx_active_page_list_lock);
 
-static __init bool sgx_is_enabled(void)
+		if (epc_page->impl->ops->reclaim(epc_page)) {
+			cluster[j++] = epc_page;
+		} else {
+			spin_lock(&sgx_active_page_list_lock);
+			list_add_tail(&epc_page->list, &sgx_active_page_list);
+			spin_unlock(&sgx_active_page_list_lock);
+			epc_page->impl->ops->put(epc_page);
+		}
+	}
+
+	for (i = 0; cluster[i]; i++) {
+		epc_page = cluster[i];
+		epc_page->impl->ops->block(epc_page);
+	}
+
+	for (i = 0; cluster[i]; i++) {
+		epc_page = cluster[i];
+		epc_page->impl->ops->write(epc_page);
+		epc_page->impl->ops->put(epc_page);
+		sgx_free_page(epc_page);
+	}
+}
+
+static int ksgxswapd(void *p)
+{
+	set_freezable();
+
+	while (!kthread_should_stop()) {
+		if (try_to_freeze())
+			continue;
+
+		wait_event_freezable(ksgxswapd_waitq, kthread_should_stop() ||
+				     atomic_read(&sgx_nr_free_pages) <
+				     SGX_NR_HIGH_PAGES);
+
+		if (atomic_read(&sgx_nr_free_pages) < SGX_NR_HIGH_PAGES)
+			sgx_swap_cluster();
+	}
+
+	pr_info("%s: done\n", __func__);
+	return 0;
+}
+
+/**
+ * sgx_try_alloc_page - try to allocate an EPC page
+ * @impl:	implementation for the struct sgx_epc_page
+ *
+ * Try to grab a page from the free EPC page list. If there is a free page
+ * available, it is returned to the caller.
+ *
+ * Return:
+ *   a &struct sgx_epc_page instace,
+ *   NULL otherwise
+ */
+struct sgx_epc_page *sgx_try_alloc_page(struct sgx_epc_page_impl *impl)
+{
+	struct sgx_epc_bank *bank;
+	struct sgx_epc_page *page = NULL;
+	int i;
+
+	for (i = 0; i < sgx_nr_epc_banks; i++) {
+		bank = &sgx_epc_banks[i];
+
+		down_write(&bank->lock);
+
+		if (atomic_read(&bank->free_cnt))
+			page = bank->pages[atomic_dec_return(&bank->free_cnt)];
+
+		up_write(&bank->lock);
+
+		if (page)
+			break;
+	}
+
+	if (page) {
+		atomic_dec(&sgx_nr_free_pages);
+		page->impl = impl;
+	}
+
+	return page;
+}
+EXPORT_SYMBOL(sgx_try_alloc_page);
+
+/**
+ * sgx_alloc_page - allocate an EPC page
+ * @flags:	allocation flags
+ * @impl:	implementation for the struct sgx_epc_page
+ *
+ * Try to grab a page from the free EPC page list. If there is a free page
+ * available, it is returned to the caller. If called with SGX_ALLOC_ATOMIC,
+ * the function will return immediately if the list is empty. Otherwise, it
+ * will swap pages up until there is a free page available. Upon returning the
+ * low watermark is checked and ksgxswapd is waken up if we are below it.
+ *
+ * Return:
+ *   a &struct sgx_epc_page instace,
+ *   -ENOMEM if all pages are unreclaimable,
+ *   -EBUSY when called with SGX_ALLOC_ATOMIC and out of free pages
+ */
+struct sgx_epc_page *sgx_alloc_page(struct sgx_epc_page_impl *impl,
+				    unsigned int flags)
+{
+	struct sgx_epc_page *entry;
+
+	for ( ; ; ) {
+		entry = sgx_try_alloc_page(impl);
+		if (entry)
+			break;
+
+		if (list_empty(&sgx_active_page_list))
+			return ERR_PTR(-ENOMEM);
+
+		if (flags & SGX_ALLOC_ATOMIC) {
+			entry = ERR_PTR(-EBUSY);
+			break;
+		}
+
+		if (signal_pending(current)) {
+			entry = ERR_PTR(-ERESTARTSYS);
+			break;
+		}
+
+		sgx_swap_cluster();
+		schedule();
+	}
+
+	if (atomic_read(&sgx_nr_free_pages) < SGX_NR_LOW_PAGES)
+		wake_up(&ksgxswapd_waitq);
+
+	return entry;
+}
+EXPORT_SYMBOL(sgx_alloc_page);
+
+/**
+ * sgx_free_page - free an EPC page
+ *
+ * @page:	any EPC page
+ *
+ * Remove an EPC page and insert it back to the list of free pages.
+ *
+ * Return: SGX error code
+ */
+int sgx_free_page(struct sgx_epc_page *page)
+{
+	struct sgx_epc_bank *bank = SGX_EPC_BANK(page);
+	int ret;
+
+	ret = sgx_eremove(page);
+	if (ret) {
+		pr_debug("EREMOVE returned %d\n", ret);
+		return ret;
+	}
+
+	down_read(&bank->lock);
+	bank->pages[atomic_inc_return(&bank->free_cnt) - 1] = page;
+	atomic_inc(&sgx_nr_free_pages);
+	up_read(&bank->lock);
+
+	return 0;
+}
+EXPORT_SYMBOL(sgx_free_page);
+
+/**
+ * sgx_get_page - pin an EPC page
+ * @page:	an EPC page
+ *
+ * Return: a pointer to the pinned EPC page
+ */
+void *sgx_get_page(struct sgx_epc_page *page)
+{
+	struct sgx_epc_bank *bank = SGX_EPC_BANK(page);
+
+	if (IS_ENABLED(CONFIG_X86_64))
+		return (void *)(bank->va + SGX_EPC_ADDR(page) - bank->pa);
+
+	return kmap_atomic_pfn(SGX_EPC_PFN(page));
+}
+EXPORT_SYMBOL(sgx_get_page);
+
+/**
+ * sgx_put_page - unpin an EPC page
+ * @ptr:	a pointer to the pinned EPC page
+ */
+void sgx_put_page(void *ptr)
+{
+	if (IS_ENABLED(CONFIG_X86_64))
+		return;
+
+	kunmap_atomic(ptr);
+}
+EXPORT_SYMBOL(sgx_put_page);
+
+struct page *sgx_get_backing(struct file *file, pgoff_t index)
+{
+	struct inode *inode = file->f_path.dentry->d_inode;
+	struct address_space *mapping = inode->i_mapping;
+	gfp_t gfpmask = mapping_gfp_mask(mapping);
+
+	return shmem_read_mapping_page_gfp(mapping, index, gfpmask);
+}
+EXPORT_SYMBOL(sgx_get_backing);
+
+void sgx_put_backing(struct page *backing_page, bool write)
+{
+	if (write)
+		set_page_dirty(backing_page);
+
+	put_page(backing_page);
+}
+EXPORT_SYMBOL(sgx_put_backing);
+
+/**
+ * sgx_einit - EINIT an enclave with the appropriate LE pubkey hash
+ * @sigstruct:		a pointer to the enclave's sigstruct
+ * @token:		a pointer to the enclave's EINIT token
+ * @secs_page:		a pointer to the enclave's SECS EPC page
+ * @le_pubkey_hash:	the desired LE pubkey hash for EINIT
+ */
+int sgx_einit(struct sgx_sigstruct *sigstruct, struct sgx_einittoken *token,
+	      struct sgx_epc_page *secs_page, u64 le_pubkey_hash[4])
+{
+	u64 __percpu *cache;
+	void *secs;
+	int i, ret;
+
+	secs = sgx_get_page(secs_page);
+
+	if (!sgx_lc_enabled) {
+		ret = __einit(sigstruct, token, secs);
+		goto out;
+	}
+
+	cache = per_cpu(sgx_le_pubkey_hash_cache, smp_processor_id());
+
+	preempt_disable();
+	for (i = 0; i < 4; i++) {
+		if (le_pubkey_hash[i] == cache[i])
+			continue;
+
+		wrmsrl(MSR_IA32_SGXLEPUBKEYHASH0 + i, le_pubkey_hash[i]);
+		cache[i] = le_pubkey_hash[i];
+	}
+	ret = __einit(sigstruct, token, secs);
+	preempt_enable();
+
+out:
+	sgx_put_page(secs);
+	return ret;
+}
+EXPORT_SYMBOL(sgx_einit);
+
+static __init int sgx_init_epc_bank(unsigned long addr, unsigned long size,
+				    unsigned long index,
+				    struct sgx_epc_bank *bank)
+{
+	unsigned long nr_pages = size >> PAGE_SHIFT;
+	unsigned long i;
+	void *va;
+
+	if (IS_ENABLED(CONFIG_X86_64)) {
+		va = ioremap_cache(addr, size);
+		if (!va)
+			return -ENOMEM;
+	}
+
+	bank->pages_data = kzalloc(nr_pages * sizeof(struct sgx_epc_page),
+				   GFP_KERNEL);
+	if (!bank->pages_data) {
+		if (IS_ENABLED(CONFIG_X86_64))
+			iounmap(va);
+
+		return -ENOMEM;
+	}
+
+	bank->pages = kzalloc(nr_pages * sizeof(struct sgx_epc_page *),
+			      GFP_KERNEL);
+	if (!bank->pages) {
+		if (IS_ENABLED(CONFIG_X86_64))
+			iounmap(va);
+		kfree(bank->pages_data);
+		bank->pages_data = NULL;
+		return -ENOMEM;
+	}
+
+	for (i = 0; i < nr_pages; i++) {
+		bank->pages[i] = &bank->pages_data[i];
+		bank->pages[i]->desc = (addr + (i << PAGE_SHIFT)) | index;
+	}
+
+	bank->pa = addr;
+	bank->size = size;
+	if (IS_ENABLED(CONFIG_X86_64))
+		bank->va = (unsigned long)va;
+
+	atomic_set(&bank->free_cnt, nr_pages);
+	init_rwsem(&bank->lock);
+	atomic_add(nr_pages, &sgx_nr_free_pages);
+	return 0;
+}
+
+static __init void sgx_page_cache_teardown(void)
+{
+	struct sgx_epc_bank *bank;
+	int i;
+
+	for (i = 0; i < sgx_nr_epc_banks; i++) {
+		bank = &sgx_epc_banks[i];
+
+		if (IS_ENABLED(CONFIG_X86_64))
+			iounmap((void *)bank->va);
+
+		kfree(bank->pages);
+		kfree(bank->pages_data);
+	}
+
+	if (ksgxswapd_tsk) {
+		kthread_stop(ksgxswapd_tsk);
+		ksgxswapd_tsk = NULL;
+	}
+}
+
+static __init int sgx_page_cache_init(void)
+{
+	struct task_struct *tsk;
+	unsigned long size;
+	unsigned int eax;
+	unsigned int ebx;
+	unsigned int ecx;
+	unsigned int edx;
+	unsigned long pa;
+	int i;
+	int ret;
+
+	for (i = 0; i < SGX_MAX_EPC_BANKS; i++) {
+		cpuid_count(SGX_CPUID, i + SGX_CPUID_EPC_BANKS, &eax, &ebx,
+			    &ecx, &edx);
+		if (!(eax & 0xf))
+			break;
+
+		pa = ((u64)(ebx & 0xfffff) << 32) + (u64)(eax & 0xfffff000);
+		size = ((u64)(edx & 0xfffff) << 32) + (u64)(ecx & 0xfffff000);
+
+		pr_info("EPC bank 0x%lx-0x%lx\n", pa, pa + size);
+
+		ret = sgx_init_epc_bank(pa, size, i, &sgx_epc_banks[i]);
+		if (ret) {
+			sgx_page_cache_teardown();
+			return ret;
+		}
+
+		sgx_nr_epc_banks++;
+	}
+
+	tsk = kthread_run(ksgxswapd, NULL, "ksgxswapd");
+	if (IS_ERR(tsk)) {
+		sgx_page_cache_teardown();
+		return PTR_ERR(tsk);
+	}
+	ksgxswapd_tsk = tsk;
+	return 0;
+}
+
+static __init bool sgx_is_enabled(bool *lc_enabled)
 {
 	unsigned long fc;
 
@@ -41,12 +466,26 @@ static __init bool sgx_is_enabled(void)
 	if (!(fc & FEATURE_CONTROL_SGX_ENABLE))
 		return false;
 
+	*lc_enabled = !!(fc & FEATURE_CONTROL_SGX_LE_WR);
+
 	return true;
 }
 
 static __init int sgx_init(void)
 {
-	sgx_enabled = sgx_is_enabled();
+	bool lc_enabled;
+	int ret;
+
+	if (!sgx_is_enabled(&lc_enabled))
+		return 0;
+
+	ret = sgx_page_cache_init();
+	if (ret)
+		return ret;
+
+	sgx_enabled = true;
+	sgx_lc_enabled = lc_enabled;
+
 	return 0;
 }
 
-- 
2.17.0

[PATCH v11 09/13] x86, sgx: basic routines for enclave page cache

[Jarkko Sakkinen]

SGX has a set of data structures to maintain information about the enclaves
and their security properties. BIOS reserves a fixed size region of
physical memory for these structures by setting Processor Reserved Memory
Range Registers (PRMRR). This memory area is called Enclave Page Cache
(EPC).

This commit implements the basic routines to allocate and free pages from
different EPC banks. There is also a swapper thread ksgxswapd for EPC pages
that gets woken up by sgx_alloc_page() when we run below the low watermark.
The swapper thread continues swapping pages up until it reaches the high
watermark.

Each subsystem that uses SGX must provide a set of callbacks for EPC
pages that are used to reclaim, block and write an EPC page. Kernel
takes the responsibility of maintaining LRU cache for them.

Signed-off-by: Jarkko Sakkinen <jarkko.sakkinen@linux.intel.com>
---
 arch/x86/include/asm/sgx.h      |  67 +++++
 arch/x86/include/asm/sgx_arch.h | 224 ++++++++++++++++
 arch/x86/kernel/cpu/intel_sgx.c | 443 +++++++++++++++++++++++++++++++-
 3 files changed, 732 insertions(+), 2 deletions(-)
 create mode 100644 arch/x86/include/asm/sgx_arch.h

diff --git a/arch/x86/include/asm/sgx.h b/arch/x86/include/asm/sgx.h
index a2f727f85b91..ae738e16ba6c 100644
--- a/arch/x86/include/asm/sgx.h
+++ b/arch/x86/include/asm/sgx.h
@@ -14,6 +14,7 @@
 #include <asm/asm.h>
 #include <linux/bitops.h>
 #include <linux/err.h>
+#include <linux/rwsem.h>
 #include <linux/types.h>
 
 #define SGX_CPUID 0x12
@@ -193,7 +194,73 @@ static inline int __emodt(struct sgx_secinfo *secinfo, void *epc)
 	return __encls_ret_2(EMODT, secinfo, epc);
 }
 
+#define SGX_MAX_EPC_BANKS 8
+
+#define SGX_EPC_BANK(epc_page) \
+	(&sgx_epc_banks[(unsigned long)(epc_page->desc) & ~PAGE_MASK])
+#define SGX_EPC_PFN(epc_page) PFN_DOWN((unsigned long)(epc_page->desc))
+#define SGX_EPC_ADDR(epc_page) ((unsigned long)(epc_page->desc) & PAGE_MASK)
+
+struct sgx_epc_page;
+
+struct sgx_epc_page_ops {
+	bool (*get)(struct sgx_epc_page *epc_page);
+	void (*put)(struct sgx_epc_page *epc_page);
+	bool (*reclaim)(struct sgx_epc_page *epc_page);
+	void (*block)(struct sgx_epc_page *epc_page);
+	void (*write)(struct sgx_epc_page *epc_page);
+};
+
+struct sgx_epc_page_impl {
+	const struct sgx_epc_page_ops *ops;
+};
+
+struct sgx_epc_page {
+	unsigned long desc;
+	struct sgx_epc_page_impl *impl;
+	struct list_head list;
+};
+
+struct sgx_epc_bank {
+	unsigned long pa;
+	unsigned long va;
+	unsigned long size;
+	struct sgx_epc_page *pages_data;
+	struct sgx_epc_page **pages;
+	atomic_t free_cnt;
+	struct rw_semaphore lock;
+};
+
 extern bool sgx_enabled;
+extern bool sgx_lc_enabled;
+extern atomic_t sgx_nr_free_pages;
+extern struct sgx_epc_bank sgx_epc_banks[];
+extern int sgx_nr_epc_banks;
+extern struct list_head sgx_active_page_list;
+extern struct spinlock sgx_active_page_list_lock;
+
+enum sgx_alloc_flags {
+	SGX_ALLOC_ATOMIC	= BIT(0),
+};
+
+struct sgx_epc_page *sgx_try_alloc_page(struct sgx_epc_page_impl *impl);
+struct sgx_epc_page *sgx_alloc_page(struct sgx_epc_page_impl *impl,
+				    unsigned int flags);
+int sgx_free_page(struct sgx_epc_page *page);
+void *sgx_get_page(struct sgx_epc_page *ptr);
+void sgx_put_page(void *epc_page_ptr);
+struct page *sgx_get_backing(struct file *file, pgoff_t index);
+void sgx_put_backing(struct page *backing_page, bool write);
+int sgx_einit(struct sgx_sigstruct *sigstruct, struct sgx_einittoken *token,
+	      struct sgx_epc_page *secs_page, u64 le_pubkey_hash[4]);
+
+struct sgx_launch_request {
+	u8 mrenclave[32];
+	u8 mrsigner[32];
+	uint64_t attributes;
+	uint64_t xfrm;
+	struct sgx_einittoken token;
+};
 
 #define SGX_FN(name, params...)		\
 {					\
diff --git a/arch/x86/include/asm/sgx_arch.h b/arch/x86/include/asm/sgx_arch.h
new file mode 100644
index 000000000000..eb64bd92f84e
--- /dev/null
+++ b/arch/x86/include/asm/sgx_arch.h
@@ -0,0 +1,224 @@
+// SPDX-License-Identifier: (GPL-2.0 OR BSD-3-Clause)
+// Copyright(c) 2016-17 Intel Corporation.
+//
+// Authors:
+//
+// Jarkko Sakkinen <jarkko.sakkinen@linux.intel.com>
+// Suresh Siddha <suresh.b.siddha@intel.com>
+
+#ifndef _ASM_X86_SGX_ARCH_H
+#define _ASM_X86_SGX_ARCH_H
+
+#include <linux/types.h>
+
+#define SGX_SSA_GPRS_SIZE		182
+#define SGX_SSA_MISC_EXINFO_SIZE	16
+
+enum sgx_misc {
+	SGX_MISC_EXINFO		= 0x01,
+};
+
+#define SGX_MISC_RESERVED_MASK 0xFFFFFFFFFFFFFFFEL
+
+enum sgx_attribute {
+	SGX_ATTR_DEBUG		= 0x02,
+	SGX_ATTR_MODE64BIT	= 0x04,
+	SGX_ATTR_PROVISIONKEY	= 0x10,
+	SGX_ATTR_EINITTOKENKEY	= 0x20,
+};
+
+#define SGX_ATTR_RESERVED_MASK 0xFFFFFFFFFFFFFFC9L
+
+#define SGX_SECS_RESERVED1_SIZE 24
+#define SGX_SECS_RESERVED2_SIZE 32
+#define SGX_SECS_RESERVED3_SIZE 96
+#define SGX_SECS_RESERVED4_SIZE 3836
+
+struct sgx_secs {
+	uint64_t size;
+	uint64_t base;
+	uint32_t ssaframesize;
+	uint32_t miscselect;
+	uint8_t reserved1[SGX_SECS_RESERVED1_SIZE];
+	uint64_t attributes;
+	uint64_t xfrm;
+	uint32_t mrenclave[8];
+	uint8_t reserved2[SGX_SECS_RESERVED2_SIZE];
+	uint32_t mrsigner[8];
+	uint8_t	reserved3[SGX_SECS_RESERVED3_SIZE];
+	uint16_t isvvprodid;
+	uint16_t isvsvn;
+	uint8_t reserved4[SGX_SECS_RESERVED4_SIZE];
+};
+
+enum sgx_tcs_flags {
+	SGX_TCS_DBGOPTIN	= 0x01, /* cleared on EADD */
+};
+
+#define SGX_TCS_RESERVED_MASK 0xFFFFFFFFFFFFFFFEL
+
+struct sgx_tcs {
+	uint64_t state;
+	uint64_t flags;
+	uint64_t ossa;
+	uint32_t cssa;
+	uint32_t nssa;
+	uint64_t oentry;
+	uint64_t aep;
+	uint64_t ofsbase;
+	uint64_t ogsbase;
+	uint32_t fslimit;
+	uint32_t gslimit;
+	uint64_t reserved[503];
+};
+
+struct sgx_pageinfo {
+	uint64_t linaddr;
+	uint64_t srcpge;
+	union {
+		uint64_t secinfo;
+		uint64_t pcmd;
+	};
+	uint64_t secs;
+} __attribute__((aligned(32)));
+
+
+#define SGX_SECINFO_PERMISSION_MASK	0x0000000000000007L
+#define SGX_SECINFO_PAGE_TYPE_MASK	0x000000000000FF00L
+#define SGX_SECINFO_RESERVED_MASK	0xFFFFFFFFFFFF00F8L
+
+enum sgx_page_type {
+	SGX_PAGE_TYPE_SECS	= 0x00,
+	SGX_PAGE_TYPE_TCS	= 0x01,
+	SGX_PAGE_TYPE_REG	= 0x02,
+	SGX_PAGE_TYPE_VA	= 0x03,
+	SGX_PAGE_TYPE_TRIM	= 0x04,
+};
+
+enum sgx_secinfo_flags {
+	SGX_SECINFO_R		= 0x01,
+	SGX_SECINFO_W		= 0x02,
+	SGX_SECINFO_X		= 0x04,
+	SGX_SECINFO_SECS	= (SGX_PAGE_TYPE_SECS << 8),
+	SGX_SECINFO_TCS		= (SGX_PAGE_TYPE_TCS << 8),
+	SGX_SECINFO_REG		= (SGX_PAGE_TYPE_REG << 8),
+	SGX_SECINFO_TRIM	= (SGX_PAGE_TYPE_TRIM << 8),
+};
+
+struct sgx_secinfo {
+	uint64_t flags;
+	uint64_t reserved[7];
+} __attribute__((aligned(64)));
+
+struct sgx_pcmd {
+	struct sgx_secinfo secinfo;
+	uint64_t enclave_id;
+	uint8_t reserved[40];
+	uint8_t mac[16];
+};
+
+#define SGX_MODULUS_SIZE 384
+
+struct sgx_sigstruct_header {
+	uint64_t header1[2];
+	uint32_t vendor;
+	uint32_t date;
+	uint64_t header2[2];
+	uint32_t swdefined;
+	uint8_t reserved1[84];
+};
+
+struct sgx_sigstruct_body {
+	uint32_t miscselect;
+	uint32_t miscmask;
+	uint8_t reserved2[20];
+	uint64_t attributes;
+	uint64_t xfrm;
+	uint8_t attributemask[16];
+	uint8_t mrenclave[32];
+	uint8_t reserved3[32];
+	uint16_t isvprodid;
+	uint16_t isvsvn;
+} __attribute__((__packed__));
+
+struct sgx_sigstruct {
+	struct sgx_sigstruct_header header;
+	uint8_t modulus[SGX_MODULUS_SIZE];
+	uint32_t exponent;
+	uint8_t signature[SGX_MODULUS_SIZE];
+	struct sgx_sigstruct_body body;
+	uint8_t reserved4[12];
+	uint8_t q1[SGX_MODULUS_SIZE];
+	uint8_t q2[SGX_MODULUS_SIZE];
+};
+
+struct sgx_sigstruct_payload {
+	struct sgx_sigstruct_header header;
+	struct sgx_sigstruct_body body;
+};
+
+struct sgx_einittoken_payload {
+	uint32_t valid;
+	uint32_t reserved1[11];
+	uint64_t attributes;
+	uint64_t xfrm;
+	uint8_t mrenclave[32];
+	uint8_t reserved2[32];
+	uint8_t mrsigner[32];
+	uint8_t reserved3[32];
+};
+
+struct sgx_einittoken {
+	struct sgx_einittoken_payload payload;
+	uint8_t cpusvnle[16];
+	uint16_t isvprodidle;
+	uint16_t isvsvnle;
+	uint8_t reserved2[24];
+	uint32_t maskedmiscselectle;
+	uint64_t maskedattributesle;
+	uint64_t maskedxfrmle;
+	uint8_t keyid[32];
+	uint8_t mac[16];
+};
+
+struct sgx_report {
+	uint8_t cpusvn[16];
+	uint32_t miscselect;
+	uint8_t reserved1[28];
+	uint64_t attributes;
+	uint64_t xfrm;
+	uint8_t mrenclave[32];
+	uint8_t reserved2[32];
+	uint8_t mrsigner[32];
+	uint8_t reserved3[96];
+	uint16_t isvprodid;
+	uint16_t isvsvn;
+	uint8_t reserved4[60];
+	uint8_t reportdata[64];
+	uint8_t keyid[32];
+	uint8_t mac[16];
+};
+
+struct sgx_targetinfo {
+	uint8_t mrenclave[32];
+	uint64_t attributes;
+	uint64_t xfrm;
+	uint8_t reserved1[4];
+	uint32_t miscselect;
+	uint8_t reserved2[456];
+};
+
+struct sgx_keyrequest {
+	uint16_t keyname;
+	uint16_t keypolicy;
+	uint16_t isvsvn;
+	uint16_t reserved1;
+	uint8_t cpusvn[16];
+	uint64_t attributemask;
+	uint64_t xfrmmask;
+	uint8_t keyid[32];
+	uint32_t miscmask;
+	uint8_t reserved2[436];
+};
+
+#endif /* _ASM_X86_SGX_ARCH_H */
diff --git a/arch/x86/kernel/cpu/intel_sgx.c b/arch/x86/kernel/cpu/intel_sgx.c
index db6b315334f4..ae2b5c5b455f 100644
--- a/arch/x86/kernel/cpu/intel_sgx.c
+++ b/arch/x86/kernel/cpu/intel_sgx.c
@@ -14,14 +14,439 @@
 #include <linux/freezer.h>
 #include <linux/highmem.h>
 #include <linux/kthread.h>
+#include <linux/pagemap.h>
 #include <linux/ratelimit.h>
 #include <linux/sched/signal.h>
+#include <linux/shmem_fs.h>
 #include <linux/slab.h>
 
+#define SGX_NR_TO_SCAN	16
+#define SGX_NR_LOW_PAGES 32
+#define SGX_NR_HIGH_PAGES 64
+
 bool sgx_enabled __ro_after_init = false;
 EXPORT_SYMBOL(sgx_enabled);
+bool sgx_lc_enabled __ro_after_init;
+EXPORT_SYMBOL(sgx_lc_enabled);
+atomic_t sgx_nr_free_pages = ATOMIC_INIT(0);
+EXPORT_SYMBOL(sgx_nr_free_pages);
+struct sgx_epc_bank sgx_epc_banks[SGX_MAX_EPC_BANKS];
+EXPORT_SYMBOL(sgx_epc_banks);
+int sgx_nr_epc_banks;
+EXPORT_SYMBOL(sgx_nr_epc_banks);
+LIST_HEAD(sgx_active_page_list);
+EXPORT_SYMBOL(sgx_active_page_list);
+DEFINE_SPINLOCK(sgx_active_page_list_lock);
+EXPORT_SYMBOL(sgx_active_page_list_lock);
+
+static struct task_struct *ksgxswapd_tsk;
+static DECLARE_WAIT_QUEUE_HEAD(ksgxswapd_waitq);
+
+/*
+ * Writing the LE hash MSRs is extraordinarily expensive, e.g.
+ * 3-4x slower than normal MSRs, so we use a per-cpu cache to
+ * track the last known value of the MSRs to avoid unnecessarily
+ * writing the MSRs with the current value.  Because most Linux
+ * kernels will use an LE that is signed with a non-Intel key,
+ * i.e. the first EINIT will need to write the MSRs regardless
+ * of the cache, the cache is intentionally left uninitialized
+ * during boot as initializing the cache would be pure overhead
+ * for the majority of systems.  Furthermore, the MSRs are per-cpu
+ * and the boot-time values aren't guaranteed to be identical
+ * across cpus, so we'd have to run code all all cpus to properly
+ * init the cache.  All in all, the complexity and overhead of
+ * initializing the cache is not justified.
+ */
+static DEFINE_PER_CPU(u64 [4], sgx_le_pubkey_hash_cache);
+
+static void sgx_swap_cluster(void)
+{
+	struct sgx_epc_page *cluster[SGX_NR_TO_SCAN + 1];
+	struct sgx_epc_page *epc_page;
+	int i;
+	int j;
+
+	memset(cluster, 0, sizeof(cluster));
+
+	for (i = 0, j = 0; i < SGX_NR_TO_SCAN; i++) {
+		spin_lock(&sgx_active_page_list_lock);
+		if (list_empty(&sgx_active_page_list)) {
+			spin_unlock(&sgx_active_page_list_lock);
+			break;
+		}
+		epc_page = list_first_entry(&sgx_active_page_list,
+					    struct sgx_epc_page, list);
+		if (!epc_page->impl->ops->get(epc_page)) {
+			list_move_tail(&epc_page->list, &sgx_active_page_list);
+			spin_unlock(&sgx_active_page_list_lock);
+			continue;
+		}
+		list_del(&epc_page->list);
+		spin_unlock(&sgx_active_page_list_lock);
 
-static __init bool sgx_is_enabled(void)
+		if (epc_page->impl->ops->reclaim(epc_page)) {
+			cluster[j++] = epc_page;
+		} else {
+			spin_lock(&sgx_active_page_list_lock);
+			list_add_tail(&epc_page->list, &sgx_active_page_list);
+			spin_unlock(&sgx_active_page_list_lock);
+			epc_page->impl->ops->put(epc_page);
+		}
+	}
+
+	for (i = 0; cluster[i]; i++) {
+		epc_page = cluster[i];
+		epc_page->impl->ops->block(epc_page);
+	}
+
+	for (i = 0; cluster[i]; i++) {
+		epc_page = cluster[i];
+		epc_page->impl->ops->write(epc_page);
+		epc_page->impl->ops->put(epc_page);
+		sgx_free_page(epc_page);
+	}
+}
+
+static int ksgxswapd(void *p)
+{
+	set_freezable();
+
+	while (!kthread_should_stop()) {
+		if (try_to_freeze())
+			continue;
+
+		wait_event_freezable(ksgxswapd_waitq, kthread_should_stop() ||
+				     atomic_read(&sgx_nr_free_pages) <
+				     SGX_NR_HIGH_PAGES);
+
+		if (atomic_read(&sgx_nr_free_pages) < SGX_NR_HIGH_PAGES)
+			sgx_swap_cluster();
+	}
+
+	pr_info("%s: done\n", __func__);
+	return 0;
+}
+
+/**
+ * sgx_try_alloc_page - try to allocate an EPC page
+ * @impl:	implementation for the struct sgx_epc_page
+ *
+ * Try to grab a page from the free EPC page list. If there is a free page
+ * available, it is returned to the caller.
+ *
+ * Return:
+ *   a &struct sgx_epc_page instace,
+ *   NULL otherwise
+ */
+struct sgx_epc_page *sgx_try_alloc_page(struct sgx_epc_page_impl *impl)
+{
+	struct sgx_epc_bank *bank;
+	struct sgx_epc_page *page = NULL;
+	int i;
+
+	for (i = 0; i < sgx_nr_epc_banks; i++) {
+		bank = &sgx_epc_banks[i];
+
+		down_write(&bank->lock);
+
+		if (atomic_read(&bank->free_cnt))
+			page = bank->pages[atomic_dec_return(&bank->free_cnt)];
+
+		up_write(&bank->lock);
+
+		if (page)
+			break;
+	}
+
+	if (page) {
+		atomic_dec(&sgx_nr_free_pages);
+		page->impl = impl;
+	}
+
+	return page;
+}
+EXPORT_SYMBOL(sgx_try_alloc_page);
+
+/**
+ * sgx_alloc_page - allocate an EPC page
+ * @flags:	allocation flags
+ * @impl:	implementation for the struct sgx_epc_page
+ *
+ * Try to grab a page from the free EPC page list. If there is a free page
+ * available, it is returned to the caller. If called with SGX_ALLOC_ATOMIC,
+ * the function will return immediately if the list is empty. Otherwise, it
+ * will swap pages up until there is a free page available. Upon returning the
+ * low watermark is checked and ksgxswapd is waken up if we are below it.
+ *
+ * Return:
+ *   a &struct sgx_epc_page instace,
+ *   -ENOMEM if all pages are unreclaimable,
+ *   -EBUSY when called with SGX_ALLOC_ATOMIC and out of free pages
+ */
+struct sgx_epc_page *sgx_alloc_page(struct sgx_epc_page_impl *impl,
+				    unsigned int flags)
+{
+	struct sgx_epc_page *entry;
+
+	for ( ; ; ) {
+		entry = sgx_try_alloc_page(impl);
+		if (entry)
+			break;
+
+		if (list_empty(&sgx_active_page_list))
+			return ERR_PTR(-ENOMEM);
+
+		if (flags & SGX_ALLOC_ATOMIC) {
+			entry = ERR_PTR(-EBUSY);
+			break;
+		}
+
+		if (signal_pending(current)) {
+			entry = ERR_PTR(-ERESTARTSYS);
+			break;
+		}
+
+		sgx_swap_cluster();
+		schedule();
+	}
+
+	if (atomic_read(&sgx_nr_free_pages) < SGX_NR_LOW_PAGES)
+		wake_up(&ksgxswapd_waitq);
+
+	return entry;
+}
+EXPORT_SYMBOL(sgx_alloc_page);
+
+/**
+ * sgx_free_page - free an EPC page
+ *
+ * @page:	any EPC page
+ *
+ * Remove an EPC page and insert it back to the list of free pages.
+ *
+ * Return: SGX error code
+ */
+int sgx_free_page(struct sgx_epc_page *page)
+{
+	struct sgx_epc_bank *bank = SGX_EPC_BANK(page);
+	int ret;
+
+	ret = sgx_eremove(page);
+	if (ret) {
+		pr_debug("EREMOVE returned %d\n", ret);
+		return ret;
+	}
+
+	down_read(&bank->lock);
+	bank->pages[atomic_inc_return(&bank->free_cnt) - 1] = page;
+	atomic_inc(&sgx_nr_free_pages);
+	up_read(&bank->lock);
+
+	return 0;
+}
+EXPORT_SYMBOL(sgx_free_page);
+
+/**
+ * sgx_get_page - pin an EPC page
+ * @page:	an EPC page
+ *
+ * Return: a pointer to the pinned EPC page
+ */
+void *sgx_get_page(struct sgx_epc_page *page)
+{
+	struct sgx_epc_bank *bank = SGX_EPC_BANK(page);
+
+	if (IS_ENABLED(CONFIG_X86_64))
+		return (void *)(bank->va + SGX_EPC_ADDR(page) - bank->pa);
+
+	return kmap_atomic_pfn(SGX_EPC_PFN(page));
+}
+EXPORT_SYMBOL(sgx_get_page);
+
+/**
+ * sgx_put_page - unpin an EPC page
+ * @ptr:	a pointer to the pinned EPC page
+ */
+void sgx_put_page(void *ptr)
+{
+	if (IS_ENABLED(CONFIG_X86_64))
+		return;
+
+	kunmap_atomic(ptr);
+}
+EXPORT_SYMBOL(sgx_put_page);
+
+struct page *sgx_get_backing(struct file *file, pgoff_t index)
+{
+	struct inode *inode = file->f_path.dentry->d_inode;
+	struct address_space *mapping = inode->i_mapping;
+	gfp_t gfpmask = mapping_gfp_mask(mapping);
+
+	return shmem_read_mapping_page_gfp(mapping, index, gfpmask);
+}
+EXPORT_SYMBOL(sgx_get_backing);
+
+void sgx_put_backing(struct page *backing_page, bool write)
+{
+	if (write)
+		set_page_dirty(backing_page);
+
+	put_page(backing_page);
+}
+EXPORT_SYMBOL(sgx_put_backing);
+
+/**
+ * sgx_einit - EINIT an enclave with the appropriate LE pubkey hash
+ * @sigstruct:		a pointer to the enclave's sigstruct
+ * @token:		a pointer to the enclave's EINIT token
+ * @secs_page:		a pointer to the enclave's SECS EPC page
+ * @le_pubkey_hash:	the desired LE pubkey hash for EINIT
+ */
+int sgx_einit(struct sgx_sigstruct *sigstruct, struct sgx_einittoken *token,
+	      struct sgx_epc_page *secs_page, u64 le_pubkey_hash[4])
+{
+	u64 __percpu *cache;
+	void *secs;
+	int i, ret;
+
+	secs = sgx_get_page(secs_page);
+
+	if (!sgx_lc_enabled) {
+		ret = __einit(sigstruct, token, secs);
+		goto out;
+	}
+
+	cache = per_cpu(sgx_le_pubkey_hash_cache, smp_processor_id());
+
+	preempt_disable();
+	for (i = 0; i < 4; i++) {
+		if (le_pubkey_hash[i] == cache[i])
+			continue;
+
+		wrmsrl(MSR_IA32_SGXLEPUBKEYHASH0 + i, le_pubkey_hash[i]);
+		cache[i] = le_pubkey_hash[i];
+	}
+	ret = __einit(sigstruct, token, secs);
+	preempt_enable();
+
+out:
+	sgx_put_page(secs);
+	return ret;
+}
+EXPORT_SYMBOL(sgx_einit);
+
+static __init int sgx_init_epc_bank(unsigned long addr, unsigned long size,
+				    unsigned long index,
+				    struct sgx_epc_bank *bank)
+{
+	unsigned long nr_pages = size >> PAGE_SHIFT;
+	unsigned long i;
+	void *va;
+
+	if (IS_ENABLED(CONFIG_X86_64)) {
+		va = ioremap_cache(addr, size);
+		if (!va)
+			return -ENOMEM;
+	}
+
+	bank->pages_data = kzalloc(nr_pages * sizeof(struct sgx_epc_page),
+				   GFP_KERNEL);
+	if (!bank->pages_data) {
+		if (IS_ENABLED(CONFIG_X86_64))
+			iounmap(va);
+
+		return -ENOMEM;
+	}
+
+	bank->pages = kzalloc(nr_pages * sizeof(struct sgx_epc_page *),
+			      GFP_KERNEL);
+	if (!bank->pages) {
+		if (IS_ENABLED(CONFIG_X86_64))
+			iounmap(va);
+		kfree(bank->pages_data);
+		bank->pages_data = NULL;
+		return -ENOMEM;
+	}
+
+	for (i = 0; i < nr_pages; i++) {
+		bank->pages[i] = &bank->pages_data[i];
+		bank->pages[i]->desc = (addr + (i << PAGE_SHIFT)) | index;
+	}
+
+	bank->pa = addr;
+	bank->size = size;
+	if (IS_ENABLED(CONFIG_X86_64))
+		bank->va = (unsigned long)va;
+
+	atomic_set(&bank->free_cnt, nr_pages);
+	init_rwsem(&bank->lock);
+	atomic_add(nr_pages, &sgx_nr_free_pages);
+	return 0;
+}
+
+static __init void sgx_page_cache_teardown(void)
+{
+	struct sgx_epc_bank *bank;
+	int i;
+
+	for (i = 0; i < sgx_nr_epc_banks; i++) {
+		bank = &sgx_epc_banks[i];
+
+		if (IS_ENABLED(CONFIG_X86_64))
+			iounmap((void *)bank->va);
+
+		kfree(bank->pages);
+		kfree(bank->pages_data);
+	}
+
+	if (ksgxswapd_tsk) {
+		kthread_stop(ksgxswapd_tsk);
+		ksgxswapd_tsk = NULL;
+	}
+}
+
+static __init int sgx_page_cache_init(void)
+{
+	struct task_struct *tsk;
+	unsigned long size;
+	unsigned int eax;
+	unsigned int ebx;
+	unsigned int ecx;
+	unsigned int edx;
+	unsigned long pa;
+	int i;
+	int ret;
+
+	for (i = 0; i < SGX_MAX_EPC_BANKS; i++) {
+		cpuid_count(SGX_CPUID, i + SGX_CPUID_EPC_BANKS, &eax, &ebx,
+			    &ecx, &edx);
+		if (!(eax & 0xf))
+			break;
+
+		pa = ((u64)(ebx & 0xfffff) << 32) + (u64)(eax & 0xfffff000);
+		size = ((u64)(edx & 0xfffff) << 32) + (u64)(ecx & 0xfffff000);
+
+		pr_info("EPC bank 0x%lx-0x%lx\n", pa, pa + size);
+
+		ret = sgx_init_epc_bank(pa, size, i, &sgx_epc_banks[i]);
+		if (ret) {
+			sgx_page_cache_teardown();
+			return ret;
+		}
+
+		sgx_nr_epc_banks++;
+	}
+
+	tsk = kthread_run(ksgxswapd, NULL, "ksgxswapd");
+	if (IS_ERR(tsk)) {
+		sgx_page_cache_teardown();
+		return PTR_ERR(tsk);
+	}
+	ksgxswapd_tsk = tsk;
+	return 0;
+}
+
+static __init bool sgx_is_enabled(bool *lc_enabled)
 {
 	unsigned long fc;
 
@@ -41,12 +466,26 @@ static __init bool sgx_is_enabled(void)
 	if (!(fc & FEATURE_CONTROL_SGX_ENABLE))
 		return false;
 
+	*lc_enabled = !!(fc & FEATURE_CONTROL_SGX_LE_WR);
+
 	return true;
 }
 
 static __init int sgx_init(void)
 {
-	sgx_enabled = sgx_is_enabled();
+	bool lc_enabled;
+	int ret;
+
+	if (!sgx_is_enabled(&lc_enabled))
+		return 0;
+
+	ret = sgx_page_cache_init();
+	if (ret)
+		return ret;
+
+	sgx_enabled = true;
+	sgx_lc_enabled = lc_enabled;
+
 	return 0;
 }
 
-- 
2.17.0

[PATCH v11 10/13] intel_sgx: driver for Intel Software Guard Extensions

[Jarkko Sakkinen]

Intel SGX is a set of CPU instructions that can be used by applications to
set aside private regions of code and data.  The code outside the enclave
is disallowed to access the memory inside the enclave by the CPU access
control.

SGX driver provides a ioctl API for loading and initializing enclaves.
Address range for enclaves is reserved with mmap() and they are destroyed
with munmap(). Enclave construction, measurement and initialization is done
with the provided the ioctl API.

Signed-off-by: Jarkko Sakkinen <jarkko.sakkinen@linux.intel.com>
Tested-by: Serge Ayoun <serge.ayoun@intel.com>
---
 arch/x86/include/asm/sgx_pr.h                 |   16 +
 arch/x86/include/uapi/asm/sgx.h               |  138 +++
 drivers/platform/x86/Kconfig                  |    2 +
 drivers/platform/x86/Makefile                 |    1 +
 drivers/platform/x86/intel_sgx/Kconfig        |   20 +
 drivers/platform/x86/intel_sgx/Makefile       |   13 +
 drivers/platform/x86/intel_sgx/sgx.h          |  164 +++
 drivers/platform/x86/intel_sgx/sgx_encl.c     | 1002 +++++++++++++++++
 .../platform/x86/intel_sgx/sgx_encl_page.c    |  294 +++++
 drivers/platform/x86/intel_sgx/sgx_fault.c    |  159 +++
 drivers/platform/x86/intel_sgx/sgx_ioctl.c    |  233 ++++
 drivers/platform/x86/intel_sgx/sgx_main.c     |  306 +++++
 drivers/platform/x86/intel_sgx/sgx_vma.c      |   66 ++
 13 files changed, 2414 insertions(+)
 create mode 100644 arch/x86/include/uapi/asm/sgx.h
 create mode 100644 drivers/platform/x86/intel_sgx/Kconfig
 create mode 100644 drivers/platform/x86/intel_sgx/Makefile
 create mode 100644 drivers/platform/x86/intel_sgx/sgx.h
 create mode 100644 drivers/platform/x86/intel_sgx/sgx_encl.c
 create mode 100644 drivers/platform/x86/intel_sgx/sgx_encl_page.c
 create mode 100644 drivers/platform/x86/intel_sgx/sgx_fault.c
 create mode 100644 drivers/platform/x86/intel_sgx/sgx_ioctl.c
 create mode 100644 drivers/platform/x86/intel_sgx/sgx_main.c
 create mode 100644 drivers/platform/x86/intel_sgx/sgx_vma.c

diff --git a/arch/x86/include/asm/sgx_pr.h b/arch/x86/include/asm/sgx_pr.h
index 876dc44c2ccc..9d7c3cc1ea1b 100644
--- a/arch/x86/include/asm/sgx_pr.h
+++ b/arch/x86/include/asm/sgx_pr.h
@@ -17,4 +17,20 @@
 #undef pr_fmt
 #define pr_fmt(fmt) "intel_sgx: " fmt
 
+#define sgx_pr_ratelimited(level, encl, fmt, ...)			\
+	pr_ ## level ## _ratelimited("[%d:0x%p] " fmt,			\
+				     pid_nr((encl)->tgid),		\
+				     (void *)(encl)->base, ##__VA_ARGS__)
+
+#define sgx_dbg(encl, fmt, ...) \
+	sgx_pr_ratelimited(debug, encl, fmt, ##__VA_ARGS__)
+#define sgx_info(encl, fmt, ...) \
+	sgx_pr_ratelimited(info, encl, fmt, ##__VA_ARGS__)
+#define sgx_warn(encl, fmt, ...) \
+	sgx_pr_ratelimited(warn, encl, fmt, ##__VA_ARGS__)
+#define sgx_err(encl, fmt, ...) \
+	sgx_pr_ratelimited(err, encl, fmt, ##__VA_ARGS__)
+#define sgx_crit(encl, fmt, ...) \
+	sgx_pr_ratelimited(crit, encl, fmt, ##__VA_ARGS__)
+
 #endif /* _ASM_X86_SGX_PR_H */
diff --git a/arch/x86/include/uapi/asm/sgx.h b/arch/x86/include/uapi/asm/sgx.h
new file mode 100644
index 000000000000..9bd8907efdaf
--- /dev/null
+++ b/arch/x86/include/uapi/asm/sgx.h
@@ -0,0 +1,138 @@
+/*
+ * This file is provided under a dual BSD/GPLv2 license.  When using or
+ * redistributing this file, you may do so under either license.
+ *
+ * GPL LICENSE SUMMARY
+ *
+ * Copyright(c) 2016-2017 Intel Corporation.
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of version 2 of the GNU General Public License as
+ * published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful, but
+ * WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+ * General Public License for more details.
+ *
+ * Contact Information:
+ * Jarkko Sakkinen <jarkko.sakkinen@linux.intel.com>
+ * Intel Finland Oy - BIC 0357606-4 - Westendinkatu 7, 02160 Espoo
+ *
+ * BSD LICENSE
+ *
+ * Copyright(c) 2016-2017 Intel Corporation.
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions
+ * are met:
+ *
+ *   * Redistributions of source code must retain the above copyright
+ *     notice, this list of conditions and the following disclaimer.
+ *   * Redistributions in binary form must reproduce the above copyright
+ *     notice, this list of conditions and the following disclaimer in
+ *     the documentation and/or other materials provided with the
+ *     distribution.
+ *   * Neither the name of Intel Corporation nor the names of its
+ *     contributors may be used to endorse or promote products derived
+ *     from this software without specific prior written permission.
+ *
+ * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
+ * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
+ * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
+ * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
+ * OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+ * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+ * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+ * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
+ * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+ * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+ * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+ *
+ * Authors:
+ *
+ * Jarkko Sakkinen <jarkko.sakkinen@linux.intel.com>
+ * Suresh Siddha <suresh.b.siddha@intel.com>
+ */
+
+#ifndef _UAPI_ASM_X86_SGX_H
+#define _UAPI_ASM_X86_SGX_H
+
+#include <linux/types.h>
+#include <linux/ioctl.h>
+
+#define SGX_MAGIC 0xA4
+
+#define SGX_IOC_ENCLAVE_CREATE \
+	_IOW(SGX_MAGIC, 0x00, struct sgx_enclave_create)
+#define SGX_IOC_ENCLAVE_ADD_PAGE \
+	_IOW(SGX_MAGIC, 0x01, struct sgx_enclave_add_page)
+#define SGX_IOC_ENCLAVE_INIT \
+	_IOW(SGX_MAGIC, 0x02, struct sgx_enclave_init)
+
+/* SGX leaf instruction return values */
+#define SGX_SUCCESS			0
+#define SGX_INVALID_SIG_STRUCT		1
+#define SGX_INVALID_ATTRIBUTE		2
+#define SGX_BLKSTATE			3
+#define SGX_INVALID_MEASUREMENT		4
+#define SGX_NOTBLOCKABLE		5
+#define SGX_PG_INVLD			6
+#define SGX_LOCKFAIL			7
+#define SGX_INVALID_SIGNATURE		8
+#define SGX_MAC_COMPARE_FAIL		9
+#define SGX_PAGE_NOT_BLOCKED		10
+#define SGX_NOT_TRACKED			11
+#define SGX_VA_SLOT_OCCUPIED		12
+#define SGX_CHILD_PRESENT		13
+#define SGX_ENCLAVE_ACT			14
+#define SGX_ENTRYEPOCH_LOCKED		15
+#define SGX_INVALID_EINITTOKEN		16
+#define SGX_PREV_TRK_INCMPL		17
+#define SGX_PG_IS_SECS			18
+#define SGX_INVALID_CPUSVN		32
+#define SGX_INVALID_ISVSVN		64
+#define SGX_UNMASKED_EVENT		128
+#define SGX_INVALID_KEYNAME		256
+
+/* IOCTL return values */
+#define SGX_POWER_LOST_ENCLAVE		0x40000000
+#define SGX_LE_ROLLBACK			0x40000001
+
+/**
+ * struct sgx_enclave_create - parameter structure for the
+ *                             %SGX_IOC_ENCLAVE_CREATE ioctl
+ * @src:	address for the SECS page data
+ */
+struct sgx_enclave_create  {
+	__u64	src;
+};
+
+/**
+ * struct sgx_enclave_add_page - parameter structure for the
+ *                               %SGX_IOC_ENCLAVE_ADD_PAGE ioctl
+ * @addr:	address within the ELRANGE
+ * @src:	address for the page data
+ * @secinfo:	address for the SECINFO data
+ * @mrmask:	bitmask for the measured 256 byte chunks
+ */
+struct sgx_enclave_add_page {
+	__u64	addr;
+	__u64	src;
+	__u64	secinfo;
+	__u16	mrmask;
+} __attribute__((__packed__));
+
+
+/**
+ * struct sgx_enclave_init - parameter structure for the
+ *                           %SGX_IOC_ENCLAVE_INIT ioctl
+ * @addr:	address within the ELRANGE
+ * @sigstruct:	address for the SIGSTRUCT data
+ */
+struct sgx_enclave_init {
+	__u64	addr;
+	__u64	sigstruct;
+};
+
+#endif /* _UAPI_ASM_X86_SGX_H */
diff --git a/drivers/platform/x86/Kconfig b/drivers/platform/x86/Kconfig
index 566644bb496a..645888c2049d 100644
--- a/drivers/platform/x86/Kconfig
+++ b/drivers/platform/x86/Kconfig
@@ -1218,6 +1218,8 @@ config INTEL_CHTDC_TI_PWRBTN
 	  To compile this driver as a module, choose M here: the module
 	  will be called intel_chtdc_ti_pwrbtn.
 
+source "drivers/platform/x86/intel_sgx/Kconfig"
+
 endif # X86_PLATFORM_DEVICES
 
 config PMC_ATOM
diff --git a/drivers/platform/x86/Makefile b/drivers/platform/x86/Makefile
index 2ba6cb795338..729208a9c958 100644
--- a/drivers/platform/x86/Makefile
+++ b/drivers/platform/x86/Makefile
@@ -91,3 +91,4 @@ obj-$(CONFIG_PMC_ATOM)		+= pmc_atom.o
 obj-$(CONFIG_MLX_PLATFORM)	+= mlx-platform.o
 obj-$(CONFIG_INTEL_TURBO_MAX_3) += intel_turbo_max_3.o
 obj-$(CONFIG_INTEL_CHTDC_TI_PWRBTN)	+= intel_chtdc_ti_pwrbtn.o
+obj-$(CONFIG_INTEL_SGX) += intel_sgx/
diff --git a/drivers/platform/x86/intel_sgx/Kconfig b/drivers/platform/x86/intel_sgx/Kconfig
new file mode 100644
index 000000000000..4cd88310617e
--- /dev/null
+++ b/drivers/platform/x86/intel_sgx/Kconfig
@@ -0,0 +1,20 @@
+#
+# Intel SGX
+#
+
+config INTEL_SGX
+	tristate "Intel(R) SGX Driver"
+	default n
+	depends on X86_64 && CPU_SUP_INTEL
+	select INTEL_SGX_CORE
+	select MMU_NOTIFIER
+	help
+	Intel(R) SGX is a set of CPU instructions that can be used by
+	applications to set aside private regions of code and data.  The code
+	outside the enclave is disallowed to access the memory inside the
+	enclave by the CPU access control.
+
+	The firmware uses PRMRR registers to reserve an area of physical memory
+	called Enclave Page Cache (EPC). There is a hardware unit in the
+	processor called Memory Encryption Engine. The MEE encrypts and decrypts
+	the EPC pages as they enter and leave the processor package.
diff --git a/drivers/platform/x86/intel_sgx/Makefile b/drivers/platform/x86/intel_sgx/Makefile
new file mode 100644
index 000000000000..95f254e30a8b
--- /dev/null
+++ b/drivers/platform/x86/intel_sgx/Makefile
@@ -0,0 +1,13 @@
+#
+# Intel SGX
+#
+
+obj-$(CONFIG_INTEL_SGX) += intel_sgx.o
+
+intel_sgx-$(CONFIG_INTEL_SGX) += \
+	sgx_ioctl.o \
+	sgx_encl.o \
+	sgx_encl_page.o \
+	sgx_main.o \
+	sgx_fault.o \
+	sgx_vma.o \
diff --git a/drivers/platform/x86/intel_sgx/sgx.h b/drivers/platform/x86/intel_sgx/sgx.h
new file mode 100644
index 000000000000..193f5e5b2338
--- /dev/null
+++ b/drivers/platform/x86/intel_sgx/sgx.h
@@ -0,0 +1,164 @@
+// SPDX-License-Identifier: (GPL-2.0 OR BSD-3-Clause)
+// Copyright(c) 2016-17 Intel Corporation.
+//
+// Authors:
+//
+// Jarkko Sakkinen <jarkko.sakkinen@linux.intel.com>
+// Suresh Siddha <suresh.b.siddha@intel.com>
+// Serge Ayoun <serge.ayoun@intel.com>
+// Shay Katz-zamir <shay.katz-zamir@intel.com>
+
+#ifndef __ARCH_INTEL_SGX_H__
+#define __ARCH_INTEL_SGX_H__
+
+#include <crypto/hash.h>
+#include <linux/kref.h>
+#include <linux/mmu_notifier.h>
+#include <linux/mmu_notifier.h>
+#include <linux/radix-tree.h>
+#include <linux/radix-tree.h>
+#include <linux/rbtree.h>
+#include <linux/rwsem.h>
+#include <linux/sched.h>
+#include <linux/workqueue.h>
+#include <asm/sgx.h>
+#include <asm/sgx_pr.h>
+#include <uapi/asm/sgx.h>
+
+#define SGX_EINIT_SPIN_COUNT	20
+#define SGX_EINIT_SLEEP_COUNT	50
+#define SGX_EINIT_SLEEP_TIME	20
+
+#define SGX_VA_SLOT_COUNT 512
+#define SGX_VA_OFFSET_MASK ((SGX_VA_SLOT_COUNT - 1) << 3)
+
+struct sgx_va_page {
+	struct sgx_epc_page *epc_page;
+	DECLARE_BITMAP(slots, SGX_VA_SLOT_COUNT);
+	struct list_head list;
+};
+
+enum sgx_encl_page_flags {
+	SGX_ENCL_PAGE_TCS	= BIT(0),
+	SGX_ENCL_PAGE_RESERVED	= BIT(1),
+	SGX_ENCL_PAGE_LOADED	= BIT(2),
+};
+
+#define SGX_ENCL_PAGE_ADDR(encl_page) ((encl_page)->desc & PAGE_MASK)
+#define SGX_ENCL_PAGE_VA_OFFSET(encl_page) \
+	((encl_page)->desc & SGX_VA_OFFSET_MASK)
+#define SGX_ENCL_PAGE_BACKING_INDEX(encl_page, encl)		\
+({								\
+	pgoff_t index;						\
+	if (!PFN_DOWN(encl_page->desc))				\
+		index = PFN_DOWN(encl->size);			\
+	else							\
+		index = PFN_DOWN(encl_page->desc - encl->base);	\
+	index;							\
+})
+#define SGX_ENCL_PAGE_PCMD_OFFSET(encl_page, encl)		\
+({								\
+	unsigned long ret;					\
+	ret = SGX_ENCL_PAGE_BACKING_INDEX(encl_page, encl);	\
+	((ret & 31) * 128);					\
+})
+
+struct sgx_encl_page {
+	unsigned long desc;
+	union {
+		struct sgx_epc_page *epc_page;
+		struct sgx_va_page *va_page;
+	};
+	struct sgx_encl *encl;
+	struct sgx_epc_page_impl impl;
+};
+
+enum sgx_encl_flags {
+	SGX_ENCL_INITIALIZED	= BIT(0),
+	SGX_ENCL_DEBUG		= BIT(1),
+	SGX_ENCL_SUSPEND	= BIT(2),
+	SGX_ENCL_DEAD		= BIT(3),
+};
+
+struct sgx_encl {
+	unsigned int flags;
+	uint64_t attributes;
+	uint64_t xfrm;
+	unsigned int page_cnt;
+	unsigned int secs_child_cnt;
+	struct mutex lock;
+	struct mm_struct *mm;
+	struct file *backing;
+	struct file *pcmd;
+	struct kref refcount;
+	unsigned long base;
+	unsigned long size;
+	unsigned long ssaframesize;
+	struct list_head va_pages;
+	struct radix_tree_root page_tree;
+	struct list_head add_page_reqs;
+	struct work_struct add_page_work;
+	struct sgx_encl_page secs;
+	struct pid *tgid;
+	struct mmu_notifier mmu_notifier;
+};
+
+extern struct workqueue_struct *sgx_add_page_wq;
+extern u64 sgx_encl_size_max_32;
+extern u64 sgx_encl_size_max_64;
+extern u64 sgx_xfrm_mask;
+extern u32 sgx_misc_reserved;
+extern u32 sgx_xsave_size_tbl[64];
+
+extern const struct vm_operations_struct sgx_vm_ops;
+
+int sgx_encl_find(struct mm_struct *mm, unsigned long addr,
+		  struct vm_area_struct **vma);
+void sgx_invalidate(struct sgx_encl *encl, bool flush_cpus);
+#define SGX_INVD(ret, encl, fmt, ...)			\
+do {							\
+	if (unlikely(ret)) {				\
+		sgx_err(encl, fmt, ##__VA_ARGS__);	\
+		sgx_invalidate(encl, true);		\
+	}						\
+} while (0)
+
+struct sgx_encl *sgx_encl_alloc(struct sgx_secs *secs);
+int sgx_encl_create(struct sgx_encl *encl, struct sgx_secs *secs);
+struct sgx_encl_page *sgx_encl_alloc_page(struct sgx_encl *encl,
+					  unsigned long addr);
+void sgx_encl_free_page(struct sgx_encl_page *encl_page);
+int sgx_encl_add_page(struct sgx_encl *encl, unsigned long addr, void *data,
+		      struct sgx_secinfo *secinfo, unsigned int mrmask);
+int sgx_encl_init(struct sgx_encl *encl, struct sgx_sigstruct *sigstruct,
+		  struct sgx_einittoken *einittoken);
+void sgx_encl_block(struct sgx_encl_page *encl_page);
+void sgx_encl_track(struct sgx_encl *encl);
+int sgx_encl_load_page(struct sgx_encl_page *encl_page,
+		       struct sgx_epc_page *epc_page);
+void sgx_encl_release(struct kref *ref);
+
+long sgx_ioctl(struct file *filep, unsigned int cmd, unsigned long arg);
+#ifdef CONFIG_COMPAT
+long sgx_compat_ioctl(struct file *filep, unsigned int cmd, unsigned long arg);
+#endif
+
+/* Utility functions */
+int sgx_test_and_clear_young(struct sgx_encl_page *page);
+void sgx_flush_cpus(struct sgx_encl *encl);
+
+struct sgx_encl_page *sgx_fault_page(struct vm_area_struct *vma,
+				     unsigned long addr,
+				     bool do_reserve);
+
+extern const struct sgx_epc_page_ops sgx_encl_page_ops;
+
+void sgx_set_epc_page(struct sgx_encl_page *encl_page,
+		      struct sgx_epc_page *epc_page);
+void sgx_set_page_reclaimable(struct sgx_encl_page *encl_page);
+struct sgx_epc_page *sgx_alloc_va_page(unsigned int flags);
+unsigned int sgx_alloc_va_slot(struct sgx_va_page *va_page);
+void sgx_free_va_slot(struct sgx_va_page *va_page, unsigned int offset);
+bool sgx_va_page_full(struct sgx_va_page *va_page);
+
+#endif /* __ARCH_X86_INTEL_SGX_H__ */
diff --git a/drivers/platform/x86/intel_sgx/sgx_encl.c b/drivers/platform/x86/intel_sgx/sgx_encl.c
new file mode 100644
index 000000000000..562d4ce412d4
--- /dev/null
+++ b/drivers/platform/x86/intel_sgx/sgx_encl.c
@@ -0,0 +1,1002 @@
+// SPDX-License-Identifier: (GPL-2.0 OR BSD-3-Clause)
+// Copyright(c) 2016-17 Intel Corporation.
+//
+// Authors:
+//
+// Jarkko Sakkinen <jarkko.sakkinen@linux.intel.com>
+// Suresh Siddha <suresh.b.siddha@intel.com>
+// Serge Ayoun <serge.ayoun@intel.com>
+// Shay Katz-zamir <shay.katz-zamir@intel.com>
+// Sean Christopherson <sean.j.christopherson@intel.com>
+
+#include <asm/mman.h>
+#include <linux/delay.h>
+#include <linux/file.h>
+#include <linux/hashtable.h>
+#include <linux/highmem.h>
+#include <linux/ratelimit.h>
+#include <linux/sched/signal.h>
+#include <linux/shmem_fs.h>
+#include <linux/slab.h>
+#include "sgx.h"
+
+struct sgx_add_page_req {
+	struct sgx_encl *encl;
+	struct sgx_encl_page *encl_page;
+	struct sgx_secinfo secinfo;
+	u16 mrmask;
+	struct list_head list;
+};
+
+/**
+ * sgx_encl_find - find an enclave
+ * @mm:		mm struct of the current process
+ * @addr:	address in the ELRANGE
+ * @vma:	the resulting VMA
+ *
+ * Finds an enclave identified by the given address. Gives back the VMA, that
+ * is part of the enclave, located in that address. The VMA is given back if it
+ * is a proper enclave VMA even if an &sgx_encl instance does not exist
+ * yet (enclave creation has not been performed).
+ *
+ * Return:
+ *   0 on success,
+ *   -EINVAL if an enclave was not found,
+ *   -ENOENT if the enclave has not been created yet
+ */
+int sgx_encl_find(struct mm_struct *mm, unsigned long addr,
+		  struct vm_area_struct **vma)
+{
+	struct vm_area_struct *result;
+	struct sgx_encl *encl;
+
+	result = find_vma(mm, addr);
+	if (!result || result->vm_ops != &sgx_vm_ops || addr < result->vm_start)
+		return -EINVAL;
+
+	encl = result->vm_private_data;
+	*vma = result;
+
+	return encl ? 0 : -ENOENT;
+}
+
+static void sgx_encl_free_va_pages(struct sgx_encl *encl)
+{
+	struct sgx_va_page *va_page;
+
+	while (!list_empty(&encl->va_pages)) {
+		va_page = list_first_entry(&encl->va_pages, struct sgx_va_page,
+					   list);
+		list_del(&va_page->list);
+		sgx_free_page(va_page->epc_page);
+		kfree(va_page);
+	}
+}
+
+/**
+ * sgx_invalidate - kill an active enclave
+ *
+ * @encl:	an &sgx_encl instance
+ * @flush_cpus	whether to kick threads out of the enclave
+ *
+ * Clear PTEs for all the entry points, kick threads out of the enclave if
+ * requested and mark enclave state as dead.
+ */
+void sgx_invalidate(struct sgx_encl *encl, bool flush_cpus)
+{
+	struct sgx_encl_page *entry;
+	struct radix_tree_iter iter;
+	struct vm_area_struct *vma;
+	unsigned long addr;
+	void **slot;
+
+	if (encl->flags & SGX_ENCL_DEAD)
+		return;
+
+	radix_tree_for_each_slot(slot, &encl->page_tree, &iter, 0) {
+		entry = *slot;
+		addr = SGX_ENCL_PAGE_ADDR(entry);
+
+		if ((entry->desc & SGX_ENCL_PAGE_LOADED) &&
+		    (entry->desc & SGX_ENCL_PAGE_TCS) &&
+		    !sgx_encl_find(encl->mm, addr, &vma))
+			zap_vma_ptes(vma, addr, PAGE_SIZE);
+	}
+
+	encl->flags |= SGX_ENCL_DEAD;
+
+	if (flush_cpus)
+		sgx_flush_cpus(encl);
+
+	sgx_encl_free_va_pages(encl);
+}
+
+static int sgx_measure(struct sgx_epc_page *secs_page,
+		       struct sgx_epc_page *epc_page,
+		       u16 mrmask)
+{
+	int ret = 0;
+	void *secs;
+	void *epc;
+	int i;
+	int j;
+
+	if (!mrmask)
+		return ret;
+
+	secs = sgx_get_page(secs_page);
+	epc = sgx_get_page(epc_page);
+
+	for (i = 0, j = 1; i < 0x1000 && !ret; i += 0x100, j <<= 1) {
+		if (!(j & mrmask))
+			continue;
+
+		ret = __eextend(secs, (void *)((unsigned long)epc + i));
+	}
+
+	sgx_put_page(epc);
+	sgx_put_page(secs);
+
+	return ret;
+}
+
+static int sgx_eadd(struct sgx_epc_page *secs_page,
+		    struct sgx_epc_page *epc_page,
+		    unsigned long linaddr,
+		    struct sgx_secinfo *secinfo,
+		    struct page *backing)
+{
+	struct sgx_pageinfo pginfo;
+	void *epc_page_vaddr;
+	int ret;
+
+	pginfo.srcpge = (unsigned long)kmap_atomic(backing);
+	pginfo.secs = (unsigned long)sgx_get_page(secs_page);
+	epc_page_vaddr = sgx_get_page(epc_page);
+
+	pginfo.linaddr = linaddr;
+	pginfo.secinfo = (unsigned long)secinfo;
+	ret = __eadd(&pginfo, epc_page_vaddr);
+
+	sgx_put_page(epc_page_vaddr);
+	sgx_put_page((void *)(unsigned long)pginfo.secs);
+	kunmap_atomic((void *)(unsigned long)pginfo.srcpge);
+
+	return ret;
+}
+
+static bool sgx_process_add_page_req(struct sgx_add_page_req *req,
+				     struct sgx_epc_page *epc_page)
+{
+	struct sgx_encl_page *encl_page = req->encl_page;
+	struct sgx_encl *encl = req->encl;
+	struct vm_area_struct *vma;
+	pgoff_t backing_index;
+	struct page *backing;
+	unsigned long addr;
+	int ret;
+
+	if (encl->flags & (SGX_ENCL_SUSPEND | SGX_ENCL_DEAD))
+		return false;
+
+	addr = SGX_ENCL_PAGE_ADDR(encl_page);
+	ret = sgx_encl_find(encl->mm, addr, &vma);
+	if (ret)
+		return false;
+
+	backing_index = SGX_ENCL_PAGE_BACKING_INDEX(encl_page, encl);
+	backing = sgx_get_backing(encl->backing, backing_index);
+	if (IS_ERR(backing))
+		return false;
+
+	ret = vm_insert_pfn(vma, addr, SGX_EPC_PFN(epc_page));
+	if (ret) {
+		sgx_err(encl, "%s: vm_insert_pfn() returned %d\n", __func__,
+			ret);
+		sgx_put_backing(backing, false);
+		return false;
+	}
+
+	ret = sgx_eadd(encl->secs.epc_page, epc_page, addr, &req->secinfo,
+		       backing);
+
+	sgx_put_backing(backing, false);
+	if (ret) {
+		sgx_err(encl, "EADD returned %d\n", ret);
+		zap_vma_ptes(vma, addr, PAGE_SIZE);
+		return false;
+	}
+
+	ret = sgx_measure(encl->secs.epc_page, epc_page, req->mrmask);
+	if (ret) {
+		sgx_err(encl, "EEXTEND returned %d\n", ret);
+		zap_vma_ptes(vma, addr, PAGE_SIZE);
+		return false;
+	}
+
+	encl_page->encl = encl;
+	encl->secs_child_cnt++;
+	sgx_set_epc_page(encl_page, epc_page);
+	sgx_set_page_reclaimable(encl_page);
+	return true;
+}
+
+static void sgx_add_page_worker(struct work_struct *work)
+{
+	struct sgx_add_page_req *req;
+	bool skip_rest = false;
+	bool is_empty = false;
+	struct sgx_encl *encl;
+	struct sgx_epc_page *epc_page;
+
+	encl = container_of(work, struct sgx_encl, add_page_work);
+
+	do {
+		schedule();
+
+		if (encl->flags & SGX_ENCL_DEAD)
+			skip_rest = true;
+
+		mutex_lock(&encl->lock);
+		req = list_first_entry(&encl->add_page_reqs,
+				       struct sgx_add_page_req, list);
+		list_del(&req->list);
+		is_empty = list_empty(&encl->add_page_reqs);
+		mutex_unlock(&encl->lock);
+
+		if (skip_rest)
+			goto next;
+
+		epc_page = sgx_alloc_page(&req->encl_page->impl, 0);
+		down_read(&encl->mm->mmap_sem);
+		mutex_lock(&encl->lock);
+
+		if (IS_ERR(epc_page)) {
+			sgx_invalidate(encl, false);
+			skip_rest = true;
+		} else	if (!sgx_process_add_page_req(req, epc_page)) {
+			sgx_free_page(epc_page);
+			sgx_invalidate(encl, false);
+			skip_rest = true;
+		}
+
+		mutex_unlock(&encl->lock);
+		up_read(&encl->mm->mmap_sem);
+
+next:
+		kfree(req);
+	} while (!kref_put(&encl->refcount, sgx_encl_release) && !is_empty);
+}
+
+static u32 sgx_calc_ssaframesize(u32 miscselect, u64 xfrm)
+{
+	u32 size_max = PAGE_SIZE;
+	u32 size;
+	int i;
+
+	for (i = 2; i < 64; i++) {
+		if (!((1 << i) & xfrm))
+			continue;
+
+		size = SGX_SSA_GPRS_SIZE + sgx_xsave_size_tbl[i];
+		if (miscselect & SGX_MISC_EXINFO)
+			size += SGX_SSA_MISC_EXINFO_SIZE;
+
+		if (size > size_max)
+			size_max = size;
+	}
+
+	return (size_max + PAGE_SIZE - 1) >> PAGE_SHIFT;
+}
+
+static int sgx_validate_secs(const struct sgx_secs *secs,
+			     unsigned long ssaframesize)
+{
+	int i;
+
+	if (secs->size < (2 * PAGE_SIZE) ||
+	    (secs->size & (secs->size - 1)) != 0)
+		return -EINVAL;
+
+	if (secs->base & (secs->size - 1))
+		return -EINVAL;
+
+	if (secs->attributes & SGX_ATTR_RESERVED_MASK ||
+	    secs->miscselect & sgx_misc_reserved)
+		return -EINVAL;
+
+	if (secs->attributes & SGX_ATTR_MODE64BIT) {
+#ifdef CONFIG_X86_64
+		if (secs->size > sgx_encl_size_max_64)
+			return -EINVAL;
+#else
+		return -EINVAL;
+#endif
+	} else {
+		/* On 64-bit architecture allow 32-bit encls only in
+		 * the compatibility mode.
+		 */
+#ifdef CONFIG_X86_64
+		if (!test_thread_flag(TIF_ADDR32))
+			return -EINVAL;
+#endif
+		if (secs->size > sgx_encl_size_max_32)
+			return -EINVAL;
+	}
+
+	if ((secs->xfrm & 0x3) != 0x3 || (secs->xfrm & ~sgx_xfrm_mask))
+		return -EINVAL;
+
+	/* Check that BNDREGS and BNDCSR are equal. */
+	if (((secs->xfrm >> 3) & 1) != ((secs->xfrm >> 4) & 1))
+		return -EINVAL;
+
+	if (!secs->ssaframesize || ssaframesize > secs->ssaframesize)
+		return -EINVAL;
+
+	for (i = 0; i < SGX_SECS_RESERVED1_SIZE; i++)
+		if (secs->reserved1[i])
+			return -EINVAL;
+
+	for (i = 0; i < SGX_SECS_RESERVED2_SIZE; i++)
+		if (secs->reserved2[i])
+			return -EINVAL;
+
+	for (i = 0; i < SGX_SECS_RESERVED3_SIZE; i++)
+		if (secs->reserved3[i])
+			return -EINVAL;
+
+	for (i = 0; i < SGX_SECS_RESERVED4_SIZE; i++)
+		if (secs->reserved4[i])
+			return -EINVAL;
+
+	return 0;
+}
+
+static void sgx_mmu_notifier_release(struct mmu_notifier *mn,
+				     struct mm_struct *mm)
+{
+	struct sgx_encl *encl =
+		container_of(mn, struct sgx_encl, mmu_notifier);
+
+	mutex_lock(&encl->lock);
+	encl->flags |= SGX_ENCL_DEAD;
+	mutex_unlock(&encl->lock);
+}
+
+static const struct mmu_notifier_ops sgx_mmu_notifier_ops = {
+	.release	= sgx_mmu_notifier_release,
+};
+
+static int sgx_encl_grow(struct sgx_encl *encl)
+{
+	struct sgx_va_page *va_page;
+	int ret;
+
+	mutex_lock(&encl->lock);
+	if (!(encl->page_cnt % SGX_VA_SLOT_COUNT)) {
+		mutex_unlock(&encl->lock);
+
+		va_page = kzalloc(sizeof(*va_page), GFP_KERNEL);
+		if (!va_page)
+			return -ENOMEM;
+		va_page->epc_page = sgx_alloc_va_page(0);
+		if (IS_ERR(va_page->epc_page)) {
+			ret = PTR_ERR(va_page->epc_page);
+			kfree(va_page);
+			return ret;
+		}
+
+		mutex_lock(&encl->lock);
+		if (encl->page_cnt % SGX_VA_SLOT_COUNT) {
+			sgx_free_page(va_page->epc_page);
+			kfree(va_page);
+		} else {
+			list_add(&va_page->list, &encl->va_pages);
+		}
+	}
+	encl->page_cnt++;
+	mutex_unlock(&encl->lock);
+	return 0;
+}
+
+/**
+ * sgx_encl_alloc - allocate memory for an enclave and set attributes
+ *
+ * @secs:	SECS data (must be page aligned)
+ *
+ * Allocates a new &sgx_encl instance. Validates SECS attributes, creates
+ * backing storage for the enclave and sets enclave attributes to sane initial
+ * values.
+ *
+ * Return:
+ *   an &sgx_encl instance,
+ *   -errno otherwise
+ */
+struct sgx_encl *sgx_encl_alloc(struct sgx_secs *secs)
+{
+	unsigned long ssaframesize;
+	struct sgx_encl *encl;
+	struct file *backing;
+	struct file *pcmd;
+
+	ssaframesize = sgx_calc_ssaframesize(secs->miscselect, secs->xfrm);
+	if (sgx_validate_secs(secs, ssaframesize))
+		return ERR_PTR(-EINVAL);
+
+	backing = shmem_file_setup("[dev/sgx]", secs->size + PAGE_SIZE,
+				   VM_NORESERVE);
+	if (IS_ERR(backing))
+		return (void *)backing;
+
+	pcmd = shmem_file_setup("[dev/sgx]", (secs->size + PAGE_SIZE) >> 5,
+				VM_NORESERVE);
+	if (IS_ERR(pcmd)) {
+		fput(backing);
+		return (void *)pcmd;
+	}
+
+	encl = kzalloc(sizeof(*encl), GFP_KERNEL);
+	if (!encl) {
+		fput(backing);
+		fput(pcmd);
+		return ERR_PTR(-ENOMEM);
+	}
+
+	encl->attributes = secs->attributes;
+	encl->xfrm = secs->xfrm;
+
+	kref_init(&encl->refcount);
+	INIT_LIST_HEAD(&encl->add_page_reqs);
+	INIT_LIST_HEAD(&encl->va_pages);
+	INIT_RADIX_TREE(&encl->page_tree, GFP_KERNEL);
+	mutex_init(&encl->lock);
+	INIT_WORK(&encl->add_page_work, sgx_add_page_worker);
+
+	encl->mm = current->mm;
+	encl->base = secs->base;
+	encl->size = secs->size;
+	encl->ssaframesize = secs->ssaframesize;
+	encl->backing = backing;
+	encl->pcmd = pcmd;
+
+	return encl;
+}
+
+
+/**
+ * sgx_encl_create - create an enclave
+ *
+ * @encl:	an enclave
+ * @secs:	page aligned SECS data
+ *
+ * Validates SECS attributes, allocates an EPC page for the SECS and creates
+ * the enclave by performing ECREATE.
+ *
+ * Return:
+ *   0 on success,
+ *   -errno otherwise
+ */
+int sgx_encl_create(struct sgx_encl *encl, struct sgx_secs *secs)
+{
+	struct vm_area_struct *vma;
+	struct sgx_pageinfo pginfo;
+	struct sgx_secinfo secinfo;
+	struct sgx_epc_page *secs_epc;
+	void *secs_vaddr;
+	long ret;
+
+	secs_epc = sgx_alloc_page(&encl->secs.impl, 0);
+	if (IS_ERR(secs_epc)) {
+		ret = PTR_ERR(secs_epc);
+		return ret;
+	}
+
+	sgx_set_epc_page(&encl->secs, secs_epc);
+	encl->secs.encl = encl;
+	encl->secs.impl.ops = &sgx_encl_page_ops;
+	encl->tgid = get_pid(task_tgid(current));
+
+	ret = sgx_encl_grow(encl);
+	if (ret)
+		return ret;
+
+	secs_vaddr = sgx_get_page(secs_epc);
+
+	pginfo.srcpge = (unsigned long)secs;
+	pginfo.linaddr = 0;
+	pginfo.secinfo = (unsigned long)&secinfo;
+	pginfo.secs = 0;
+	memset(&secinfo, 0, sizeof(secinfo));
+	ret = __ecreate((void *)&pginfo, secs_vaddr);
+
+	sgx_put_page(secs_vaddr);
+
+	if (ret) {
+		sgx_dbg(encl, "ECREATE returned %ld\n", ret);
+		return ret;
+	}
+
+	if (secs->attributes & SGX_ATTR_DEBUG)
+		encl->flags |= SGX_ENCL_DEBUG;
+
+	encl->mmu_notifier.ops = &sgx_mmu_notifier_ops;
+	ret = mmu_notifier_register(&encl->mmu_notifier, encl->mm);
+	if (ret) {
+		if (ret == -EINTR)
+			ret = -ERESTARTSYS;
+		encl->mmu_notifier.ops = NULL;
+		return ret;
+	}
+
+	down_read(&current->mm->mmap_sem);
+	ret = sgx_encl_find(current->mm, secs->base, &vma);
+	if (ret != -ENOENT) {
+		if (!ret)
+			ret = -EINVAL;
+		up_read(&current->mm->mmap_sem);
+		return ret;
+	}
+
+	if (vma->vm_start != secs->base ||
+	    vma->vm_end != (secs->base + secs->size) ||
+	    vma->vm_pgoff != 0) {
+		ret = -EINVAL;
+		up_read(&current->mm->mmap_sem);
+		return ret;
+	}
+
+	vma->vm_private_data = encl;
+	up_read(&current->mm->mmap_sem);
+	return 0;
+}
+
+static int sgx_validate_secinfo(struct sgx_secinfo *secinfo)
+{
+	u64 page_type = secinfo->flags & SGX_SECINFO_PAGE_TYPE_MASK;
+	u64 perm = secinfo->flags & SGX_SECINFO_PERMISSION_MASK;
+	int i;
+
+	if ((secinfo->flags & SGX_SECINFO_RESERVED_MASK) ||
+	    ((perm & SGX_SECINFO_W) && !(perm & SGX_SECINFO_R)) ||
+	    (page_type != SGX_SECINFO_TCS &&
+	     page_type != SGX_SECINFO_REG))
+		return -EINVAL;
+
+	for (i = 0; i < sizeof(secinfo->reserved) / sizeof(u64); i++)
+		if (secinfo->reserved[i])
+			return -EINVAL;
+
+	return 0;
+}
+
+static bool sgx_validate_offset(struct sgx_encl *encl, unsigned long offset)
+{
+	if (offset & (PAGE_SIZE - 1))
+		return false;
+
+	if (offset >= encl->size)
+		return false;
+
+	return true;
+}
+
+static int sgx_validate_tcs(struct sgx_encl *encl, struct sgx_tcs *tcs)
+{
+	int i;
+
+	if (tcs->flags & SGX_TCS_RESERVED_MASK) {
+		sgx_dbg(encl, "%s: invalid TCS flags = 0x%lx\n",
+			__func__, (unsigned long)tcs->flags);
+		return -EINVAL;
+	}
+
+	if (tcs->flags & SGX_TCS_DBGOPTIN) {
+		sgx_dbg(encl, "%s: DBGOPTIN TCS flag is set, EADD will clear it\n",
+			__func__);
+		return -EINVAL;
+	}
+
+	if (!sgx_validate_offset(encl, tcs->ossa)) {
+		sgx_dbg(encl, "%s: invalid OSSA: 0x%lx\n", __func__,
+			(unsigned long)tcs->ossa);
+		return -EINVAL;
+	}
+
+	if (!sgx_validate_offset(encl, tcs->ofsbase)) {
+		sgx_dbg(encl, "%s: invalid OFSBASE: 0x%lx\n", __func__,
+			(unsigned long)tcs->ofsbase);
+		return -EINVAL;
+	}
+
+	if (!sgx_validate_offset(encl, tcs->ogsbase)) {
+		sgx_dbg(encl, "%s: invalid OGSBASE: 0x%lx\n", __func__,
+			(unsigned long)tcs->ogsbase);
+		return -EINVAL;
+	}
+
+	if ((tcs->fslimit & 0xFFF) != 0xFFF) {
+		sgx_dbg(encl, "%s: invalid FSLIMIT: 0x%x\n", __func__,
+			tcs->fslimit);
+		return -EINVAL;
+	}
+
+	if ((tcs->gslimit & 0xFFF) != 0xFFF) {
+		sgx_dbg(encl, "%s: invalid GSLIMIT: 0x%x\n", __func__,
+			tcs->gslimit);
+		return -EINVAL;
+	}
+
+	for (i = 0; i < sizeof(tcs->reserved) / sizeof(u64); i++)
+		if (tcs->reserved[i])
+			return -EINVAL;
+
+	return 0;
+}
+
+static int __sgx_encl_add_page(struct sgx_encl *encl,
+			       struct sgx_encl_page *encl_page,
+			       void *data,
+			       struct sgx_secinfo *secinfo,
+			       unsigned int mrmask)
+{
+	u64 page_type = secinfo->flags & SGX_SECINFO_PAGE_TYPE_MASK;
+	struct sgx_add_page_req *req = NULL;
+	pgoff_t backing_index;
+	struct page *backing;
+	void *backing_ptr;
+	int empty;
+
+	req = kzalloc(sizeof(*req), GFP_KERNEL);
+	if (!req)
+		return -ENOMEM;
+
+	backing_index = SGX_ENCL_PAGE_BACKING_INDEX(encl_page, encl);
+	backing = sgx_get_backing(encl->backing, backing_index);
+	if (IS_ERR(backing)) {
+		kfree(req);
+		return PTR_ERR(backing);
+	}
+	backing_ptr = kmap(backing);
+	memcpy(backing_ptr, data, PAGE_SIZE);
+	kunmap(backing);
+	if (page_type == SGX_SECINFO_TCS)
+		encl_page->desc |= SGX_ENCL_PAGE_TCS;
+	memcpy(&req->secinfo, secinfo, sizeof(*secinfo));
+	req->encl = encl;
+	req->encl_page = encl_page;
+	req->mrmask = mrmask;
+	empty = list_empty(&encl->add_page_reqs);
+	kref_get(&encl->refcount);
+	list_add_tail(&req->list, &encl->add_page_reqs);
+	if (empty)
+		queue_work(sgx_add_page_wq, &encl->add_page_work);
+	sgx_put_backing(backing, true /* write */);
+	return 0;
+}
+
+/**
+ * sgx_encl_alloc_page - allocate a new enclave page
+ * @encl:	an enclave
+ * @addr:	page address in the ELRANGE
+ *
+ * Return:
+ *   an &sgx_encl_page instance on success,
+ *   -errno otherwise
+ */
+struct sgx_encl_page *sgx_encl_alloc_page(struct sgx_encl *encl,
+					  unsigned long addr)
+{
+	struct sgx_encl_page *encl_page;
+	int ret;
+
+	if (radix_tree_lookup(&encl->page_tree, PFN_DOWN(addr)))
+		return ERR_PTR(-EEXIST);
+	encl_page = kzalloc(sizeof(*encl_page), GFP_KERNEL);
+	if (!encl_page)
+		return ERR_PTR(-ENOMEM);
+	encl_page->desc = addr;
+	encl_page->impl.ops = &sgx_encl_page_ops;
+	encl_page->encl = encl;
+	ret = radix_tree_insert(&encl->page_tree, PFN_DOWN(encl_page->desc),
+				encl_page);
+	if (ret) {
+		kfree(encl_page);
+		return ERR_PTR(ret);
+	}
+	return encl_page;
+}
+
+/**
+ * sgx_encl_free_page - free an enclave page
+ * @encl_page:	an enclave page
+ */
+void sgx_encl_free_page(struct sgx_encl_page *encl_page)
+{
+	radix_tree_delete(&encl_page->encl->page_tree,
+			  PFN_DOWN(encl_page->desc));
+	if (encl_page->desc & SGX_ENCL_PAGE_LOADED) {
+		spin_lock(&sgx_active_page_list_lock);
+		list_del(&encl_page->epc_page->list);
+		spin_unlock(&sgx_active_page_list_lock);
+		sgx_free_page(encl_page->epc_page);
+	}
+	kfree(encl_page);
+}
+
+/**
+ * sgx_encl_add_page - add a page to the enclave
+ *
+ * @encl:	an enclave
+ * @addr:	page address in the ELRANGE
+ * @data:	page data
+ * @secinfo:	page permissions
+ * @mrmask:	bitmask to select the 256 byte chunks to be measured
+ *
+ * Creates a new enclave page and enqueues an EADD operation that will be
+ * processed by a worker thread later on.
+ *
+ * Return:
+ *   0 on success,
+ *   -errno otherwise
+ */
+int sgx_encl_add_page(struct sgx_encl *encl, unsigned long addr, void *data,
+		      struct sgx_secinfo *secinfo, unsigned int mrmask)
+{
+	u64 page_type = secinfo->flags & SGX_SECINFO_PAGE_TYPE_MASK;
+	struct sgx_encl_page *encl_page;
+	int ret;
+
+	if (sgx_validate_secinfo(secinfo))
+		return -EINVAL;
+	if (page_type == SGX_SECINFO_TCS) {
+		ret = sgx_validate_tcs(encl, data);
+		if (ret)
+			return ret;
+	}
+	ret = sgx_encl_grow(encl);
+	if (ret)
+		return ret;
+	mutex_lock(&encl->lock);
+	if (encl->flags & (SGX_ENCL_INITIALIZED | SGX_ENCL_DEAD)) {
+		mutex_unlock(&encl->lock);
+		return -EINVAL;
+	}
+	encl_page = sgx_encl_alloc_page(encl, addr);
+	if (IS_ERR(encl_page)) {
+		mutex_unlock(&encl->lock);
+		return PTR_ERR(encl_page);
+	}
+	ret = __sgx_encl_add_page(encl, encl_page, data, secinfo, mrmask);
+	if (ret)
+		sgx_encl_free_page(encl_page);
+	mutex_unlock(&encl->lock);
+	return ret;
+}
+
+static int sgx_einit(struct sgx_encl *encl, struct sgx_sigstruct *sigstruct,
+		     struct sgx_einittoken *token)
+{
+	struct sgx_epc_page *secs_epc = encl->secs.epc_page;
+	void *secs_va;
+	int ret;
+
+	secs_va = sgx_get_page(secs_epc);
+	ret = __einit(sigstruct, token, secs_va);
+	sgx_put_page(secs_va);
+
+	return ret;
+}
+
+/**
+ * sgx_encl_init - perform EINIT for the given enclave
+ *
+ * @encl:	an enclave
+ * @sigstruct:	SIGSTRUCT for the enclave
+ * @token:	EINITTOKEN for the enclave
+ *
+ * Retries a few times in order to perform EINIT operation on an enclave
+ * because there could be potentially an interrupt storm.
+ *
+ * Return:
+ *   0 on success,
+ *   SGX error code on EINIT failure,
+ *   -errno otherwise
+ */
+int sgx_encl_init(struct sgx_encl *encl, struct sgx_sigstruct *sigstruct,
+		  struct sgx_einittoken *token)
+{
+	int ret;
+	int i;
+	int j;
+
+	flush_work(&encl->add_page_work);
+
+	mutex_lock(&encl->lock);
+
+	if (encl->flags & SGX_ENCL_INITIALIZED) {
+		mutex_unlock(&encl->lock);
+		return 0;
+	}
+
+	for (i = 0; i < SGX_EINIT_SLEEP_COUNT; i++) {
+		for (j = 0; j < SGX_EINIT_SPIN_COUNT; j++) {
+			ret = sgx_einit(encl, sigstruct, token);
+
+			if (ret == SGX_UNMASKED_EVENT)
+				continue;
+			else
+				break;
+		}
+
+		if (ret != SGX_UNMASKED_EVENT)
+			break;
+
+		msleep_interruptible(SGX_EINIT_SLEEP_TIME);
+		if (signal_pending(current)) {
+			mutex_unlock(&encl->lock);
+			return -ERESTARTSYS;
+		}
+	}
+
+	if (ret > 0)
+		sgx_dbg(encl, "EINIT returned %d\n", ret);
+	else if (!ret)
+		encl->flags |= SGX_ENCL_INITIALIZED;
+	mutex_unlock(&encl->lock);
+
+	return ret;
+}
+
+/**
+ * sgx_encl_block - block an enclave page
+ * @encl_page:	an enclave page
+ *
+ * Changes the state of the associated EPC page to blocked.
+ */
+void sgx_encl_block(struct sgx_encl_page *encl_page)
+{
+	unsigned long addr = SGX_ENCL_PAGE_ADDR(encl_page);
+	struct sgx_encl *encl = encl_page->encl;
+	struct vm_area_struct *vma;
+	int ret;
+
+	if (encl->flags & SGX_ENCL_DEAD)
+		return;
+
+	ret = sgx_encl_find(encl->mm, addr, &vma);
+	if (ret || encl != vma->vm_private_data)
+		return;
+
+	zap_vma_ptes(vma, addr, PAGE_SIZE);
+	ret = sgx_eblock(encl_page->epc_page);
+	SGX_INVD(ret, encl, "EBLOCK returned %d\n", ret);
+}
+
+/**
+ * sgx_encl_track - start tracking pages in the blocked state
+ * @encl:	an enclave
+ *
+ * Start blocking accesses for pages in the blocked state for threads that enter
+ * inside the enclave by executing the ETRACK leaf instruction. This starts a
+ * shootdown sequence for threads that entered before ETRACK.
+ *
+ * The caller must take care (with an IPI when necessary) to make sure that the
+ * previous shootdown sequence was completed before calling this function.  If
+ * this is not the case, the callee prints a critical error to the klog and
+ * kills the enclave.
+ */
+void sgx_encl_track(struct sgx_encl *encl)
+{
+	int ret = sgx_etrack(encl->secs.epc_page);
+
+	SGX_INVD(ret, encl, "ETRACK returned %d\n", ret);
+}
+
+/**
+ * sgx_encl_load_page - load an enclave page
+ * @encl_page:	a &sgx_encl_page
+ * @epc_page:	a &sgx_epc_page
+ *
+ * Loads an enclave page from the regular memory to the EPC. The pages, which
+ * are not children of a SECS (eg SECS itself and VA pages) should set their
+ * address to zero.
+ */
+int sgx_encl_load_page(struct sgx_encl_page *encl_page,
+		       struct sgx_epc_page *epc_page)
+{
+	unsigned long addr = SGX_ENCL_PAGE_ADDR(encl_page);
+	struct sgx_encl *encl = encl_page->encl;
+	struct sgx_pageinfo pginfo;
+	unsigned long pcmd_offset;
+	unsigned long va_offset;
+	void *secs_ptr = NULL;
+	pgoff_t backing_index;
+	struct page *backing;
+	struct page *pcmd;
+	void *epc_ptr;
+	void *va_ptr;
+	int ret;
+
+	backing_index = SGX_ENCL_PAGE_BACKING_INDEX(encl_page, encl);
+	pcmd_offset = SGX_ENCL_PAGE_PCMD_OFFSET(encl_page, encl);
+	va_offset = SGX_ENCL_PAGE_VA_OFFSET(encl_page);
+
+	backing = sgx_get_backing(encl->backing, backing_index);
+	if (IS_ERR(backing))
+		return PTR_ERR(backing);
+
+	pcmd = sgx_get_backing(encl->pcmd, backing_index >> 5);
+	if (IS_ERR(pcmd)) {
+		sgx_put_backing(backing, false);
+		return PTR_ERR(pcmd);
+	}
+
+	if (addr)
+		secs_ptr = sgx_get_page(encl->secs.epc_page);
+
+	epc_ptr = sgx_get_page(epc_page);
+	va_ptr = sgx_get_page(encl_page->va_page->epc_page);
+	pginfo.srcpge = (unsigned long)kmap_atomic(backing);
+	pginfo.pcmd = (unsigned long)kmap_atomic(pcmd) + pcmd_offset;
+	pginfo.linaddr = addr;
+	pginfo.secs = (unsigned long)secs_ptr;
+
+	ret = __eldu(&pginfo, epc_ptr, va_ptr + va_offset);
+	if (ret) {
+		sgx_err(encl, "ELDU returned %d\n", ret);
+		ret = -EFAULT;
+	}
+
+	kunmap_atomic((void *)(unsigned long)(pginfo.pcmd - pcmd_offset));
+	kunmap_atomic((void *)(unsigned long)pginfo.srcpge);
+	sgx_put_page(va_ptr);
+	sgx_put_page(epc_ptr);
+
+	if (addr)
+		sgx_put_page(secs_ptr);
+
+	sgx_put_backing(pcmd, false);
+	sgx_put_backing(backing, false);
+	return ret;
+}
+
+/**
+ * sgx_encl_release - destroy an enclave
+ *
+ * @kref:	address of the kref inside &sgx_encl
+ *
+ * Used togethere with kref_put(). Frees all the resources associated with
+ * the enclave.
+ */
+void sgx_encl_release(struct kref *ref)
+{
+	struct sgx_encl *encl = container_of(ref, struct sgx_encl, refcount);
+	struct sgx_encl_page *entry;
+	struct radix_tree_iter iter;
+	void **slot;
+
+	radix_tree_for_each_slot(slot, &encl->page_tree, &iter, 0) {
+		entry = *slot;
+		sgx_encl_free_page(entry);
+	}
+
+	put_pid(encl->tgid);
+
+	if (encl->mmu_notifier.ops)
+		mmu_notifier_unregister_no_release(&encl->mmu_notifier,
+						   encl->mm);
+
+	sgx_encl_free_va_pages(encl);
+
+	if (encl->secs.desc & SGX_ENCL_PAGE_LOADED)
+		sgx_free_page(encl->secs.epc_page);
+
+	if (encl->backing)
+		fput(encl->backing);
+
+	if (encl->pcmd)
+		fput(encl->pcmd);
+
+	kfree(encl);
+}
diff --git a/drivers/platform/x86/intel_sgx/sgx_encl_page.c b/drivers/platform/x86/intel_sgx/sgx_encl_page.c
new file mode 100644
index 000000000000..61bd2425640b
--- /dev/null
+++ b/drivers/platform/x86/intel_sgx/sgx_encl_page.c
@@ -0,0 +1,294 @@
+// SPDX-License-Identifier: (GPL-2.0 OR BSD-3-Clause)
+// Copyright(c) 2016-17 Intel Corporation.
+//
+// Authors:
+//
+// Jarkko Sakkinen <jarkko.sakkinen@linux.intel.com>
+// Suresh Siddha <suresh.b.siddha@intel.com>
+// Serge Ayoun <serge.ayoun@intel.com>
+// Shay Katz-zamir <shay.katz-zamir@intel.com>
+// Sean Christopherson <sean.j.christopherson@intel.com>
+
+#include <linux/device.h>
+#include <linux/freezer.h>
+#include <linux/highmem.h>
+#include <linux/kthread.h>
+#include <linux/ratelimit.h>
+#include <linux/sched/signal.h>
+#include <linux/slab.h>
+#include "sgx.h"
+
+static bool sgx_encl_page_get(struct sgx_epc_page *epc_page)
+{
+	struct sgx_encl_page *encl_page = container_of(epc_page->impl,
+						       struct sgx_encl_page,
+						       impl);
+	struct sgx_encl *encl = encl_page->encl;
+
+	return kref_get_unless_zero(&encl->refcount) != 0;
+}
+
+static void sgx_encl_page_put(struct sgx_epc_page *epc_page)
+{
+	struct sgx_encl_page *encl_page = container_of(epc_page->impl,
+						       struct sgx_encl_page,
+						       impl);
+	struct sgx_encl *encl = encl_page->encl;
+
+	kref_put(&encl->refcount, sgx_encl_release);
+}
+
+static bool sgx_encl_page_reclaim(struct sgx_epc_page *epc_page)
+{
+	struct sgx_encl_page *encl_page = container_of(epc_page->impl,
+						       struct sgx_encl_page,
+						       impl);
+	struct sgx_encl *encl = encl_page->encl;
+	bool ret = false;
+
+	down_read(&encl->mm->mmap_sem);
+	mutex_lock(&encl->lock);
+	if ((encl->flags & SGX_ENCL_DEAD) ||
+	    (!sgx_test_and_clear_young(encl_page) &&
+	     !(encl_page->desc & SGX_ENCL_PAGE_RESERVED))) {
+		encl_page->desc |= SGX_ENCL_PAGE_RESERVED;
+		ret = true;
+	}
+	mutex_unlock(&encl->lock);
+	up_read(&encl->mm->mmap_sem);
+	return ret;
+}
+
+static void sgx_encl_page_block(struct sgx_epc_page *epc_page)
+{
+	struct sgx_encl_page *encl_page = container_of(epc_page->impl,
+						       struct sgx_encl_page,
+						       impl);
+	struct sgx_encl *encl = encl_page->encl;
+
+	down_read(&encl->mm->mmap_sem);
+	mutex_lock(&encl->lock);
+	sgx_encl_block(encl_page);
+	mutex_unlock(&encl->lock);
+	up_read(&encl->mm->mmap_sem);
+}
+
+static int sgx_ewb(struct sgx_encl *encl, struct sgx_epc_page *epc_page,
+		   struct sgx_va_page *va_page, unsigned int va_offset)
+{
+	struct sgx_encl_page *encl_page = container_of(epc_page->impl,
+						       struct sgx_encl_page,
+						       impl);
+	unsigned long pcmd_offset = SGX_ENCL_PAGE_PCMD_OFFSET(encl_page, encl);
+	struct sgx_pageinfo pginfo;
+	pgoff_t backing_index;
+	struct page *backing;
+	struct page *pcmd;
+	void *epc;
+	void *va;
+	int ret;
+
+	backing_index = SGX_ENCL_PAGE_BACKING_INDEX(encl_page, encl);
+
+	backing = sgx_get_backing(encl->backing, backing_index);
+	if (IS_ERR(backing)) {
+		ret = PTR_ERR(backing);
+		return ret;
+	}
+
+	pcmd = sgx_get_backing(encl->pcmd, backing_index >> 5);
+	if (IS_ERR(pcmd)) {
+		ret = PTR_ERR(pcmd);
+		sgx_put_backing(backing, true);
+		return ret;
+	}
+
+	epc = sgx_get_page(epc_page);
+	va = sgx_get_page(va_page->epc_page);
+
+	pginfo.srcpge = (unsigned long)kmap_atomic(backing);
+	pginfo.pcmd = (unsigned long)kmap_atomic(pcmd) + pcmd_offset;
+	pginfo.linaddr = 0;
+	pginfo.secs = 0;
+	ret = __ewb(&pginfo, epc, (void *)((unsigned long)va + va_offset));
+	kunmap_atomic((void *)(unsigned long)(pginfo.pcmd - pcmd_offset));
+	kunmap_atomic((void *)(unsigned long)pginfo.srcpge);
+
+	sgx_put_page(va);
+	sgx_put_page(epc);
+	sgx_put_backing(pcmd, true);
+	sgx_put_backing(backing, true);
+
+	return ret;
+}
+
+/**
+ * sgx_write_page - write a page to the regular memory
+ *
+ * Writes an EPC page to the shmem file associated with the enclave. Flushes
+ * CPUs and retries if there are hardware threads that can potentially have TLB
+ * entries to the page (indicated by SGX_NOT_TRACKED). Clears the reserved flag
+ * after the page is swapped.
+ *
+ * @epc_page:	an EPC page
+ */
+static void sgx_write_page(struct sgx_epc_page *epc_page, bool do_free)
+{
+	struct sgx_encl_page *encl_page = container_of(epc_page->impl,
+						       struct sgx_encl_page,
+						       impl);
+	struct sgx_encl *encl = encl_page->encl;
+	struct sgx_va_page *va_page;
+	unsigned int va_offset;
+	int ret;
+
+	if (!(encl->flags & SGX_ENCL_DEAD)) {
+		va_page = list_first_entry(&encl->va_pages, struct sgx_va_page,
+					   list);
+		va_offset = sgx_alloc_va_slot(va_page);
+		if (sgx_va_page_full(va_page))
+			list_move_tail(&va_page->list, &encl->va_pages);
+
+		ret = sgx_ewb(encl, epc_page, va_page, va_offset);
+		if (ret == SGX_NOT_TRACKED) {
+			sgx_encl_track(encl);
+			ret = sgx_ewb(encl, epc_page, va_page, va_offset);
+			if (ret == SGX_NOT_TRACKED) {
+				/* slow path, IPI needed */
+				sgx_flush_cpus(encl);
+				ret = sgx_ewb(encl, epc_page, va_page,
+					      va_offset);
+			}
+		}
+		SGX_INVD(ret, encl, "EWB returned %d\n", ret);
+
+		encl_page->desc |= va_offset;
+		encl_page->va_page = va_page;
+		encl_page->desc &= ~SGX_ENCL_PAGE_RESERVED;
+	}
+	encl_page->desc &= ~SGX_ENCL_PAGE_LOADED;
+	if (do_free)
+		sgx_free_page(epc_page);
+}
+
+static void sgx_encl_page_write(struct sgx_epc_page *epc_page)
+{
+	struct sgx_encl_page *encl_page = container_of(epc_page->impl,
+						       struct sgx_encl_page,
+						       impl);
+	struct sgx_encl *encl = encl_page->encl;
+
+	down_read(&encl->mm->mmap_sem);
+	mutex_lock(&encl->lock);
+	sgx_write_page(epc_page, false);
+	encl->secs_child_cnt--;
+	if (!encl->secs_child_cnt && (encl->flags & SGX_ENCL_INITIALIZED))
+		sgx_write_page(encl->secs.epc_page, true);
+	mutex_unlock(&encl->lock);
+	up_read(&encl->mm->mmap_sem);
+}
+
+const struct sgx_epc_page_ops sgx_encl_page_ops = {
+	.get = sgx_encl_page_get,
+	.put = sgx_encl_page_put,
+	.reclaim = sgx_encl_page_reclaim,
+	.block = sgx_encl_page_block,
+	.write = sgx_encl_page_write,
+};
+
+/**
+ * sgx_set_page_reclaimable - associated an EPC page with an enclave page
+ * @encl_page:	an enclave page
+ * @epc_page:	the EPC page to attach to @encl_page
+ */
+void sgx_set_epc_page(struct sgx_encl_page *encl_page,
+		      struct sgx_epc_page *epc_page)
+{
+	encl_page->desc |= SGX_ENCL_PAGE_LOADED;
+	encl_page->epc_page = epc_page;
+}
+
+/**
+ * sgx_set_page_reclaimable - mark an EPC page reclaimable
+ * @encl_page:	an enclave page with a loaded EPC page
+ */
+void sgx_set_page_reclaimable(struct sgx_encl_page *encl_page)
+{
+	sgx_test_and_clear_young(encl_page);
+
+	spin_lock(&sgx_active_page_list_lock);
+	list_add_tail(&encl_page->epc_page->list, &sgx_active_page_list);
+	spin_unlock(&sgx_active_page_list_lock);
+}
+
+/**
+ * sgx_alloc_page - allocate a VA page
+ * @flags:	allocation flags
+ *
+ * Allocates an &sgx_epc_page instance and converts it to a VA page.
+ *
+ * Return:
+ *   a &struct sgx_va_page instance,
+ *   -errno otherwise
+ */
+struct sgx_epc_page *sgx_alloc_va_page(unsigned int flags)
+{
+	struct sgx_epc_page *epc_page;
+	int ret;
+
+	epc_page = sgx_alloc_page(NULL, flags);
+	if (IS_ERR(epc_page))
+		return (void *)epc_page;
+
+	ret = sgx_epa(epc_page);
+	if (ret) {
+		pr_crit("EPA failed\n");
+		sgx_free_page(epc_page);
+		return ERR_PTR(ret);
+	}
+
+	return epc_page;
+}
+
+/**
+ * sgx_alloc_va_slot - allocate a VA slot
+ * @va_page:	a &struct sgx_va_page instance
+ *
+ * Allocates a slot from a &struct sgx_va_page instance.
+ *
+ * Return: offset of the slot inside the VA page
+ */
+unsigned int sgx_alloc_va_slot(struct sgx_va_page *va_page)
+{
+	int slot = find_first_zero_bit(va_page->slots, SGX_VA_SLOT_COUNT);
+
+	if (slot < SGX_VA_SLOT_COUNT)
+		set_bit(slot, va_page->slots);
+
+	return slot << 3;
+}
+
+/**
+ * sgx_free_va_slot - free a VA slot
+ * @va_page:	a &struct sgx_va_page instance
+ * @offset:	offset of the slot inside the VA page
+ *
+ * Frees a slot from a &struct sgx_va_page instance.
+ */
+void sgx_free_va_slot(struct sgx_va_page *va_page, unsigned int offset)
+{
+	clear_bit(offset >> 3, va_page->slots);
+}
+
+/**
+ * sgx_va_page_full - is the VA page full?
+ * @va_page:	a &struct sgx_va_page instance
+ *
+ * Return: true if all slots have been taken
+ */
+bool sgx_va_page_full(struct sgx_va_page *va_page)
+{
+	int slot = find_first_zero_bit(va_page->slots, SGX_VA_SLOT_COUNT);
+
+	return slot == SGX_VA_SLOT_COUNT;
+}
diff --git a/drivers/platform/x86/intel_sgx/sgx_fault.c b/drivers/platform/x86/intel_sgx/sgx_fault.c
new file mode 100644
index 000000000000..e5b2786ce42b
--- /dev/null
+++ b/drivers/platform/x86/intel_sgx/sgx_fault.c
@@ -0,0 +1,159 @@
+// SPDX-License-Identifier: (GPL-2.0 OR BSD-3-Clause)
+// Copyright(c) 2016-17 Intel Corporation.
+//
+// Authors:
+//
+// Jarkko Sakkinen <jarkko.sakkinen@linux.intel.com>
+// Suresh Siddha <suresh.b.siddha@intel.com>
+// Serge Ayoun <serge.ayoun@intel.com>
+// Shay Katz-zamir <shay.katz-zamir@intel.com>
+// Sean Christopherson <sean.j.christopherson@intel.com>
+
+#include <linux/highmem.h>
+#include <linux/sched/mm.h>
+#include "sgx.h"
+
+static int sgx_test_and_clear_young_cb(pte_t *ptep, pgtable_t token,
+				       unsigned long addr, void *data)
+{
+	pte_t pte;
+	int ret;
+
+	ret = pte_young(*ptep);
+	if (ret) {
+		pte = pte_mkold(*ptep);
+		set_pte_at((struct mm_struct *)data, addr, ptep, pte);
+	}
+
+	return ret;
+}
+
+/**
+ * sgx_test_and_clear_young() - Test and reset the accessed bit
+ * @page:	enclave page to be tested for recent access
+ *
+ * Checks the Access (A) bit from the PTE corresponding to the
+ * enclave page and clears it.  Returns 1 if the page has been
+ * recently accessed and 0 if not.
+ */
+int sgx_test_and_clear_young(struct sgx_encl_page *page)
+{
+	unsigned long addr = SGX_ENCL_PAGE_ADDR(page);
+	struct sgx_encl *encl = page->encl;
+	struct vm_area_struct *vma;
+	int ret;
+
+	ret = sgx_encl_find(encl->mm, addr, &vma);
+	if (ret)
+		return 0;
+
+	if (encl != vma->vm_private_data)
+		return 0;
+
+	return apply_to_page_range(vma->vm_mm, addr, PAGE_SIZE,
+				   sgx_test_and_clear_young_cb, vma->vm_mm);
+}
+
+static void sgx_ipi_cb(void *info)
+{
+}
+
+void sgx_flush_cpus(struct sgx_encl *encl)
+{
+	on_each_cpu_mask(mm_cpumask(encl->mm), sgx_ipi_cb, NULL, 1);
+}
+
+static struct sgx_epc_page *__sgx_load_faulted_page(
+	struct sgx_encl_page *encl_page)
+{
+	unsigned long va_offset = SGX_ENCL_PAGE_VA_OFFSET(encl_page);
+	struct sgx_encl *encl = encl_page->encl;
+	struct sgx_epc_page *epc_page;
+	int ret;
+
+	epc_page = sgx_alloc_page(&encl_page->impl, SGX_ALLOC_ATOMIC);
+	if (IS_ERR(epc_page))
+		return epc_page;
+	ret = sgx_encl_load_page(encl_page, epc_page);
+	if (ret) {
+		sgx_free_page(epc_page);
+		return ERR_PTR(ret);
+	}
+	sgx_free_va_slot(encl_page->va_page, va_offset);
+	list_move(&encl_page->va_page->list, &encl->va_pages);
+	encl_page->desc &= ~SGX_VA_OFFSET_MASK;
+	sgx_set_epc_page(encl_page, epc_page);
+	return epc_page;
+}
+
+static struct sgx_encl_page *__sgx_fault_page(struct vm_area_struct *vma,
+					      unsigned long addr,
+					      bool do_reserve)
+{
+	struct sgx_encl *encl = vma->vm_private_data;
+	struct sgx_epc_page *epc_page;
+	struct sgx_encl_page *entry;
+	int rc = 0;
+
+	if ((encl->flags & SGX_ENCL_DEAD) ||
+	    !(encl->flags & SGX_ENCL_INITIALIZED))
+		return ERR_PTR(-EFAULT);
+
+	entry = radix_tree_lookup(&encl->page_tree, addr >> PAGE_SHIFT);
+	if (!entry)
+		return ERR_PTR(-EFAULT);
+	if (entry->desc & SGX_ENCL_PAGE_RESERVED) {
+		sgx_dbg(encl, "0x%p is reserved\n",
+			(void *)SGX_ENCL_PAGE_ADDR(entry));
+		return ERR_PTR(-EBUSY);
+	}
+	/* Page was faulted by another thread. */
+	if (entry->desc & SGX_ENCL_PAGE_LOADED) {
+		if (do_reserve)
+			entry->desc |= SGX_ENCL_PAGE_RESERVED;
+		return entry;
+	}
+
+	if (!(encl->secs.desc & SGX_ENCL_PAGE_LOADED)) {
+		epc_page = __sgx_load_faulted_page(&encl->secs);
+		if (IS_ERR(epc_page))
+			return (void *)epc_page;
+	}
+	epc_page = __sgx_load_faulted_page(entry);
+	if (IS_ERR(epc_page))
+		return (void *)epc_page;
+
+	encl->secs_child_cnt++;
+	sgx_set_page_reclaimable(entry);
+	if (do_reserve)
+		entry->desc |= SGX_ENCL_PAGE_RESERVED;
+
+	rc = vm_insert_pfn(vma, addr, SGX_EPC_PFN(entry->epc_page));
+	SGX_INVD(rc, encl, "%s: vm_insert_pfn() returned %d\n", __func__, rc);
+	if (rc)
+		return ERR_PTR(rc);
+
+	return entry;
+}
+
+struct sgx_encl_page *sgx_fault_page(struct vm_area_struct *vma,
+				     unsigned long addr, bool do_reserve)
+{
+	struct sgx_encl *encl = vma->vm_private_data;
+	struct sgx_encl_page *entry;
+
+	/* If process was forked, VMA is still there but vm_private_data is set
+	 * to NULL.
+	 */
+	if (!encl)
+		return ERR_PTR(-EFAULT);
+	do {
+		mutex_lock(&encl->lock);
+		entry = __sgx_fault_page(vma, addr, do_reserve);
+		mutex_unlock(&encl->lock);
+		if (!do_reserve)
+			break;
+	} while (PTR_ERR(entry) == -EBUSY);
+
+	return entry;
+}
diff --git a/drivers/platform/x86/intel_sgx/sgx_ioctl.c b/drivers/platform/x86/intel_sgx/sgx_ioctl.c
new file mode 100644
index 000000000000..68b04893d66a
--- /dev/null
+++ b/drivers/platform/x86/intel_sgx/sgx_ioctl.c
@@ -0,0 +1,233 @@
+// SPDX-License-Identifier: (GPL-2.0 OR BSD-3-Clause)
+// Copyright(c) 2016-17 Intel Corporation.
+//
+// Authors:
+//
+// Jarkko Sakkinen <jarkko.sakkinen@linux.intel.com>
+// Suresh Siddha <suresh.b.siddha@intel.com>
+// Serge Ayoun <serge.ayoun@intel.com>
+// Shay Katz-zamir <shay.katz-zamir@intel.com>
+// Sean Christopherson <sean.j.christopherson@intel.com>
+
+#include <asm/mman.h>
+#include <linux/delay.h>
+#include <linux/file.h>
+#include <linux/hashtable.h>
+#include <linux/highmem.h>
+#include <linux/ratelimit.h>
+#include <linux/sched/signal.h>
+#include <linux/slab.h>
+#include "sgx.h"
+
+static int sgx_encl_get(unsigned long addr, struct sgx_encl **encl)
+{
+	struct mm_struct *mm = current->mm;
+	struct vm_area_struct *vma;
+	int ret;
+
+	if (addr & (PAGE_SIZE - 1))
+		return -EINVAL;
+
+	down_read(&mm->mmap_sem);
+
+	ret = sgx_encl_find(mm, addr, &vma);
+	if (!ret) {
+		*encl = vma->vm_private_data;
+
+		if ((*encl)->flags & SGX_ENCL_SUSPEND)
+			ret = SGX_POWER_LOST_ENCLAVE;
+		else
+			kref_get(&(*encl)->refcount);
+	}
+
+	up_read(&mm->mmap_sem);
+	return ret;
+}
+
+/**
+ * sgx_ioc_enclave_create - handler for %SGX_IOC_ENCLAVE_CREATE
+ * @filep:	open file to /dev/sgx
+ * @cmd:	the command value
+ * @arg:	pointer to an &sgx_enclave_create instance
+ *
+ * Validates SECS attributes, allocates an EPC page for the SECS and performs
+ * ECREATE.
+ *
+ * Return:
+ *   0 on success,
+ *   -errno otherwise
+ */
+static long sgx_ioc_enclave_create(struct file *filep, unsigned int cmd,
+				   unsigned long arg)
+{
+	struct sgx_enclave_create *createp = (struct sgx_enclave_create *)arg;
+	struct sgx_secs *secs;
+	struct sgx_encl *encl;
+	int ret;
+
+	secs = kzalloc(sizeof(*secs),  GFP_KERNEL);
+	if (!secs)
+		return -ENOMEM;
+
+	ret = copy_from_user(secs, (void __user *)createp->src, sizeof(*secs));
+	if (ret)
+		goto out;
+
+	encl = sgx_encl_alloc(secs);
+	if (IS_ERR(encl)) {
+		ret = PTR_ERR(encl);
+		goto out;
+	}
+
+	ret = sgx_encl_create(encl, secs);
+	if (ret)
+		kref_put(&encl->refcount, sgx_encl_release);
+
+out:
+	kfree(secs);
+	return ret;
+}
+
+/**
+ * sgx_ioc_enclave_add_page - handler for %SGX_IOC_ENCLAVE_ADD_PAGE
+ *
+ * @filep:	open file to /dev/sgx
+ * @cmd:	the command value
+ * @arg:	pointer to an &sgx_enclave_add_page instance
+ *
+ * Creates a new enclave page and enqueues an EADD operation that will be
+ * processed by a worker thread later on.
+ *
+ * Return:
+ *   0 on success,
+ *   -errno otherwise
+ */
+static long sgx_ioc_enclave_add_page(struct file *filep, unsigned int cmd,
+				     unsigned long arg)
+{
+	struct sgx_enclave_add_page *addp = (void *)arg;
+	struct sgx_secinfo secinfo;
+	struct sgx_encl *encl;
+	struct page *data_page;
+	void *data;
+	int ret;
+
+	ret = sgx_encl_get(addp->addr, &encl);
+	if (ret)
+		return ret;
+
+	if (copy_from_user(&secinfo, (void __user *)addp->secinfo,
+			   sizeof(secinfo))) {
+		kref_put(&encl->refcount, sgx_encl_release);
+		return -EFAULT;
+	}
+
+	data_page = alloc_page(GFP_HIGHUSER);
+	if (!data_page) {
+		kref_put(&encl->refcount, sgx_encl_release);
+		return -ENOMEM;
+	}
+
+	data = kmap(data_page);
+
+	ret = copy_from_user((void *)data, (void __user *)addp->src, PAGE_SIZE);
+	if (ret)
+		goto out;
+
+	ret = sgx_encl_add_page(encl, addp->addr, data, &secinfo, addp->mrmask);
+	if (ret)
+		goto out;
+
+out:
+	kref_put(&encl->refcount, sgx_encl_release);
+	kunmap(data_page);
+	__free_page(data_page);
+	return ret;
+}
+
+/**
+ * sgx_ioc_enclave_init - handler for %SGX_IOC_ENCLAVE_INIT
+ *
+ * @filep:	open file to /dev/sgx
+ * @cmd:	the command value
+ * @arg:	pointer to an &sgx_enclave_init instance
+ *
+ * Flushes the remaining enqueued EADD operations and performs EINIT.
+ *
+ * Return:
+ *   0 on success,
+ *   SGX error code on EINIT failure,
+ *   -errno otherwise
+ */
+static long sgx_ioc_enclave_init(struct file *filep, unsigned int cmd,
+				 unsigned long arg)
+{
+	struct sgx_enclave_init *initp = (struct sgx_enclave_init *)arg;
+	struct sgx_sigstruct *sigstruct;
+	struct sgx_einittoken *einittoken;
+	struct sgx_encl *encl;
+	struct page *initp_page;
+	int ret;
+
+	initp_page = alloc_page(GFP_HIGHUSER);
+	if (!initp_page)
+		return -ENOMEM;
+
+	sigstruct = kmap(initp_page);
+	einittoken = (struct sgx_einittoken *)
+		((unsigned long)sigstruct + PAGE_SIZE / 2);
+
+	ret = copy_from_user(sigstruct, (void __user *)initp->sigstruct,
+			     sizeof(*sigstruct));
+	if (ret)
+		goto out;
+
+	ret = sgx_encl_get(initp->addr, &encl);
+	if (ret)
+		goto out;
+
+	ret = sgx_encl_init(encl, sigstruct, einittoken);
+
+	kref_put(&encl->refcount, sgx_encl_release);
+
+out:
+	kunmap(initp_page);
+	__free_page(initp_page);
+	return ret;
+}
+
+typedef long (*sgx_ioc_t)(struct file *filep, unsigned int cmd,
+			  unsigned long arg);
+
+long sgx_ioctl(struct file *filep, unsigned int cmd, unsigned long arg)
+{
+	char data[256];
+	sgx_ioc_t handler = NULL;
+	long ret;
+
+	switch (cmd) {
+	case SGX_IOC_ENCLAVE_CREATE:
+		handler = sgx_ioc_enclave_create;
+		break;
+	case SGX_IOC_ENCLAVE_ADD_PAGE:
+		handler = sgx_ioc_enclave_add_page;
+		break;
+	case SGX_IOC_ENCLAVE_INIT:
+		handler = sgx_ioc_enclave_init;
+		break;
+	default:
+		return -ENOIOCTLCMD;
+	}
+
+	if (copy_from_user(data, (void __user *)arg, _IOC_SIZE(cmd)))
+		return -EFAULT;
+
+	ret = handler(filep, cmd, (unsigned long)((void *)data));
+	if (!ret && (cmd & IOC_OUT)) {
+		if (copy_to_user((void __user *)arg, data, _IOC_SIZE(cmd)))
+			return -EFAULT;
+	}
+	if (IS_ENCLS_FAULT(ret))
+		return -EFAULT;
+	return ret;
+}
diff --git a/drivers/platform/x86/intel_sgx/sgx_main.c b/drivers/platform/x86/intel_sgx/sgx_main.c
new file mode 100644
index 000000000000..79d772847026
--- /dev/null
+++ b/drivers/platform/x86/intel_sgx/sgx_main.c
@@ -0,0 +1,306 @@
+// SPDX-License-Identifier: (GPL-2.0 OR BSD-3-Clause)
+// Copyright(c) 2016-17 Intel Corporation.
+//
+// Authors:
+//
+// Jarkko Sakkinen <jarkko.sakkinen@linux.intel.com>
+// Suresh Siddha <suresh.b.siddha@intel.com>
+// Serge Ayoun <serge.ayoun@intel.com>
+// Shay Katz-zamir <shay.katz-zamir@intel.com>
+// Sean Christopherson <sean.j.christopherson@intel.com>
+
+#include <linux/acpi.h>
+#include <linux/cdev.h>
+#include <linux/platform_device.h>
+#include <linux/suspend.h>
+#include "sgx.h"
+
+#define DRV_DESCRIPTION "Intel SGX Driver"
+
+MODULE_DESCRIPTION("Intel SGX Driver");
+MODULE_AUTHOR("Jarkko Sakkinen <jarkko.sakkinen@linux.intel.com>");
+MODULE_LICENSE("Dual BSD/GPL");
+
+/*
+ * Global data.
+ */
+
+struct workqueue_struct *sgx_add_page_wq;
+u64 sgx_encl_size_max_32;
+u64 sgx_encl_size_max_64;
+u64 sgx_xfrm_mask = 0x3;
+u32 sgx_misc_reserved;
+u32 sgx_xsave_size_tbl[64];
+
+#ifdef CONFIG_COMPAT
+long sgx_compat_ioctl(struct file *filep, unsigned int cmd, unsigned long arg)
+{
+	return sgx_ioctl(filep, cmd, arg);
+}
+#endif
+
+static int sgx_mmap(struct file *file, struct vm_area_struct *vma)
+{
+	vma->vm_ops = &sgx_vm_ops;
+	vma->vm_flags |= VM_PFNMAP | VM_DONTEXPAND | VM_DONTDUMP | VM_IO |
+			 VM_DONTCOPY;
+
+	return 0;
+}
+
+static unsigned long sgx_get_unmapped_area(struct file *file,
+					   unsigned long addr,
+					   unsigned long len,
+					   unsigned long pgoff,
+					   unsigned long flags)
+{
+	if (len < 2 * PAGE_SIZE || (len & (len - 1)))
+		return -EINVAL;
+
+	/* On 64-bit architecture, allow mmap() to exceed 32-bit encl
+	 * limit only if the task is not running in 32-bit compatibility
+	 * mode.
+	 */
+	if (len > sgx_encl_size_max_32)
+#ifdef CONFIG_X86_64
+		if (test_thread_flag(TIF_ADDR32))
+			return -EINVAL;
+#else
+		return -EINVAL;
+#endif
+
+#ifdef CONFIG_X86_64
+	if (len > sgx_encl_size_max_64)
+		return -EINVAL;
+#endif
+
+	addr = current->mm->get_unmapped_area(file, addr, 2 * len, pgoff,
+					      flags);
+	if (IS_ERR_VALUE(addr))
+		return addr;
+
+	addr = (addr + (len - 1)) & ~(len - 1);
+
+	return addr;
+}
+
+static const struct file_operations sgx_fops = {
+	.owner			= THIS_MODULE,
+	.unlocked_ioctl		= sgx_ioctl,
+#ifdef CONFIG_COMPAT
+	.compat_ioctl		= sgx_compat_ioctl,
+#endif
+	.mmap			= sgx_mmap,
+	.get_unmapped_area	= sgx_get_unmapped_area,
+};
+
+static int sgx_pm_suspend(struct device *dev)
+{
+	struct sgx_encl_page *encl_page;
+	struct sgx_epc_page *epc_page;
+	struct sgx_encl *encl;
+
+	list_for_each_entry(epc_page, &sgx_active_page_list, list) {
+		encl_page = container_of(epc_page->impl, struct sgx_encl_page,
+					 impl);
+		encl = encl_page->encl;
+		sgx_invalidate(encl, false);
+		encl->flags |= SGX_ENCL_SUSPEND;
+		flush_work(&encl->add_page_work);
+	}
+
+	return 0;
+}
+
+static SIMPLE_DEV_PM_OPS(sgx_drv_pm, sgx_pm_suspend, NULL);
+
+static struct bus_type sgx_bus_type = {
+	.name	= "sgx",
+};
+
+struct sgx_context {
+	struct device dev;
+	struct cdev cdev;
+};
+
+static dev_t sgx_devt;
+
+static void sgx_dev_release(struct device *dev)
+{
+	struct sgx_context *ctx = container_of(dev, struct sgx_context, dev);
+
+	kfree(ctx);
+}
+
+static struct sgx_context *sgx_ctx_alloc(struct device *parent)
+{
+	struct sgx_context *ctx;
+
+	ctx = kzalloc(sizeof(*ctx), GFP_KERNEL);
+	if (!ctx)
+		return ERR_PTR(-ENOMEM);
+
+	device_initialize(&ctx->dev);
+
+	ctx->dev.bus = &sgx_bus_type;
+	ctx->dev.parent = parent;
+	ctx->dev.devt = MKDEV(MAJOR(sgx_devt), 0);
+	ctx->dev.release = sgx_dev_release;
+
+	dev_set_name(&ctx->dev, "sgx");
+
+	cdev_init(&ctx->cdev, &sgx_fops);
+	ctx->cdev.owner = THIS_MODULE;
+
+	dev_set_drvdata(parent, ctx);
+
+	return ctx;
+}
+
+static struct sgx_context *sgxm_ctx_alloc(struct device *parent)
+{
+	struct sgx_context *ctx;
+	int rc;
+
+	ctx = sgx_ctx_alloc(parent);
+	if (IS_ERR(ctx))
+		return ctx;
+
+	rc = devm_add_action_or_reset(parent, (void (*)(void *))put_device,
+				      &ctx->dev);
+	if (rc) {
+		kfree(ctx);
+		return ERR_PTR(rc);
+	}
+
+	return ctx;
+}
+
+static int sgx_dev_init(struct device *parent)
+{
+	struct sgx_context *sgx_dev;
+	unsigned int eax;
+	unsigned int ebx;
+	unsigned int ecx;
+	unsigned int edx;
+	int ret;
+	int i;
+
+	sgx_dev = sgxm_ctx_alloc(parent);
+
+	cpuid_count(SGX_CPUID, SGX_CPUID_CAPABILITIES, &eax, &ebx, &ecx, &edx);
+	/* Only allow misc bits supported by the driver. */
+	sgx_misc_reserved = ~ebx | SGX_MISC_RESERVED_MASK;
+#ifdef CONFIG_X86_64
+	sgx_encl_size_max_64 = 1ULL << ((edx >> 8) & 0xFF);
+#endif
+	sgx_encl_size_max_32 = 1ULL << (edx & 0xFF);
+
+	if (boot_cpu_has(X86_FEATURE_OSXSAVE)) {
+		cpuid_count(SGX_CPUID, SGX_CPUID_ATTRIBUTES, &eax, &ebx, &ecx,
+			    &edx);
+		sgx_xfrm_mask = (((u64)edx) << 32) + (u64)ecx;
+
+		for (i = 2; i < 64; i++) {
+			cpuid_count(0x0D, i, &eax, &ebx, &ecx, &edx);
+			if ((1 << i) & sgx_xfrm_mask)
+				sgx_xsave_size_tbl[i] = eax + ebx;
+		}
+	}
+
+	sgx_add_page_wq = alloc_workqueue("intel_sgx-add-page-wq",
+					  WQ_UNBOUND | WQ_FREEZABLE, 1);
+	if (!sgx_add_page_wq)
+		return -ENOMEM;
+
+	ret = cdev_device_add(&sgx_dev->cdev, &sgx_dev->dev);
+	if (ret)
+		goto out_workqueue;
+
+	return 0;
+out_workqueue:
+	destroy_workqueue(sgx_add_page_wq);
+	return ret;
+}
+
+static int sgx_drv_probe(struct platform_device *pdev)
+{
+	if (!sgx_enabled || !boot_cpu_has(X86_FEATURE_SGX_LC))
+		return -ENODEV;
+
+	return sgx_dev_init(&pdev->dev);
+}
+
+static int sgx_drv_remove(struct platform_device *pdev)
+{
+	struct sgx_context *ctx = dev_get_drvdata(&pdev->dev);
+
+	cdev_device_del(&ctx->cdev, &ctx->dev);
+	destroy_workqueue(sgx_add_page_wq);
+
+	return 0;
+}
+
+#ifdef CONFIG_ACPI
+static struct acpi_device_id sgx_device_ids[] = {
+	{"INT0E0C", 0},
+	{"", 0},
+};
+MODULE_DEVICE_TABLE(acpi, sgx_device_ids);
+#endif
+
+static struct platform_driver sgx_drv = {
+	.probe = sgx_drv_probe,
+	.remove = sgx_drv_remove,
+	.driver = {
+		.name			= "intel_sgx",
+		.pm			= &sgx_drv_pm,
+		.acpi_match_table	= ACPI_PTR(sgx_device_ids),
+	},
+};
+
+static int __init sgx_drv_subsys_init(void)
+{
+	int ret;
+
+	ret = bus_register(&sgx_bus_type);
+	if (ret)
+		return ret;
+
+	ret = alloc_chrdev_region(&sgx_devt, 0, 1, "sgx");
+	if (ret < 0) {
+		bus_unregister(&sgx_bus_type);
+		return ret;
+	}
+
+	return 0;
+}
+
+static void sgx_drv_subsys_exit(void)
+{
+	bus_unregister(&sgx_bus_type);
+	unregister_chrdev_region(sgx_devt, 1);
+}
+
+static int __init sgx_drv_init(void)
+{
+	int ret;
+
+	ret = sgx_drv_subsys_init();
+	if (ret)
+		return ret;
+
+	ret = platform_driver_register(&sgx_drv);
+	if (ret)
+		sgx_drv_subsys_exit();
+
+	return ret;
+}
+module_init(sgx_drv_init);
+
+static void __exit sgx_drv_exit(void)
+{
+	platform_driver_unregister(&sgx_drv);
+	sgx_drv_subsys_exit();
+}
+module_exit(sgx_drv_exit);
diff --git a/drivers/platform/x86/intel_sgx/sgx_vma.c b/drivers/platform/x86/intel_sgx/sgx_vma.c
new file mode 100644
index 000000000000..ad47383ea7f5
--- /dev/null
+++ b/drivers/platform/x86/intel_sgx/sgx_vma.c
@@ -0,0 +1,66 @@
+// SPDX-License-Identifier: (GPL-2.0 OR BSD-3-Clause)
+// Copyright(c) 2016-17 Intel Corporation.
+//
+// Authors:
+//
+// Jarkko Sakkinen <jarkko.sakkinen@linux.intel.com>
+// Suresh Siddha <suresh.b.siddha@intel.com>
+// Serge Ayoun <serge.ayoun@intel.com>
+// Shay Katz-zamir <shay.katz-zamir@intel.com>
+// Sean Christopherson <sean.j.christopherson@intel.com>
+
+#include <asm/mman.h>
+#include <linux/delay.h>
+#include <linux/file.h>
+#include <linux/hashtable.h>
+#include <linux/highmem.h>
+#include <linux/mm.h>
+#include <linux/ratelimit.h>
+#include <linux/slab.h>
+#include "sgx.h"
+
+static void sgx_vma_open(struct vm_area_struct *vma)
+{
+	struct sgx_encl *encl = vma->vm_private_data;
+
+	if (!encl)
+		return;
+
+	/* kref cannot underflow because ECREATE ioctl checks that there is only
+	 * one single VMA for the enclave before proceeding.
+	 */
+	kref_get(&encl->refcount);
+}
+
+static void sgx_vma_close(struct vm_area_struct *vma)
+{
+	struct sgx_encl *encl = vma->vm_private_data;
+
+	if (!encl)
+		return;
+
+	mutex_lock(&encl->lock);
+	sgx_invalidate(encl, true);
+	mutex_unlock(&encl->lock);
+	kref_put(&encl->refcount, sgx_encl_release);
+}
+
+static int sgx_vma_fault(struct vm_fault *vmf)
+{
+	unsigned long addr = (unsigned long)vmf->address;
+	struct vm_area_struct *vma = vmf->vma;
+	struct sgx_encl_page *entry;
+
+	entry = sgx_fault_page(vma, addr, 0);
+
+	if (!IS_ERR(entry) || PTR_ERR(entry) == -EBUSY)
+		return VM_FAULT_NOPAGE;
+	else
+		return VM_FAULT_SIGBUS;
+}
+
+const struct vm_operations_struct sgx_vm_ops = {
+	.close = sgx_vma_close,
+	.open = sgx_vma_open,
+	.fault = sgx_vma_fault,
+};
-- 
2.17.0

[PATCH v11 10/13] intel_sgx: driver for Intel Software Guard Extensions

[Jarkko Sakkinen]

Intel SGX is a set of CPU instructions that can be used by applications to
set aside private regions of code and data.  The code outside the enclave
is disallowed to access the memory inside the enclave by the CPU access
control.

SGX driver provides a ioctl API for loading and initializing enclaves.
Address range for enclaves is reserved with mmap() and they are destroyed
with munmap(). Enclave construction, measurement and initialization is done
with the provided the ioctl API.

Signed-off-by: Jarkko Sakkinen <jarkko.sakkinen@linux.intel.com>
Tested-by: Serge Ayoun <serge.ayoun@intel.com>
---
 arch/x86/include/asm/sgx_pr.h                 |   16 +
 arch/x86/include/uapi/asm/sgx.h               |  138 +++
 drivers/platform/x86/Kconfig                  |    2 +
 drivers/platform/x86/Makefile                 |    1 +
 drivers/platform/x86/intel_sgx/Kconfig        |   20 +
 drivers/platform/x86/intel_sgx/Makefile       |   13 +
 drivers/platform/x86/intel_sgx/sgx.h          |  164 +++
 drivers/platform/x86/intel_sgx/sgx_encl.c     | 1002 +++++++++++++++++
 .../platform/x86/intel_sgx/sgx_encl_page.c    |  294 +++++
 drivers/platform/x86/intel_sgx/sgx_fault.c    |  159 +++
 drivers/platform/x86/intel_sgx/sgx_ioctl.c    |  233 ++++
 drivers/platform/x86/intel_sgx/sgx_main.c     |  306 +++++
 drivers/platform/x86/intel_sgx/sgx_vma.c      |   66 ++
 13 files changed, 2414 insertions(+)
 create mode 100644 arch/x86/include/uapi/asm/sgx.h
 create mode 100644 drivers/platform/x86/intel_sgx/Kconfig
 create mode 100644 drivers/platform/x86/intel_sgx/Makefile
 create mode 100644 drivers/platform/x86/intel_sgx/sgx.h
 create mode 100644 drivers/platform/x86/intel_sgx/sgx_encl.c
 create mode 100644 drivers/platform/x86/intel_sgx/sgx_encl_page.c
 create mode 100644 drivers/platform/x86/intel_sgx/sgx_fault.c
 create mode 100644 drivers/platform/x86/intel_sgx/sgx_ioctl.c
 create mode 100644 drivers/platform/x86/intel_sgx/sgx_main.c
 create mode 100644 drivers/platform/x86/intel_sgx/sgx_vma.c

diff --git a/arch/x86/include/asm/sgx_pr.h b/arch/x86/include/asm/sgx_pr.h
index 876dc44c2ccc..9d7c3cc1ea1b 100644
--- a/arch/x86/include/asm/sgx_pr.h
+++ b/arch/x86/include/asm/sgx_pr.h
@@ -17,4 +17,20 @@
 #undef pr_fmt
 #define pr_fmt(fmt) "intel_sgx: " fmt
 
+#define sgx_pr_ratelimited(level, encl, fmt, ...)			\
+	pr_ ## level ## _ratelimited("[%d:0x%p] " fmt,			\
+				     pid_nr((encl)->tgid),		\
+				     (void *)(encl)->base, ##__VA_ARGS__)
+
+#define sgx_dbg(encl, fmt, ...) \
+	sgx_pr_ratelimited(debug, encl, fmt, ##__VA_ARGS__)
+#define sgx_info(encl, fmt, ...) \
+	sgx_pr_ratelimited(info, encl, fmt, ##__VA_ARGS__)
+#define sgx_warn(encl, fmt, ...) \
+	sgx_pr_ratelimited(warn, encl, fmt, ##__VA_ARGS__)
+#define sgx_err(encl, fmt, ...) \
+	sgx_pr_ratelimited(err, encl, fmt, ##__VA_ARGS__)
+#define sgx_crit(encl, fmt, ...) \
+	sgx_pr_ratelimited(crit, encl, fmt, ##__VA_ARGS__)
+
 #endif /* _ASM_X86_SGX_PR_H */
diff --git a/arch/x86/include/uapi/asm/sgx.h b/arch/x86/include/uapi/asm/sgx.h
new file mode 100644
index 000000000000..9bd8907efdaf
--- /dev/null
+++ b/arch/x86/include/uapi/asm/sgx.h
@@ -0,0 +1,138 @@
+/*
+ * This file is provided under a dual BSD/GPLv2 license.  When using or
+ * redistributing this file, you may do so under either license.
+ *
+ * GPL LICENSE SUMMARY
+ *
+ * Copyright(c) 2016-2017 Intel Corporation.
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of version 2 of the GNU General Public License as
+ * published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful, but
+ * WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+ * General Public License for more details.
+ *
+ * Contact Information:
+ * Jarkko Sakkinen <jarkko.sakkinen@linux.intel.com>
+ * Intel Finland Oy - BIC 0357606-4 - Westendinkatu 7, 02160 Espoo
+ *
+ * BSD LICENSE
+ *
+ * Copyright(c) 2016-2017 Intel Corporation.
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions
+ * are met:
+ *
+ *   * Redistributions of source code must retain the above copyright
+ *     notice, this list of conditions and the following disclaimer.
+ *   * Redistributions in binary form must reproduce the above copyright
+ *     notice, this list of conditions and the following disclaimer in
+ *     the documentation and/or other materials provided with the
+ *     distribution.
+ *   * Neither the name of Intel Corporation nor the names of its
+ *     contributors may be used to endorse or promote products derived
+ *     from this software without specific prior written permission.
+ *
+ * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
+ * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
+ * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
+ * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
+ * OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+ * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+ * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+ * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
+ * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+ * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+ * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+ *
+ * Authors:
+ *
+ * Jarkko Sakkinen <jarkko.sakkinen@linux.intel.com>
+ * Suresh Siddha <suresh.b.siddha@intel.com>
+ */
+
+#ifndef _UAPI_ASM_X86_SGX_H
+#define _UAPI_ASM_X86_SGX_H
+
+#include <linux/types.h>
+#include <linux/ioctl.h>
+
+#define SGX_MAGIC 0xA4
+
+#define SGX_IOC_ENCLAVE_CREATE \
+	_IOW(SGX_MAGIC, 0x00, struct sgx_enclave_create)
+#define SGX_IOC_ENCLAVE_ADD_PAGE \
+	_IOW(SGX_MAGIC, 0x01, struct sgx_enclave_add_page)
+#define SGX_IOC_ENCLAVE_INIT \
+	_IOW(SGX_MAGIC, 0x02, struct sgx_enclave_init)
+
+/* SGX leaf instruction return values */
+#define SGX_SUCCESS			0
+#define SGX_INVALID_SIG_STRUCT		1
+#define SGX_INVALID_ATTRIBUTE		2
+#define SGX_BLKSTATE			3
+#define SGX_INVALID_MEASUREMENT		4
+#define SGX_NOTBLOCKABLE		5
+#define SGX_PG_INVLD			6
+#define SGX_LOCKFAIL			7
+#define SGX_INVALID_SIGNATURE		8
+#define SGX_MAC_COMPARE_FAIL		9
+#define SGX_PAGE_NOT_BLOCKED		10
+#define SGX_NOT_TRACKED			11
+#define SGX_VA_SLOT_OCCUPIED		12
+#define SGX_CHILD_PRESENT		13
+#define SGX_ENCLAVE_ACT			14
+#define SGX_ENTRYEPOCH_LOCKED		15
+#define SGX_INVALID_EINITTOKEN		16
+#define SGX_PREV_TRK_INCMPL		17
+#define SGX_PG_IS_SECS			18
+#define SGX_INVALID_CPUSVN		32
+#define SGX_INVALID_ISVSVN		64
+#define SGX_UNMASKED_EVENT		128
+#define SGX_INVALID_KEYNAME		256
+
+/* IOCTL return values */
+#define SGX_POWER_LOST_ENCLAVE		0x40000000
+#define SGX_LE_ROLLBACK			0x40000001
+
+/**
+ * struct sgx_enclave_create - parameter structure for the
+ *                             %SGX_IOC_ENCLAVE_CREATE ioctl
+ * @src:	address for the SECS page data
+ */
+struct sgx_enclave_create  {
+	__u64	src;
+};
+
+/**
+ * struct sgx_enclave_add_page - parameter structure for the
+ *                               %SGX_IOC_ENCLAVE_ADD_PAGE ioctl
+ * @addr:	address within the ELRANGE
+ * @src:	address for the page data
+ * @secinfo:	address for the SECINFO data
+ * @mrmask:	bitmask for the measured 256 byte chunks
+ */
+struct sgx_enclave_add_page {
+	__u64	addr;
+	__u64	src;
+	__u64	secinfo;
+	__u16	mrmask;
+} __attribute__((__packed__));
+
+
+/**
+ * struct sgx_enclave_init - parameter structure for the
+ *                           %SGX_IOC_ENCLAVE_INIT ioctl
+ * @addr:	address within the ELRANGE
+ * @sigstruct:	address for the SIGSTRUCT data
+ */
+struct sgx_enclave_init {
+	__u64	addr;
+	__u64	sigstruct;
+};
+
+#endif /* _UAPI_ASM_X86_SGX_H */
diff --git a/drivers/platform/x86/Kconfig b/drivers/platform/x86/Kconfig
index 566644bb496a..645888c2049d 100644
--- a/drivers/platform/x86/Kconfig
+++ b/drivers/platform/x86/Kconfig
@@ -1218,6 +1218,8 @@ config INTEL_CHTDC_TI_PWRBTN
 	  To compile this driver as a module, choose M here: the module
 	  will be called intel_chtdc_ti_pwrbtn.
 
+source "drivers/platform/x86/intel_sgx/Kconfig"
+
 endif # X86_PLATFORM_DEVICES
 
 config PMC_ATOM
diff --git a/drivers/platform/x86/Makefile b/drivers/platform/x86/Makefile
index 2ba6cb795338..729208a9c958 100644
--- a/drivers/platform/x86/Makefile
+++ b/drivers/platform/x86/Makefile
@@ -91,3 +91,4 @@ obj-$(CONFIG_PMC_ATOM)		+= pmc_atom.o
 obj-$(CONFIG_MLX_PLATFORM)	+= mlx-platform.o
 obj-$(CONFIG_INTEL_TURBO_MAX_3) += intel_turbo_max_3.o
 obj-$(CONFIG_INTEL_CHTDC_TI_PWRBTN)	+= intel_chtdc_ti_pwrbtn.o
+obj-$(CONFIG_INTEL_SGX) += intel_sgx/
diff --git a/drivers/platform/x86/intel_sgx/Kconfig b/drivers/platform/x86/intel_sgx/Kconfig
new file mode 100644
index 000000000000..4cd88310617e
--- /dev/null
+++ b/drivers/platform/x86/intel_sgx/Kconfig
@@ -0,0 +1,20 @@
+#
+# Intel SGX
+#
+
+config INTEL_SGX
+	tristate "Intel(R) SGX Driver"
+	default n
+	depends on X86_64 && CPU_SUP_INTEL
+	select INTEL_SGX_CORE
+	select MMU_NOTIFIER
+	help
+	Intel(R) SGX is a set of CPU instructions that can be used by
+	applications to set aside private regions of code and data.  The code
+	outside the enclave is disallowed to access the memory inside the
+	enclave by the CPU access control.
+
+	The firmware uses PRMRR registers to reserve an area of physical memory
+	called Enclave Page Cache (EPC). There is a hardware unit in the
+	processor called Memory Encryption Engine. The MEE encrypts and decrypts
+	the EPC pages as they enter and leave the processor package.
diff --git a/drivers/platform/x86/intel_sgx/Makefile b/drivers/platform/x86/intel_sgx/Makefile
new file mode 100644
index 000000000000..95f254e30a8b
--- /dev/null
+++ b/drivers/platform/x86/intel_sgx/Makefile
@@ -0,0 +1,13 @@
+#
+# Intel SGX
+#
+
+obj-$(CONFIG_INTEL_SGX) += intel_sgx.o
+
+intel_sgx-$(CONFIG_INTEL_SGX) += \
+	sgx_ioctl.o \
+	sgx_encl.o \
+	sgx_encl_page.o \
+	sgx_main.o \
+	sgx_fault.o \
+	sgx_vma.o \
diff --git a/drivers/platform/x86/intel_sgx/sgx.h b/drivers/platform/x86/intel_sgx/sgx.h
new file mode 100644
index 000000000000..193f5e5b2338
--- /dev/null
+++ b/drivers/platform/x86/intel_sgx/sgx.h
@@ -0,0 +1,164 @@
+// SPDX-License-Identifier: (GPL-2.0 OR BSD-3-Clause)
+// Copyright(c) 2016-17 Intel Corporation.
+//
+// Authors:
+//
+// Jarkko Sakkinen <jarkko.sakkinen@linux.intel.com>
+// Suresh Siddha <suresh.b.siddha@intel.com>
+// Serge Ayoun <serge.ayoun@intel.com>
+// Shay Katz-zamir <shay.katz-zamir@intel.com>
+
+#ifndef __ARCH_INTEL_SGX_H__
+#define __ARCH_INTEL_SGX_H__
+
+#include <crypto/hash.h>
+#include <linux/kref.h>
+#include <linux/mmu_notifier.h>
+#include <linux/mmu_notifier.h>
+#include <linux/radix-tree.h>
+#include <linux/radix-tree.h>
+#include <linux/rbtree.h>
+#include <linux/rwsem.h>
+#include <linux/sched.h>
+#include <linux/workqueue.h>
+#include <asm/sgx.h>
+#include <asm/sgx_pr.h>
+#include <uapi/asm/sgx.h>
+
+#define SGX_EINIT_SPIN_COUNT	20
+#define SGX_EINIT_SLEEP_COUNT	50
+#define SGX_EINIT_SLEEP_TIME	20
+
+#define SGX_VA_SLOT_COUNT 512
+#define SGX_VA_OFFSET_MASK ((SGX_VA_SLOT_COUNT - 1) << 3)
+
+struct sgx_va_page {
+	struct sgx_epc_page *epc_page;
+	DECLARE_BITMAP(slots, SGX_VA_SLOT_COUNT);
+	struct list_head list;
+};
+
+enum sgx_encl_page_flags {
+	SGX_ENCL_PAGE_TCS	= BIT(0),
+	SGX_ENCL_PAGE_RESERVED	= BIT(1),
+	SGX_ENCL_PAGE_LOADED	= BIT(2),
+};
+
+#define SGX_ENCL_PAGE_ADDR(encl_page) ((encl_page)->desc & PAGE_MASK)
+#define SGX_ENCL_PAGE_VA_OFFSET(encl_page) \
+	((encl_page)->desc & SGX_VA_OFFSET_MASK)
+#define SGX_ENCL_PAGE_BACKING_INDEX(encl_page, encl)		\
+({								\
+	pgoff_t index;						\
+	if (!PFN_DOWN(encl_page->desc))				\
+		index = PFN_DOWN(encl->size);			\
+	else							\
+		index = PFN_DOWN(encl_page->desc - encl->base);	\
+	index;							\
+})
+#define SGX_ENCL_PAGE_PCMD_OFFSET(encl_page, encl)		\
+({								\
+	unsigned long ret;					\
+	ret = SGX_ENCL_PAGE_BACKING_INDEX(encl_page, encl);	\
+	((ret & 31) * 128);					\
+})
+
+struct sgx_encl_page {
+	unsigned long desc;
+	union {
+		struct sgx_epc_page *epc_page;
+		struct sgx_va_page *va_page;
+	};
+	struct sgx_encl *encl;
+	struct sgx_epc_page_impl impl;
+};
+
+enum sgx_encl_flags {
+	SGX_ENCL_INITIALIZED	= BIT(0),
+	SGX_ENCL_DEBUG		= BIT(1),
+	SGX_ENCL_SUSPEND	= BIT(2),
+	SGX_ENCL_DEAD		= BIT(3),
+};
+
+struct sgx_encl {
+	unsigned int flags;
+	uint64_t attributes;
+	uint64_t xfrm;
+	unsigned int page_cnt;
+	unsigned int secs_child_cnt;
+	struct mutex lock;
+	struct mm_struct *mm;
+	struct file *backing;
+	struct file *pcmd;
+	struct kref refcount;
+	unsigned long base;
+	unsigned long size;
+	unsigned long ssaframesize;
+	struct list_head va_pages;
+	struct radix_tree_root page_tree;
+	struct list_head add_page_reqs;
+	struct work_struct add_page_work;
+	struct sgx_encl_page secs;
+	struct pid *tgid;
+	struct mmu_notifier mmu_notifier;
+};
+
+extern struct workqueue_struct *sgx_add_page_wq;
+extern u64 sgx_encl_size_max_32;
+extern u64 sgx_encl_size_max_64;
+extern u64 sgx_xfrm_mask;
+extern u32 sgx_misc_reserved;
+extern u32 sgx_xsave_size_tbl[64];
+
+extern const struct vm_operations_struct sgx_vm_ops;
+
+int sgx_encl_find(struct mm_struct *mm, unsigned long addr,
+		  struct vm_area_struct **vma);
+void sgx_invalidate(struct sgx_encl *encl, bool flush_cpus);
+#define SGX_INVD(ret, encl, fmt, ...)			\
+do {							\
+	if (unlikely(ret)) {				\
+		sgx_err(encl, fmt, ##__VA_ARGS__);	\
+		sgx_invalidate(encl, true);		\
+	}						\
+} while (0)
+
+struct sgx_encl *sgx_encl_alloc(struct sgx_secs *secs);
+int sgx_encl_create(struct sgx_encl *encl, struct sgx_secs *secs);
+struct sgx_encl_page *sgx_encl_alloc_page(struct sgx_encl *encl,
+					  unsigned long addr);
+void sgx_encl_free_page(struct sgx_encl_page *encl_page);
+int sgx_encl_add_page(struct sgx_encl *encl, unsigned long addr, void *data,
+		      struct sgx_secinfo *secinfo, unsigned int mrmask);
+int sgx_encl_init(struct sgx_encl *encl, struct sgx_sigstruct *sigstruct,
+		  struct sgx_einittoken *einittoken);
+void sgx_encl_block(struct sgx_encl_page *encl_page);
+void sgx_encl_track(struct sgx_encl *encl);
+int sgx_encl_load_page(struct sgx_encl_page *encl_page,
+		       struct sgx_epc_page *epc_page);
+void sgx_encl_release(struct kref *ref);
+
+long sgx_ioctl(struct file *filep, unsigned int cmd, unsigned long arg);
+#ifdef CONFIG_COMPAT
+long sgx_compat_ioctl(struct file *filep, unsigned int cmd, unsigned long arg);
+#endif
+
+/* Utility functions */
+int sgx_test_and_clear_young(struct sgx_encl_page *page);
+void sgx_flush_cpus(struct sgx_encl *encl);
+
+struct sgx_encl_page *sgx_fault_page(struct vm_area_struct *vma,
+				     unsigned long addr,
+				     bool do_reserve);
+
+extern const struct sgx_epc_page_ops sgx_encl_page_ops;
+
+void sgx_set_epc_page(struct sgx_encl_page *encl_page,
+		      struct sgx_epc_page *epc_page);
+void sgx_set_page_reclaimable(struct sgx_encl_page *encl_page);
+struct sgx_epc_page *sgx_alloc_va_page(unsigned int flags);
+unsigned int sgx_alloc_va_slot(struct sgx_va_page *va_page);
+void sgx_free_va_slot(struct sgx_va_page *va_page, unsigned int offset);
+bool sgx_va_page_full(struct sgx_va_page *va_page);
+
+#endif /* __ARCH_X86_INTEL_SGX_H__ */
diff --git a/drivers/platform/x86/intel_sgx/sgx_encl.c b/drivers/platform/x86/intel_sgx/sgx_encl.c
new file mode 100644
index 000000000000..562d4ce412d4
--- /dev/null
+++ b/drivers/platform/x86/intel_sgx/sgx_encl.c
@@ -0,0 +1,1002 @@
+// SPDX-License-Identifier: (GPL-2.0 OR BSD-3-Clause)
+// Copyright(c) 2016-17 Intel Corporation.
+//
+// Authors:
+//
+// Jarkko Sakkinen <jarkko.sakkinen@linux.intel.com>
+// Suresh Siddha <suresh.b.siddha@intel.com>
+// Serge Ayoun <serge.ayoun@intel.com>
+// Shay Katz-zamir <shay.katz-zamir@intel.com>
+// Sean Christopherson <sean.j.christopherson@intel.com>
+
+#include <asm/mman.h>
+#include <linux/delay.h>
+#include <linux/file.h>
+#include <linux/hashtable.h>
+#include <linux/highmem.h>
+#include <linux/ratelimit.h>
+#include <linux/sched/signal.h>
+#include <linux/shmem_fs.h>
+#include <linux/slab.h>
+#include "sgx.h"
+
+struct sgx_add_page_req {
+	struct sgx_encl *encl;
+	struct sgx_encl_page *encl_page;
+	struct sgx_secinfo secinfo;
+	u16 mrmask;
+	struct list_head list;
+};
+
+/**
+ * sgx_encl_find - find an enclave
+ * @mm:		mm struct of the current process
+ * @addr:	address in the ELRANGE
+ * @vma:	the resulting VMA
+ *
+ * Finds an enclave identified by the given address. Gives back the VMA, that
+ * is part of the enclave, located in that address. The VMA is given back if it
+ * is a proper enclave VMA even if an &sgx_encl instance does not exist
+ * yet (enclave creation has not been performed).
+ *
+ * Return:
+ *   0 on success,
+ *   -EINVAL if an enclave was not found,
+ *   -ENOENT if the enclave has not been created yet
+ */
+int sgx_encl_find(struct mm_struct *mm, unsigned long addr,
+		  struct vm_area_struct **vma)
+{
+	struct vm_area_struct *result;
+	struct sgx_encl *encl;
+
+	result = find_vma(mm, addr);
+	if (!result || result->vm_ops != &sgx_vm_ops || addr < result->vm_start)
+		return -EINVAL;
+
+	encl = result->vm_private_data;
+	*vma = result;
+
+	return encl ? 0 : -ENOENT;
+}
+
+static void sgx_encl_free_va_pages(struct sgx_encl *encl)
+{
+	struct sgx_va_page *va_page;
+
+	while (!list_empty(&encl->va_pages)) {
+		va_page = list_first_entry(&encl->va_pages, struct sgx_va_page,
+					   list);
+		list_del(&va_page->list);
+		sgx_free_page(va_page->epc_page);
+		kfree(va_page);
+	}
+}
+
+/**
+ * sgx_invalidate - kill an active enclave
+ *
+ * @encl:	an &sgx_encl instance
+ * @flush_cpus	whether to kick threads out of the enclave
+ *
+ * Clear PTEs for all the entry points, kick threads out of the enclave if
+ * requested and mark enclave state as dead.
+ */
+void sgx_invalidate(struct sgx_encl *encl, bool flush_cpus)
+{
+	struct sgx_encl_page *entry;
+	struct radix_tree_iter iter;
+	struct vm_area_struct *vma;
+	unsigned long addr;
+	void **slot;
+
+	if (encl->flags & SGX_ENCL_DEAD)
+		return;
+
+	radix_tree_for_each_slot(slot, &encl->page_tree, &iter, 0) {
+		entry = *slot;
+		addr = SGX_ENCL_PAGE_ADDR(entry);
+
+		if ((entry->desc & SGX_ENCL_PAGE_LOADED) &&
+		    (entry->desc & SGX_ENCL_PAGE_TCS) &&
+		    !sgx_encl_find(encl->mm, addr, &vma))
+			zap_vma_ptes(vma, addr, PAGE_SIZE);
+	}
+
+	encl->flags |= SGX_ENCL_DEAD;
+
+	if (flush_cpus)
+		sgx_flush_cpus(encl);
+
+	sgx_encl_free_va_pages(encl);
+}
+
+static int sgx_measure(struct sgx_epc_page *secs_page,
+		       struct sgx_epc_page *epc_page,
+		       u16 mrmask)
+{
+	int ret = 0;
+	void *secs;
+	void *epc;
+	int i;
+	int j;
+
+	if (!mrmask)
+		return ret;
+
+	secs = sgx_get_page(secs_page);
+	epc = sgx_get_page(epc_page);
+
+	for (i = 0, j = 1; i < 0x1000 && !ret; i += 0x100, j <<= 1) {
+		if (!(j & mrmask))
+			continue;
+
+		ret = __eextend(secs, (void *)((unsigned long)epc + i));
+	}
+
+	sgx_put_page(epc);
+	sgx_put_page(secs);
+
+	return ret;
+}
+
+static int sgx_eadd(struct sgx_epc_page *secs_page,
+		    struct sgx_epc_page *epc_page,
+		    unsigned long linaddr,
+		    struct sgx_secinfo *secinfo,
+		    struct page *backing)
+{
+	struct sgx_pageinfo pginfo;
+	void *epc_page_vaddr;
+	int ret;
+
+	pginfo.srcpge = (unsigned long)kmap_atomic(backing);
+	pginfo.secs = (unsigned long)sgx_get_page(secs_page);
+	epc_page_vaddr = sgx_get_page(epc_page);
+
+	pginfo.linaddr = linaddr;
+	pginfo.secinfo = (unsigned long)secinfo;
+	ret = __eadd(&pginfo, epc_page_vaddr);
+
+	sgx_put_page(epc_page_vaddr);
+	sgx_put_page((void *)(unsigned long)pginfo.secs);
+	kunmap_atomic((void *)(unsigned long)pginfo.srcpge);
+
+	return ret;
+}
+
+static bool sgx_process_add_page_req(struct sgx_add_page_req *req,
+				     struct sgx_epc_page *epc_page)
+{
+	struct sgx_encl_page *encl_page = req->encl_page;
+	struct sgx_encl *encl = req->encl;
+	struct vm_area_struct *vma;
+	pgoff_t backing_index;
+	struct page *backing;
+	unsigned long addr;
+	int ret;
+
+	if (encl->flags & (SGX_ENCL_SUSPEND | SGX_ENCL_DEAD))
+		return false;
+
+	addr = SGX_ENCL_PAGE_ADDR(encl_page);
+	ret = sgx_encl_find(encl->mm, addr, &vma);
+	if (ret)
+		return false;
+
+	backing_index = SGX_ENCL_PAGE_BACKING_INDEX(encl_page, encl);
+	backing = sgx_get_backing(encl->backing, backing_index);
+	if (IS_ERR(backing))
+		return false;
+
+	ret = vm_insert_pfn(vma, addr, SGX_EPC_PFN(epc_page));
+	if (ret) {
+		sgx_err(encl, "%s: vm_insert_pfn() returned %d\n", __func__,
+			ret);
+		sgx_put_backing(backing, false);
+		return false;
+	}
+
+	ret = sgx_eadd(encl->secs.epc_page, epc_page, addr, &req->secinfo,
+		       backing);
+
+	sgx_put_backing(backing, false);
+	if (ret) {
+		sgx_err(encl, "EADD returned %d\n", ret);
+		zap_vma_ptes(vma, addr, PAGE_SIZE);
+		return false;
+	}
+
+	ret = sgx_measure(encl->secs.epc_page, epc_page, req->mrmask);
+	if (ret) {
+		sgx_err(encl, "EEXTEND returned %d\n", ret);
+		zap_vma_ptes(vma, addr, PAGE_SIZE);
+		return false;
+	}
+
+	encl_page->encl = encl;
+	encl->secs_child_cnt++;
+	sgx_set_epc_page(encl_page, epc_page);
+	sgx_set_page_reclaimable(encl_page);
+	return true;
+}
+
+static void sgx_add_page_worker(struct work_struct *work)
+{
+	struct sgx_add_page_req *req;
+	bool skip_rest = false;
+	bool is_empty = false;
+	struct sgx_encl *encl;
+	struct sgx_epc_page *epc_page;
+
+	encl = container_of(work, struct sgx_encl, add_page_work);
+
+	do {
+		schedule();
+
+		if (encl->flags & SGX_ENCL_DEAD)
+			skip_rest = true;
+
+		mutex_lock(&encl->lock);
+		req = list_first_entry(&encl->add_page_reqs,
+				       struct sgx_add_page_req, list);
+		list_del(&req->list);
+		is_empty = list_empty(&encl->add_page_reqs);
+		mutex_unlock(&encl->lock);
+
+		if (skip_rest)
+			goto next;
+
+		epc_page = sgx_alloc_page(&req->encl_page->impl, 0);
+		down_read(&encl->mm->mmap_sem);
+		mutex_lock(&encl->lock);
+
+		if (IS_ERR(epc_page)) {
+			sgx_invalidate(encl, false);
+			skip_rest = true;
+		} else	if (!sgx_process_add_page_req(req, epc_page)) {
+			sgx_free_page(epc_page);
+			sgx_invalidate(encl, false);
+			skip_rest = true;
+		}
+
+		mutex_unlock(&encl->lock);
+		up_read(&encl->mm->mmap_sem);
+
+next:
+		kfree(req);
+	} while (!kref_put(&encl->refcount, sgx_encl_release) && !is_empty);
+}
+
+static u32 sgx_calc_ssaframesize(u32 miscselect, u64 xfrm)
+{
+	u32 size_max = PAGE_SIZE;
+	u32 size;
+	int i;
+
+	for (i = 2; i < 64; i++) {
+		if (!((1 << i) & xfrm))
+			continue;
+
+		size = SGX_SSA_GPRS_SIZE + sgx_xsave_size_tbl[i];
+		if (miscselect & SGX_MISC_EXINFO)
+			size += SGX_SSA_MISC_EXINFO_SIZE;
+
+		if (size > size_max)
+			size_max = size;
+	}
+
+	return (size_max + PAGE_SIZE - 1) >> PAGE_SHIFT;
+}
+
+static int sgx_validate_secs(const struct sgx_secs *secs,
+			     unsigned long ssaframesize)
+{
+	int i;
+
+	if (secs->size < (2 * PAGE_SIZE) ||
+	    (secs->size & (secs->size - 1)) != 0)
+		return -EINVAL;
+
+	if (secs->base & (secs->size - 1))
+		return -EINVAL;
+
+	if (secs->attributes & SGX_ATTR_RESERVED_MASK ||
+	    secs->miscselect & sgx_misc_reserved)
+		return -EINVAL;
+
+	if (secs->attributes & SGX_ATTR_MODE64BIT) {
+#ifdef CONFIG_X86_64
+		if (secs->size > sgx_encl_size_max_64)
+			return -EINVAL;
+#else
+		return -EINVAL;
+#endif
+	} else {
+		/* On 64-bit architecture allow 32-bit encls only in
+		 * the compatibility mode.
+		 */
+#ifdef CONFIG_X86_64
+		if (!test_thread_flag(TIF_ADDR32))
+			return -EINVAL;
+#endif
+		if (secs->size > sgx_encl_size_max_32)
+			return -EINVAL;
+	}
+
+	if ((secs->xfrm & 0x3) != 0x3 || (secs->xfrm & ~sgx_xfrm_mask))
+		return -EINVAL;
+
+	/* Check that BNDREGS and BNDCSR are equal. */
+	if (((secs->xfrm >> 3) & 1) != ((secs->xfrm >> 4) & 1))
+		return -EINVAL;
+
+	if (!secs->ssaframesize || ssaframesize > secs->ssaframesize)
+		return -EINVAL;
+
+	for (i = 0; i < SGX_SECS_RESERVED1_SIZE; i++)
+		if (secs->reserved1[i])
+			return -EINVAL;
+
+	for (i = 0; i < SGX_SECS_RESERVED2_SIZE; i++)
+		if (secs->reserved2[i])
+			return -EINVAL;
+
+	for (i = 0; i < SGX_SECS_RESERVED3_SIZE; i++)
+		if (secs->reserved3[i])
+			return -EINVAL;
+
+	for (i = 0; i < SGX_SECS_RESERVED4_SIZE; i++)
+		if (secs->reserved4[i])
+			return -EINVAL;
+
+	return 0;
+}
+
+static void sgx_mmu_notifier_release(struct mmu_notifier *mn,
+				     struct mm_struct *mm)
+{
+	struct sgx_encl *encl =
+		container_of(mn, struct sgx_encl, mmu_notifier);
+
+	mutex_lock(&encl->lock);
+	encl->flags |= SGX_ENCL_DEAD;
+	mutex_unlock(&encl->lock);
+}
+
+static const struct mmu_notifier_ops sgx_mmu_notifier_ops = {
+	.release	= sgx_mmu_notifier_release,
+};
+
+static int sgx_encl_grow(struct sgx_encl *encl)
+{
+	struct sgx_va_page *va_page;
+	int ret;
+
+	mutex_lock(&encl->lock);
+	if (!(encl->page_cnt % SGX_VA_SLOT_COUNT)) {
+		mutex_unlock(&encl->lock);
+
+		va_page = kzalloc(sizeof(*va_page), GFP_KERNEL);
+		if (!va_page)
+			return -ENOMEM;
+		va_page->epc_page = sgx_alloc_va_page(0);
+		if (IS_ERR(va_page->epc_page)) {
+			ret = PTR_ERR(va_page->epc_page);
+			kfree(va_page);
+			return ret;
+		}
+
+		mutex_lock(&encl->lock);
+		if (encl->page_cnt % SGX_VA_SLOT_COUNT) {
+			sgx_free_page(va_page->epc_page);
+			kfree(va_page);
+		} else {
+			list_add(&va_page->list, &encl->va_pages);
+		}
+	}
+	encl->page_cnt++;
+	mutex_unlock(&encl->lock);
+	return 0;
+}
+
+/**
+ * sgx_encl_alloc - allocate memory for an enclave and set attributes
+ *
+ * @secs:	SECS data (must be page aligned)
+ *
+ * Allocates a new &sgx_encl instance. Validates SECS attributes, creates
+ * backing storage for the enclave and sets enclave attributes to sane initial
+ * values.
+ *
+ * Return:
+ *   an &sgx_encl instance,
+ *   -errno otherwise
+ */
+struct sgx_encl *sgx_encl_alloc(struct sgx_secs *secs)
+{
+	unsigned long ssaframesize;
+	struct sgx_encl *encl;
+	struct file *backing;
+	struct file *pcmd;
+
+	ssaframesize = sgx_calc_ssaframesize(secs->miscselect, secs->xfrm);
+	if (sgx_validate_secs(secs, ssaframesize))
+		return ERR_PTR(-EINVAL);
+
+	backing = shmem_file_setup("[dev/sgx]", secs->size + PAGE_SIZE,
+				   VM_NORESERVE);
+	if (IS_ERR(backing))
+		return (void *)backing;
+
+	pcmd = shmem_file_setup("[dev/sgx]", (secs->size + PAGE_SIZE) >> 5,
+				VM_NORESERVE);
+	if (IS_ERR(pcmd)) {
+		fput(backing);
+		return (void *)pcmd;
+	}
+
+	encl = kzalloc(sizeof(*encl), GFP_KERNEL);
+	if (!encl) {
+		fput(backing);
+		fput(pcmd);
+		return ERR_PTR(-ENOMEM);
+	}
+
+	encl->attributes = secs->attributes;
+	encl->xfrm = secs->xfrm;
+
+	kref_init(&encl->refcount);
+	INIT_LIST_HEAD(&encl->add_page_reqs);
+	INIT_LIST_HEAD(&encl->va_pages);
+	INIT_RADIX_TREE(&encl->page_tree, GFP_KERNEL);
+	mutex_init(&encl->lock);
+	INIT_WORK(&encl->add_page_work, sgx_add_page_worker);
+
+	encl->mm = current->mm;
+	encl->base = secs->base;
+	encl->size = secs->size;
+	encl->ssaframesize = secs->ssaframesize;
+	encl->backing = backing;
+	encl->pcmd = pcmd;
+
+	return encl;
+}
+
+
+/**
+ * sgx_encl_create - create an enclave
+ *
+ * @encl:	an enclave
+ * @secs:	page aligned SECS data
+ *
+ * Validates SECS attributes, allocates an EPC page for the SECS and creates
+ * the enclave by performing ECREATE.
+ *
+ * Return:
+ *   0 on success,
+ *   -errno otherwise
+ */
+int sgx_encl_create(struct sgx_encl *encl, struct sgx_secs *secs)
+{
+	struct vm_area_struct *vma;
+	struct sgx_pageinfo pginfo;
+	struct sgx_secinfo secinfo;
+	struct sgx_epc_page *secs_epc;
+	void *secs_vaddr;
+	long ret;
+
+	secs_epc = sgx_alloc_page(&encl->secs.impl, 0);
+	if (IS_ERR(secs_epc)) {
+		ret = PTR_ERR(secs_epc);
+		return ret;
+	}
+
+	sgx_set_epc_page(&encl->secs, secs_epc);
+	encl->secs.encl = encl;
+	encl->secs.impl.ops = &sgx_encl_page_ops;
+	encl->tgid = get_pid(task_tgid(current));
+
+	ret = sgx_encl_grow(encl);
+	if (ret)
+		return ret;
+
+	secs_vaddr = sgx_get_page(secs_epc);
+
+	pginfo.srcpge = (unsigned long)secs;
+	pginfo.linaddr = 0;
+	pginfo.secinfo = (unsigned long)&secinfo;
+	pginfo.secs = 0;
+	memset(&secinfo, 0, sizeof(secinfo));
+	ret = __ecreate((void *)&pginfo, secs_vaddr);
+
+	sgx_put_page(secs_vaddr);
+
+	if (ret) {
+		sgx_dbg(encl, "ECREATE returned %ld\n", ret);
+		return ret;
+	}
+
+	if (secs->attributes & SGX_ATTR_DEBUG)
+		encl->flags |= SGX_ENCL_DEBUG;
+
+	encl->mmu_notifier.ops = &sgx_mmu_notifier_ops;
+	ret = mmu_notifier_register(&encl->mmu_notifier, encl->mm);
+	if (ret) {
+		if (ret == -EINTR)
+			ret = -ERESTARTSYS;
+		encl->mmu_notifier.ops = NULL;
+		return ret;
+	}
+
+	down_read(&current->mm->mmap_sem);
+	ret = sgx_encl_find(current->mm, secs->base, &vma);
+	if (ret != -ENOENT) {
+		if (!ret)
+			ret = -EINVAL;
+		up_read(&current->mm->mmap_sem);
+		return ret;
+	}
+
+	if (vma->vm_start != secs->base ||
+	    vma->vm_end != (secs->base + secs->size) ||
+	    vma->vm_pgoff != 0) {
+		ret = -EINVAL;
+		up_read(&current->mm->mmap_sem);
+		return ret;
+	}
+
+	vma->vm_private_data = encl;
+	up_read(&current->mm->mmap_sem);
+	return 0;
+}
+
+static int sgx_validate_secinfo(struct sgx_secinfo *secinfo)
+{
+	u64 page_type = secinfo->flags & SGX_SECINFO_PAGE_TYPE_MASK;
+	u64 perm = secinfo->flags & SGX_SECINFO_PERMISSION_MASK;
+	int i;
+
+	if ((secinfo->flags & SGX_SECINFO_RESERVED_MASK) ||
+	    ((perm & SGX_SECINFO_W) && !(perm & SGX_SECINFO_R)) ||
+	    (page_type != SGX_SECINFO_TCS &&
+	     page_type != SGX_SECINFO_REG))
+		return -EINVAL;
+
+	for (i = 0; i < sizeof(secinfo->reserved) / sizeof(u64); i++)
+		if (secinfo->reserved[i])
+			return -EINVAL;
+
+	return 0;
+}
+
+static bool sgx_validate_offset(struct sgx_encl *encl, unsigned long offset)
+{
+	if (offset & (PAGE_SIZE - 1))
+		return false;
+
+	if (offset >= encl->size)
+		return false;
+
+	return true;
+}
+
+static int sgx_validate_tcs(struct sgx_encl *encl, struct sgx_tcs *tcs)
+{
+	int i;
+
+	if (tcs->flags & SGX_TCS_RESERVED_MASK) {
+		sgx_dbg(encl, "%s: invalid TCS flags = 0x%lx\n",
+			__func__, (unsigned long)tcs->flags);
+		return -EINVAL;
+	}
+
+	if (tcs->flags & SGX_TCS_DBGOPTIN) {
+		sgx_dbg(encl, "%s: DBGOPTIN TCS flag is set, EADD will clear it\n",
+			__func__);
+		return -EINVAL;
+	}
+
+	if (!sgx_validate_offset(encl, tcs->ossa)) {
+		sgx_dbg(encl, "%s: invalid OSSA: 0x%lx\n", __func__,
+			(unsigned long)tcs->ossa);
+		return -EINVAL;
+	}
+
+	if (!sgx_validate_offset(encl, tcs->ofsbase)) {
+		sgx_dbg(encl, "%s: invalid OFSBASE: 0x%lx\n", __func__,
+			(unsigned long)tcs->ofsbase);
+		return -EINVAL;
+	}
+
+	if (!sgx_validate_offset(encl, tcs->ogsbase)) {
+		sgx_dbg(encl, "%s: invalid OGSBASE: 0x%lx\n", __func__,
+			(unsigned long)tcs->ogsbase);
+		return -EINVAL;
+	}
+
+	if ((tcs->fslimit & 0xFFF) != 0xFFF) {
+		sgx_dbg(encl, "%s: invalid FSLIMIT: 0x%x\n", __func__,
+			tcs->fslimit);
+		return -EINVAL;
+	}
+
+	if ((tcs->gslimit & 0xFFF) != 0xFFF) {
+		sgx_dbg(encl, "%s: invalid GSLIMIT: 0x%x\n", __func__,
+			tcs->gslimit);
+		return -EINVAL;
+	}
+
+	for (i = 0; i < sizeof(tcs->reserved) / sizeof(u64); i++)
+		if (tcs->reserved[i])
+			return -EINVAL;
+
+	return 0;
+}
+
+static int __sgx_encl_add_page(struct sgx_encl *encl,
+			       struct sgx_encl_page *encl_page,
+			       void *data,
+			       struct sgx_secinfo *secinfo,
+			       unsigned int mrmask)
+{
+	u64 page_type = secinfo->flags & SGX_SECINFO_PAGE_TYPE_MASK;
+	struct sgx_add_page_req *req = NULL;
+	pgoff_t backing_index;
+	struct page *backing;
+	void *backing_ptr;
+	int empty;
+
+	req = kzalloc(sizeof(*req), GFP_KERNEL);
+	if (!req)
+		return -ENOMEM;
+
+	backing_index = SGX_ENCL_PAGE_BACKING_INDEX(encl_page, encl);
+	backing = sgx_get_backing(encl->backing, backing_index);
+	if (IS_ERR(backing)) {
+		kfree(req);
+		return PTR_ERR(backing);
+	}
+	backing_ptr = kmap(backing);
+	memcpy(backing_ptr, data, PAGE_SIZE);
+	kunmap(backing);
+	if (page_type == SGX_SECINFO_TCS)
+		encl_page->desc |= SGX_ENCL_PAGE_TCS;
+	memcpy(&req->secinfo, secinfo, sizeof(*secinfo));
+	req->encl = encl;
+	req->encl_page = encl_page;
+	req->mrmask = mrmask;
+	empty = list_empty(&encl->add_page_reqs);
+	kref_get(&encl->refcount);
+	list_add_tail(&req->list, &encl->add_page_reqs);
+	if (empty)
+		queue_work(sgx_add_page_wq, &encl->add_page_work);
+	sgx_put_backing(backing, true /* write */);
+	return 0;
+}
+
+/**
+ * sgx_encl_alloc_page - allocate a new enclave page
+ * @encl:	an enclave
+ * @addr:	page address in the ELRANGE
+ *
+ * Return:
+ *   an &sgx_encl_page instance on success,
+ *   -errno otherwise
+ */
+struct sgx_encl_page *sgx_encl_alloc_page(struct sgx_encl *encl,
+					  unsigned long addr)
+{
+	struct sgx_encl_page *encl_page;
+	int ret;
+
+	if (radix_tree_lookup(&encl->page_tree, PFN_DOWN(addr)))
+		return ERR_PTR(-EEXIST);
+	encl_page = kzalloc(sizeof(*encl_page), GFP_KERNEL);
+	if (!encl_page)
+		return ERR_PTR(-ENOMEM);
+	encl_page->desc = addr;
+	encl_page->impl.ops = &sgx_encl_page_ops;
+	encl_page->encl = encl;
+	ret = radix_tree_insert(&encl->page_tree, PFN_DOWN(encl_page->desc),
+				encl_page);
+	if (ret) {
+		kfree(encl_page);
+		return ERR_PTR(ret);
+	}
+	return encl_page;
+}
+
+/**
+ * sgx_encl_free_page - free an enclave page
+ * @encl_page:	an enclave page
+ */
+void sgx_encl_free_page(struct sgx_encl_page *encl_page)
+{
+	radix_tree_delete(&encl_page->encl->page_tree,
+			  PFN_DOWN(encl_page->desc));
+	if (encl_page->desc & SGX_ENCL_PAGE_LOADED) {
+		spin_lock(&sgx_active_page_list_lock);
+		list_del(&encl_page->epc_page->list);
+		spin_unlock(&sgx_active_page_list_lock);
+		sgx_free_page(encl_page->epc_page);
+	}
+	kfree(encl_page);
+}
+
+/**
+ * sgx_encl_add_page - add a page to the enclave
+ *
+ * @encl:	an enclave
+ * @addr:	page address in the ELRANGE
+ * @data:	page data
+ * @secinfo:	page permissions
+ * @mrmask:	bitmask to select the 256 byte chunks to be measured
+ *
+ * Creates a new enclave page and enqueues an EADD operation that will be
+ * processed by a worker thread later on.
+ *
+ * Return:
+ *   0 on success,
+ *   -errno otherwise
+ */
+int sgx_encl_add_page(struct sgx_encl *encl, unsigned long addr, void *data,
+		      struct sgx_secinfo *secinfo, unsigned int mrmask)
+{
+	u64 page_type = secinfo->flags & SGX_SECINFO_PAGE_TYPE_MASK;
+	struct sgx_encl_page *encl_page;
+	int ret;
+
+	if (sgx_validate_secinfo(secinfo))
+		return -EINVAL;
+	if (page_type == SGX_SECINFO_TCS) {
+		ret = sgx_validate_tcs(encl, data);
+		if (ret)
+			return ret;
+	}
+	ret = sgx_encl_grow(encl);
+	if (ret)
+		return ret;
+	mutex_lock(&encl->lock);
+	if (encl->flags & (SGX_ENCL_INITIALIZED | SGX_ENCL_DEAD)) {
+		mutex_unlock(&encl->lock);
+		return -EINVAL;
+	}
+	encl_page = sgx_encl_alloc_page(encl, addr);
+	if (IS_ERR(encl_page)) {
+		mutex_unlock(&encl->lock);
+		return PTR_ERR(encl_page);
+	}
+	ret = __sgx_encl_add_page(encl, encl_page, data, secinfo, mrmask);
+	if (ret)
+		sgx_encl_free_page(encl_page);
+	mutex_unlock(&encl->lock);
+	return ret;
+}
+
+static int sgx_einit(struct sgx_encl *encl, struct sgx_sigstruct *sigstruct,
+		     struct sgx_einittoken *token)
+{
+	struct sgx_epc_page *secs_epc = encl->secs.epc_page;
+	void *secs_va;
+	int ret;
+
+	secs_va = sgx_get_page(secs_epc);
+	ret = __einit(sigstruct, token, secs_va);
+	sgx_put_page(secs_va);
+
+	return ret;
+}
+
+/**
+ * sgx_encl_init - perform EINIT for the given enclave
+ *
+ * @encl:	an enclave
+ * @sigstruct:	SIGSTRUCT for the enclave
+ * @token:	EINITTOKEN for the enclave
+ *
+ * Retries a few times in order to perform EINIT operation on an enclave
+ * because there could be potentially an interrupt storm.
+ *
+ * Return:
+ *   0 on success,
+ *   SGX error code on EINIT failure,
+ *   -errno otherwise
+ */
+int sgx_encl_init(struct sgx_encl *encl, struct sgx_sigstruct *sigstruct,
+		  struct sgx_einittoken *token)
+{
+	int ret;
+	int i;
+	int j;
+
+	flush_work(&encl->add_page_work);
+
+	mutex_lock(&encl->lock);
+
+	if (encl->flags & SGX_ENCL_INITIALIZED) {
+		mutex_unlock(&encl->lock);
+		return 0;
+	}
+
+	for (i = 0; i < SGX_EINIT_SLEEP_COUNT; i++) {
+		for (j = 0; j < SGX_EINIT_SPIN_COUNT; j++) {
+			ret = sgx_einit(encl, sigstruct, token);
+
+			if (ret == SGX_UNMASKED_EVENT)
+				continue;
+			else
+				break;
+		}
+
+		if (ret != SGX_UNMASKED_EVENT)
+			break;
+
+		msleep_interruptible(SGX_EINIT_SLEEP_TIME);
+		if (signal_pending(current)) {
+			mutex_unlock(&encl->lock);
+			return -ERESTARTSYS;
+		}
+	}
+
+	if (ret > 0)
+		sgx_dbg(encl, "EINIT returned %d\n", ret);
+	else if (!ret)
+		encl->flags |= SGX_ENCL_INITIALIZED;
+	mutex_unlock(&encl->lock);
+
+	return ret;
+}
+
+/**
+ * sgx_encl_block - block an enclave page
+ * @encl_page:	an enclave page
+ *
+ * Changes the state of the associated EPC page to blocked.
+ */
+void sgx_encl_block(struct sgx_encl_page *encl_page)
+{
+	unsigned long addr = SGX_ENCL_PAGE_ADDR(encl_page);
+	struct sgx_encl *encl = encl_page->encl;
+	struct vm_area_struct *vma;
+	int ret;
+
+	if (encl->flags & SGX_ENCL_DEAD)
+		return;
+
+	ret = sgx_encl_find(encl->mm, addr, &vma);
+	if (ret || encl != vma->vm_private_data)
+		return;
+
+	zap_vma_ptes(vma, addr, PAGE_SIZE);
+	ret = sgx_eblock(encl_page->epc_page);
+	SGX_INVD(ret, encl, "EBLOCK returned %d\n", ret);
+}
+
+/**
+ * sgx_encl_track - start tracking pages in the blocked state
+ * @encl:	an enclave
+ *
+ * Start blocking accesses for pages in the blocked state for threads that enter
+ * inside the enclave by executing the ETRACK leaf instruction. This starts a
+ * shootdown sequence for threads that entered before ETRACK.
+ *
+ * The caller must take care (with an IPI when necessary) to make sure that the
+ * previous shootdown sequence was completed before calling this function.  If
+ * this is not the case, the callee prints a critical error to the klog and
+ * kills the enclave.
+ */
+void sgx_encl_track(struct sgx_encl *encl)
+{
+	int ret = sgx_etrack(encl->secs.epc_page);
+
+	SGX_INVD(ret, encl, "ETRACK returned %d\n", ret);
+}
+
+/**
+ * sgx_encl_load_page - load an enclave page
+ * @encl_page:	a &sgx_encl_page
+ * @epc_page:	a &sgx_epc_page
+ *
+ * Loads an enclave page from the regular memory to the EPC. The pages, which
+ * are not children of a SECS (eg SECS itself and VA pages) should set their
+ * address to zero.
+ */
+int sgx_encl_load_page(struct sgx_encl_page *encl_page,
+		       struct sgx_epc_page *epc_page)
+{
+	unsigned long addr = SGX_ENCL_PAGE_ADDR(encl_page);
+	struct sgx_encl *encl = encl_page->encl;
+	struct sgx_pageinfo pginfo;
+	unsigned long pcmd_offset;
+	unsigned long va_offset;
+	void *secs_ptr = NULL;
+	pgoff_t backing_index;
+	struct page *backing;
+	struct page *pcmd;
+	void *epc_ptr;
+	void *va_ptr;
+	int ret;
+
+	backing_index = SGX_ENCL_PAGE_BACKING_INDEX(encl_page, encl);
+	pcmd_offset = SGX_ENCL_PAGE_PCMD_OFFSET(encl_page, encl);
+	va_offset = SGX_ENCL_PAGE_VA_OFFSET(encl_page);
+
+	backing = sgx_get_backing(encl->backing, backing_index);
+	if (IS_ERR(backing))
+		return PTR_ERR(backing);
+
+	pcmd = sgx_get_backing(encl->pcmd, backing_index >> 5);
+	if (IS_ERR(pcmd)) {
+		sgx_put_backing(backing, false);
+		return PTR_ERR(pcmd);
+	}
+
+	if (addr)
+		secs_ptr = sgx_get_page(encl->secs.epc_page);
+
+	epc_ptr = sgx_get_page(epc_page);
+	va_ptr = sgx_get_page(encl_page->va_page->epc_page);
+	pginfo.srcpge = (unsigned long)kmap_atomic(backing);
+	pginfo.pcmd = (unsigned long)kmap_atomic(pcmd) + pcmd_offset;
+	pginfo.linaddr = addr;
+	pginfo.secs = (unsigned long)secs_ptr;
+
+	ret = __eldu(&pginfo, epc_ptr, va_ptr + va_offset);
+	if (ret) {
+		sgx_err(encl, "ELDU returned %d\n", ret);
+		ret = -EFAULT;
+	}
+
+	kunmap_atomic((void *)(unsigned long)(pginfo.pcmd - pcmd_offset));
+	kunmap_atomic((void *)(unsigned long)pginfo.srcpge);
+	sgx_put_page(va_ptr);
+	sgx_put_page(epc_ptr);
+
+	if (addr)
+		sgx_put_page(secs_ptr);
+
+	sgx_put_backing(pcmd, false);
+	sgx_put_backing(backing, false);
+	return ret;
+}
+
+/**
+ * sgx_encl_release - destroy an enclave
+ *
+ * @kref:	address of the kref inside &sgx_encl
+ *
+ * Used togethere with kref_put(). Frees all the resources associated with
+ * the enclave.
+ */
+void sgx_encl_release(struct kref *ref)
+{
+	struct sgx_encl *encl = container_of(ref, struct sgx_encl, refcount);
+	struct sgx_encl_page *entry;
+	struct radix_tree_iter iter;
+	void **slot;
+
+	radix_tree_for_each_slot(slot, &encl->page_tree, &iter, 0) {
+		entry = *slot;
+		sgx_encl_free_page(entry);
+	}
+
+	put_pid(encl->tgid);
+
+	if (encl->mmu_notifier.ops)
+		mmu_notifier_unregister_no_release(&encl->mmu_notifier,
+						   encl->mm);
+
+	sgx_encl_free_va_pages(encl);
+
+	if (encl->secs.desc & SGX_ENCL_PAGE_LOADED)
+		sgx_free_page(encl->secs.epc_page);
+
+	if (encl->backing)
+		fput(encl->backing);
+
+	if (encl->pcmd)
+		fput(encl->pcmd);
+
+	kfree(encl);
+}
diff --git a/drivers/platform/x86/intel_sgx/sgx_encl_page.c b/drivers/platform/x86/intel_sgx/sgx_encl_page.c
new file mode 100644
index 000000000000..61bd2425640b
--- /dev/null
+++ b/drivers/platform/x86/intel_sgx/sgx_encl_page.c
@@ -0,0 +1,294 @@
+// SPDX-License-Identifier: (GPL-2.0 OR BSD-3-Clause)
+// Copyright(c) 2016-17 Intel Corporation.
+//
+// Authors:
+//
+// Jarkko Sakkinen <jarkko.sakkinen@linux.intel.com>
+// Suresh Siddha <suresh.b.siddha@intel.com>
+// Serge Ayoun <serge.ayoun@intel.com>
+// Shay Katz-zamir <shay.katz-zamir@intel.com>
+// Sean Christopherson <sean.j.christopherson@intel.com>
+
+#include <linux/device.h>
+#include <linux/freezer.h>
+#include <linux/highmem.h>
+#include <linux/kthread.h>
+#include <linux/ratelimit.h>
+#include <linux/sched/signal.h>
+#include <linux/slab.h>
+#include "sgx.h"
+
+static bool sgx_encl_page_get(struct sgx_epc_page *epc_page)
+{
+	struct sgx_encl_page *encl_page = container_of(epc_page->impl,
+						       struct sgx_encl_page,
+						       impl);
+	struct sgx_encl *encl = encl_page->encl;
+
+	return kref_get_unless_zero(&encl->refcount) != 0;
+}
+
+static void sgx_encl_page_put(struct sgx_epc_page *epc_page)
+{
+	struct sgx_encl_page *encl_page = container_of(epc_page->impl,
+						       struct sgx_encl_page,
+						       impl);
+	struct sgx_encl *encl = encl_page->encl;
+
+	kref_put(&encl->refcount, sgx_encl_release);
+}
+
+static bool sgx_encl_page_reclaim(struct sgx_epc_page *epc_page)
+{
+	struct sgx_encl_page *encl_page = container_of(epc_page->impl,
+						       struct sgx_encl_page,
+						       impl);
+	struct sgx_encl *encl = encl_page->encl;
+	bool ret = false;
+
+	down_read(&encl->mm->mmap_sem);
+	mutex_lock(&encl->lock);
+	if ((encl->flags & SGX_ENCL_DEAD) ||
+	    (!sgx_test_and_clear_young(encl_page) &&
+	     !(encl_page->desc & SGX_ENCL_PAGE_RESERVED))) {
+		encl_page->desc |= SGX_ENCL_PAGE_RESERVED;
+		ret = true;
+	}
+	mutex_unlock(&encl->lock);
+	up_read(&encl->mm->mmap_sem);
+	return ret;
+}
+
+static void sgx_encl_page_block(struct sgx_epc_page *epc_page)
+{
+	struct sgx_encl_page *encl_page = container_of(epc_page->impl,
+						       struct sgx_encl_page,
+						       impl);
+	struct sgx_encl *encl = encl_page->encl;
+
+	down_read(&encl->mm->mmap_sem);
+	mutex_lock(&encl->lock);
+	sgx_encl_block(encl_page);
+	mutex_unlock(&encl->lock);
+	up_read(&encl->mm->mmap_sem);
+}
+
+static int sgx_ewb(struct sgx_encl *encl, struct sgx_epc_page *epc_page,
+		   struct sgx_va_page *va_page, unsigned int va_offset)
+{
+	struct sgx_encl_page *encl_page = container_of(epc_page->impl,
+						       struct sgx_encl_page,
+						       impl);
+	unsigned long pcmd_offset = SGX_ENCL_PAGE_PCMD_OFFSET(encl_page, encl);
+	struct sgx_pageinfo pginfo;
+	pgoff_t backing_index;
+	struct page *backing;
+	struct page *pcmd;
+	void *epc;
+	void *va;
+	int ret;
+
+	backing_index = SGX_ENCL_PAGE_BACKING_INDEX(encl_page, encl);
+
+	backing = sgx_get_backing(encl->backing, backing_index);
+	if (IS_ERR(backing)) {
+		ret = PTR_ERR(backing);
+		return ret;
+	}
+
+	pcmd = sgx_get_backing(encl->pcmd, backing_index >> 5);
+	if (IS_ERR(pcmd)) {
+		ret = PTR_ERR(pcmd);
+		sgx_put_backing(backing, true);
+		return ret;
+	}
+
+	epc = sgx_get_page(epc_page);
+	va = sgx_get_page(va_page->epc_page);
+
+	pginfo.srcpge = (unsigned long)kmap_atomic(backing);
+	pginfo.pcmd = (unsigned long)kmap_atomic(pcmd) + pcmd_offset;
+	pginfo.linaddr = 0;
+	pginfo.secs = 0;
+	ret = __ewb(&pginfo, epc, (void *)((unsigned long)va + va_offset));
+	kunmap_atomic((void *)(unsigned long)(pginfo.pcmd - pcmd_offset));
+	kunmap_atomic((void *)(unsigned long)pginfo.srcpge);
+
+	sgx_put_page(va);
+	sgx_put_page(epc);
+	sgx_put_backing(pcmd, true);
+	sgx_put_backing(backing, true);
+
+	return ret;
+}
+
+/**
+ * sgx_write_page - write a page to the regular memory
+ *
+ * Writes an EPC page to the shmem file associated with the enclave. Flushes
+ * CPUs and retries if there are hardware threads that can potentially have TLB
+ * entries to the page (indicated by SGX_NOT_TRACKED). Clears the reserved flag
+ * after the page is swapped.
+ *
+ * @epc_page:	an EPC page
+ */
+static void sgx_write_page(struct sgx_epc_page *epc_page, bool do_free)
+{
+	struct sgx_encl_page *encl_page = container_of(epc_page->impl,
+						       struct sgx_encl_page,
+						       impl);
+	struct sgx_encl *encl = encl_page->encl;
+	struct sgx_va_page *va_page;
+	unsigned int va_offset;
+	int ret;
+
+	if (!(encl->flags & SGX_ENCL_DEAD)) {
+		va_page = list_first_entry(&encl->va_pages, struct sgx_va_page,
+					   list);
+		va_offset = sgx_alloc_va_slot(va_page);
+		if (sgx_va_page_full(va_page))
+			list_move_tail(&va_page->list, &encl->va_pages);
+
+		ret = sgx_ewb(encl, epc_page, va_page, va_offset);
+		if (ret == SGX_NOT_TRACKED) {
+			sgx_encl_track(encl);
+			ret = sgx_ewb(encl, epc_page, va_page, va_offset);
+			if (ret == SGX_NOT_TRACKED) {
+				/* slow path, IPI needed */
+				sgx_flush_cpus(encl);
+				ret = sgx_ewb(encl, epc_page, va_page,
+					      va_offset);
+			}
+		}
+		SGX_INVD(ret, encl, "EWB returned %d\n", ret);
+
+		encl_page->desc |= va_offset;
+		encl_page->va_page = va_page;
+		encl_page->desc &= ~SGX_ENCL_PAGE_RESERVED;
+	}
+	encl_page->desc &= ~SGX_ENCL_PAGE_LOADED;
+	if (do_free)
+		sgx_free_page(epc_page);
+}
+
+static void sgx_encl_page_write(struct sgx_epc_page *epc_page)
+{
+	struct sgx_encl_page *encl_page = container_of(epc_page->impl,
+						       struct sgx_encl_page,
+						       impl);
+	struct sgx_encl *encl = encl_page->encl;
+
+	down_read(&encl->mm->mmap_sem);
+	mutex_lock(&encl->lock);
+	sgx_write_page(epc_page, false);
+	encl->secs_child_cnt--;
+	if (!encl->secs_child_cnt && (encl->flags & SGX_ENCL_INITIALIZED))
+		sgx_write_page(encl->secs.epc_page, true);
+	mutex_unlock(&encl->lock);
+	up_read(&encl->mm->mmap_sem);
+}
+
+const struct sgx_epc_page_ops sgx_encl_page_ops = {
+	.get = sgx_encl_page_get,
+	.put = sgx_encl_page_put,
+	.reclaim = sgx_encl_page_reclaim,
+	.block = sgx_encl_page_block,
+	.write = sgx_encl_page_write,
+};
+
+/**
+ * sgx_set_page_reclaimable - associated an EPC page with an enclave page
+ * @encl_page:	an enclave page
+ * @epc_page:	the EPC page to attach to @encl_page
+ */
+void sgx_set_epc_page(struct sgx_encl_page *encl_page,
+		      struct sgx_epc_page *epc_page)
+{
+	encl_page->desc |= SGX_ENCL_PAGE_LOADED;
+	encl_page->epc_page = epc_page;
+}
+
+/**
+ * sgx_set_page_reclaimable - mark an EPC page reclaimable
+ * @encl_page:	an enclave page with a loaded EPC page
+ */
+void sgx_set_page_reclaimable(struct sgx_encl_page *encl_page)
+{
+	sgx_test_and_clear_young(encl_page);
+
+	spin_lock(&sgx_active_page_list_lock);
+	list_add_tail(&encl_page->epc_page->list, &sgx_active_page_list);
+	spin_unlock(&sgx_active_page_list_lock);
+}
+
+/**
+ * sgx_alloc_page - allocate a VA page
+ * @flags:	allocation flags
+ *
+ * Allocates an &sgx_epc_page instance and converts it to a VA page.
+ *
+ * Return:
+ *   a &struct sgx_va_page instance,
+ *   -errno otherwise
+ */
+struct sgx_epc_page *sgx_alloc_va_page(unsigned int flags)
+{
+	struct sgx_epc_page *epc_page;
+	int ret;
+
+	epc_page = sgx_alloc_page(NULL, flags);
+	if (IS_ERR(epc_page))
+		return (void *)epc_page;
+
+	ret = sgx_epa(epc_page);
+	if (ret) {
+		pr_crit("EPA failed\n");
+		sgx_free_page(epc_page);
+		return ERR_PTR(ret);
+	}
+
+	return epc_page;
+}
+
+/**
+ * sgx_alloc_va_slot - allocate a VA slot
+ * @va_page:	a &struct sgx_va_page instance
+ *
+ * Allocates a slot from a &struct sgx_va_page instance.
+ *
+ * Return: offset of the slot inside the VA page
+ */
+unsigned int sgx_alloc_va_slot(struct sgx_va_page *va_page)
+{
+	int slot = find_first_zero_bit(va_page->slots, SGX_VA_SLOT_COUNT);
+
+	if (slot < SGX_VA_SLOT_COUNT)
+		set_bit(slot, va_page->slots);
+
+	return slot << 3;
+}
+
+/**
+ * sgx_free_va_slot - free a VA slot
+ * @va_page:	a &struct sgx_va_page instance
+ * @offset:	offset of the slot inside the VA page
+ *
+ * Frees a slot from a &struct sgx_va_page instance.
+ */
+void sgx_free_va_slot(struct sgx_va_page *va_page, unsigned int offset)
+{
+	clear_bit(offset >> 3, va_page->slots);
+}
+
+/**
+ * sgx_va_page_full - is the VA page full?
+ * @va_page:	a &struct sgx_va_page instance
+ *
+ * Return: true if all slots have been taken
+ */
+bool sgx_va_page_full(struct sgx_va_page *va_page)
+{
+	int slot = find_first_zero_bit(va_page->slots, SGX_VA_SLOT_COUNT);
+
+	return slot == SGX_VA_SLOT_COUNT;
+}
diff --git a/drivers/platform/x86/intel_sgx/sgx_fault.c b/drivers/platform/x86/intel_sgx/sgx_fault.c
new file mode 100644
index 000000000000..e5b2786ce42b
--- /dev/null
+++ b/drivers/platform/x86/intel_sgx/sgx_fault.c
@@ -0,0 +1,159 @@
+// SPDX-License-Identifier: (GPL-2.0 OR BSD-3-Clause)
+// Copyright(c) 2016-17 Intel Corporation.
+//
+// Authors:
+//
+// Jarkko Sakkinen <jarkko.sakkinen@linux.intel.com>
+// Suresh Siddha <suresh.b.siddha@intel.com>
+// Serge Ayoun <serge.ayoun@intel.com>
+// Shay Katz-zamir <shay.katz-zamir@intel.com>
+// Sean Christopherson <sean.j.christopherson@intel.com>
+
+#include <linux/highmem.h>
+#include <linux/sched/mm.h>
+#include "sgx.h"
+
+static int sgx_test_and_clear_young_cb(pte_t *ptep, pgtable_t token,
+				       unsigned long addr, void *data)
+{
+	pte_t pte;
+	int ret;
+
+	ret = pte_young(*ptep);
+	if (ret) {
+		pte = pte_mkold(*ptep);
+		set_pte_at((struct mm_struct *)data, addr, ptep, pte);
+	}
+
+	return ret;
+}
+
+/**
+ * sgx_test_and_clear_young() - Test and reset the accessed bit
+ * @page:	enclave page to be tested for recent access
+ *
+ * Checks the Access (A) bit from the PTE corresponding to the
+ * enclave page and clears it.  Returns 1 if the page has been
+ * recently accessed and 0 if not.
+ */
+int sgx_test_and_clear_young(struct sgx_encl_page *page)
+{
+	unsigned long addr = SGX_ENCL_PAGE_ADDR(page);
+	struct sgx_encl *encl = page->encl;
+	struct vm_area_struct *vma;
+	int ret;
+
+	ret = sgx_encl_find(encl->mm, addr, &vma);
+	if (ret)
+		return 0;
+
+	if (encl != vma->vm_private_data)
+		return 0;
+
+	return apply_to_page_range(vma->vm_mm, addr, PAGE_SIZE,
+				   sgx_test_and_clear_young_cb, vma->vm_mm);
+}
+
+static void sgx_ipi_cb(void *info)
+{
+}
+
+void sgx_flush_cpus(struct sgx_encl *encl)
+{
+	on_each_cpu_mask(mm_cpumask(encl->mm), sgx_ipi_cb, NULL, 1);
+}
+
+static struct sgx_epc_page *__sgx_load_faulted_page(
+	struct sgx_encl_page *encl_page)
+{
+	unsigned long va_offset = SGX_ENCL_PAGE_VA_OFFSET(encl_page);
+	struct sgx_encl *encl = encl_page->encl;
+	struct sgx_epc_page *epc_page;
+	int ret;
+
+	epc_page = sgx_alloc_page(&encl_page->impl, SGX_ALLOC_ATOMIC);
+	if (IS_ERR(epc_page))
+		return epc_page;
+	ret = sgx_encl_load_page(encl_page, epc_page);
+	if (ret) {
+		sgx_free_page(epc_page);
+		return ERR_PTR(ret);
+	}
+	sgx_free_va_slot(encl_page->va_page, va_offset);
+	list_move(&encl_page->va_page->list, &encl->va_pages);
+	encl_page->desc &= ~SGX_VA_OFFSET_MASK;
+	sgx_set_epc_page(encl_page, epc_page);
+	return epc_page;
+}
+
+static struct sgx_encl_page *__sgx_fault_page(struct vm_area_struct *vma,
+					      unsigned long addr,
+					      bool do_reserve)
+{
+	struct sgx_encl *encl = vma->vm_private_data;
+	struct sgx_epc_page *epc_page;
+	struct sgx_encl_page *entry;
+	int rc = 0;
+
+	if ((encl->flags & SGX_ENCL_DEAD) ||
+	    !(encl->flags & SGX_ENCL_INITIALIZED))
+		return ERR_PTR(-EFAULT);
+
+	entry = radix_tree_lookup(&encl->page_tree, addr >> PAGE_SHIFT);
+	if (!entry)
+		return ERR_PTR(-EFAULT);
+	if (entry->desc & SGX_ENCL_PAGE_RESERVED) {
+		sgx_dbg(encl, "0x%p is reserved\n",
+			(void *)SGX_ENCL_PAGE_ADDR(entry));
+		return ERR_PTR(-EBUSY);
+	}
+	/* Page was faulted by another thread. */
+	if (entry->desc & SGX_ENCL_PAGE_LOADED) {
+		if (do_reserve)
+			entry->desc |= SGX_ENCL_PAGE_RESERVED;
+		return entry;
+	}
+
+	if (!(encl->secs.desc & SGX_ENCL_PAGE_LOADED)) {
+		epc_page = __sgx_load_faulted_page(&encl->secs);
+		if (IS_ERR(epc_page))
+			return (void *)epc_page;
+	}
+	epc_page = __sgx_load_faulted_page(entry);
+	if (IS_ERR(epc_page))
+		return (void *)epc_page;
+
+	encl->secs_child_cnt++;
+	sgx_set_page_reclaimable(entry);
+	if (do_reserve)
+		entry->desc |= SGX_ENCL_PAGE_RESERVED;
+
+	rc = vm_insert_pfn(vma, addr, SGX_EPC_PFN(entry->epc_page));
+	SGX_INVD(rc, encl, "%s: vm_insert_pfn() returned %d\n", __func__, rc);
+	if (rc)
+		return ERR_PTR(rc);
+
+	return entry;
+}
+
+struct sgx_encl_page *sgx_fault_page(struct vm_area_struct *vma,
+				     unsigned long addr, bool do_reserve)
+{
+	struct sgx_encl *encl = vma->vm_private_data;
+	struct sgx_encl_page *entry;
+
+	/* If process was forked, VMA is still there but vm_private_data is set
+	 * to NULL.
+	 */
+	if (!encl)
+		return ERR_PTR(-EFAULT);
+	do {
+		mutex_lock(&encl->lock);
+		entry = __sgx_fault_page(vma, addr, do_reserve);
+		mutex_unlock(&encl->lock);
+		if (!do_reserve)
+			break;
+	} while (PTR_ERR(entry) == -EBUSY);
+
+	return entry;
+}
diff --git a/drivers/platform/x86/intel_sgx/sgx_ioctl.c b/drivers/platform/x86/intel_sgx/sgx_ioctl.c
new file mode 100644
index 000000000000..68b04893d66a
--- /dev/null
+++ b/drivers/platform/x86/intel_sgx/sgx_ioctl.c
@@ -0,0 +1,233 @@
+// SPDX-License-Identifier: (GPL-2.0 OR BSD-3-Clause)
+// Copyright(c) 2016-17 Intel Corporation.
+//
+// Authors:
+//
+// Jarkko Sakkinen <jarkko.sakkinen@linux.intel.com>
+// Suresh Siddha <suresh.b.siddha@intel.com>
+// Serge Ayoun <serge.ayoun@intel.com>
+// Shay Katz-zamir <shay.katz-zamir@intel.com>
+// Sean Christopherson <sean.j.christopherson@intel.com>
+
+#include <asm/mman.h>
+#include <linux/delay.h>
+#include <linux/file.h>
+#include <linux/hashtable.h>
+#include <linux/highmem.h>
+#include <linux/ratelimit.h>
+#include <linux/sched/signal.h>
+#include <linux/slab.h>
+#include "sgx.h"
+
+static int sgx_encl_get(unsigned long addr, struct sgx_encl **encl)
+{
+	struct mm_struct *mm = current->mm;
+	struct vm_area_struct *vma;
+	int ret;
+
+	if (addr & (PAGE_SIZE - 1))
+		return -EINVAL;
+
+	down_read(&mm->mmap_sem);
+
+	ret = sgx_encl_find(mm, addr, &vma);
+	if (!ret) {
+		*encl = vma->vm_private_data;
+
+		if ((*encl)->flags & SGX_ENCL_SUSPEND)
+			ret = SGX_POWER_LOST_ENCLAVE;
+		else
+			kref_get(&(*encl)->refcount);
+	}
+
+	up_read(&mm->mmap_sem);
+	return ret;
+}
+
+/**
+ * sgx_ioc_enclave_create - handler for %SGX_IOC_ENCLAVE_CREATE
+ * @filep:	open file to /dev/sgx
+ * @cmd:	the command value
+ * @arg:	pointer to an &sgx_enclave_create instance
+ *
+ * Validates SECS attributes, allocates an EPC page for the SECS and performs
+ * ECREATE.
+ *
+ * Return:
+ *   0 on success,
+ *   -errno otherwise
+ */
+static long sgx_ioc_enclave_create(struct file *filep, unsigned int cmd,
+				   unsigned long arg)
+{
+	struct sgx_enclave_create *createp = (struct sgx_enclave_create *)arg;
+	struct sgx_secs *secs;
+	struct sgx_encl *encl;
+	int ret;
+
+	secs = kzalloc(sizeof(*secs),  GFP_KERNEL);
+	if (!secs)
+		return -ENOMEM;
+
+	ret = copy_from_user(secs, (void __user *)createp->src, sizeof(*secs));
+	if (ret)
+		goto out;
+
+	encl = sgx_encl_alloc(secs);
+	if (IS_ERR(encl)) {
+		ret = PTR_ERR(encl);
+		goto out;
+	}
+
+	ret = sgx_encl_create(encl, secs);
+	if (ret)
+		kref_put(&encl->refcount, sgx_encl_release);
+
+out:
+	kfree(secs);
+	return ret;
+}
+
+/**
+ * sgx_ioc_enclave_add_page - handler for %SGX_IOC_ENCLAVE_ADD_PAGE
+ *
+ * @filep:	open file to /dev/sgx
+ * @cmd:	the command value
+ * @arg:	pointer to an &sgx_enclave_add_page instance
+ *
+ * Creates a new enclave page and enqueues an EADD operation that will be
+ * processed by a worker thread later on.
+ *
+ * Return:
+ *   0 on success,
+ *   -errno otherwise
+ */
+static long sgx_ioc_enclave_add_page(struct file *filep, unsigned int cmd,
+				     unsigned long arg)
+{
+	struct sgx_enclave_add_page *addp = (void *)arg;
+	struct sgx_secinfo secinfo;
+	struct sgx_encl *encl;
+	struct page *data_page;
+	void *data;
+	int ret;
+
+	ret = sgx_encl_get(addp->addr, &encl);
+	if (ret)
+		return ret;
+
+	if (copy_from_user(&secinfo, (void __user *)addp->secinfo,
+			   sizeof(secinfo))) {
+		kref_put(&encl->refcount, sgx_encl_release);
+		return -EFAULT;
+	}
+
+	data_page = alloc_page(GFP_HIGHUSER);
+	if (!data_page) {
+		kref_put(&encl->refcount, sgx_encl_release);
+		return -ENOMEM;
+	}
+
+	data = kmap(data_page);
+
+	ret = copy_from_user((void *)data, (void __user *)addp->src, PAGE_SIZE);
+	if (ret)
+		goto out;
+
+	ret = sgx_encl_add_page(encl, addp->addr, data, &secinfo, addp->mrmask);
+	if (ret)
+		goto out;
+
+out:
+	kref_put(&encl->refcount, sgx_encl_release);
+	kunmap(data_page);
+	__free_page(data_page);
+	return ret;
+}
+
+/**
+ * sgx_ioc_enclave_init - handler for %SGX_IOC_ENCLAVE_INIT
+ *
+ * @filep:	open file to /dev/sgx
+ * @cmd:	the command value
+ * @arg:	pointer to an &sgx_enclave_init instance
+ *
+ * Flushes the remaining enqueued EADD operations and performs EINIT.
+ *
+ * Return:
+ *   0 on success,
+ *   SGX error code on EINIT failure,
+ *   -errno otherwise
+ */
+static long sgx_ioc_enclave_init(struct file *filep, unsigned int cmd,
+				 unsigned long arg)
+{
+	struct sgx_enclave_init *initp = (struct sgx_enclave_init *)arg;
+	struct sgx_sigstruct *sigstruct;
+	struct sgx_einittoken *einittoken;
+	struct sgx_encl *encl;
+	struct page *initp_page;
+	int ret;
+
+	initp_page = alloc_page(GFP_HIGHUSER);
+	if (!initp_page)
+		return -ENOMEM;
+
+	sigstruct = kmap(initp_page);
+	einittoken = (struct sgx_einittoken *)
+		((unsigned long)sigstruct + PAGE_SIZE / 2);
+
+	ret = copy_from_user(sigstruct, (void __user *)initp->sigstruct,
+			     sizeof(*sigstruct));
+	if (ret)
+		goto out;
+
+	ret = sgx_encl_get(initp->addr, &encl);
+	if (ret)
+		goto out;
+
+	ret = sgx_encl_init(encl, sigstruct, einittoken);
+
+	kref_put(&encl->refcount, sgx_encl_release);
+
+out:
+	kunmap(initp_page);
+	__free_page(initp_page);
+	return ret;
+}
+
+typedef long (*sgx_ioc_t)(struct file *filep, unsigned int cmd,
+			  unsigned long arg);
+
+long sgx_ioctl(struct file *filep, unsigned int cmd, unsigned long arg)
+{
+	char data[256];
+	sgx_ioc_t handler = NULL;
+	long ret;
+
+	switch (cmd) {
+	case SGX_IOC_ENCLAVE_CREATE:
+		handler = sgx_ioc_enclave_create;
+		break;
+	case SGX_IOC_ENCLAVE_ADD_PAGE:
+		handler = sgx_ioc_enclave_add_page;
+		break;
+	case SGX_IOC_ENCLAVE_INIT:
+		handler = sgx_ioc_enclave_init;
+		break;
+	default:
+		return -ENOIOCTLCMD;
+	}
+
+	if (copy_from_user(data, (void __user *)arg, _IOC_SIZE(cmd)))
+		return -EFAULT;
+
+	ret = handler(filep, cmd, (unsigned long)((void *)data));
+	if (!ret && (cmd & IOC_OUT)) {
+		if (copy_to_user((void __user *)arg, data, _IOC_SIZE(cmd)))
+			return -EFAULT;
+	}
+	if (IS_ENCLS_FAULT(ret))
+		return -EFAULT;
+	return ret;
+}
diff --git a/drivers/platform/x86/intel_sgx/sgx_main.c b/drivers/platform/x86/intel_sgx/sgx_main.c
new file mode 100644
index 000000000000..79d772847026
--- /dev/null
+++ b/drivers/platform/x86/intel_sgx/sgx_main.c
@@ -0,0 +1,306 @@
+// SPDX-License-Identifier: (GPL-2.0 OR BSD-3-Clause)
+// Copyright(c) 2016-17 Intel Corporation.
+//
+// Authors:
+//
+// Jarkko Sakkinen <jarkko.sakkinen@linux.intel.com>
+// Suresh Siddha <suresh.b.siddha@intel.com>
+// Serge Ayoun <serge.ayoun@intel.com>
+// Shay Katz-zamir <shay.katz-zamir@intel.com>
+// Sean Christopherson <sean.j.christopherson@intel.com>
+
+#include <linux/acpi.h>
+#include <linux/cdev.h>
+#include <linux/platform_device.h>
+#include <linux/suspend.h>
+#include "sgx.h"
+
+#define DRV_DESCRIPTION "Intel SGX Driver"
+
+MODULE_DESCRIPTION("Intel SGX Driver");
+MODULE_AUTHOR("Jarkko Sakkinen <jarkko.sakkinen@linux.intel.com>");
+MODULE_LICENSE("Dual BSD/GPL");
+
+/*
+ * Global data.
+ */
+
+struct workqueue_struct *sgx_add_page_wq;
+u64 sgx_encl_size_max_32;
+u64 sgx_encl_size_max_64;
+u64 sgx_xfrm_mask = 0x3;
+u32 sgx_misc_reserved;
+u32 sgx_xsave_size_tbl[64];
+
+#ifdef CONFIG_COMPAT
+long sgx_compat_ioctl(struct file *filep, unsigned int cmd, unsigned long arg)
+{
+	return sgx_ioctl(filep, cmd, arg);
+}
+#endif
+
+static int sgx_mmap(struct file *file, struct vm_area_struct *vma)
+{
+	vma->vm_ops = &sgx_vm_ops;
+	vma->vm_flags |= VM_PFNMAP | VM_DONTEXPAND | VM_DONTDUMP | VM_IO |
+			 VM_DONTCOPY;
+
+	return 0;
+}
+
+static unsigned long sgx_get_unmapped_area(struct file *file,
+					   unsigned long addr,
+					   unsigned long len,
+					   unsigned long pgoff,
+					   unsigned long flags)
+{
+	if (len < 2 * PAGE_SIZE || (len & (len - 1)))
+		return -EINVAL;
+
+	/* On 64-bit architecture, allow mmap() to exceed 32-bit encl
+	 * limit only if the task is not running in 32-bit compatibility
+	 * mode.
+	 */
+	if (len > sgx_encl_size_max_32)
+#ifdef CONFIG_X86_64
+		if (test_thread_flag(TIF_ADDR32))
+			return -EINVAL;
+#else
+		return -EINVAL;
+#endif
+
+#ifdef CONFIG_X86_64
+	if (len > sgx_encl_size_max_64)
+		return -EINVAL;
+#endif
+
+	addr = current->mm->get_unmapped_area(file, addr, 2 * len, pgoff,
+					      flags);
+	if (IS_ERR_VALUE(addr))
+		return addr;
+
+	addr = (addr + (len - 1)) & ~(len - 1);
+
+	return addr;
+}
+
+static const struct file_operations sgx_fops = {
+	.owner			= THIS_MODULE,
+	.unlocked_ioctl		= sgx_ioctl,
+#ifdef CONFIG_COMPAT
+	.compat_ioctl		= sgx_compat_ioctl,
+#endif
+	.mmap			= sgx_mmap,
+	.get_unmapped_area	= sgx_get_unmapped_area,
+};
+
+static int sgx_pm_suspend(struct device *dev)
+{
+	struct sgx_encl_page *encl_page;
+	struct sgx_epc_page *epc_page;
+	struct sgx_encl *encl;
+
+	list_for_each_entry(epc_page, &sgx_active_page_list, list) {
+		encl_page = container_of(epc_page->impl, struct sgx_encl_page,
+					 impl);
+		encl = encl_page->encl;
+		sgx_invalidate(encl, false);
+		encl->flags |= SGX_ENCL_SUSPEND;
+		flush_work(&encl->add_page_work);
+	}
+
+	return 0;
+}
+
+static SIMPLE_DEV_PM_OPS(sgx_drv_pm, sgx_pm_suspend, NULL);
+
+static struct bus_type sgx_bus_type = {
+	.name	= "sgx",
+};
+
+struct sgx_context {
+	struct device dev;
+	struct cdev cdev;
+};
+
+static dev_t sgx_devt;
+
+static void sgx_dev_release(struct device *dev)
+{
+	struct sgx_context *ctx = container_of(dev, struct sgx_context, dev);
+
+	kfree(ctx);
+}
+
+static struct sgx_context *sgx_ctx_alloc(struct device *parent)
+{
+	struct sgx_context *ctx;
+
+	ctx = kzalloc(sizeof(*ctx), GFP_KERNEL);
+	if (!ctx)
+		return ERR_PTR(-ENOMEM);
+
+	device_initialize(&ctx->dev);
+
+	ctx->dev.bus = &sgx_bus_type;
+	ctx->dev.parent = parent;
+	ctx->dev.devt = MKDEV(MAJOR(sgx_devt), 0);
+	ctx->dev.release = sgx_dev_release;
+
+	dev_set_name(&ctx->dev, "sgx");
+
+	cdev_init(&ctx->cdev, &sgx_fops);
+	ctx->cdev.owner = THIS_MODULE;
+
+	dev_set_drvdata(parent, ctx);
+
+	return ctx;
+}
+
+static struct sgx_context *sgxm_ctx_alloc(struct device *parent)
+{
+	struct sgx_context *ctx;
+	int rc;
+
+	ctx = sgx_ctx_alloc(parent);
+	if (IS_ERR(ctx))
+		return ctx;
+
+	rc = devm_add_action_or_reset(parent, (void (*)(void *))put_device,
+				      &ctx->dev);
+	if (rc) {
+		kfree(ctx);
+		return ERR_PTR(rc);
+	}
+
+	return ctx;
+}
+
+static int sgx_dev_init(struct device *parent)
+{
+	struct sgx_context *sgx_dev;
+	unsigned int eax;
+	unsigned int ebx;
+	unsigned int ecx;
+	unsigned int edx;
+	int ret;
+	int i;
+
+	sgx_dev = sgxm_ctx_alloc(parent);
+
+	cpuid_count(SGX_CPUID, SGX_CPUID_CAPABILITIES, &eax, &ebx, &ecx, &edx);
+	/* Only allow misc bits supported by the driver. */
+	sgx_misc_reserved = ~ebx | SGX_MISC_RESERVED_MASK;
+#ifdef CONFIG_X86_64
+	sgx_encl_size_max_64 = 1ULL << ((edx >> 8) & 0xFF);
+#endif
+	sgx_encl_size_max_32 = 1ULL << (edx & 0xFF);
+
+	if (boot_cpu_has(X86_FEATURE_OSXSAVE)) {
+		cpuid_count(SGX_CPUID, SGX_CPUID_ATTRIBUTES, &eax, &ebx, &ecx,
+			    &edx);
+		sgx_xfrm_mask = (((u64)edx) << 32) + (u64)ecx;
+
+		for (i = 2; i < 64; i++) {
+			cpuid_count(0x0D, i, &eax, &ebx, &ecx, &edx);
+			if ((1 << i) & sgx_xfrm_mask)
+				sgx_xsave_size_tbl[i] = eax + ebx;
+		}
+	}
+
+	sgx_add_page_wq = alloc_workqueue("intel_sgx-add-page-wq",
+					  WQ_UNBOUND | WQ_FREEZABLE, 1);
+	if (!sgx_add_page_wq)
+		return -ENOMEM;
+
+	ret = cdev_device_add(&sgx_dev->cdev, &sgx_dev->dev);
+	if (ret)
+		goto out_workqueue;
+
+	return 0;
+out_workqueue:
+	destroy_workqueue(sgx_add_page_wq);
+	return ret;
+}
+
+static int sgx_drv_probe(struct platform_device *pdev)
+{
+	if (!sgx_enabled || !boot_cpu_has(X86_FEATURE_SGX_LC))
+		return -ENODEV;
+
+	return sgx_dev_init(&pdev->dev);
+}
+
+static int sgx_drv_remove(struct platform_device *pdev)
+{
+	struct sgx_context *ctx = dev_get_drvdata(&pdev->dev);
+
+	cdev_device_del(&ctx->cdev, &ctx->dev);
+	destroy_workqueue(sgx_add_page_wq);
+
+	return 0;
+}
+
+#ifdef CONFIG_ACPI
+static struct acpi_device_id sgx_device_ids[] = {
+	{"INT0E0C", 0},
+	{"", 0},
+};
+MODULE_DEVICE_TABLE(acpi, sgx_device_ids);
+#endif
+
+static struct platform_driver sgx_drv = {
+	.probe = sgx_drv_probe,
+	.remove = sgx_drv_remove,
+	.driver = {
+		.name			= "intel_sgx",
+		.pm			= &sgx_drv_pm,
+		.acpi_match_table	= ACPI_PTR(sgx_device_ids),
+	},
+};
+
+static int __init sgx_drv_subsys_init(void)
+{
+	int ret;
+
+	ret = bus_register(&sgx_bus_type);
+	if (ret)
+		return ret;
+
+	ret = alloc_chrdev_region(&sgx_devt, 0, 1, "sgx");
+	if (ret < 0) {
+		bus_unregister(&sgx_bus_type);
+		return ret;
+	}
+
+	return 0;
+}
+
+static void sgx_drv_subsys_exit(void)
+{
+	bus_unregister(&sgx_bus_type);
+	unregister_chrdev_region(sgx_devt, 1);
+}
+
+static int __init sgx_drv_init(void)
+{
+	int ret;
+
+	ret = sgx_drv_subsys_init();
+	if (ret)
+		return ret;
+
+	ret = platform_driver_register(&sgx_drv);
+	if (ret)
+		sgx_drv_subsys_exit();
+
+	return ret;
+}
+module_init(sgx_drv_init);
+
+static void __exit sgx_drv_exit(void)
+{
+	platform_driver_unregister(&sgx_drv);
+	sgx_drv_subsys_exit();
+}
+module_exit(sgx_drv_exit);
diff --git a/drivers/platform/x86/intel_sgx/sgx_vma.c b/drivers/platform/x86/intel_sgx/sgx_vma.c
new file mode 100644
index 000000000000..ad47383ea7f5
--- /dev/null
+++ b/drivers/platform/x86/intel_sgx/sgx_vma.c
@@ -0,0 +1,66 @@
+// SPDX-License-Identifier: (GPL-2.0 OR BSD-3-Clause)
+// Copyright(c) 2016-17 Intel Corporation.
+//
+// Authors:
+//
+// Jarkko Sakkinen <jarkko.sakkinen@linux.intel.com>
+// Suresh Siddha <suresh.b.siddha@intel.com>
+// Serge Ayoun <serge.ayoun@intel.com>
+// Shay Katz-zamir <shay.katz-zamir@intel.com>
+// Sean Christopherson <sean.j.christopherson@intel.com>
+
+#include <asm/mman.h>
+#include <linux/delay.h>
+#include <linux/file.h>
+#include <linux/hashtable.h>
+#include <linux/highmem.h>
+#include <linux/mm.h>
+#include <linux/ratelimit.h>
+#include <linux/slab.h>
+#include "sgx.h"
+
+static void sgx_vma_open(struct vm_area_struct *vma)
+{
+	struct sgx_encl *encl = vma->vm_private_data;
+
+	if (!encl)
+		return;
+
+	/* kref cannot underflow because ECREATE ioctl checks that there is only
+	 * one single VMA for the enclave before proceeding.
+	 */
+	kref_get(&encl->refcount);
+}
+
+static void sgx_vma_close(struct vm_area_struct *vma)
+{
+	struct sgx_encl *encl = vma->vm_private_data;
+
+	if (!encl)
+		return;
+
+	mutex_lock(&encl->lock);
+	sgx_invalidate(encl, true);
+	mutex_unlock(&encl->lock);
+	kref_put(&encl->refcount, sgx_encl_release);
+}
+
+static int sgx_vma_fault(struct vm_fault *vmf)
+{
+	unsigned long addr = (unsigned long)vmf->address;
+	struct vm_area_struct *vma = vmf->vma;
+	struct sgx_encl_page *entry;
+
+	entry = sgx_fault_page(vma, addr, 0);
+
+	if (!IS_ERR(entry) || PTR_ERR(entry) == -EBUSY)
+		return VM_FAULT_NOPAGE;
+	else
+		return VM_FAULT_SIGBUS;
+}
+
+const struct vm_operations_struct sgx_vm_ops = {
+	.close = sgx_vma_close,
+	.open = sgx_vma_open,
+	.fault = sgx_vma_fault,
+};
-- 
2.17.0

[PATCH v11 11/13] intel_sgx: ptrace() support

[Jarkko Sakkinen]

Implemented VMA callbacks in order to ptrace() debug enclaves. With
debug enclaves data can be read and write the memory word at a time by
using ENCLS(EDBGRD) and ENCLS(EDBGWR) leaf instructions.

Signed-off-by: Jarkko Sakkinen <jarkko.sakkinen@linux.intel.com>
Tested-by: Serge Ayoun <serge.ayoun@intel.com>
---
 drivers/platform/x86/intel_sgx/sgx_encl.c |   2 +-
 drivers/platform/x86/intel_sgx/sgx_vma.c  | 116 ++++++++++++++++++++++
 2 files changed, 117 insertions(+), 1 deletion(-)

diff --git a/drivers/platform/x86/intel_sgx/sgx_encl.c b/drivers/platform/x86/intel_sgx/sgx_encl.c
index 562d4ce412d4..35436497530b 100644
--- a/drivers/platform/x86/intel_sgx/sgx_encl.c
+++ b/drivers/platform/x86/intel_sgx/sgx_encl.c
@@ -945,7 +945,7 @@ int sgx_encl_load_page(struct sgx_encl_page *encl_page,
 	ret = __eldu(&pginfo, epc_ptr, va_ptr + va_offset);
 	if (ret) {
 		sgx_err(encl, "ELDU returned %d\n", ret);
-		ret = -EFAULT;
+		ret = ENCLS_TO_ERR(ret);
 	}
 
 	kunmap_atomic((void *)(unsigned long)(pginfo.pcmd - pcmd_offset));
diff --git a/drivers/platform/x86/intel_sgx/sgx_vma.c b/drivers/platform/x86/intel_sgx/sgx_vma.c
index ad47383ea7f5..afc02d70c618 100644
--- a/drivers/platform/x86/intel_sgx/sgx_vma.c
+++ b/drivers/platform/x86/intel_sgx/sgx_vma.c
@@ -59,8 +59,124 @@ static int sgx_vma_fault(struct vm_fault *vmf)
 		return VM_FAULT_SIGBUS;
 }
 
+static int sgx_edbgrd(struct sgx_encl *encl, struct sgx_encl_page *page,
+		      unsigned long addr, void *data)
+{
+	unsigned long offset;
+	void *ptr;
+	int ret;
+
+	offset = addr & ~PAGE_MASK;
+
+	if ((page->desc & SGX_ENCL_PAGE_TCS) &&
+            offset > offsetof(struct sgx_tcs, gslimit))
+		return -ECANCELED;
+
+	ptr = sgx_get_page(page->epc_page);
+	ret = __edbgrd((unsigned long)ptr + offset, data);
+	sgx_put_page(ptr);
+	if (ret) {
+		sgx_dbg(encl, "EDBGRD returned %d\n", ret);
+		return ENCLS_TO_ERR(ret);
+	}
+
+	return 0;
+}
+
+static int sgx_edbgwr(struct sgx_encl *encl, struct sgx_encl_page *page,
+		      unsigned long addr, void *data)
+{
+	unsigned long offset;
+	void *ptr;
+	int ret;
+
+	offset = addr & ~PAGE_MASK;
+
+	/* Writing anything else than flags will cause #GP */
+	if ((page->desc & SGX_ENCL_PAGE_TCS) &&
+	    offset != offsetof(struct sgx_tcs, flags))
+		return -ECANCELED;
+
+	ptr = sgx_get_page(page->epc_page);
+	ret = __edbgwr((unsigned long)ptr + offset, data);
+	sgx_put_page(ptr);
+	if (ret) {
+		sgx_dbg(encl, "EDBGWR returned %d\n", ret);
+		return ENCLS_TO_ERR(ret);
+	}
+
+	return 0;
+}
+
+static int sgx_vma_access(struct vm_area_struct *vma, unsigned long addr,
+			  void *buf, int len, int write)
+{
+	struct sgx_encl *encl = vma->vm_private_data;
+	struct sgx_encl_page *entry = NULL;
+	unsigned long align;
+	char data[sizeof(unsigned long)];
+	int offset;
+	int cnt;
+	int ret = 0;
+	int i;
+
+	/* If process was forked, VMA is still there but vm_private_data is set
+	 * to NULL.
+	 */
+	if (!encl)
+		return -EFAULT;
+
+	if (!(encl->flags & SGX_ENCL_DEBUG) ||
+	    !(encl->flags & SGX_ENCL_INITIALIZED) ||
+	    (encl->flags & SGX_ENCL_DEAD))
+		return -EFAULT;
+
+	for (i = 0; i < len; i += cnt) {
+		if (!entry || !((addr + i) & (PAGE_SIZE - 1))) {
+			if (entry)
+				entry->desc &= ~SGX_ENCL_PAGE_RESERVED;
+
+			entry = sgx_fault_page(vma, (addr + i) & PAGE_MASK,
+					       true);
+			if (IS_ERR(entry)) {
+				ret = PTR_ERR(entry);
+				entry = NULL;
+				break;
+			}
+		}
+
+		/* Locking is not needed because only immutable fields of the
+		 * page are accessed and page itself is reserved so that it
+		 * cannot be swapped out in the middle.
+		 */
+
+		align = ALIGN_DOWN(addr + i, sizeof(unsigned long));
+		offset = (addr + i) & (sizeof(unsigned long) - 1);
+		cnt = sizeof(unsigned long) - offset;
+		cnt = min(cnt, len - i);
+
+		ret = sgx_edbgrd(encl, entry, align, data);
+		if (ret)
+			break;
+		if (write) {
+			memcpy(data + offset, buf + i, cnt);
+			ret = sgx_edbgwr(encl, entry, align, data);
+			if (ret)
+				break;
+		}
+		else
+			memcpy(buf + i,data + offset, cnt);
+	}
+
+	if (entry)
+		entry->desc &= ~SGX_ENCL_PAGE_RESERVED;
+
+	return (ret < 0 && ret != -ECANCELED) ? ret : i;
+}
+
 const struct vm_operations_struct sgx_vm_ops = {
 	.close = sgx_vma_close,
 	.open = sgx_vma_open,
 	.fault = sgx_vma_fault,
+	.access = sgx_vma_access,
 };
-- 
2.17.0

[PATCH v11 11/13] intel_sgx: ptrace() support

[Jarkko Sakkinen]

Implemented VMA callbacks in order to ptrace() debug enclaves. With
debug enclaves data can be read and write the memory word at a time by
using ENCLS(EDBGRD) and ENCLS(EDBGWR) leaf instructions.

Signed-off-by: Jarkko Sakkinen <jarkko.sakkinen@linux.intel.com>
Tested-by: Serge Ayoun <serge.ayoun@intel.com>
---
 drivers/platform/x86/intel_sgx/sgx_encl.c |   2 +-
 drivers/platform/x86/intel_sgx/sgx_vma.c  | 116 ++++++++++++++++++++++
 2 files changed, 117 insertions(+), 1 deletion(-)

diff --git a/drivers/platform/x86/intel_sgx/sgx_encl.c b/drivers/platform/x86/intel_sgx/sgx_encl.c
index 562d4ce412d4..35436497530b 100644
--- a/drivers/platform/x86/intel_sgx/sgx_encl.c
+++ b/drivers/platform/x86/intel_sgx/sgx_encl.c
@@ -945,7 +945,7 @@ int sgx_encl_load_page(struct sgx_encl_page *encl_page,
 	ret = __eldu(&pginfo, epc_ptr, va_ptr + va_offset);
 	if (ret) {
 		sgx_err(encl, "ELDU returned %d\n", ret);
-		ret = -EFAULT;
+		ret = ENCLS_TO_ERR(ret);
 	}
 
 	kunmap_atomic((void *)(unsigned long)(pginfo.pcmd - pcmd_offset));
diff --git a/drivers/platform/x86/intel_sgx/sgx_vma.c b/drivers/platform/x86/intel_sgx/sgx_vma.c
index ad47383ea7f5..afc02d70c618 100644
--- a/drivers/platform/x86/intel_sgx/sgx_vma.c
+++ b/drivers/platform/x86/intel_sgx/sgx_vma.c
@@ -59,8 +59,124 @@ static int sgx_vma_fault(struct vm_fault *vmf)
 		return VM_FAULT_SIGBUS;
 }
 
+static int sgx_edbgrd(struct sgx_encl *encl, struct sgx_encl_page *page,
+		      unsigned long addr, void *data)
+{
+	unsigned long offset;
+	void *ptr;
+	int ret;
+
+	offset = addr & ~PAGE_MASK;
+
+	if ((page->desc & SGX_ENCL_PAGE_TCS) &&
+            offset > offsetof(struct sgx_tcs, gslimit))
+		return -ECANCELED;
+
+	ptr = sgx_get_page(page->epc_page);
+	ret = __edbgrd((unsigned long)ptr + offset, data);
+	sgx_put_page(ptr);
+	if (ret) {
+		sgx_dbg(encl, "EDBGRD returned %d\n", ret);
+		return ENCLS_TO_ERR(ret);
+	}
+
+	return 0;
+}
+
+static int sgx_edbgwr(struct sgx_encl *encl, struct sgx_encl_page *page,
+		      unsigned long addr, void *data)
+{
+	unsigned long offset;
+	void *ptr;
+	int ret;
+
+	offset = addr & ~PAGE_MASK;
+
+	/* Writing anything else than flags will cause #GP */
+	if ((page->desc & SGX_ENCL_PAGE_TCS) &&
+	    offset != offsetof(struct sgx_tcs, flags))
+		return -ECANCELED;
+
+	ptr = sgx_get_page(page->epc_page);
+	ret = __edbgwr((unsigned long)ptr + offset, data);
+	sgx_put_page(ptr);
+	if (ret) {
+		sgx_dbg(encl, "EDBGWR returned %d\n", ret);
+		return ENCLS_TO_ERR(ret);
+	}
+
+	return 0;
+}
+
+static int sgx_vma_access(struct vm_area_struct *vma, unsigned long addr,
+			  void *buf, int len, int write)
+{
+	struct sgx_encl *encl = vma->vm_private_data;
+	struct sgx_encl_page *entry = NULL;
+	unsigned long align;
+	char data[sizeof(unsigned long)];
+	int offset;
+	int cnt;
+	int ret = 0;
+	int i;
+
+	/* If process was forked, VMA is still there but vm_private_data is set
+	 * to NULL.
+	 */
+	if (!encl)
+		return -EFAULT;
+
+	if (!(encl->flags & SGX_ENCL_DEBUG) ||
+	    !(encl->flags & SGX_ENCL_INITIALIZED) ||
+	    (encl->flags & SGX_ENCL_DEAD))
+		return -EFAULT;
+
+	for (i = 0; i < len; i += cnt) {
+		if (!entry || !((addr + i) & (PAGE_SIZE - 1))) {
+			if (entry)
+				entry->desc &= ~SGX_ENCL_PAGE_RESERVED;
+
+			entry = sgx_fault_page(vma, (addr + i) & PAGE_MASK,
+					       true);
+			if (IS_ERR(entry)) {
+				ret = PTR_ERR(entry);
+				entry = NULL;
+				break;
+			}
+		}
+
+		/* Locking is not needed because only immutable fields of the
+		 * page are accessed and page itself is reserved so that it
+		 * cannot be swapped out in the middle.
+		 */
+
+		align = ALIGN_DOWN(addr + i, sizeof(unsigned long));
+		offset = (addr + i) & (sizeof(unsigned long) - 1);
+		cnt = sizeof(unsigned long) - offset;
+		cnt = min(cnt, len - i);
+
+		ret = sgx_edbgrd(encl, entry, align, data);
+		if (ret)
+			break;
+		if (write) {
+			memcpy(data + offset, buf + i, cnt);
+			ret = sgx_edbgwr(encl, entry, align, data);
+			if (ret)
+				break;
+		}
+		else
+			memcpy(buf + i,data + offset, cnt);
+	}
+
+	if (entry)
+		entry->desc &= ~SGX_ENCL_PAGE_RESERVED;
+
+	return (ret < 0 && ret != -ECANCELED) ? ret : i;
+}
+
 const struct vm_operations_struct sgx_vm_ops = {
 	.close = sgx_vma_close,
 	.open = sgx_vma_open,
 	.fault = sgx_vma_fault,
+	.access = sgx_vma_access,
 };
-- 
2.17.0

[PATCH v11 12/13] intel_sgx: driver documentation

[Jarkko Sakkinen]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain; charset=y, Size: 9660 bytes --]

Documentation of the features of the  Software Guard eXtensions usable
for the Linux kernel and how the driver internals uses these features.
In addition, contains documentation for the ioctl API.

Signed-off-by: Jarkko Sakkinen <jarkko.sakkinen@linux.intel.com>
---
 Documentation/index.rst         |   1 +
 Documentation/x86/intel_sgx.rst | 195 ++++++++++++++++++++++++++++++++
 2 files changed, 196 insertions(+)
 create mode 100644 Documentation/x86/intel_sgx.rst

diff --git a/Documentation/index.rst b/Documentation/index.rst
index 3b99ab931d41..b9fb92928e8c 100644
--- a/Documentation/index.rst
+++ b/Documentation/index.rst
@@ -100,6 +100,7 @@ implementation.
    :maxdepth: 2
 
    sh/index
+   x86/index
 
 Korean translations
 -------------------
diff --git a/Documentation/x86/intel_sgx.rst b/Documentation/x86/intel_sgx.rst
new file mode 100644
index 000000000000..ecbe544eb2cb
--- /dev/null
+++ b/Documentation/x86/intel_sgx.rst
@@ -0,0 +1,195 @@
+===================
+Intel(R) SGX driver
+===================
+
+Introduction
+============
+
+Intel(R) SGX is a set of CPU instructions that can be used by applications to
+set aside private regions of code and data. The code outside the enclave is
+disallowed to access the memory inside the enclave by the CPU access control.
+In a way you can think that SGX provides inverted sandbox. It protects the
+application from a malicious host.
+
+You can tell if your CPU supports SGX by looking into ``/proc/cpuinfo``:
+
+	``cat /proc/cpuinfo  | grep sgx``
+
+Overview of SGX
+===============
+
+SGX has a set of data structures to maintain information about the enclaves and
+their security properties. BIOS reserves a fixed size region of physical memory
+for these structures by setting Processor Reserved Memory Range Registers
+(PRMRR).
+
+This memory range is protected from outside access by the CPU and all the data
+coming in and out of the CPU package is encrypted by a key that is generated for
+each boot cycle.
+
+Enclaves execute in ring-3 in a special enclave submode using pages from the
+reserved memory range. A fixed logical address range for the enclave is reserved
+by ENCLS(ECREATE), a leaf instruction used to create enclaves. It is referred in
+the documentation commonly as the ELRANGE.
+
+Every memory access to the ELRANGE is asserted by the CPU. If the CPU is not
+executing in the enclave mode inside the enclave, #GP is raised. On the other
+hand enclave code can make memory accesses both inside and outside of the
+ELRANGE.
+
+Enclave can only execute code inside the ELRANGE. Instructions that may cause
+VMEXIT, IO instructions and instructions that require a privilege change are
+prohibited inside the enclave. Interrupts and exceptions always cause enclave
+to exit and jump to an address outside the enclave given when the enclave is
+entered by using the leaf instruction ENCLS(EENTER).
+
+Data types
+----------
+
+The protected memory range contains the following data:
+
+* **Enclave Page Cache (EPC):** protected pages
+* **Enclave Page Cache Map (EPCM):** a database that describes the state of the
+  pages and link them to an enclave.
+
+EPC has a number of different types of pages:
+
+* **SGX Enclave Control Structure (SECS)**: describes the global
+  properties of an enclave.
+* **Regular (REG):** code and data pages in the ELRANGE.
+* **Thread Control Structure (TCS):** pages that define entry points inside an
+  enclave. The enclave can only be entered through these entry points and each
+  can host a single hardware thread at a time.
+* **Version Array (VA)**: 64-bit version numbers for pages that have been
+  swapped outside the enclave. Each page contains 512 version numbers.
+
+Launch control
+--------------
+
+To launch an enclave, two structures must be provided for ENCLS(EINIT):
+
+1. **SIGSTRUCT:** signed measurement of the enclave binary.
+2. **EINITTOKEN:** a cryptographic token CMAC-signed with a AES256-key called
+   *launch key*, which is re-generated for each boot cycle.
+
+The CPU holds a SHA256 hash of a 3072-bit RSA public key inside
+IA32_SGXLEPUBKEYHASHn MSRs. Enclaves with a SIGSTRUCT that is signed with this
+key do not require a valid EINITTOKEN and can be authorized with special
+privileges. One of those privileges is ability to acquire the launch key with
+ENCLS(EGETKEY).
+
+**IA32_FEATURE_CONTROL[17]** is used by to BIOS configure whether
+IA32_SGXLEPUBKEYHASH MSRs are read-only or read-write before locking the
+feature control register and handing over control to the operating system.
+
+Enclave construction
+--------------------
+
+The construction is started by filling out the SECS that contains enclave
+address range, privileged attributes and measurement of TCS and REG pages (pages
+that will be mapped to the address range) among the other things. This structure
+is passed out to the ENCLS(ECREATE) together with a physical address of a page
+in EPC that will hold the SECS.
+
+Then pages are added with ENCLS(EADD) and measured with ENCLS(EEXTEND).  Finally
+enclave is initialized with ENCLS(EINIT). ENCLS(INIT) checks that the SIGSTRUCT
+is signed with the contained public key and that the supplied EINITTOKEN is
+valid (CMAC'd with the launch key). If these hold, the enclave is successfully
+initialized.
+
+Swapping pages
+--------------
+
+Enclave pages can be swapped out with ENCLS(EWB) to the unprotected memory. In
+addition to the EPC page, ENCLS(EWB) takes in a VA page and address for PCMD
+structure (Page Crypto MetaData) as input. The VA page will seal a version
+number for the page. PCMD is 128 byte structure that contains tracking
+information for the page, most importantly its MAC. With these structures the
+enclave is sealed and rollback protected while it resides in the unprotected
+memory.
+
+Before the page can be swapped out it must not have any active TLB references.
+By using ENCLS(EBLOCK) instructions no new TLB entries can be created to it.
+After this the a counter called *epoch* associated hardware threads inside the
+enclave is increased with ENCLS(ETRACK). After all the threads from the previous
+epoch have exited the page can be safely swapped out.
+
+An enclave memory access to a swapped out pages will cause #PF. #PF handler can
+fault the page back by using ENCLS(ELDU).
+
+Kernel internals
+================
+
+Requirements
+------------
+
+Because SGX has an ever evolving and expanding feature set, it's possible for
+a BIOS or VMM to configure a system in such a way that not all cpus are equal,
+e.g. where Launch Control is only enabled on a subset of cpus.  Linux does
+*not* support such a heterogenous system configuration, nor does it even
+attempt to play nice in the face of a misconfigured system.  With the exception
+of Launch Control's hash MSRs, which can vary per cpu, Linux assumes that all
+cpus have a configuration that is identical to the boot cpu.
+
+
+Roles and responsibilities
+--------------------------
+
+SGX introduces system resources, e.g. EPC memory, that must be accessible to
+multiple entities, e.g. the native kernel driver (to expose SGX to userspace)
+and KVM (to expose SGX to VMs), ideally without introducing any dependencies
+between each SGX entity.  To that end, the kernel owns and manages the shared
+system resources, i.e. the EPC and Launch Control MSRs, and defines functions
+that provide appropriate access to the shared resources.  SGX support for
+userpace and VMs is left to the SGX platform driver and KVM respectively.
+
+Launching enclaves
+------------------
+
+For privileged enclaves the launch is performed simply by submitting the
+SIGSTRUCT for that enclave to ENCLS(EINIT). For unprivileged enclaves the
+driver hosts a process in ring-3 that hosts a launch enclave signed with a key
+supplied for kbuild.
+
+The current implementation of the launch enclave generates a token for any
+enclave. In the future it could be potentially extended to have ways to
+configure policy what can be lauched.
+
+The driver will fail to initialize if it cannot start its own launch enclave.
+A user space application can submit a SIGSTRUCT instance through the ioctl API.
+The kernel will take care of the rest.
+
+This design assures that the Linux kernel has always full control, which
+enclaves get to launch and which do not, even if the public key MSRs are
+read-only. Having launch intrinsics inside the kernel also enables easy
+development of enclaves without necessarily needing any heavy weight SDK.
+Having a low-barrier to implement enclaves could make sense for example for
+system daemons where amount of dependecies ought to be minimized.
+
+EPC management
+--------------
+
+Due to the unique requirements for swapping EPC pages, and because EPC pages
+(currently) do not have associated page structures, management of the EPC is
+not handled by the standard Linux swapper.  SGX directly handles swapping
+of EPC pages, including a kthread to initiate reclaim and a rudimentary LRU
+mechanism.  Consumsers of EPC pages, e.g. the SGX driver, are required to
+implement function callbacks that can be invoked by the kernel to age,
+swap, and/or forcefully reclaim a target EPC page.  In effect, the kernel
+controls what happens and when, while the consumers (driver, KVM, etc..) do
+the actual work.
+
+SGX uapi
+========
+
+.. kernel-doc:: drivers/platform/x86/intel_sgx/sgx_ioctl.c
+   :functions: sgx_ioc_enclave_create
+               sgx_ioc_enclave_add_page
+               sgx_ioc_enclave_init
+
+.. kernel-doc:: arch/x86/include/uapi/asm/sgx.h
+
+References
+==========
+
+* System Programming Manual: 39.1.4 Intel® SGX Launch Control Configuration
-- 
2.17.0

[PATCH v11 12/13] intel_sgx: driver documentation

[Jarkko Sakkinen]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain; charset=y, Size: 9854 bytes --]

Documentation of the features of the  Software Guard eXtensions usable
for the Linux kernel and how the driver internals uses these features.
In addition, contains documentation for the ioctl API.

Signed-off-by: Jarkko Sakkinen <jarkko.sakkinen@linux.intel.com>
---
 Documentation/index.rst         |   1 +
 Documentation/x86/intel_sgx.rst | 195 ++++++++++++++++++++++++++++++++
 2 files changed, 196 insertions(+)
 create mode 100644 Documentation/x86/intel_sgx.rst

diff --git a/Documentation/index.rst b/Documentation/index.rst
index 3b99ab931d41..b9fb92928e8c 100644
--- a/Documentation/index.rst
+++ b/Documentation/index.rst
@@ -100,6 +100,7 @@ implementation.
    :maxdepth: 2
 
    sh/index
+   x86/index
 
 Korean translations
 -------------------
diff --git a/Documentation/x86/intel_sgx.rst b/Documentation/x86/intel_sgx.rst
new file mode 100644
index 000000000000..ecbe544eb2cb
--- /dev/null
+++ b/Documentation/x86/intel_sgx.rst
@@ -0,0 +1,195 @@
+===================
+Intel(R) SGX driver
+===================
+
+Introduction
+============
+
+Intel(R) SGX is a set of CPU instructions that can be used by applications to
+set aside private regions of code and data. The code outside the enclave is
+disallowed to access the memory inside the enclave by the CPU access control.
+In a way you can think that SGX provides inverted sandbox. It protects the
+application from a malicious host.
+
+You can tell if your CPU supports SGX by looking into ``/proc/cpuinfo``:
+
+	``cat /proc/cpuinfo  | grep sgx``
+
+Overview of SGX
+===============
+
+SGX has a set of data structures to maintain information about the enclaves and
+their security properties. BIOS reserves a fixed size region of physical memory
+for these structures by setting Processor Reserved Memory Range Registers
+(PRMRR).
+
+This memory range is protected from outside access by the CPU and all the data
+coming in and out of the CPU package is encrypted by a key that is generated for
+each boot cycle.
+
+Enclaves execute in ring-3 in a special enclave submode using pages from the
+reserved memory range. A fixed logical address range for the enclave is reserved
+by ENCLS(ECREATE), a leaf instruction used to create enclaves. It is referred in
+the documentation commonly as the ELRANGE.
+
+Every memory access to the ELRANGE is asserted by the CPU. If the CPU is not
+executing in the enclave mode inside the enclave, #GP is raised. On the other
+hand enclave code can make memory accesses both inside and outside of the
+ELRANGE.
+
+Enclave can only execute code inside the ELRANGE. Instructions that may cause
+VMEXIT, IO instructions and instructions that require a privilege change are
+prohibited inside the enclave. Interrupts and exceptions always cause enclave
+to exit and jump to an address outside the enclave given when the enclave is
+entered by using the leaf instruction ENCLS(EENTER).
+
+Data types
+----------
+
+The protected memory range contains the following data:
+
+* **Enclave Page Cache (EPC):** protected pages
+* **Enclave Page Cache Map (EPCM):** a database that describes the state of the
+  pages and link them to an enclave.
+
+EPC has a number of different types of pages:
+
+* **SGX Enclave Control Structure (SECS)**: describes the global
+  properties of an enclave.
+* **Regular (REG):** code and data pages in the ELRANGE.
+* **Thread Control Structure (TCS):** pages that define entry points inside an
+  enclave. The enclave can only be entered through these entry points and each
+  can host a single hardware thread at a time.
+* **Version Array (VA)**: 64-bit version numbers for pages that have been
+  swapped outside the enclave. Each page contains 512 version numbers.
+
+Launch control
+--------------
+
+To launch an enclave, two structures must be provided for ENCLS(EINIT):
+
+1. **SIGSTRUCT:** signed measurement of the enclave binary.
+2. **EINITTOKEN:** a cryptographic token CMAC-signed with a AES256-key called
+   *launch key*, which is re-generated for each boot cycle.
+
+The CPU holds a SHA256 hash of a 3072-bit RSA public key inside
+IA32_SGXLEPUBKEYHASHn MSRs. Enclaves with a SIGSTRUCT that is signed with this
+key do not require a valid EINITTOKEN and can be authorized with special
+privileges. One of those privileges is ability to acquire the launch key with
+ENCLS(EGETKEY).
+
+**IA32_FEATURE_CONTROL[17]** is used by to BIOS configure whether
+IA32_SGXLEPUBKEYHASH MSRs are read-only or read-write before locking the
+feature control register and handing over control to the operating system.
+
+Enclave construction
+--------------------
+
+The construction is started by filling out the SECS that contains enclave
+address range, privileged attributes and measurement of TCS and REG pages (pages
+that will be mapped to the address range) among the other things. This structure
+is passed out to the ENCLS(ECREATE) together with a physical address of a page
+in EPC that will hold the SECS.
+
+Then pages are added with ENCLS(EADD) and measured with ENCLS(EEXTEND).  Finally
+enclave is initialized with ENCLS(EINIT). ENCLS(INIT) checks that the SIGSTRUCT
+is signed with the contained public key and that the supplied EINITTOKEN is
+valid (CMAC'd with the launch key). If these hold, the enclave is successfully
+initialized.
+
+Swapping pages
+--------------
+
+Enclave pages can be swapped out with ENCLS(EWB) to the unprotected memory. In
+addition to the EPC page, ENCLS(EWB) takes in a VA page and address for PCMD
+structure (Page Crypto MetaData) as input. The VA page will seal a version
+number for the page. PCMD is 128 byte structure that contains tracking
+information for the page, most importantly its MAC. With these structures the
+enclave is sealed and rollback protected while it resides in the unprotected
+memory.
+
+Before the page can be swapped out it must not have any active TLB references.
+By using ENCLS(EBLOCK) instructions no new TLB entries can be created to it.
+After this the a counter called *epoch* associated hardware threads inside the
+enclave is increased with ENCLS(ETRACK). After all the threads from the previous
+epoch have exited the page can be safely swapped out.
+
+An enclave memory access to a swapped out pages will cause #PF. #PF handler can
+fault the page back by using ENCLS(ELDU).
+
+Kernel internals
+================
+
+Requirements
+------------
+
+Because SGX has an ever evolving and expanding feature set, it's possible for
+a BIOS or VMM to configure a system in such a way that not all cpus are equal,
+e.g. where Launch Control is only enabled on a subset of cpus.  Linux does
+*not* support such a heterogenous system configuration, nor does it even
+attempt to play nice in the face of a misconfigured system.  With the exception
+of Launch Control's hash MSRs, which can vary per cpu, Linux assumes that all
+cpus have a configuration that is identical to the boot cpu.
+
+
+Roles and responsibilities
+--------------------------
+
+SGX introduces system resources, e.g. EPC memory, that must be accessible to
+multiple entities, e.g. the native kernel driver (to expose SGX to userspace)
+and KVM (to expose SGX to VMs), ideally without introducing any dependencies
+between each SGX entity.  To that end, the kernel owns and manages the shared
+system resources, i.e. the EPC and Launch Control MSRs, and defines functions
+that provide appropriate access to the shared resources.  SGX support for
+userpace and VMs is left to the SGX platform driver and KVM respectively.
+
+Launching enclaves
+------------------
+
+For privileged enclaves the launch is performed simply by submitting the
+SIGSTRUCT for that enclave to ENCLS(EINIT). For unprivileged enclaves the
+driver hosts a process in ring-3 that hosts a launch enclave signed with a key
+supplied for kbuild.
+
+The current implementation of the launch enclave generates a token for any
+enclave. In the future it could be potentially extended to have ways to
+configure policy what can be lauched.
+
+The driver will fail to initialize if it cannot start its own launch enclave.
+A user space application can submit a SIGSTRUCT instance through the ioctl API.
+The kernel will take care of the rest.
+
+This design assures that the Linux kernel has always full control, which
+enclaves get to launch and which do not, even if the public key MSRs are
+read-only. Having launch intrinsics inside the kernel also enables easy
+development of enclaves without necessarily needing any heavy weight SDK.
+Having a low-barrier to implement enclaves could make sense for example for
+system daemons where amount of dependecies ought to be minimized.
+
+EPC management
+--------------
+
+Due to the unique requirements for swapping EPC pages, and because EPC pages
+(currently) do not have associated page structures, management of the EPC is
+not handled by the standard Linux swapper.  SGX directly handles swapping
+of EPC pages, including a kthread to initiate reclaim and a rudimentary LRU
+mechanism.  Consumsers of EPC pages, e.g. the SGX driver, are required to
+implement function callbacks that can be invoked by the kernel to age,
+swap, and/or forcefully reclaim a target EPC page.  In effect, the kernel
+controls what happens and when, while the consumers (driver, KVM, etc..) do
+the actual work.
+
+SGX uapi
+========
+
+.. kernel-doc:: drivers/platform/x86/intel_sgx/sgx_ioctl.c
+   :functions: sgx_ioc_enclave_create
+               sgx_ioc_enclave_add_page
+               sgx_ioc_enclave_init
+
+.. kernel-doc:: arch/x86/include/uapi/asm/sgx.h
+
+References
+==========
+
+* System Programming Manual: 39.1.4 Intel® SGX Launch Control Configuration
-- 
2.17.0

--
To unsubscribe from this list: send the line "unsubscribe linux-doc" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[PATCH v11 12/13] intel_sgx: driver documentation

[Jarkko Sakkinen]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain; charset=y, Size: 9660 bytes --]

Documentation of the features of the  Software Guard eXtensions usable
for the Linux kernel and how the driver internals uses these features.
In addition, contains documentation for the ioctl API.

Signed-off-by: Jarkko Sakkinen <jarkko.sakkinen@linux.intel.com>
---
 Documentation/index.rst         |   1 +
 Documentation/x86/intel_sgx.rst | 195 ++++++++++++++++++++++++++++++++
 2 files changed, 196 insertions(+)
 create mode 100644 Documentation/x86/intel_sgx.rst

diff --git a/Documentation/index.rst b/Documentation/index.rst
index 3b99ab931d41..b9fb92928e8c 100644
--- a/Documentation/index.rst
+++ b/Documentation/index.rst
@@ -100,6 +100,7 @@ implementation.
    :maxdepth: 2
 
    sh/index
+   x86/index
 
 Korean translations
 -------------------
diff --git a/Documentation/x86/intel_sgx.rst b/Documentation/x86/intel_sgx.rst
new file mode 100644
index 000000000000..ecbe544eb2cb
--- /dev/null
+++ b/Documentation/x86/intel_sgx.rst
@@ -0,0 +1,195 @@
+===================
+Intel(R) SGX driver
+===================
+
+Introduction
+============
+
+Intel(R) SGX is a set of CPU instructions that can be used by applications to
+set aside private regions of code and data. The code outside the enclave is
+disallowed to access the memory inside the enclave by the CPU access control.
+In a way you can think that SGX provides inverted sandbox. It protects the
+application from a malicious host.
+
+You can tell if your CPU supports SGX by looking into ``/proc/cpuinfo``:
+
+	``cat /proc/cpuinfo  | grep sgx``
+
+Overview of SGX
+===============
+
+SGX has a set of data structures to maintain information about the enclaves and
+their security properties. BIOS reserves a fixed size region of physical memory
+for these structures by setting Processor Reserved Memory Range Registers
+(PRMRR).
+
+This memory range is protected from outside access by the CPU and all the data
+coming in and out of the CPU package is encrypted by a key that is generated for
+each boot cycle.
+
+Enclaves execute in ring-3 in a special enclave submode using pages from the
+reserved memory range. A fixed logical address range for the enclave is reserved
+by ENCLS(ECREATE), a leaf instruction used to create enclaves. It is referred in
+the documentation commonly as the ELRANGE.
+
+Every memory access to the ELRANGE is asserted by the CPU. If the CPU is not
+executing in the enclave mode inside the enclave, #GP is raised. On the other
+hand enclave code can make memory accesses both inside and outside of the
+ELRANGE.
+
+Enclave can only execute code inside the ELRANGE. Instructions that may cause
+VMEXIT, IO instructions and instructions that require a privilege change are
+prohibited inside the enclave. Interrupts and exceptions always cause enclave
+to exit and jump to an address outside the enclave given when the enclave is
+entered by using the leaf instruction ENCLS(EENTER).
+
+Data types
+----------
+
+The protected memory range contains the following data:
+
+* **Enclave Page Cache (EPC):** protected pages
+* **Enclave Page Cache Map (EPCM):** a database that describes the state of the
+  pages and link them to an enclave.
+
+EPC has a number of different types of pages:
+
+* **SGX Enclave Control Structure (SECS)**: describes the global
+  properties of an enclave.
+* **Regular (REG):** code and data pages in the ELRANGE.
+* **Thread Control Structure (TCS):** pages that define entry points inside an
+  enclave. The enclave can only be entered through these entry points and each
+  can host a single hardware thread at a time.
+* **Version Array (VA)**: 64-bit version numbers for pages that have been
+  swapped outside the enclave. Each page contains 512 version numbers.
+
+Launch control
+--------------
+
+To launch an enclave, two structures must be provided for ENCLS(EINIT):
+
+1. **SIGSTRUCT:** signed measurement of the enclave binary.
+2. **EINITTOKEN:** a cryptographic token CMAC-signed with a AES256-key called
+   *launch key*, which is re-generated for each boot cycle.
+
+The CPU holds a SHA256 hash of a 3072-bit RSA public key inside
+IA32_SGXLEPUBKEYHASHn MSRs. Enclaves with a SIGSTRUCT that is signed with this
+key do not require a valid EINITTOKEN and can be authorized with special
+privileges. One of those privileges is ability to acquire the launch key with
+ENCLS(EGETKEY).
+
+**IA32_FEATURE_CONTROL[17]** is used by to BIOS configure whether
+IA32_SGXLEPUBKEYHASH MSRs are read-only or read-write before locking the
+feature control register and handing over control to the operating system.
+
+Enclave construction
+--------------------
+
+The construction is started by filling out the SECS that contains enclave
+address range, privileged attributes and measurement of TCS and REG pages (pages
+that will be mapped to the address range) among the other things. This structure
+is passed out to the ENCLS(ECREATE) together with a physical address of a page
+in EPC that will hold the SECS.
+
+Then pages are added with ENCLS(EADD) and measured with ENCLS(EEXTEND).  Finally
+enclave is initialized with ENCLS(EINIT). ENCLS(INIT) checks that the SIGSTRUCT
+is signed with the contained public key and that the supplied EINITTOKEN is
+valid (CMAC'd with the launch key). If these hold, the enclave is successfully
+initialized.
+
+Swapping pages
+--------------
+
+Enclave pages can be swapped out with ENCLS(EWB) to the unprotected memory. In
+addition to the EPC page, ENCLS(EWB) takes in a VA page and address for PCMD
+structure (Page Crypto MetaData) as input. The VA page will seal a version
+number for the page. PCMD is 128 byte structure that contains tracking
+information for the page, most importantly its MAC. With these structures the
+enclave is sealed and rollback protected while it resides in the unprotected
+memory.
+
+Before the page can be swapped out it must not have any active TLB references.
+By using ENCLS(EBLOCK) instructions no new TLB entries can be created to it.
+After this the a counter called *epoch* associated hardware threads inside the
+enclave is increased with ENCLS(ETRACK). After all the threads from the previous
+epoch have exited the page can be safely swapped out.
+
+An enclave memory access to a swapped out pages will cause #PF. #PF handler can
+fault the page back by using ENCLS(ELDU).
+
+Kernel internals
+================
+
+Requirements
+------------
+
+Because SGX has an ever evolving and expanding feature set, it's possible for
+a BIOS or VMM to configure a system in such a way that not all cpus are equal,
+e.g. where Launch Control is only enabled on a subset of cpus.  Linux does
+*not* support such a heterogenous system configuration, nor does it even
+attempt to play nice in the face of a misconfigured system.  With the exception
+of Launch Control's hash MSRs, which can vary per cpu, Linux assumes that all
+cpus have a configuration that is identical to the boot cpu.
+
+
+Roles and responsibilities
+--------------------------
+
+SGX introduces system resources, e.g. EPC memory, that must be accessible to
+multiple entities, e.g. the native kernel driver (to expose SGX to userspace)
+and KVM (to expose SGX to VMs), ideally without introducing any dependencies
+between each SGX entity.  To that end, the kernel owns and manages the shared
+system resources, i.e. the EPC and Launch Control MSRs, and defines functions
+that provide appropriate access to the shared resources.  SGX support for
+userpace and VMs is left to the SGX platform driver and KVM respectively.
+
+Launching enclaves
+------------------
+
+For privileged enclaves the launch is performed simply by submitting the
+SIGSTRUCT for that enclave to ENCLS(EINIT). For unprivileged enclaves the
+driver hosts a process in ring-3 that hosts a launch enclave signed with a key
+supplied for kbuild.
+
+The current implementation of the launch enclave generates a token for any
+enclave. In the future it could be potentially extended to have ways to
+configure policy what can be lauched.
+
+The driver will fail to initialize if it cannot start its own launch enclave.
+A user space application can submit a SIGSTRUCT instance through the ioctl API.
+The kernel will take care of the rest.
+
+This design assures that the Linux kernel has always full control, which
+enclaves get to launch and which do not, even if the public key MSRs are
+read-only. Having launch intrinsics inside the kernel also enables easy
+development of enclaves without necessarily needing any heavy weight SDK.
+Having a low-barrier to implement enclaves could make sense for example for
+system daemons where amount of dependecies ought to be minimized.
+
+EPC management
+--------------
+
+Due to the unique requirements for swapping EPC pages, and because EPC pages
+(currently) do not have associated page structures, management of the EPC is
+not handled by the standard Linux swapper.  SGX directly handles swapping
+of EPC pages, including a kthread to initiate reclaim and a rudimentary LRU
+mechanism.  Consumsers of EPC pages, e.g. the SGX driver, are required to
+implement function callbacks that can be invoked by the kernel to age,
+swap, and/or forcefully reclaim a target EPC page.  In effect, the kernel
+controls what happens and when, while the consumers (driver, KVM, etc..) do
+the actual work.
+
+SGX uapi
+========
+
+.. kernel-doc:: drivers/platform/x86/intel_sgx/sgx_ioctl.c
+   :functions: sgx_ioc_enclave_create
+               sgx_ioc_enclave_add_page
+               sgx_ioc_enclave_init
+
+.. kernel-doc:: arch/x86/include/uapi/asm/sgx.h
+
+References
+==========
+
+* System Programming Manual: 39.1.4 Intel® SGX Launch Control Configuration
-- 
2.17.0

[PATCH v11 13/13] intel_sgx: in-kernel launch enclave

[Jarkko Sakkinen]

The Launch Enclave (LE) generates cryptographic launch tokens for user
enclaves. A launch token is used by EINIT to check whether the enclave
is authorized to launch or not. By having its own launch enclave, Linux
has full control of the enclave launch process.

LE is wrapped into a user space proxy program that reads enclave
signatures outputs launch tokens. The kernel-side glue code is
implemented by using the user space helper framework.  The IPC between
the LE proxy program and kernel is handled with an anonymous inode.

The commit also adds enclave signing tool that is used by kbuild to
measure and sign the launch enclave. CONFIG_INTEL_SGX_SIGNING_KEY points
to a PEM-file for the 3072-bit RSA key that is used as the LE public key
pair. The default location is:

  drivers/platform/x86/intel_sgx/sgx_signing_key.pem

If the default key does not exist kbuild will generate a random key and
place it to this location. KBUILD_SGX_SIGN_PIN can be used to specify
the passphrase for the LE public key.

The CMAC implementation has been derived from TinyCrypt. The kernel
AES-NI implementation is used for AES.

[1] https://github.com/01org/tinycrypt

Signed-off-by: Jarkko Sakkinen <jarkko.sakkinen@linux.intel.com>
Tested-by: Serge Ayoun <serge.ayoun@intel.com>
---
 arch/x86/include/asm/sgx.h                    |   1 -
 arch/x86/include/asm/sgx_le.h                 |  17 +
 drivers/platform/x86/intel_sgx/Kconfig        |  14 +
 drivers/platform/x86/intel_sgx/Makefile       |  19 +
 drivers/platform/x86/intel_sgx/le/Makefile    |  34 ++
 .../x86/intel_sgx/le/enclave/Makefile         |  54 ++
 .../intel_sgx/le/enclave/aesni-intel_asm.S    |   1 +
 .../x86/intel_sgx/le/enclave/cmac_mode.c      | 209 +++++++
 .../x86/intel_sgx/le/enclave/cmac_mode.h      |  54 ++
 .../x86/intel_sgx/le/enclave/encl_bootstrap.S | 116 ++++
 .../platform/x86/intel_sgx/le/enclave/main.c  | 146 +++++
 .../platform/x86/intel_sgx/le/enclave/main.h  |  19 +
 .../x86/intel_sgx/le/enclave/sgx_le.lds       |  33 ++
 .../x86/intel_sgx/le/enclave/sgxsign.c        | 551 ++++++++++++++++++
 .../x86/intel_sgx/le/enclave/string.c         |   1 +
 drivers/platform/x86/intel_sgx/le/entry.S     |  70 +++
 .../x86/intel_sgx/le/include/sgx_asm.h        |  15 +
 drivers/platform/x86/intel_sgx/le/main.c      | 140 +++++
 drivers/platform/x86/intel_sgx/le/main.h      |  30 +
 .../platform/x86/intel_sgx/le/sgx_le_piggy.S  |  22 +
 drivers/platform/x86/intel_sgx/le/string.c    |  39 ++
 drivers/platform/x86/intel_sgx/sgx.h          |  17 +
 drivers/platform/x86/intel_sgx/sgx_encl.c     |  18 +-
 drivers/platform/x86/intel_sgx/sgx_ioctl.c    |   4 +-
 drivers/platform/x86/intel_sgx/sgx_le.c       | 303 ++++++++++
 .../x86/intel_sgx/sgx_le_proxy_piggy.S        |  22 +
 drivers/platform/x86/intel_sgx/sgx_main.c     |  71 ++-
 27 files changed, 2000 insertions(+), 20 deletions(-)
 create mode 100644 arch/x86/include/asm/sgx_le.h
 create mode 100644 drivers/platform/x86/intel_sgx/le/Makefile
 create mode 100644 drivers/platform/x86/intel_sgx/le/enclave/Makefile
 create mode 120000 drivers/platform/x86/intel_sgx/le/enclave/aesni-intel_asm.S
 create mode 100644 drivers/platform/x86/intel_sgx/le/enclave/cmac_mode.c
 create mode 100644 drivers/platform/x86/intel_sgx/le/enclave/cmac_mode.h
 create mode 100644 drivers/platform/x86/intel_sgx/le/enclave/encl_bootstrap.S
 create mode 100644 drivers/platform/x86/intel_sgx/le/enclave/main.c
 create mode 100644 drivers/platform/x86/intel_sgx/le/enclave/main.h
 create mode 100644 drivers/platform/x86/intel_sgx/le/enclave/sgx_le.lds
 create mode 100644 drivers/platform/x86/intel_sgx/le/enclave/sgxsign.c
 create mode 120000 drivers/platform/x86/intel_sgx/le/enclave/string.c
 create mode 100644 drivers/platform/x86/intel_sgx/le/entry.S
 create mode 100644 drivers/platform/x86/intel_sgx/le/include/sgx_asm.h
 create mode 100644 drivers/platform/x86/intel_sgx/le/main.c
 create mode 100644 drivers/platform/x86/intel_sgx/le/main.h
 create mode 100644 drivers/platform/x86/intel_sgx/le/sgx_le_piggy.S
 create mode 100644 drivers/platform/x86/intel_sgx/le/string.c
 create mode 100644 drivers/platform/x86/intel_sgx/sgx_le.c
 create mode 100644 drivers/platform/x86/intel_sgx/sgx_le_proxy_piggy.S

diff --git a/arch/x86/include/asm/sgx.h b/arch/x86/include/asm/sgx.h
index ae738e16ba6c..07c16945076e 100644
--- a/arch/x86/include/asm/sgx.h
+++ b/arch/x86/include/asm/sgx.h
@@ -259,7 +259,6 @@ struct sgx_launch_request {
 	u8 mrsigner[32];
 	uint64_t attributes;
 	uint64_t xfrm;
-	struct sgx_einittoken token;
 };
 
 #define SGX_FN(name, params...)		\
diff --git a/arch/x86/include/asm/sgx_le.h b/arch/x86/include/asm/sgx_le.h
new file mode 100644
index 000000000000..61f73fa71ba2
--- /dev/null
+++ b/arch/x86/include/asm/sgx_le.h
@@ -0,0 +1,17 @@
+// SPDX-License-Identifier: (GPL-2.0 OR BSD-3-Clause)
+// Copyright(c) 2016-17 Intel Corporation.
+//
+// Authors:
+//
+// Jarkko Sakkinen <jarkko.sakkinen@linux.intel.com>
+
+#ifndef _ASM_X86_SGX_LE_H
+#define _ASM_X86_SGX_LE_H
+
+#define SGX_LE_EXE_PATH "/proc/self/fd/3"
+
+#define SGX_LE_EXE_FD 3
+#define SGX_LE_DEV_FD 4
+#define SGX_LE_PIPE_FD 5
+
+#endif /* _ASM_X86_SGX_LE_H */
diff --git a/drivers/platform/x86/intel_sgx/Kconfig b/drivers/platform/x86/intel_sgx/Kconfig
index 4cd88310617e..4db0297b55ab 100644
--- a/drivers/platform/x86/intel_sgx/Kconfig
+++ b/drivers/platform/x86/intel_sgx/Kconfig
@@ -2,12 +2,16 @@
 # Intel SGX
 #
 
+menu "Intel SGX"
+
 config INTEL_SGX
 	tristate "Intel(R) SGX Driver"
 	default n
 	depends on X86_64 && CPU_SUP_INTEL
 	select INTEL_SGX_CORE
 	select MMU_NOTIFIER
+	select CRYPTO
+	select CRYPTO_SHA256
 	help
 	Intel(R) SGX is a set of CPU instructions that can be used by
 	applications to set aside private regions of code and data.  The code
@@ -18,3 +22,13 @@ config INTEL_SGX
 	called Enclave Page Cache (EPC). There is a hardware unit in the
 	processor called Memory Encryption Engine. The MEE encrypts and decrypts
 	the EPC pages as they enter and leave the processor package.
+
+config INTEL_SGX_SIGNING_KEY
+	string "Path to the Intel SGX LE signing key"
+	default "drivers/platform/x86/intel_sgx/signing_key.pem"
+	depends on INTEL_SGX
+	---help---
+	Provide a path to a 3072-bit RSA private key that will be used to
+	sign the launch enclave.
+
+endmenu
diff --git a/drivers/platform/x86/intel_sgx/Makefile b/drivers/platform/x86/intel_sgx/Makefile
index 95f254e30a8b..a5b729310ee2 100644
--- a/drivers/platform/x86/intel_sgx/Makefile
+++ b/drivers/platform/x86/intel_sgx/Makefile
@@ -11,3 +11,22 @@ intel_sgx-$(CONFIG_INTEL_SGX) += \
 	sgx_main.o \
 	sgx_fault.o \
 	sgx_vma.o \
+	sgx_le.o \
+	sgx_le_proxy_piggy.o
+
+$(eval $(call config_filename,INTEL_SGX_SIGNING_KEY))
+
+INTEL_SGX_SIGNING_KEY_PATH := \
+	$(INTEL_SGX_SIGNING_KEY_SRCPREFIX)$(INTEL_SGX_SIGNING_KEY_FILENAME)
+
+ifeq ($(CONFIG_INTEL_SGX_SIGNING_KEY),"drivers/platform/x86/intel_sgx/signing_key.pem")
+$(INTEL_SGX_SIGNING_KEY_PATH):
+	$(Q)openssl genrsa -3 -out $(INTEL_SGX_SIGNING_KEY_PATH) 3072
+endif
+
+$(obj)/sgx_le_proxy_piggy.o: $(INTEL_SGX_SIGNING_KEY_PATH) \
+			     $(obj)/le/sgx_le_proxy
+$(obj)/le/sgx_le_proxy: FORCE
+	$(Q)$(MAKE) -j1 $(build)=$(obj)/le $@
+
+export INTEL_SGX_SIGNING_KEY_PATH
diff --git a/drivers/platform/x86/intel_sgx/le/Makefile b/drivers/platform/x86/intel_sgx/le/Makefile
new file mode 100644
index 000000000000..5adaddc71bd6
--- /dev/null
+++ b/drivers/platform/x86/intel_sgx/le/Makefile
@@ -0,0 +1,34 @@
+# SPDX-License-Identifier: (GPL-2.0 OR BSD-3-Clause)
+# Copyright(c) 2016-17 Intel Corporation.
+#
+# Authors:
+#
+# Jarkko Sakkinen <jarkko.sakkinen@linux.intel.com>
+
+KASAN_SANITIZE := n
+OBJECT_FILES_NON_STANDARD := y
+KCOV_INSTRUMENT := n
+KBUILD_CFLAGS := -Wall -Werror -static -nostdlib -nostartfiles -fno-builtin \
+		 -I$(obj)/include
+KBUILD_AFLAGS += -I$(obj)/include
+
+subdir- := enclave
+
+always := sgx_le_proxy
+clean-files := sgx_le_proxy
+
+#
+# sgx_le_proxy
+#
+
+sgx_le_proxy-y += main.o entry.o sgx_le_piggy.o string.o
+targets += $(sgx_le_proxy-y)
+SGX_LE_PROXY_OBJS = $(addprefix $(obj)/,$(sgx_le_proxy-y))
+
+$(obj)/sgx_le_piggy.o: $(obj)/enclave/sgx_le.bin $(obj)/enclave/sgx_le.ss
+$(obj)/enclave/sgx_le.bin $(obj)/enclave/sgx_le.ss: FORCE
+	$(Q)$(MAKE) -j1 $(build)=$(obj)/enclave $@
+
+targets += sgx_le_proxy
+$(obj)/sgx_le_proxy: $(SGX_LE_PROXY_OBJS)
+	$(call if_changed,ld)
diff --git a/drivers/platform/x86/intel_sgx/le/enclave/Makefile b/drivers/platform/x86/intel_sgx/le/enclave/Makefile
new file mode 100644
index 000000000000..aef95d00223c
--- /dev/null
+++ b/drivers/platform/x86/intel_sgx/le/enclave/Makefile
@@ -0,0 +1,54 @@
+# SPDX-License-Identifier: (GPL-2.0 OR BSD-3-Clause)
+# Copyright(c) 2016-17 Intel Corporation.
+#
+# Authors:
+#
+# Jarkko Sakkinen <jarkko.sakkinen@linux.intel.com>
+
+KASAN_SANITIZE := n
+OBJECT_FILES_NON_STANDARD := y
+KCOV_INSTRUMENT := n
+KBUILD_CFLAGS := -Wall -Werror -static -nostdlib -nostartfiles -fPIC \
+		 -fno-stack-protector -mrdrnd  -I$(obj)/../include \
+		 -I$(srctree)/arch/x86/include
+KBUILD_AFLAGS := $(KBUILD_CFLAGS) -D__ASSEMBLY__ -DAESNI_INTEL_MINIMAL
+
+always := sgx_le.elf sgx_le.bin sgx_le.ss
+clean-files := sgx_le.elf sgx_le.bin sgx_le.ss
+
+#
+# sgx_le.ss
+#
+
+HOST_EXTRACFLAGS += -I$(srctree)/arch/x86/include
+HOSTLOADLIBES_sgxsign = -lcrypto
+hostprogs-y += sgxsign
+quiet_cmd_sgxsign = SGXSIGN $@
+      cmd_sgxsign = drivers/platform/x86/intel_sgx/le/enclave/sgxsign \
+		    $(INTEL_SGX_SIGNING_KEY_PATH) $< $@
+
+targets += sgx_le.ss
+$(obj)/sgx_le.ss: $(obj)/sgx_le.bin $(obj)/sgxsign FORCE
+	$(call if_changed,sgxsign)
+
+#
+# sgx_le.bin
+#
+
+targets += sgx_le.bin
+OBJCOPYFLAGS_sgx_le.bin := --remove-section=.got.plt -O binary
+$(obj)/sgx_le.bin: $(obj)/sgx_le.elf FORCE
+	$(call if_changed,objcopy)
+
+#
+# sgx_le.elf
+#
+
+sgx_le-y +=  main.o encl_bootstrap.o cmac_mode.o aesni-intel_asm.o string.o
+targets += $(sgx_le-y)
+SGX_LE_OBJS = $(addprefix $(obj)/,$(sgx_le-y))
+
+targets += sgx_le.elf
+LDFLAGS_sgx_le.elf := -T
+$(obj)/sgx_le.elf: $(obj)/sgx_le.lds $(SGX_LE_OBJS)
+	$(call if_changed,ld)
diff --git a/drivers/platform/x86/intel_sgx/le/enclave/aesni-intel_asm.S b/drivers/platform/x86/intel_sgx/le/enclave/aesni-intel_asm.S
new file mode 120000
index 000000000000..61971c8f7b3f
--- /dev/null
+++ b/drivers/platform/x86/intel_sgx/le/enclave/aesni-intel_asm.S
@@ -0,0 +1 @@
+../../../../../../arch/x86/crypto/aesni-intel_asm.S
\ No newline at end of file
diff --git a/drivers/platform/x86/intel_sgx/le/enclave/cmac_mode.c b/drivers/platform/x86/intel_sgx/le/enclave/cmac_mode.c
new file mode 100644
index 000000000000..f1f72b1679ee
--- /dev/null
+++ b/drivers/platform/x86/intel_sgx/le/enclave/cmac_mode.c
@@ -0,0 +1,209 @@
+// SPDX-License-Identifier: (GPL-2.0 OR BSD-3-Clause)
+// Copyright(c) 2016-17 Intel Corporation.
+//
+// Derived from TinyCrypt CMAC implementation.
+//
+// Authors:
+//
+// Jarkko Sakkinen <jarkko.sakkinen@linux.intel.com>
+
+#include <asm/sgx.h>
+#include "cmac_mode.h"
+
+/* max number of calls until change the key (2^48).*/
+const static uint64_t MAX_CALLS = ((uint64_t)1 << 48);
+
+/*
+ *  gf_wrap -- In our implementation, GF(2^128) is represented as a 16 byte
+ *  array with byte 0 the most significant and byte 15 the least significant.
+ *  High bit carry reduction is based on the primitive polynomial
+ *
+ *                     X^128 + X^7 + X^2 + X + 1,
+ *
+ *  which leads to the reduction formula X^128 = X^7 + X^2 + X + 1. Indeed,
+ *  since 0 = (X^128 + X^7 + X^2 + 1) mod (X^128 + X^7 + X^2 + X + 1) and since
+ *  addition of polynomials with coefficients in Z/Z(2) is just XOR, we can
+ *  add X^128 to both sides to get
+ *
+ *       X^128 = (X^7 + X^2 + X + 1) mod (X^128 + X^7 + X^2 + X + 1)
+ *
+ *  and the coefficients of the polynomial on the right hand side form the
+ *  string 1000 0111 = 0x87, which is the value of gf_wrap.
+ *
+ *  This gets used in the following way. Doubling in GF(2^128) is just a left
+ *  shift by 1 bit, except when the most significant bit is 1. In the latter
+ *  case, the relation X^128 = X^7 + X^2 + X + 1 says that the high order bit
+ *  that overflows beyond 128 bits can be replaced by addition of
+ *  X^7 + X^2 + X + 1 <--> 0x87 to the low order 128 bits. Since addition
+ *  in GF(2^128) is represented by XOR, we therefore only have to XOR 0x87
+ *  into the low order byte after a left shift when the starting high order
+ *  bit is 1.
+ */
+const unsigned char gf_wrap = 0x87;
+
+/*
+ *  assumes: out != NULL and points to a GF(2^n) value to receive the
+ *            doubled value;
+ *           in != NULL and points to a 16 byte GF(2^n) value
+ *            to double;
+ *           the in and out buffers do not overlap.
+ *  effects: doubles the GF(2^n) value pointed to by "in" and places
+ *           the result in the GF(2^n) value pointed to by "out."
+ */
+void gf_double(uint8_t *out, uint8_t *in)
+{
+	/* start with low order byte */
+	uint8_t *x = in + (AES_BLOCK_SIZE - 1);
+
+	/* if msb == 1, we need to add the gf_wrap value, otherwise add 0 */
+	uint8_t carry = (in[0] >> 7) ? gf_wrap : 0;
+
+	out += (AES_BLOCK_SIZE - 1);
+	for (;;) {
+		*out-- = (*x << 1) ^ carry;
+		if (x == in)
+			break;
+		carry = *x-- >> 7;
+	}
+}
+
+/**
+ * tc_cmac_setup - configures the CMAC state to use the given AES key
+ *
+ * @s: the state to set up
+ * @key: the key to use:w
+ * @ctx: AES context
+ */
+void tc_cmac_setup(struct tc_cmac_struct *s, const uint8_t *key,
+		   struct crypto_aes_ctx *ctx)
+{
+	/* put s into a known state */
+	tc_cmac_erase(s);
+	s->ctx = ctx;
+
+	/* configure the encryption key used by the underlying block cipher */
+	aesni_set_key(ctx, key, AES_KEYSIZE_128);
+
+	/* compute s->K1 and s->K2 from s->iv using s->keyid */
+	memset(s->iv, 0, AES_BLOCK_SIZE);
+	aesni_enc(ctx, s->iv, s->iv);
+
+	gf_double (s->K1, s->iv);
+	gf_double (s->K2, s->K1);
+
+	/* reset s->iv to 0 in case someone wants to compute now */
+	tc_cmac_init(s);
+}
+
+/**
+ * tc_cmac_erase - erases the CMAC state
+ *
+ * @s:	the state to erase
+ */
+void tc_cmac_erase(struct tc_cmac_struct *s)
+{
+	memset(s, 0, sizeof(*s));
+}
+
+/**
+ * tc_cmac_init - initializes a new CMAC computation
+ *
+ * @s:	the state to initialize
+ */
+void tc_cmac_init(struct tc_cmac_struct *s)
+{
+	/* CMAC starts with an all zero initialization vector */
+	memset(s->iv, 0, AES_BLOCK_SIZE);
+
+	/* and the leftover buffer is empty */
+	memset(s->leftover, 0, AES_BLOCK_SIZE);
+	s->leftover_offset = 0;
+
+	/* Set countdown to max number of calls allowed before re-keying: */
+	s->countdown = MAX_CALLS;
+}
+
+/**
+ * tc_cmac_update - incrementally computes CMAC over the next data segment
+ *
+ * s:		the CMAC state
+ * data:	the next data segment to MAC
+ * dlen:	the length of data in bytes
+ */
+void tc_cmac_update(struct tc_cmac_struct *s, const uint8_t *data, size_t dlen)
+{
+	uint32_t i;
+
+	s->countdown--;
+
+	if (s->leftover_offset > 0) {
+		/* last data added to s didn't end on a AES_BLOCK_SIZE byte
+		 * boundary
+		 */
+		size_t remaining_space = AES_BLOCK_SIZE - s->leftover_offset;
+
+		if (dlen < remaining_space) {
+			/* still not enough data to encrypt this time either */
+			memcpy(&s->leftover[s->leftover_offset], data,
+			       dlen);
+			s->leftover_offset += dlen;
+			return;
+		}
+		/* leftover block is now full; encrypt it first */
+		memcpy(&s->leftover[s->leftover_offset], data, remaining_space);
+		dlen -= remaining_space;
+		data += remaining_space;
+		s->leftover_offset = 0;
+
+		for (i = 0; i < AES_BLOCK_SIZE; ++i)
+			s->iv[i] ^= s->leftover[i];
+
+		aesni_enc(s->ctx, s->iv, s->iv);
+	}
+
+	/* CBC encrypt each (except the last) of the data blocks */
+	while (dlen > AES_BLOCK_SIZE) {
+		for (i = 0; i < AES_BLOCK_SIZE; ++i)
+			s->iv[i] ^= data[i];
+		aesni_enc(s->ctx, s->iv, s->iv);
+		data += AES_BLOCK_SIZE;
+		dlen  -= AES_BLOCK_SIZE;
+	}
+
+	if (dlen > 0) {
+		/* save leftover data for next time */
+		memcpy(s->leftover, data, dlen);
+		s->leftover_offset = dlen;
+	}
+}
+
+/**
+ * tc_cmac_final - generates the tag from the CMAC state
+ *
+ * @tag:	the CMAC tag
+ * @s:		CMAC state
+ */
+void tc_cmac_final(uint8_t *tag, struct tc_cmac_struct *s)
+{
+	uint8_t *k;
+	uint32_t i;
+
+	if (s->leftover_offset == AES_BLOCK_SIZE) {
+		/* the last message block is a full-sized block */
+		k = (uint8_t *) s->K1;
+	} else {
+		/* the final message block is not a full-sized  block */
+		size_t remaining = AES_BLOCK_SIZE - s->leftover_offset;
+
+		memset(&s->leftover[s->leftover_offset], 0, remaining);
+		s->leftover[s->leftover_offset] = TC_CMAC_PADDING;
+		k = (uint8_t *) s->K2;
+	}
+	for (i = 0; i < AES_BLOCK_SIZE; ++i)
+		s->iv[i] ^= s->leftover[i] ^ k[i];
+
+	aesni_enc(s->ctx, tag, s->iv);
+
+	/* erasing state: */
+	tc_cmac_erase(s);
+}
diff --git a/drivers/platform/x86/intel_sgx/le/enclave/cmac_mode.h b/drivers/platform/x86/intel_sgx/le/enclave/cmac_mode.h
new file mode 100644
index 000000000000..18c9223bbd4f
--- /dev/null
+++ b/drivers/platform/x86/intel_sgx/le/enclave/cmac_mode.h
@@ -0,0 +1,54 @@
+// SPDX-License-Identifier: (GPL-2.0 OR BSD-3-Clause)
+// Copyright(c) 2016-17 Intel Corporation.
+//
+// Derived from TinyCrypt CMAC implementation.
+//
+// Authors:
+//
+// Jarkko Sakkinen <jarkko.sakkinen@linux.intel.com>
+
+#ifndef CMAC_MODE_H
+#define CMAC_MODE_H
+
+#include <stddef.h>
+#include <crypto/aes.h>
+
+/* padding for last message block */
+#define TC_CMAC_PADDING 0x80
+
+/* struct tc_cmac_struct represents the state of a CMAC computation */
+struct tc_cmac_struct {
+	/* initialization vector */
+	uint8_t iv[AES_BLOCK_SIZE];
+	/* used if message length is a multiple of block_size bytes */
+	uint8_t K1[AES_BLOCK_SIZE];
+	/* used if message length isn't a multiple block_size bytes */
+	uint8_t K2[AES_BLOCK_SIZE];
+	/* where to put bytes that didn't fill a block */
+	uint8_t leftover[AES_BLOCK_SIZE];
+	/* identifies the encryption key */
+	uint32_t keyid;
+	/* next available leftover location */
+	uint32_t leftover_offset;
+	/* AES key schedule */
+	struct crypto_aes_ctx *ctx;
+	/* calls to tc_cmac_update left before re-key */
+	uint64_t countdown;
+};
+
+void tc_cmac_setup(struct tc_cmac_struct *s, const uint8_t *key,
+		   struct crypto_aes_ctx *ctx);
+
+void tc_cmac_erase(struct tc_cmac_struct *s);
+
+void tc_cmac_init(struct tc_cmac_struct *s);
+
+void tc_cmac_update(struct tc_cmac_struct *s, const uint8_t *data, size_t dlen);
+
+void tc_cmac_final(uint8_t *tag, struct tc_cmac_struct *s);
+
+asmlinkage int aesni_set_key(struct crypto_aes_ctx *ctx, const u8 *in_key,
+			     unsigned int key_len);
+asmlinkage void aesni_enc(struct crypto_aes_ctx *ctx, u8 *out, const u8 *in);
+
+#endif /* CMAC_MODE_H */
diff --git a/drivers/platform/x86/intel_sgx/le/enclave/encl_bootstrap.S b/drivers/platform/x86/intel_sgx/le/enclave/encl_bootstrap.S
new file mode 100644
index 000000000000..603c4e167761
--- /dev/null
+++ b/drivers/platform/x86/intel_sgx/le/enclave/encl_bootstrap.S
@@ -0,0 +1,116 @@
+// SPDX-License-Identifier: (GPL-2.0 OR BSD-3-Clause)
+// Copyright(c) 2016-17 Intel Corporation.
+//
+// Authors:
+//
+// Jarkko Sakkinen <jarkko.sakkinen@linux.intel.com>
+// Haim Cohen <haim.cohen@intel.com>
+// Sean Christopherson <sean.j.christopherson@intel.com>
+
+#include <sgx_asm.h>
+
+	.section ".tcs", "a"
+	.balign	4096
+
+	.fill	1, 8, 0			# STATE (set by CPU)
+	.fill	1, 8, 0			# FLAGS
+	.long	encl_ssa		# OSSA
+	.fill	1, 4, 0
+	.fill	1, 4, 0			# CSSA (set by CPU)
+	.fill	1, 4, 1			# NSSA
+	.long	encl_entry		# OENTRY
+	.fill	1, 4, 0
+	.fill	1, 8, 0			# AEP (set by EENTER and ERESUME)
+	.fill	1, 8, 0			# OFSBASE
+	.fill	1, 8, 0			# OGSBASE
+	.fill	1, 4, 0xFFFFFFFF 	# FSLIMIT
+	.fill	1, 4, 0xFFFFFFFF	# GSLIMIT
+	.fill	503, 8, 0		# Reserved
+
+	.text
+
+encl_entry:
+	# %rbx contains the base address for TCS, which is also the first
+	# address inside the enclave. By adding $le_stack_end to it, we get the
+	# absolute address for the stack.
+	lea	(encl_stack)(%rbx), %rax
+	xchg	%rsp, %rax
+	push	%rax
+
+	push	%rcx # push the address after EENTER
+	push	%rbx # push the enclave base address
+
+	call	encl_body
+
+	pop	%rbx # pop the enclave base address
+
+	# Restore XSAVE registers to a synthetic state.
+	mov     $0xFFFFFFFF, %rax
+	mov     $0xFFFFFFFF, %rdx
+	lea	(xsave_area)(%rbx), %rdi
+	fxrstor	(%rdi)
+
+	# Clear GPRs
+	xor     %rcx, %rcx
+	xor     %rdx, %rdx
+	xor     %rdi, %rdi
+	xor     %rsi, %rsi
+	xor     %r8, %r8
+	xor     %r9, %r9
+	xor     %r10, %r10
+	xor     %r11, %r11
+	xor     %r12, %r12
+	xor     %r13, %r13
+	xor     %r14, %r14
+	xor     %r15, %r15
+
+	# Reset status flags
+	add     %rdx, %rdx # OF = SF = AF = CF = 0; ZF = PF = 1
+
+	pop	%rbx # pop the address after EENTER
+
+	# Restore the caller stack.
+	pop	%rax
+	mov	%rax, %rsp
+
+	# EEXIT
+	mov	$4, %rax
+	enclu
+
+	.global sgx_ereport
+sgx_ereport:
+	push	%rbx
+	xor	%rax, %rax /* EREPORT */
+	mov	%rdi, %rbx  /* TARGETINFO */
+	mov	%rsi, %rcx /* REPORTDATA */
+	ENCLU
+	pop	%rbx
+	ret
+
+	.global sgx_egetkey
+sgx_egetkey:
+	push	%rbx
+	mov	$0x01, %rax /* EGETKEY */
+	mov	%rdi, %rbx /* KEYREQUEST */
+	mov	%rsi, %rcx  /* KEY */
+	ENCLU
+	pop	%rbx
+	ret
+
+	.section ".data", "aw"
+
+encl_ssa:
+	.space 4096
+
+xsave_area:
+	.fill	1, 4, 0x037F		# FCW
+	.fill	5, 4, 0
+	.fill	1, 4, 0x1F80		# MXCSR
+	.fill	1, 4, 0xFFFF		# MXCSR_MASK
+	.fill	123, 4, 0
+	.fill	1, 4, 0x80000000	# XCOMP_BV[63] = 1, compaction mode
+	.fill	12, 4, 0
+
+	.balign 4096
+	.space 8192
+encl_stack:
diff --git a/drivers/platform/x86/intel_sgx/le/enclave/main.c b/drivers/platform/x86/intel_sgx/le/enclave/main.c
new file mode 100644
index 000000000000..f03bdad4d3b6
--- /dev/null
+++ b/drivers/platform/x86/intel_sgx/le/enclave/main.c
@@ -0,0 +1,146 @@
+// SPDX-License-Identifier: (GPL-2.0 OR BSD-3-Clause)
+// Copyright(c) 2016-17 Intel Corporation.
+//
+// Derived from TinyCrypt CMAC implementation.
+//
+// Authors:
+//
+// Jarkko Sakkinen <jarkko.sakkinen@linux.intel.com>
+
+#include <asm/sgx.h>
+#include <asm/sgx_arch.h>
+#include <linux/types.h>
+#include <uapi/asm/sgx.h>
+#include "cmac_mode.h"
+#include "main.h"
+
+static bool rdrand_uint32(uint32_t *value)
+{
+	int i;
+
+	for (i = 0; i < RAND_NR_TRIES; i++) {
+		if (__builtin_ia32_rdrand32_step((unsigned int *)value))
+			return true;
+	}
+
+	return false;
+}
+
+static bool sign_einittoken(struct sgx_einittoken *einittoken)
+{
+	struct sgx_keyrequest keyrequest __aligned(512);
+	uint8_t launch_key[16] __aligned(16);
+	struct tc_cmac_struct cmac_state;
+	struct crypto_aes_ctx ctx;
+	uint32_t *keyid_ptr;
+	int i;
+
+	memset(&ctx, 0, sizeof(ctx));
+
+	/* Despite its misleading name, the only purpose of the keyid field is
+	 * to add entropy to the token so that every token will have an unique
+	 * CMAC.
+	 */
+	keyid_ptr = (uint32_t *)einittoken->keyid;
+
+	for (i = 0; i < sizeof(einittoken->keyid) / 4; i++)
+		if (!rdrand_uint32(&keyid_ptr[i]))
+			return false;
+
+	memset(&keyrequest, 0, sizeof(keyrequest));
+	keyrequest.keyname = 0; /* LICENSE_KEY */
+	memcpy(&keyrequest.keyid, &einittoken->keyid, sizeof(keyrequest.keyid));
+	memcpy(&keyrequest.cpusvn, &(einittoken->cpusvnle),
+	       sizeof(keyrequest.cpusvn));
+	memcpy(&keyrequest.isvsvn, &(einittoken->isvsvnle),
+	       sizeof(keyrequest.isvsvn));
+
+	keyrequest.attributemask = ~SGX_ATTR_MODE64BIT;
+	keyrequest.xfrmmask = 0;
+	keyrequest.miscmask = 0xFFFFFFFF;
+
+	einittoken->maskedmiscselectle &= keyrequest.miscmask;
+	einittoken->maskedattributesle &= keyrequest.attributemask;
+	einittoken->maskedxfrmle &= keyrequest.xfrmmask;
+
+	if (sgx_egetkey(&keyrequest, launch_key))
+		return false;
+
+	tc_cmac_setup(&cmac_state, launch_key, &ctx);
+	tc_cmac_init(&cmac_state);
+	tc_cmac_update(&cmac_state, (const uint8_t *)&einittoken->payload,
+		       sizeof(einittoken->payload));
+	tc_cmac_final(einittoken->mac, &cmac_state);
+
+	memset(launch_key, 0, sizeof(launch_key));
+
+	return true;
+}
+
+static bool create_einittoken(uint8_t *mrenclave,
+			      uint8_t *mrsigner,
+			      uint64_t attributes,
+			      uint64_t xfrm,
+			      struct sgx_einittoken *einittoken)
+{
+
+	struct sgx_targetinfo tginfo __aligned(512);
+	struct sgx_report report __aligned(512);
+	uint8_t reportdata[64] __aligned(128);
+
+	if (attributes & SGX_ATTR_RESERVED_MASK)
+		return false;
+
+	memset(&tginfo, 0, sizeof(tginfo));
+	memset(reportdata, 0, sizeof(reportdata));
+	memset(&report, 0, sizeof(report));
+
+	if (sgx_ereport(&tginfo, reportdata, &report))
+		return false;
+
+	memset(einittoken, 0, sizeof(*einittoken));
+
+	einittoken->payload.valid = 1;
+
+	memcpy(einittoken->payload.mrenclave, mrenclave, 32);
+	memcpy(einittoken->payload.mrsigner, mrsigner, 32);
+	einittoken->payload.attributes = attributes;
+	einittoken->payload.xfrm = xfrm;
+
+	memcpy(&einittoken->cpusvnle, &report.cpusvn,
+		   sizeof(report.cpusvn));
+	einittoken->isvsvnle = report.isvsvn;
+	einittoken->isvprodidle = report.isvprodid;
+
+	einittoken->maskedattributesle = report.attributes;
+	einittoken->maskedxfrmle = report.xfrm;
+	einittoken->maskedmiscselectle = report.miscselect;
+
+	if (!sign_einittoken(einittoken))
+		return false;
+
+	return true;
+}
+
+void encl_body(struct sgx_launch_request *req, struct sgx_einittoken *token)
+{
+	struct sgx_einittoken tmp;
+	uint8_t mrenclave[32];
+	uint8_t mrsigner[32];
+	uint64_t attributes;
+	uint64_t xfrm;
+
+	if (!req)
+		return;
+
+	memcpy(mrenclave, req->mrenclave, sizeof(mrenclave));
+	memcpy(mrsigner, req->mrsigner, sizeof(mrsigner));
+	memcpy(&attributes, &req->attributes, sizeof(uint64_t));
+	memcpy(&xfrm, &req->xfrm, sizeof(uint64_t));
+	memset(&tmp, 0, sizeof(tmp));
+
+	if (!create_einittoken(mrenclave, mrsigner, attributes, xfrm, &tmp))
+		return;
+
+	memcpy(token, &tmp, sizeof(*token));
+}
diff --git a/drivers/platform/x86/intel_sgx/le/enclave/main.h b/drivers/platform/x86/intel_sgx/le/enclave/main.h
new file mode 100644
index 000000000000..2cec72242d0e
--- /dev/null
+++ b/drivers/platform/x86/intel_sgx/le/enclave/main.h
@@ -0,0 +1,19 @@
+// SPDX-License-Identifier: (GPL-2.0 OR BSD-3-Clause)
+// Copyright(c) 2016-17 Intel Corporation.
+//
+// Derived from TinyCrypt CMAC implementation.
+//
+// Authors:
+//
+// Jarkko Sakkinen <jarkko.sakkinen@linux.intel.com>
+
+#ifndef MAIN_H
+#define MAIN_H
+
+#define RAND_NR_TRIES 10
+
+int sgx_ereport(const void *target_info, const void *report_data,
+		void *report);
+int sgx_egetkey(void *key_request, void *key);
+
+#endif /* MAIN_H */
diff --git a/drivers/platform/x86/intel_sgx/le/enclave/sgx_le.lds b/drivers/platform/x86/intel_sgx/le/enclave/sgx_le.lds
new file mode 100644
index 000000000000..4997628813b8
--- /dev/null
+++ b/drivers/platform/x86/intel_sgx/le/enclave/sgx_le.lds
@@ -0,0 +1,33 @@
+OUTPUT_FORMAT(elf64-x86-64)
+
+SECTIONS
+{
+	. = 0;
+	.tcs : {
+		*(.tcs*)
+	}
+
+	. = ALIGN(4096);
+	.text : {
+		*(.text*)
+		*(.rodata*)
+	}
+
+	. = ALIGN(4096);
+	.data : {
+		*(.data*)
+	}
+
+	/DISCARD/ : {
+		*(.data*)
+		*(.comment*)
+		*(.note*)
+		*(.debug*)
+		*(.eh_frame*)
+	}
+}
+
+ASSERT(!DEFINED(.altinstructions), "ALTERNATIVES are not supported in the SGX LE")
+ASSERT(!DEFINED(.altinstr_replacement), "ALTERNATIVES are not supported in the SGX LE")
+ASSERT(!DEFINED(.discard.retpoline_safe), "RETPOLINE ALTERNATIVES are not supported in the SGX LE")
+ASSERT(!DEFINED(.discard.nospec), "RETPOLINE ALTERNATIVES are not supported in the SGX LE")
diff --git a/drivers/platform/x86/intel_sgx/le/enclave/sgxsign.c b/drivers/platform/x86/intel_sgx/le/enclave/sgxsign.c
new file mode 100644
index 000000000000..abbb59195fd3
--- /dev/null
+++ b/drivers/platform/x86/intel_sgx/le/enclave/sgxsign.c
@@ -0,0 +1,551 @@
+/*
+ * This file is provided under a dual BSD/GPLv2 license.  When using or
+ * redistributing this file, you may do so under either license.
+ *
+ * GPL LICENSE SUMMARY
+ *
+ * Copyright(c) 2017 Intel Corporation.
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of version 2 of the GNU General Public License as
+ * published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful, but
+ * WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+ * General Public License for more details.
+ *
+ * Contact Information:
+ * Jarkko Sakkinen <jarkko.sakkinen@linux.intel.com>
+ * Intel Finland Oy - BIC 0357606-4 - Westendinkatu 7, 02160 Espoo
+ *
+ * BSD LICENSE
+ *
+ * Copyright(c) 2017 Intel Corporation.
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions
+ * are met:
+ *
+ *   * Redistributions of source code must retain the above copyright
+ *     notice, this list of conditions and the following disclaimer.
+ *   * Redistributions in binary form must reproduce the above copyright
+ *     notice, this list of conditions and the following disclaimer in
+ *     the documentation and/or other materials provided with the
+ *     distribution.
+ *   * Neither the name of Intel Corporation nor the names of its
+ *     contributors may be used to endorse or promote products derived
+ *     from this software without specific prior written permission.
+ *
+ * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
+ * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
+ * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
+ * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
+ * OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+ * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+ * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+ * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
+ * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+ * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+ * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+ *
+ * Authors:
+ *
+ * Jarkko Sakkinen <jarkko.sakkinen@linux.intel.com>
+ */
+
+#define _GNU_SOURCE
+#include <getopt.h>
+#include <stdbool.h>
+#include <stdint.h>
+#include <stdio.h>
+#include <stdlib.h>
+#include <string.h>
+#include <sys/stat.h>
+#include <sys/types.h>
+#include <unistd.h>
+#include <asm/sgx_arch.h>
+#include <openssl/err.h>
+#include <openssl/pem.h>
+
+static const char *sign_key_pass;
+
+static bool check_crypto_errors(void)
+{
+	int err;
+	bool had_errors = false;
+	const char *filename;
+	int line;
+	char str[256];
+
+	for ( ; ; ) {
+		if (ERR_peek_error() == 0)
+			break;
+
+		had_errors = true;
+		err = ERR_get_error_line(&filename, &line);
+		ERR_error_string_n(err, str, sizeof(str));
+		fprintf(stderr, "crypto: %s: %s:%d\n", str, filename, line);
+	}
+
+	return had_errors;
+}
+
+static void exit_usage(const char *program)
+{
+	fprintf(stderr,
+		"Usage: %s/sign-le <key> <enclave> <sigstruct>\n", program);
+	exit(1);
+}
+
+static int pem_passwd_cb(char *buf, int size, int rwflag, void *u)
+{
+	if (!sign_key_pass)
+		return -1;
+
+	strncpy(buf, sign_key_pass, size);
+	/* no retry */
+	sign_key_pass = NULL;
+
+	return strlen(buf) >= size ? size - 1 : strlen(buf);
+}
+
+static inline const BIGNUM *get_modulus(RSA *key)
+{
+#if OPENSSL_VERSION_NUMBER < 0x10100000L
+	return key->n;
+#else
+	const BIGNUM *n;
+
+	RSA_get0_key(key, &n, NULL, NULL);
+	return n;
+#endif
+}
+
+static RSA *load_sign_key(const char *path)
+{
+	FILE *f;
+	RSA *key;
+
+	f = fopen(path, "rb");
+	if (!f) {
+		fprintf(stderr, "Unable to open %s\n", path);
+		return NULL;
+	}
+	key = RSA_new();
+	if (!PEM_read_RSAPrivateKey(f, &key, pem_passwd_cb, NULL))
+		return NULL;
+	fclose(f);
+
+	if (BN_num_bytes(get_modulus(key)) != SGX_MODULUS_SIZE) {
+		fprintf(stderr, "Invalid key size %d\n",
+			BN_num_bytes(get_modulus(key)));
+		RSA_free(key);
+		return NULL;
+	}
+
+	return key;
+}
+
+static void reverse_bytes(void *data, int length)
+{
+	int i = 0;
+	int j = length - 1;
+	uint8_t temp;
+	uint8_t *ptr = data;
+
+	while (i < j) {
+		temp = ptr[i];
+		ptr[i] = ptr[j];
+		ptr[j] = temp;
+		i++;
+		j--;
+	}
+}
+
+enum mrtags {
+	MRECREATE = 0x0045544145524345,
+	MREADD = 0x0000000044444145,
+	MREEXTEND = 0x00444E4554584545,
+};
+
+static bool mrenclave_update(EVP_MD_CTX *ctx, const void *data)
+{
+	if (!EVP_DigestUpdate(ctx, data, 64)) {
+		fprintf(stderr, "digest update failed\n");
+		return false;
+	}
+
+	return true;
+}
+
+static bool mrenclave_commit(EVP_MD_CTX *ctx, uint8_t *mrenclave)
+{
+	unsigned int size;
+
+	if (!EVP_DigestFinal_ex(ctx, (unsigned char *)mrenclave, &size)) {
+		fprintf(stderr, "digest commit failed\n");
+		return false;
+	}
+
+	if (size != 32) {
+		fprintf(stderr, "invalid digest size = %u\n", size);
+		return false;
+	}
+
+	return true;
+}
+
+struct mrecreate {
+	uint64_t tag;
+	uint32_t ssaframesize;
+	uint64_t size;
+	uint8_t reserved[44];
+} __attribute__((__packed__));
+
+
+static bool mrenclave_ecreate(EVP_MD_CTX *ctx, uint64_t blob_size)
+{
+	struct mrecreate mrecreate;
+	uint64_t encl_size;
+
+	for (encl_size = 0x1000; encl_size < blob_size; )
+		encl_size <<= 1;
+
+	memset(&mrecreate, 0, sizeof(mrecreate));
+	mrecreate.tag = MRECREATE;
+	mrecreate.ssaframesize = 1;
+	mrecreate.size = encl_size;
+
+	if (!EVP_DigestInit_ex(ctx, EVP_sha256(), NULL))
+		return false;
+
+	return mrenclave_update(ctx, &mrecreate);
+}
+
+struct mreadd {
+	uint64_t tag;
+	uint64_t offset;
+	uint64_t flags; /* SECINFO flags */
+	uint8_t reserved[40];
+} __attribute__((__packed__));
+
+static bool mrenclave_eadd(EVP_MD_CTX *ctx, uint64_t offset, uint64_t flags)
+{
+	struct mreadd mreadd;
+
+	memset(&mreadd, 0, sizeof(mreadd));
+	mreadd.tag = MREADD;
+	mreadd.offset = offset;
+	mreadd.flags = flags;
+
+	return mrenclave_update(ctx, &mreadd);
+}
+
+struct mreextend {
+	uint64_t tag;
+	uint64_t offset;
+	uint8_t reserved[48];
+} __attribute__((__packed__));
+
+static bool mrenclave_eextend(EVP_MD_CTX *ctx, uint64_t offset, uint8_t *data)
+{
+	struct mreextend mreextend;
+	int i;
+
+	for (i = 0; i < 0x1000; i += 0x100) {
+		memset(&mreextend, 0, sizeof(mreextend));
+		mreextend.tag = MREEXTEND;
+		mreextend.offset = offset + i;
+
+		if (!mrenclave_update(ctx, &mreextend))
+			return false;
+
+		if (!mrenclave_update(ctx, &data[i + 0x00]))
+			return false;
+
+		if (!mrenclave_update(ctx, &data[i + 0x40]))
+			return false;
+
+		if (!mrenclave_update(ctx, &data[i + 0x80]))
+			return false;
+
+		if (!mrenclave_update(ctx, &data[i + 0xC0]))
+			return false;
+	}
+
+	return true;
+}
+
+/**
+ * measure_encl - measure enclave
+ * @path: path to the enclave
+ * @mrenclave: measurement
+ *
+ * Calculates MRENCLAVE. Assumes that the very first page is a TCS page and
+ * following pages are regular pages. Does not measure the contents of the
+ * enclave as the signing tool is used at the moment only for the launch
+ * enclave, which is pass-through (everything gets a token).
+ */
+static bool measure_encl(const char *path, uint8_t *mrenclave)
+{
+	FILE *file;
+	struct stat sb;
+	EVP_MD_CTX *ctx;
+	uint64_t flags;
+	uint64_t offset;
+	uint8_t data[0x1000];
+	int rc;
+
+	ctx = EVP_MD_CTX_create();
+	if (!ctx)
+		return false;
+
+	file = fopen(path, "rb");
+	if (!file) {
+		perror("fopen");
+		EVP_MD_CTX_destroy(ctx);
+		return false;
+	}
+
+	rc = stat(path, &sb);
+	if (rc) {
+		perror("stat");
+		goto out;
+	}
+
+	if (!sb.st_size || sb.st_size & 0xfff) {
+		fprintf(stderr, "Invalid blob size %lu\n", sb.st_size);
+		goto out;
+	}
+
+	if (!mrenclave_ecreate(ctx, sb.st_size))
+		goto out;
+
+	for (offset = 0; offset < sb.st_size; offset += 0x1000) {
+		if (!offset)
+			flags = SGX_SECINFO_TCS;
+		else
+			flags = SGX_SECINFO_REG | SGX_SECINFO_R |
+				SGX_SECINFO_W | SGX_SECINFO_X;
+
+		if (!mrenclave_eadd(ctx, offset, flags))
+			goto out;
+
+		rc = fread(data, 1, 0x1000, file);
+		if (!rc)
+			break;
+		if (rc < 0x1000)
+			goto out;
+
+		if (!mrenclave_eextend(ctx, offset, data))
+			goto out;
+	}
+
+	if (!mrenclave_commit(ctx, mrenclave))
+		goto out;
+
+	fclose(file);
+	EVP_MD_CTX_destroy(ctx);
+	return true;
+out:
+	fclose(file);
+	EVP_MD_CTX_destroy(ctx);
+	return false;
+}
+
+/**
+ * sign_encl - sign enclave
+ * @sigstruct: pointer to SIGSTRUCT
+ * @key: 3072-bit RSA key
+ * @signature: byte array for the signature
+ *
+ * Calculates EMSA-PKCSv1.5 signature for the given SIGSTRUCT. The result is
+ * stored in big-endian format so that it can be further passed to OpenSSL
+ * libcrypto functions.
+ */
+static bool sign_encl(const struct sgx_sigstruct *sigstruct, RSA *key,
+		      uint8_t *signature)
+{
+	struct sgx_sigstruct_payload payload;
+	unsigned int siglen;
+	uint8_t digest[SHA256_DIGEST_LENGTH];
+	bool ret;
+
+	memcpy(&payload.header, &sigstruct->header, sizeof(sigstruct->header));
+	memcpy(&payload.body, &sigstruct->body, sizeof(sigstruct->body));
+
+	SHA256((unsigned char *)&payload, sizeof(payload), digest);
+
+	ret = RSA_sign(NID_sha256, digest, SHA256_DIGEST_LENGTH, signature,
+		       &siglen, key);
+
+	return ret;
+}
+
+struct q1q2_ctx {
+	BN_CTX *bn_ctx;
+	BIGNUM *m;
+	BIGNUM *s;
+	BIGNUM *q1;
+	BIGNUM *qr;
+	BIGNUM *q2;
+};
+
+static void free_q1q2_ctx(struct q1q2_ctx *ctx)
+{
+	BN_CTX_free(ctx->bn_ctx);
+	BN_free(ctx->m);
+	BN_free(ctx->s);
+	BN_free(ctx->q1);
+	BN_free(ctx->qr);
+	BN_free(ctx->q2);
+}
+
+static bool alloc_q1q2_ctx(const uint8_t *s, const uint8_t *m,
+			   struct q1q2_ctx *ctx)
+{
+	ctx->bn_ctx = BN_CTX_new();
+	ctx->s = BN_bin2bn(s, SGX_MODULUS_SIZE, NULL);
+	ctx->m = BN_bin2bn(m, SGX_MODULUS_SIZE, NULL);
+	ctx->q1 = BN_new();
+	ctx->qr = BN_new();
+	ctx->q2 = BN_new();
+
+	if (!ctx->bn_ctx || !ctx->s || !ctx->m || !ctx->q1 || !ctx->qr ||
+	    !ctx->q2) {
+		free_q1q2_ctx(ctx);
+		return false;
+	}
+
+	return true;
+}
+
+static bool calc_q1q2(const uint8_t *s, const uint8_t *m, uint8_t *q1,
+		      uint8_t *q2)
+{
+	struct q1q2_ctx ctx;
+
+	if (!alloc_q1q2_ctx(s, m, &ctx)) {
+		fprintf(stderr, "Not enough memory for Q1Q2 calculation\n");
+		return false;
+	}
+
+	if (!BN_mul(ctx.q1, ctx.s, ctx.s, ctx.bn_ctx))
+		goto out;
+
+	if (!BN_div(ctx.q1, ctx.qr, ctx.q1, ctx.m, ctx.bn_ctx))
+		goto out;
+
+	if (BN_num_bytes(ctx.q1) > SGX_MODULUS_SIZE) {
+		fprintf(stderr, "Too large Q1 %d bytes\n",
+			BN_num_bytes(ctx.q1));
+		goto out;
+	}
+
+	if (!BN_mul(ctx.q2, ctx.s, ctx.qr, ctx.bn_ctx))
+		goto out;
+
+	if (!BN_div(ctx.q2, NULL, ctx.q2, ctx.m, ctx.bn_ctx))
+		goto out;
+
+	if (BN_num_bytes(ctx.q2) > SGX_MODULUS_SIZE) {
+		fprintf(stderr, "Too large Q2 %d bytes\n",
+			BN_num_bytes(ctx.q2));
+		goto out;
+	}
+
+	BN_bn2bin(ctx.q1, q1);
+	BN_bn2bin(ctx.q2, q2);
+
+	free_q1q2_ctx(&ctx);
+	return true;
+out:
+	free_q1q2_ctx(&ctx);
+	return false;
+}
+
+static bool save_sigstruct(const struct sgx_sigstruct *sigstruct,
+			   const char *path)
+{
+	FILE *f = fopen(path, "wb");
+
+	if (!f) {
+		fprintf(stderr, "Unable to open %s\n", path);
+		return false;
+	}
+
+	fwrite(sigstruct, sizeof(*sigstruct), 1, f);
+	fclose(f);
+	return true;
+}
+
+int main(int argc, char **argv)
+{
+	uint64_t header1[2] = {0x000000E100000006, 0x0000000000010000};
+	uint64_t header2[2] = {0x0000006000000101, 0x0000000100000060};
+	struct sgx_sigstruct ss;
+	const char *program;
+	int opt;
+	RSA *sign_key;
+
+	memset(&ss, 0, sizeof(ss));
+	ss.header.header1[0] = header1[0];
+	ss.header.header1[1] = header1[1];
+	ss.header.header2[0] = header2[0];
+	ss.header.header2[1] = header2[1];
+	ss.exponent = 3;
+	ss.body.attributes = SGX_ATTR_MODE64BIT | SGX_ATTR_EINITTOKENKEY;
+	ss.body.xfrm = 3,
+
+	sign_key_pass = getenv("KBUILD_SGX_SIGN_PIN");
+	program = argv[0];
+
+	do {
+		opt = getopt(argc, argv, "");
+		switch (opt) {
+		case -1:
+			break;
+		default:
+			exit_usage(program);
+		}
+	} while (opt != -1);
+
+	argc -= optind;
+	argv += optind;
+
+	if (argc < 3)
+		exit_usage(program);
+
+	/* sanity check only */
+	if (check_crypto_errors())
+		exit(1);
+
+	sign_key = load_sign_key(argv[0]);
+	if (!sign_key)
+		goto out;
+
+	BN_bn2bin(get_modulus(sign_key), ss.modulus);
+
+	if (!measure_encl(argv[1], ss.body.mrenclave))
+		goto out;
+
+	if (!sign_encl(&ss, sign_key, ss.signature))
+		goto out;
+
+	if (!calc_q1q2(ss.signature, ss.modulus, ss.q1, ss.q2))
+		goto out;
+
+	/* convert to little endian */
+	reverse_bytes(ss.signature, SGX_MODULUS_SIZE);
+	reverse_bytes(ss.modulus, SGX_MODULUS_SIZE);
+	reverse_bytes(ss.q1, SGX_MODULUS_SIZE);
+	reverse_bytes(ss.q2, SGX_MODULUS_SIZE);
+
+	if (!save_sigstruct(&ss, argv[2]))
+		goto out;
+	exit(0);
+out:
+	check_crypto_errors();
+	exit(1);
+}
diff --git a/drivers/platform/x86/intel_sgx/le/enclave/string.c b/drivers/platform/x86/intel_sgx/le/enclave/string.c
new file mode 120000
index 000000000000..5ea3579f0ec6
--- /dev/null
+++ b/drivers/platform/x86/intel_sgx/le/enclave/string.c
@@ -0,0 +1 @@
+../string.c
\ No newline at end of file
diff --git a/drivers/platform/x86/intel_sgx/le/entry.S b/drivers/platform/x86/intel_sgx/le/entry.S
new file mode 100644
index 000000000000..5edab50815a2
--- /dev/null
+++ b/drivers/platform/x86/intel_sgx/le/entry.S
@@ -0,0 +1,70 @@
+// SPDX-License-Identifier: (GPL-2.0 OR BSD-3-Clause)
+// Copyright(c) 2016-17 Intel Corporation.
+//
+// Authors:
+//
+// Jarkko Sakkinen <jarkko.sakkinen@linux.intel.com>
+
+#include <asm/sgx_le.h>
+#include <sgx_asm.h>
+
+	.text
+
+	.global sgx_get_token
+sgx_get_token:
+	push	%rbx
+	mov	$0x02, %rax
+	mov	%rsi, %rbx
+	mov	%rdx, %rsi
+	mov	$sgx_async_exit, %rcx
+sgx_async_exit:
+	ENCLU
+	pop	%rbx
+	ret
+
+	.global sgx_sys_read
+sgx_sys_read:
+	mov	$0, %rax
+	mov	%rsi, %rdx /* buf */
+	mov	%rdi, %rsi /* count */
+	mov	$SGX_LE_PIPE_FD, %rdi
+	syscall
+	ret
+
+	.global sgx_sys_write
+sgx_sys_write:
+	mov	$1, %rax
+	mov	%rsi, %rdx /* buf */
+	mov	%rdi, %rsi /* count */
+	mov	$SGX_LE_PIPE_FD, %rdi
+	syscall
+	ret
+
+	.global sgx_sys_close
+sgx_sys_close:
+	mov	$3, %rax
+	syscall
+	ret
+
+	.global sgx_sys_mmap
+sgx_sys_mmap:
+	mov	$9, %rax
+	mov	%rdi, %r8 /* fd */
+	xor	%rdi, %rdi /* any address */
+
+	mov	$0x07, %rdx /* rwx */
+	mov	$0x01, %r10 /* shared */
+	mov	$0x00, %r9 /* offset */
+	syscall
+	ret
+
+	.global sgx_sys_ioctl
+sgx_sys_ioctl:
+	mov	$16, %rax
+	syscall
+	ret
+
+	.global sgx_sys_exit
+sgx_sys_exit:
+	mov	$60, %rax
+	syscall
diff --git a/drivers/platform/x86/intel_sgx/le/include/sgx_asm.h b/drivers/platform/x86/intel_sgx/le/include/sgx_asm.h
new file mode 100644
index 000000000000..9d32884846fd
--- /dev/null
+++ b/drivers/platform/x86/intel_sgx/le/include/sgx_asm.h
@@ -0,0 +1,15 @@
+// SPDX-License-Identifier: (GPL-2.0 OR BSD-3-Clause)
+// Copyright(c) 2016-17 Intel Corporation.
+//
+// Authors:
+//
+// Jarkko Sakkinen <jarkko.sakkinen@linux.intel.com>
+
+#ifndef SGX_ASM_H
+#define SGX_ASM_H
+
+.macro ENCLU
+.byte 0x0f, 0x01, 0xd7
+.endm
+
+#endif /* SGX_ASM_H */
diff --git a/drivers/platform/x86/intel_sgx/le/main.c b/drivers/platform/x86/intel_sgx/le/main.c
new file mode 100644
index 000000000000..2e2b544b5305
--- /dev/null
+++ b/drivers/platform/x86/intel_sgx/le/main.c
@@ -0,0 +1,140 @@
+// SPDX-License-Identifier: (GPL-2.0 OR BSD-3-Clause)
+// Copyright(c) 2016-17 Intel Corporation.
+//
+// Authors:
+//
+// Jarkko Sakkinen <jarkko.sakkinen@linux.intel.com>
+
+#include <asm/sgx.h>
+#include <asm/sgx_arch.h>
+#include <asm/sgx_le.h>
+#include <linux/string.h>
+#include <linux/types.h>
+#include <uapi/asm/sgx.h>
+#include "main.h"
+
+static void *start_launch_enclave(void)
+{
+	struct sgx_enclave_create create_ioc;
+	struct sgx_enclave_add_page add_ioc;
+	struct sgx_enclave_init init_ioc;
+	struct sgx_secs secs;
+	struct sgx_secinfo secinfo;
+	unsigned long blob_base;
+	unsigned long blob_size;
+	unsigned long offset;
+	int rc;
+
+	memset(&secs, 0, sizeof(secs));
+	memset(&secinfo, 0, sizeof(secinfo));
+
+	secs.ssaframesize = 1;
+	secs.attributes = SGX_ATTR_MODE64BIT | SGX_ATTR_EINITTOKENKEY;
+	secs.xfrm = 3;
+
+	blob_base = (unsigned long)&sgx_le_blob;
+	blob_size = (unsigned long)&sgx_le_blob_end - blob_base;
+
+	for (secs.size = 4096; secs.size < blob_size; )
+		secs.size <<= 1;
+
+	secs.base = (unsigned long)sgx_sys_mmap(SGX_LE_DEV_FD, secs.size);
+	if (secs.base == (unsigned long)MAP_FAILED)
+		goto out;
+
+	create_ioc.src = (unsigned long)&secs;
+	rc = sgx_sys_ioctl(SGX_LE_DEV_FD, SGX_IOC_ENCLAVE_CREATE, &create_ioc);
+	if (rc)
+		goto out;
+
+	add_ioc.secinfo = (unsigned long)&secinfo;
+	add_ioc.mrmask = 0xFFFF;
+
+	for (offset = 0; offset < blob_size; offset += 0x1000) {
+		if (!offset)
+			secinfo.flags = SGX_SECINFO_TCS;
+		else
+			secinfo.flags = SGX_SECINFO_REG | SGX_SECINFO_R |
+					SGX_SECINFO_W | SGX_SECINFO_X;
+
+		add_ioc.addr = secs.base + offset;
+		add_ioc.src = blob_base + offset;
+
+		rc = sgx_sys_ioctl(SGX_LE_DEV_FD, SGX_IOC_ENCLAVE_ADD_PAGE,
+				   &add_ioc);
+		if (rc)
+			goto out;
+	}
+
+	init_ioc.addr = secs.base;
+	init_ioc.sigstruct = (uint64_t)&sgx_le_ss;
+	rc = sgx_sys_ioctl(SGX_LE_DEV_FD, SGX_IOC_ENCLAVE_INIT, &init_ioc);
+	if (rc)
+		goto out;
+
+	return (void *)secs.base;
+out:
+	return NULL;
+}
+
+static int read_input(void *data, unsigned int len)
+{
+	uint8_t *ptr = (uint8_t *)data;
+	long i;
+	long ret;
+
+	for (i = 0; i < len; ) {
+		ret = sgx_sys_read(&ptr[i], len - i);
+		if (ret < 0)
+			return ret;
+
+		i += ret;
+	}
+
+	return 0;
+}
+
+static int write_token(const struct sgx_einittoken *token)
+{
+	const uint8_t *ptr = (const uint8_t *)token;
+	long i;
+	long ret;
+
+	for (i = 0; i < sizeof(*token); ) {
+		ret = sgx_sys_write(&ptr[i], sizeof(*token) - i);
+		if (ret < 0)
+			return ret;
+
+		i += ret;
+	}
+
+	return 0;
+}
+
+void _start(void)
+{
+	struct sgx_launch_request req;
+	struct sgx_einittoken token;
+	void *entry;
+
+	sgx_sys_close(SGX_LE_EXE_FD);
+	entry = start_launch_enclave();
+	sgx_sys_close(SGX_LE_DEV_FD);
+	if (!entry)
+		sgx_sys_exit(1);
+
+	for ( ; ; ) {
+		memset(&req, 0, sizeof(req));
+		memset(&token, 0, sizeof(token));
+
+		if (read_input(&req, sizeof(req)))
+			sgx_sys_exit(1);
+
+		sgx_get_token(&req, entry, &token);
+
+		if (write_token(&token))
+			sgx_sys_exit(1);
+	}
+
+	__builtin_unreachable();
+}
diff --git a/drivers/platform/x86/intel_sgx/le/main.h b/drivers/platform/x86/intel_sgx/le/main.h
new file mode 100644
index 000000000000..7069d416d773
--- /dev/null
+++ b/drivers/platform/x86/intel_sgx/le/main.h
@@ -0,0 +1,30 @@
+// SPDX-License-Identifier: (GPL-2.0 OR BSD-3-Clause)
+// Copyright(c) 2016-17 Intel Corporation.
+//
+// Authors:
+//
+// Jarkko Sakkinen <jarkko.sakkinen@linux.intel.com>
+
+#ifndef MAIN_H
+#define MAIN_H
+
+#ifndef NULL
+#define NULL ((void *)0)
+#endif
+
+#define MAP_FAILED ((void *)-1)
+
+extern unsigned char sgx_le_blob[];
+extern unsigned char sgx_le_blob_end[];
+extern unsigned char sgx_le_ss[];
+
+void sgx_get_token(struct sgx_launch_request *req, void *entry,
+		   struct sgx_einittoken *token);
+long sgx_sys_read(void *buf, unsigned long count);
+long sgx_sys_write(const void *buf, unsigned long count);
+long sgx_sys_close(long fd);
+long sgx_sys_mmap(long fd, unsigned long size);
+long sgx_sys_ioctl(long fd, unsigned long cmd, void *arg);
+long sgx_sys_exit(long status);
+
+#endif /* MAIN_H */
diff --git a/drivers/platform/x86/intel_sgx/le/sgx_le_piggy.S b/drivers/platform/x86/intel_sgx/le/sgx_le_piggy.S
new file mode 100644
index 000000000000..e1881066f089
--- /dev/null
+++ b/drivers/platform/x86/intel_sgx/le/sgx_le_piggy.S
@@ -0,0 +1,22 @@
+// SPDX-License-Identifier: (GPL-2.0 OR BSD-3-Clause)
+// Copyright(c) 2016-17 Intel Corporation.
+//
+// Authors:
+//
+// Jarkko Sakkinen <jarkko.sakkinen@linux.intel.com>
+
+#include <linux/linkage.h>
+#include <asm/page_types.h>
+
+	.section ".rodata","a"
+	.balign PAGE_SIZE
+
+GLOBAL(sgx_le_blob)
+	.incbin	"drivers/platform/x86/intel_sgx/le/enclave/sgx_le.bin"
+END(sgx_le_blob)
+
+GLOBAL(sgx_le_blob_end);
+
+GLOBAL(sgx_le_ss)
+	.incbin	"drivers/platform/x86/intel_sgx/le/enclave/sgx_le.ss"
+END(sgx_le_ss)
diff --git a/drivers/platform/x86/intel_sgx/le/string.c b/drivers/platform/x86/intel_sgx/le/string.c
new file mode 100644
index 000000000000..0a61977f8628
--- /dev/null
+++ b/drivers/platform/x86/intel_sgx/le/string.c
@@ -0,0 +1,39 @@
+// SPDX-License-Identifier: (GPL-2.0 OR BSD-3-Clause)
+// Copyright(c) 2016-17 Intel Corporation.
+//
+// Authors:
+//
+// Jarkko Sakkinen <jarkko.sakkinen@linux.intel.com>
+
+#include <linux/types.h>
+
+/* This might look a bit ugly but is needed because of the way asm/string_64.h
+ * redefines the symbols depending on the CONFIG_KASAN flag.
+ */
+#ifdef CONFIG_KASAN
+void *__memset(void *s, int c, size_t n)
+#else
+void *memset(void *s, int c, size_t n)
+#endif
+{
+	unsigned long i;
+
+	for (i = 0; i < n; i++)
+		((unsigned char *)s)[i] = c;
+
+	return s;
+}
+
+#ifdef CONFIG_KASAN
+void *__memcpy(void *dest, const void *src, size_t n)
+#else
+void *memcpy(void *dest, const void *src, size_t n)
+#endif
+{
+	size_t i;
+
+	for (i = 0; i < n; i++)
+		((char *)dest)[i] = ((char *)src)[i];
+
+	return dest;
+}
diff --git a/drivers/platform/x86/intel_sgx/sgx.h b/drivers/platform/x86/intel_sgx/sgx.h
index 193f5e5b2338..9b9e9d7d09dd 100644
--- a/drivers/platform/x86/intel_sgx/sgx.h
+++ b/drivers/platform/x86/intel_sgx/sgx.h
@@ -103,13 +103,18 @@ struct sgx_encl {
 	struct mmu_notifier mmu_notifier;
 };
 
+extern unsigned char sgx_le_proxy[];
+extern unsigned char sgx_le_proxy_end[];
+extern struct sgx_sigstruct sgx_le_ss;
 extern struct workqueue_struct *sgx_add_page_wq;
 extern u64 sgx_encl_size_max_32;
 extern u64 sgx_encl_size_max_64;
 extern u64 sgx_xfrm_mask;
 extern u32 sgx_misc_reserved;
 extern u32 sgx_xsave_size_tbl[64];
+extern u64 sgx_le_pubkeyhash[4];
 
+extern const struct file_operations sgx_fops;
 extern const struct vm_operations_struct sgx_vm_ops;
 
 int sgx_encl_find(struct mm_struct *mm, unsigned long addr,
@@ -161,4 +166,16 @@ unsigned int sgx_alloc_va_slot(struct sgx_va_page *va_page);
 void sgx_free_va_slot(struct sgx_va_page *va_page, unsigned int offset);
 bool sgx_va_page_full(struct sgx_va_page *va_page);
 
+extern struct sgx_le_ctx sgx_le_ctx;
+
+int sgx_get_key_hash(const void *modulus, void *hash);
+int sgx_le_init(struct sgx_le_ctx *ctx);
+void sgx_le_exit(struct sgx_le_ctx *ctx);
+void sgx_le_stop(struct sgx_le_ctx *ctx, bool update_users);
+int sgx_le_start(struct sgx_le_ctx *ctx);
+int sgx_le_get_token(struct sgx_le_ctx *ctx,
+		     const struct sgx_encl *encl,
+		     const struct sgx_sigstruct *sigstruct,
+		     struct sgx_einittoken *token);
+
 #endif /* __ARCH_X86_INTEL_SGX_H__ */
diff --git a/drivers/platform/x86/intel_sgx/sgx_encl.c b/drivers/platform/x86/intel_sgx/sgx_encl.c
index 35436497530b..043be4bb4829 100644
--- a/drivers/platform/x86/intel_sgx/sgx_encl.c
+++ b/drivers/platform/x86/intel_sgx/sgx_encl.c
@@ -774,20 +774,6 @@ int sgx_encl_add_page(struct sgx_encl *encl, unsigned long addr, void *data,
 	return ret;
 }
 
-static int sgx_einit(struct sgx_encl *encl, struct sgx_sigstruct *sigstruct,
-		     struct sgx_einittoken *token)
-{
-	struct sgx_epc_page *secs_epc = encl->secs.epc_page;
-	void *secs_va;
-	int ret;
-
-	secs_va = sgx_get_page(secs_epc);
-	ret = __einit(sigstruct, token, secs_va);
-	sgx_put_page(secs_va);
-
-	return ret;
-}
-
 /**
  * sgx_encl_init - perform EINIT for the given enclave
  *
@@ -821,8 +807,8 @@ int sgx_encl_init(struct sgx_encl *encl, struct sgx_sigstruct *sigstruct,
 
 	for (i = 0; i < SGX_EINIT_SLEEP_COUNT; i++) {
 		for (j = 0; j < SGX_EINIT_SPIN_COUNT; j++) {
-			ret = sgx_einit(encl, sigstruct, token);
-
+			ret = sgx_einit(sigstruct, token, encl->secs.epc_page,
+					sgx_le_pubkeyhash);
 			if (ret == SGX_UNMASKED_EVENT)
 				continue;
 			else
diff --git a/drivers/platform/x86/intel_sgx/sgx_ioctl.c b/drivers/platform/x86/intel_sgx/sgx_ioctl.c
index 68b04893d66a..44b7bc03c2e7 100644
--- a/drivers/platform/x86/intel_sgx/sgx_ioctl.c
+++ b/drivers/platform/x86/intel_sgx/sgx_ioctl.c
@@ -186,7 +186,9 @@ static long sgx_ioc_enclave_init(struct file *filep, unsigned int cmd,
 	if (ret)
 		goto out;
 
-	ret = sgx_encl_init(encl, sigstruct, einittoken);
+	ret = sgx_le_get_token(&sgx_le_ctx, encl, sigstruct, einittoken);
+	if (!ret)
+		ret = sgx_encl_init(encl, sigstruct, einittoken);
 
 	kref_put(&encl->refcount, sgx_encl_release);
 
diff --git a/drivers/platform/x86/intel_sgx/sgx_le.c b/drivers/platform/x86/intel_sgx/sgx_le.c
new file mode 100644
index 000000000000..ffd0fce07614
--- /dev/null
+++ b/drivers/platform/x86/intel_sgx/sgx_le.c
@@ -0,0 +1,303 @@
+// SPDX-License-Identifier: (GPL-2.0 OR BSD-3-Clause)
+// Copyright(c) 2016-17 Intel Corporation.
+//
+// Authors:
+//
+// Jarkko Sakkinen <jarkko.sakkinen@linux.intel.com>
+
+#include <asm/sgx_le.h>
+#include <linux/anon_inodes.h>
+#include <linux/file.h>
+#include <linux/fs.h>
+#include <linux/kmod.h>
+#include <linux/mutex.h>
+#include <linux/sched/signal.h>
+#include <linux/shmem_fs.h>
+#include <linux/wait.h>
+#include "sgx.h"
+
+struct sgx_le_ctx {
+	struct pid *tgid;
+	char *argv[2];
+	struct crypto_shash *tfm;
+	struct mutex hash_lock;
+	struct mutex launch_lock;
+	struct rw_semaphore users;
+	wait_queue_head_t wq;
+	bool kernel_read;
+	bool user_read;
+	struct file *pipe;
+	struct sgx_launch_request req;
+	struct sgx_einittoken token;
+};
+
+struct sgx_le_ctx sgx_le_ctx;
+
+static int __sgx_get_key_hash(struct crypto_shash *tfm, const void *modulus,
+			      void *hash)
+{
+	SHASH_DESC_ON_STACK(shash, tfm);
+
+	shash->tfm = tfm;
+	shash->flags = CRYPTO_TFM_REQ_MAY_SLEEP;
+
+	return crypto_shash_digest(shash, modulus, SGX_MODULUS_SIZE, hash);
+}
+
+/**
+ * sgx_get_key_hash - calculate SHA256 for a given RSA key
+ * @modulus:	modulus of the key
+ * @hash:	the resulting hash
+ */
+int sgx_get_key_hash(const void *modulus, void *hash)
+{
+	struct crypto_shash *tfm;
+	int ret;
+
+	tfm = crypto_alloc_shash("sha256", 0, CRYPTO_ALG_ASYNC);
+	if (IS_ERR(tfm))
+		return PTR_ERR(tfm);
+
+	ret = __sgx_get_key_hash(tfm, modulus, hash);
+
+	crypto_free_shash(tfm);
+	return ret;
+}
+
+static ssize_t sgx_le_ctx_fops_read(struct file *filp, char __user *buf,
+				    size_t count, loff_t *off)
+{
+	struct sgx_le_ctx *ctx = filp->private_data;
+	int ret;
+
+	if (count != sizeof(ctx->req)) {
+		pr_crit("%s: invalid count %lu\n", __func__, count);
+		return -EIO;
+	}
+
+	ret = wait_event_interruptible(ctx->wq, ctx->user_read);
+	if (ret)
+		return -EINTR;
+
+	ret = copy_to_user(buf, &ctx->req, count);
+	ctx->user_read = false;
+
+	return ret ? ret : count;
+}
+
+static ssize_t sgx_le_ctx_fops_write(struct file *filp, const char __user *buf,
+				     size_t count, loff_t *off)
+{
+	struct sgx_le_ctx *ctx = filp->private_data;
+	int ret;
+
+	if (count != sizeof(ctx->token)) {
+		pr_crit("%s: invalid count %lu\n", __func__, count);
+		return -EIO;
+	}
+
+	ret = copy_from_user(&ctx->token, buf, count);
+	if (!ret)
+		ctx->kernel_read = true;
+	wake_up_interruptible(&ctx->wq);
+
+	return ret ? ret : count;
+}
+
+static const struct file_operations sgx_le_ctx_fops = {
+	.owner = THIS_MODULE,
+	.llseek = no_llseek,
+	.read = sgx_le_ctx_fops_read,
+	.write = sgx_le_ctx_fops_write,
+};
+
+static int sgx_le_task_init(struct subprocess_info *subinfo, struct cred *new)
+{
+	struct sgx_le_ctx *ctx = (struct sgx_le_ctx *)subinfo->data;
+	struct file *tmp_filp;
+	unsigned long len;
+	loff_t pos = 0;
+	int ret;
+
+	len = (unsigned long)&sgx_le_proxy_end - (unsigned long)&sgx_le_proxy;
+
+	tmp_filp = shmem_file_setup("[sgx_le_proxy]", len, 0);
+	if (IS_ERR(tmp_filp)) {
+		ret = PTR_ERR(tmp_filp);
+		return ret;
+	}
+	fd_install(SGX_LE_EXE_FD, tmp_filp);
+
+	ret = kernel_write(tmp_filp, &sgx_le_proxy, len, &pos);
+	if (ret != len && ret >= 0)
+		return -ENOMEM;
+	if (ret < 0)
+		return ret;
+
+	tmp_filp = anon_inode_getfile("[/dev/sgx]", &sgx_fops, NULL, O_RDWR);
+	if (IS_ERR(tmp_filp))
+		return PTR_ERR(tmp_filp);
+	fd_install(SGX_LE_DEV_FD, tmp_filp);
+
+	tmp_filp = anon_inode_getfile("[sgx_le]", &sgx_le_ctx_fops, ctx,
+				      O_RDWR);
+	if (IS_ERR(tmp_filp))
+		return PTR_ERR(tmp_filp);
+	fd_install(SGX_LE_PIPE_FD, tmp_filp);
+
+	ctx->tgid = get_pid(task_tgid(current));
+	ctx->pipe = tmp_filp;
+
+	return 0;
+}
+
+static void __sgx_le_stop(struct sgx_le_ctx *ctx)
+{
+	if (ctx->tgid) {
+		fput(ctx->pipe);
+		kill_pid(ctx->tgid, SIGKILL, 1);
+		put_pid(ctx->tgid);
+		ctx->tgid = NULL;
+	}
+}
+
+void sgx_le_stop(struct sgx_le_ctx *ctx, bool update_users)
+{
+	if (update_users) {
+		up_read(&ctx->users);
+		if (!down_write_trylock(&ctx->users))
+			return;
+	}
+
+	mutex_lock(&ctx->launch_lock);
+	__sgx_le_stop(ctx);
+	mutex_unlock(&ctx->launch_lock);
+
+	if (update_users)
+		up_write(&ctx->users);
+}
+
+static int __sgx_le_start(struct sgx_le_ctx *ctx)
+{
+	struct subprocess_info *subinfo;
+	int ret;
+
+	if (ctx->tgid)
+		return 0;
+
+	ctx->argv[0] = SGX_LE_EXE_PATH;
+	ctx->argv[1] = NULL;
+
+	subinfo = call_usermodehelper_setup(ctx->argv[0], ctx->argv,
+					    NULL, GFP_KERNEL, sgx_le_task_init,
+					    NULL, &sgx_le_ctx);
+	if (!subinfo)
+		return -ENOMEM;
+
+	ret = call_usermodehelper_exec(subinfo, UMH_WAIT_EXEC);
+	if (ret) {
+		__sgx_le_stop(ctx);
+		return ret;
+	}
+
+	return 0;
+}
+
+int sgx_le_start(struct sgx_le_ctx *ctx)
+{
+	int ret;
+
+	down_read(&ctx->users);
+
+	mutex_lock(&ctx->launch_lock);
+	ret = __sgx_le_start(ctx);
+	mutex_unlock(&ctx->launch_lock);
+
+	if (ret)
+		up_read(&ctx->users);
+
+	return ret;
+}
+
+int sgx_le_init(struct sgx_le_ctx *ctx)
+{
+	struct crypto_shash *tfm;
+
+	tfm = crypto_alloc_shash("sha256", 0, CRYPTO_ALG_ASYNC);
+	if (IS_ERR(tfm))
+		return PTR_ERR(tfm);
+
+	ctx->tfm = tfm;
+	mutex_init(&ctx->hash_lock);
+	mutex_init(&ctx->launch_lock);
+	init_rwsem(&ctx->users);
+	init_waitqueue_head(&ctx->wq);
+
+	return 0;
+}
+
+void sgx_le_exit(struct sgx_le_ctx *ctx)
+{
+	mutex_lock(&ctx->launch_lock);
+	crypto_free_shash(ctx->tfm);
+	mutex_unlock(&ctx->launch_lock);
+}
+
+static int __sgx_le_get_token(struct sgx_le_ctx *ctx,
+			      const struct sgx_encl *encl,
+			      struct sgx_einittoken *token)
+{
+	ssize_t ret;
+
+	if (!ctx->tgid)
+		return -EIO;
+
+	ctx->user_read = true;
+	wake_up_interruptible(&ctx->wq);
+
+	ret = wait_event_interruptible(ctx->wq, ctx->kernel_read);
+	if (ret)
+		return -EINTR;
+
+	memcpy(token, &ctx->token, sizeof(*token));
+	ctx->kernel_read = false;
+
+	return 0;
+}
+
+int sgx_le_get_token(struct sgx_le_ctx *ctx,
+		     const struct sgx_encl *encl,
+		     const struct sgx_sigstruct *sigstruct,
+		     struct sgx_einittoken *token)
+{
+	u8 mrsigner[32];
+	int ret;
+
+	mutex_lock(&ctx->hash_lock);
+	ret = __sgx_get_key_hash(ctx->tfm, sigstruct->modulus, mrsigner);
+	if (ret) {
+		mutex_unlock(&ctx->hash_lock);
+		return ret;
+	}
+	if (!memcmp(mrsigner, sgx_le_pubkeyhash, 32)) {
+		token->payload.valid = false;
+		mutex_unlock(&ctx->hash_lock);
+		return 0;
+	}
+	mutex_unlock(&ctx->hash_lock);
+
+	mutex_lock(&ctx->launch_lock);
+	ret = __sgx_le_start(ctx);
+	if (ret) {
+		mutex_unlock(&ctx->launch_lock);
+		return ret;
+	}
+	memcpy(&ctx->req.mrenclave, sigstruct->body.mrenclave, 32);
+	memcpy(&ctx->req.mrsigner, mrsigner, 32);
+	ctx->req.attributes = encl->attributes;
+	ctx->req.xfrm = encl->xfrm;
+	memset(&ctx->token, 0, sizeof(ctx->token));
+	ret = __sgx_le_get_token(ctx, encl, token);
+	mutex_unlock(&ctx->launch_lock);
+	return ret;
+}
diff --git a/drivers/platform/x86/intel_sgx/sgx_le_proxy_piggy.S b/drivers/platform/x86/intel_sgx/sgx_le_proxy_piggy.S
new file mode 100644
index 000000000000..9d2057615d04
--- /dev/null
+++ b/drivers/platform/x86/intel_sgx/sgx_le_proxy_piggy.S
@@ -0,0 +1,22 @@
+// SPDX-License-Identifier: (GPL-2.0 OR BSD-3-Clause)
+// Copyright(c) 2016-17 Intel Corporation.
+//
+// Authors:
+//
+// Jarkko Sakkinen <jarkko.sakkinen@linux.intel.com>
+
+#include <linux/linkage.h>
+#include <asm/export.h>
+#include <asm/page_types.h>
+
+	.section ".rodata","a"
+
+GLOBAL(sgx_le_proxy)
+	.incbin	"drivers/platform/x86/intel_sgx/le/sgx_le_proxy"
+END(sgx_le_proxy)
+
+GLOBAL(sgx_le_proxy_end)
+
+GLOBAL(sgx_le_ss)
+	.incbin	"drivers/platform/x86/intel_sgx/le/enclave/sgx_le.ss"
+END(sgx_le_ss)
diff --git a/drivers/platform/x86/intel_sgx/sgx_main.c b/drivers/platform/x86/intel_sgx/sgx_main.c
index 79d772847026..39560ee94b43 100644
--- a/drivers/platform/x86/intel_sgx/sgx_main.c
+++ b/drivers/platform/x86/intel_sgx/sgx_main.c
@@ -31,6 +31,31 @@ u64 sgx_encl_size_max_64;
 u64 sgx_xfrm_mask = 0x3;
 u32 sgx_misc_reserved;
 u32 sgx_xsave_size_tbl[64];
+u64 sgx_le_pubkeyhash[4];
+
+static DECLARE_RWSEM(sgx_file_sem);
+
+static int sgx_open(struct inode *inode, struct file *file)
+{
+	int ret;
+
+	ret = sgx_le_start(&sgx_le_ctx);
+
+	if (!ret)
+		file->private_data = &sgx_le_ctx;
+
+	return ret;
+}
+
+static int sgx_release(struct inode *inode, struct file *file)
+{
+	if (!file->private_data)
+		return 0;
+
+	sgx_le_stop(file->private_data, true);
+
+	return 0;
+}
 
 #ifdef CONFIG_COMPAT
 long sgx_compat_ioctl(struct file *filep, unsigned int cmd, unsigned long arg)
@@ -84,8 +109,10 @@ static unsigned long sgx_get_unmapped_area(struct file *file,
 	return addr;
 }
 
-static const struct file_operations sgx_fops = {
+const struct file_operations sgx_fops = {
 	.owner			= THIS_MODULE,
+	.open			= sgx_open,
+	.release		= sgx_release,
 	.unlocked_ioctl		= sgx_ioctl,
 #ifdef CONFIG_COMPAT
 	.compat_ioctl		= sgx_compat_ioctl,
@@ -100,6 +127,7 @@ static int sgx_pm_suspend(struct device *dev)
 	struct sgx_epc_page *epc_page;
 	struct sgx_encl *encl;
 
+	sgx_le_stop(&sgx_le_ctx, false);
 	list_for_each_entry(epc_page, &sgx_active_page_list, list) {
 		encl_page = container_of(epc_page->impl, struct sgx_encl_page,
 					 impl);
@@ -176,6 +204,34 @@ static struct sgx_context *sgxm_ctx_alloc(struct device *parent)
 	return ctx;
 }
 
+static int sgx_init_msrs(void)
+{
+	u64 msrs[4];
+	int ret;
+
+	ret = sgx_get_key_hash(sgx_le_ss.modulus, sgx_le_pubkeyhash);
+	if (ret)
+		return ret;
+
+	if (sgx_lc_enabled)
+		return 0;
+
+	rdmsrl(MSR_IA32_SGXLEPUBKEYHASH0, msrs[0]);
+	rdmsrl(MSR_IA32_SGXLEPUBKEYHASH1, msrs[1]);
+	rdmsrl(MSR_IA32_SGXLEPUBKEYHASH2, msrs[2]);
+	rdmsrl(MSR_IA32_SGXLEPUBKEYHASH3, msrs[3]);
+
+	if ((sgx_le_pubkeyhash[0] != msrs[0]) ||
+	    (sgx_le_pubkeyhash[1] != msrs[1]) ||
+	    (sgx_le_pubkeyhash[2] != msrs[2]) ||
+	    (sgx_le_pubkeyhash[3] != msrs[3])) {
+		pr_err("IA32_SGXLEPUBKEYHASHn MSRs do not match to the launch enclave signing key\n");
+		return -ENODEV;
+	}
+
+	return 0;
+}
+
 static int sgx_dev_init(struct device *parent)
 {
 	struct sgx_context *sgx_dev;
@@ -186,6 +242,10 @@ static int sgx_dev_init(struct device *parent)
 	int ret;
 	int i;
 
+	ret = sgx_init_msrs();
+	if (ret)
+		return ret;
+
 	sgx_dev = sgxm_ctx_alloc(parent);
 
 	cpuid_count(SGX_CPUID, SGX_CPUID_CAPABILITIES, &eax, &ebx, &ecx, &edx);
@@ -213,11 +273,17 @@ static int sgx_dev_init(struct device *parent)
 	if (!sgx_add_page_wq)
 		return -ENOMEM;
 
-	ret = cdev_device_add(&sgx_dev->cdev, &sgx_dev->dev);
+	ret = sgx_le_init(&sgx_le_ctx);
 	if (ret)
 		goto out_workqueue;
 
+	ret = cdev_device_add(&sgx_dev->cdev, &sgx_dev->dev);
+	if (ret)
+		goto out_le;
+
 	return 0;
+out_le:
+	sgx_le_exit(&sgx_le_ctx);
 out_workqueue:
 	destroy_workqueue(sgx_add_page_wq);
 	return ret;
@@ -236,6 +302,7 @@ static int sgx_drv_remove(struct platform_device *pdev)
 	struct sgx_context *ctx = dev_get_drvdata(&pdev->dev);
 
 	cdev_device_del(&ctx->cdev, &ctx->dev);
+	sgx_le_exit(&sgx_le_ctx);
 	destroy_workqueue(sgx_add_page_wq);
 
 	return 0;
-- 
2.17.0

[PATCH v11 13/13] intel_sgx: in-kernel launch enclave

[Jarkko Sakkinen]

The Launch Enclave (LE) generates cryptographic launch tokens for user
enclaves. A launch token is used by EINIT to check whether the enclave
is authorized to launch or not. By having its own launch enclave, Linux
has full control of the enclave launch process.

LE is wrapped into a user space proxy program that reads enclave
signatures outputs launch tokens. The kernel-side glue code is
implemented by using the user space helper framework.  The IPC between
the LE proxy program and kernel is handled with an anonymous inode.

The commit also adds enclave signing tool that is used by kbuild to
measure and sign the launch enclave. CONFIG_INTEL_SGX_SIGNING_KEY points
to a PEM-file for the 3072-bit RSA key that is used as the LE public key
pair. The default location is:

  drivers/platform/x86/intel_sgx/sgx_signing_key.pem

If the default key does not exist kbuild will generate a random key and
place it to this location. KBUILD_SGX_SIGN_PIN can be used to specify
the passphrase for the LE public key.

The CMAC implementation has been derived from TinyCrypt. The kernel
AES-NI implementation is used for AES.

[1] https://github.com/01org/tinycrypt

Signed-off-by: Jarkko Sakkinen <jarkko.sakkinen@linux.intel.com>
Tested-by: Serge Ayoun <serge.ayoun@intel.com>
---
 arch/x86/include/asm/sgx.h                    |   1 -
 arch/x86/include/asm/sgx_le.h                 |  17 +
 drivers/platform/x86/intel_sgx/Kconfig        |  14 +
 drivers/platform/x86/intel_sgx/Makefile       |  19 +
 drivers/platform/x86/intel_sgx/le/Makefile    |  34 ++
 .../x86/intel_sgx/le/enclave/Makefile         |  54 ++
 .../intel_sgx/le/enclave/aesni-intel_asm.S    |   1 +
 .../x86/intel_sgx/le/enclave/cmac_mode.c      | 209 +++++++
 .../x86/intel_sgx/le/enclave/cmac_mode.h      |  54 ++
 .../x86/intel_sgx/le/enclave/encl_bootstrap.S | 116 ++++
 .../platform/x86/intel_sgx/le/enclave/main.c  | 146 +++++
 .../platform/x86/intel_sgx/le/enclave/main.h  |  19 +
 .../x86/intel_sgx/le/enclave/sgx_le.lds       |  33 ++
 .../x86/intel_sgx/le/enclave/sgxsign.c        | 551 ++++++++++++++++++
 .../x86/intel_sgx/le/enclave/string.c         |   1 +
 drivers/platform/x86/intel_sgx/le/entry.S     |  70 +++
 .../x86/intel_sgx/le/include/sgx_asm.h        |  15 +
 drivers/platform/x86/intel_sgx/le/main.c      | 140 +++++
 drivers/platform/x86/intel_sgx/le/main.h      |  30 +
 .../platform/x86/intel_sgx/le/sgx_le_piggy.S  |  22 +
 drivers/platform/x86/intel_sgx/le/string.c    |  39 ++
 drivers/platform/x86/intel_sgx/sgx.h          |  17 +
 drivers/platform/x86/intel_sgx/sgx_encl.c     |  18 +-
 drivers/platform/x86/intel_sgx/sgx_ioctl.c    |   4 +-
 drivers/platform/x86/intel_sgx/sgx_le.c       | 303 ++++++++++
 .../x86/intel_sgx/sgx_le_proxy_piggy.S        |  22 +
 drivers/platform/x86/intel_sgx/sgx_main.c     |  71 ++-
 27 files changed, 2000 insertions(+), 20 deletions(-)
 create mode 100644 arch/x86/include/asm/sgx_le.h
 create mode 100644 drivers/platform/x86/intel_sgx/le/Makefile
 create mode 100644 drivers/platform/x86/intel_sgx/le/enclave/Makefile
 create mode 120000 drivers/platform/x86/intel_sgx/le/enclave/aesni-intel_asm.S
 create mode 100644 drivers/platform/x86/intel_sgx/le/enclave/cmac_mode.c
 create mode 100644 drivers/platform/x86/intel_sgx/le/enclave/cmac_mode.h
 create mode 100644 drivers/platform/x86/intel_sgx/le/enclave/encl_bootstrap.S
 create mode 100644 drivers/platform/x86/intel_sgx/le/enclave/main.c
 create mode 100644 drivers/platform/x86/intel_sgx/le/enclave/main.h
 create mode 100644 drivers/platform/x86/intel_sgx/le/enclave/sgx_le.lds
 create mode 100644 drivers/platform/x86/intel_sgx/le/enclave/sgxsign.c
 create mode 120000 drivers/platform/x86/intel_sgx/le/enclave/string.c
 create mode 100644 drivers/platform/x86/intel_sgx/le/entry.S
 create mode 100644 drivers/platform/x86/intel_sgx/le/include/sgx_asm.h
 create mode 100644 drivers/platform/x86/intel_sgx/le/main.c
 create mode 100644 drivers/platform/x86/intel_sgx/le/main.h
 create mode 100644 drivers/platform/x86/intel_sgx/le/sgx_le_piggy.S
 create mode 100644 drivers/platform/x86/intel_sgx/le/string.c
 create mode 100644 drivers/platform/x86/intel_sgx/sgx_le.c
 create mode 100644 drivers/platform/x86/intel_sgx/sgx_le_proxy_piggy.S

diff --git a/arch/x86/include/asm/sgx.h b/arch/x86/include/asm/sgx.h
index ae738e16ba6c..07c16945076e 100644
--- a/arch/x86/include/asm/sgx.h
+++ b/arch/x86/include/asm/sgx.h
@@ -259,7 +259,6 @@ struct sgx_launch_request {
 	u8 mrsigner[32];
 	uint64_t attributes;
 	uint64_t xfrm;
-	struct sgx_einittoken token;
 };
 
 #define SGX_FN(name, params...)		\
diff --git a/arch/x86/include/asm/sgx_le.h b/arch/x86/include/asm/sgx_le.h
new file mode 100644
index 000000000000..61f73fa71ba2
--- /dev/null
+++ b/arch/x86/include/asm/sgx_le.h
@@ -0,0 +1,17 @@
+// SPDX-License-Identifier: (GPL-2.0 OR BSD-3-Clause)
+// Copyright(c) 2016-17 Intel Corporation.
+//
+// Authors:
+//
+// Jarkko Sakkinen <jarkko.sakkinen@linux.intel.com>
+
+#ifndef _ASM_X86_SGX_LE_H
+#define _ASM_X86_SGX_LE_H
+
+#define SGX_LE_EXE_PATH "/proc/self/fd/3"
+
+#define SGX_LE_EXE_FD 3
+#define SGX_LE_DEV_FD 4
+#define SGX_LE_PIPE_FD 5
+
+#endif /* _ASM_X86_SGX_LE_H */
diff --git a/drivers/platform/x86/intel_sgx/Kconfig b/drivers/platform/x86/intel_sgx/Kconfig
index 4cd88310617e..4db0297b55ab 100644
--- a/drivers/platform/x86/intel_sgx/Kconfig
+++ b/drivers/platform/x86/intel_sgx/Kconfig
@@ -2,12 +2,16 @@
 # Intel SGX
 #
 
+menu "Intel SGX"
+
 config INTEL_SGX
 	tristate "Intel(R) SGX Driver"
 	default n
 	depends on X86_64 && CPU_SUP_INTEL
 	select INTEL_SGX_CORE
 	select MMU_NOTIFIER
+	select CRYPTO
+	select CRYPTO_SHA256
 	help
 	Intel(R) SGX is a set of CPU instructions that can be used by
 	applications to set aside private regions of code and data.  The code
@@ -18,3 +22,13 @@ config INTEL_SGX
 	called Enclave Page Cache (EPC). There is a hardware unit in the
 	processor called Memory Encryption Engine. The MEE encrypts and decrypts
 	the EPC pages as they enter and leave the processor package.
+
+config INTEL_SGX_SIGNING_KEY
+	string "Path to the Intel SGX LE signing key"
+	default "drivers/platform/x86/intel_sgx/signing_key.pem"
+	depends on INTEL_SGX
+	---help---
+	Provide a path to a 3072-bit RSA private key that will be used to
+	sign the launch enclave.
+
+endmenu
diff --git a/drivers/platform/x86/intel_sgx/Makefile b/drivers/platform/x86/intel_sgx/Makefile
index 95f254e30a8b..a5b729310ee2 100644
--- a/drivers/platform/x86/intel_sgx/Makefile
+++ b/drivers/platform/x86/intel_sgx/Makefile
@@ -11,3 +11,22 @@ intel_sgx-$(CONFIG_INTEL_SGX) += \
 	sgx_main.o \
 	sgx_fault.o \
 	sgx_vma.o \
+	sgx_le.o \
+	sgx_le_proxy_piggy.o
+
+$(eval $(call config_filename,INTEL_SGX_SIGNING_KEY))
+
+INTEL_SGX_SIGNING_KEY_PATH := \
+	$(INTEL_SGX_SIGNING_KEY_SRCPREFIX)$(INTEL_SGX_SIGNING_KEY_FILENAME)
+
+ifeq ($(CONFIG_INTEL_SGX_SIGNING_KEY),"drivers/platform/x86/intel_sgx/signing_key.pem")
+$(INTEL_SGX_SIGNING_KEY_PATH):
+	$(Q)openssl genrsa -3 -out $(INTEL_SGX_SIGNING_KEY_PATH) 3072
+endif
+
+$(obj)/sgx_le_proxy_piggy.o: $(INTEL_SGX_SIGNING_KEY_PATH) \
+			     $(obj)/le/sgx_le_proxy
+$(obj)/le/sgx_le_proxy: FORCE
+	$(Q)$(MAKE) -j1 $(build)=$(obj)/le $@
+
+export INTEL_SGX_SIGNING_KEY_PATH
diff --git a/drivers/platform/x86/intel_sgx/le/Makefile b/drivers/platform/x86/intel_sgx/le/Makefile
new file mode 100644
index 000000000000..5adaddc71bd6
--- /dev/null
+++ b/drivers/platform/x86/intel_sgx/le/Makefile
@@ -0,0 +1,34 @@
+# SPDX-License-Identifier: (GPL-2.0 OR BSD-3-Clause)
+# Copyright(c) 2016-17 Intel Corporation.
+#
+# Authors:
+#
+# Jarkko Sakkinen <jarkko.sakkinen@linux.intel.com>
+
+KASAN_SANITIZE := n
+OBJECT_FILES_NON_STANDARD := y
+KCOV_INSTRUMENT := n
+KBUILD_CFLAGS := -Wall -Werror -static -nostdlib -nostartfiles -fno-builtin \
+		 -I$(obj)/include
+KBUILD_AFLAGS += -I$(obj)/include
+
+subdir- := enclave
+
+always := sgx_le_proxy
+clean-files := sgx_le_proxy
+
+#
+# sgx_le_proxy
+#
+
+sgx_le_proxy-y += main.o entry.o sgx_le_piggy.o string.o
+targets += $(sgx_le_proxy-y)
+SGX_LE_PROXY_OBJS = $(addprefix $(obj)/,$(sgx_le_proxy-y))
+
+$(obj)/sgx_le_piggy.o: $(obj)/enclave/sgx_le.bin $(obj)/enclave/sgx_le.ss
+$(obj)/enclave/sgx_le.bin $(obj)/enclave/sgx_le.ss: FORCE
+	$(Q)$(MAKE) -j1 $(build)=$(obj)/enclave $@
+
+targets += sgx_le_proxy
+$(obj)/sgx_le_proxy: $(SGX_LE_PROXY_OBJS)
+	$(call if_changed,ld)
diff --git a/drivers/platform/x86/intel_sgx/le/enclave/Makefile b/drivers/platform/x86/intel_sgx/le/enclave/Makefile
new file mode 100644
index 000000000000..aef95d00223c
--- /dev/null
+++ b/drivers/platform/x86/intel_sgx/le/enclave/Makefile
@@ -0,0 +1,54 @@
+# SPDX-License-Identifier: (GPL-2.0 OR BSD-3-Clause)
+# Copyright(c) 2016-17 Intel Corporation.
+#
+# Authors:
+#
+# Jarkko Sakkinen <jarkko.sakkinen@linux.intel.com>
+
+KASAN_SANITIZE := n
+OBJECT_FILES_NON_STANDARD := y
+KCOV_INSTRUMENT := n
+KBUILD_CFLAGS := -Wall -Werror -static -nostdlib -nostartfiles -fPIC \
+		 -fno-stack-protector -mrdrnd  -I$(obj)/../include \
+		 -I$(srctree)/arch/x86/include
+KBUILD_AFLAGS := $(KBUILD_CFLAGS) -D__ASSEMBLY__ -DAESNI_INTEL_MINIMAL
+
+always := sgx_le.elf sgx_le.bin sgx_le.ss
+clean-files := sgx_le.elf sgx_le.bin sgx_le.ss
+
+#
+# sgx_le.ss
+#
+
+HOST_EXTRACFLAGS += -I$(srctree)/arch/x86/include
+HOSTLOADLIBES_sgxsign = -lcrypto
+hostprogs-y += sgxsign
+quiet_cmd_sgxsign = SGXSIGN $@
+      cmd_sgxsign = drivers/platform/x86/intel_sgx/le/enclave/sgxsign \
+		    $(INTEL_SGX_SIGNING_KEY_PATH) $< $@
+
+targets += sgx_le.ss
+$(obj)/sgx_le.ss: $(obj)/sgx_le.bin $(obj)/sgxsign FORCE
+	$(call if_changed,sgxsign)
+
+#
+# sgx_le.bin
+#
+
+targets += sgx_le.bin
+OBJCOPYFLAGS_sgx_le.bin := --remove-section=.got.plt -O binary
+$(obj)/sgx_le.bin: $(obj)/sgx_le.elf FORCE
+	$(call if_changed,objcopy)
+
+#
+# sgx_le.elf
+#
+
+sgx_le-y +=  main.o encl_bootstrap.o cmac_mode.o aesni-intel_asm.o string.o
+targets += $(sgx_le-y)
+SGX_LE_OBJS = $(addprefix $(obj)/,$(sgx_le-y))
+
+targets += sgx_le.elf
+LDFLAGS_sgx_le.elf := -T
+$(obj)/sgx_le.elf: $(obj)/sgx_le.lds $(SGX_LE_OBJS)
+	$(call if_changed,ld)
diff --git a/drivers/platform/x86/intel_sgx/le/enclave/aesni-intel_asm.S b/drivers/platform/x86/intel_sgx/le/enclave/aesni-intel_asm.S
new file mode 120000
index 000000000000..61971c8f7b3f
--- /dev/null
+++ b/drivers/platform/x86/intel_sgx/le/enclave/aesni-intel_asm.S
@@ -0,0 +1 @@
+../../../../../../arch/x86/crypto/aesni-intel_asm.S
\ No newline at end of file
diff --git a/drivers/platform/x86/intel_sgx/le/enclave/cmac_mode.c b/drivers/platform/x86/intel_sgx/le/enclave/cmac_mode.c
new file mode 100644
index 000000000000..f1f72b1679ee
--- /dev/null
+++ b/drivers/platform/x86/intel_sgx/le/enclave/cmac_mode.c
@@ -0,0 +1,209 @@
+// SPDX-License-Identifier: (GPL-2.0 OR BSD-3-Clause)
+// Copyright(c) 2016-17 Intel Corporation.
+//
+// Derived from TinyCrypt CMAC implementation.
+//
+// Authors:
+//
+// Jarkko Sakkinen <jarkko.sakkinen@linux.intel.com>
+
+#include <asm/sgx.h>
+#include "cmac_mode.h"
+
+/* max number of calls until change the key (2^48).*/
+const static uint64_t MAX_CALLS = ((uint64_t)1 << 48);
+
+/*
+ *  gf_wrap -- In our implementation, GF(2^128) is represented as a 16 byte
+ *  array with byte 0 the most significant and byte 15 the least significant.
+ *  High bit carry reduction is based on the primitive polynomial
+ *
+ *                     X^128 + X^7 + X^2 + X + 1,
+ *
+ *  which leads to the reduction formula X^128 = X^7 + X^2 + X + 1. Indeed,
+ *  since 0 = (X^128 + X^7 + X^2 + 1) mod (X^128 + X^7 + X^2 + X + 1) and since
+ *  addition of polynomials with coefficients in Z/Z(2) is just XOR, we can
+ *  add X^128 to both sides to get
+ *
+ *       X^128 = (X^7 + X^2 + X + 1) mod (X^128 + X^7 + X^2 + X + 1)
+ *
+ *  and the coefficients of the polynomial on the right hand side form the
+ *  string 1000 0111 = 0x87, which is the value of gf_wrap.
+ *
+ *  This gets used in the following way. Doubling in GF(2^128) is just a left
+ *  shift by 1 bit, except when the most significant bit is 1. In the latter
+ *  case, the relation X^128 = X^7 + X^2 + X + 1 says that the high order bit
+ *  that overflows beyond 128 bits can be replaced by addition of
+ *  X^7 + X^2 + X + 1 <--> 0x87 to the low order 128 bits. Since addition
+ *  in GF(2^128) is represented by XOR, we therefore only have to XOR 0x87
+ *  into the low order byte after a left shift when the starting high order
+ *  bit is 1.
+ */
+const unsigned char gf_wrap = 0x87;
+
+/*
+ *  assumes: out != NULL and points to a GF(2^n) value to receive the
+ *            doubled value;
+ *           in != NULL and points to a 16 byte GF(2^n) value
+ *            to double;
+ *           the in and out buffers do not overlap.
+ *  effects: doubles the GF(2^n) value pointed to by "in" and places
+ *           the result in the GF(2^n) value pointed to by "out."
+ */
+void gf_double(uint8_t *out, uint8_t *in)
+{
+	/* start with low order byte */
+	uint8_t *x = in + (AES_BLOCK_SIZE - 1);
+
+	/* if msb == 1, we need to add the gf_wrap value, otherwise add 0 */
+	uint8_t carry = (in[0] >> 7) ? gf_wrap : 0;
+
+	out += (AES_BLOCK_SIZE - 1);
+	for (;;) {
+		*out-- = (*x << 1) ^ carry;
+		if (x == in)
+			break;
+		carry = *x-- >> 7;
+	}
+}
+
+/**
+ * tc_cmac_setup - configures the CMAC state to use the given AES key
+ *
+ * @s: the state to set up
+ * @key: the key to use:w
+ * @ctx: AES context
+ */
+void tc_cmac_setup(struct tc_cmac_struct *s, const uint8_t *key,
+		   struct crypto_aes_ctx *ctx)
+{
+	/* put s into a known state */
+	tc_cmac_erase(s);
+	s->ctx = ctx;
+
+	/* configure the encryption key used by the underlying block cipher */
+	aesni_set_key(ctx, key, AES_KEYSIZE_128);
+
+	/* compute s->K1 and s->K2 from s->iv using s->keyid */
+	memset(s->iv, 0, AES_BLOCK_SIZE);
+	aesni_enc(ctx, s->iv, s->iv);
+
+	gf_double (s->K1, s->iv);
+	gf_double (s->K2, s->K1);
+
+	/* reset s->iv to 0 in case someone wants to compute now */
+	tc_cmac_init(s);
+}
+
+/**
+ * tc_cmac_erase - erases the CMAC state
+ *
+ * @s:	the state to erase
+ */
+void tc_cmac_erase(struct tc_cmac_struct *s)
+{
+	memset(s, 0, sizeof(*s));
+}
+
+/**
+ * tc_cmac_init - initializes a new CMAC computation
+ *
+ * @s:	the state to initialize
+ */
+void tc_cmac_init(struct tc_cmac_struct *s)
+{
+	/* CMAC starts with an all zero initialization vector */
+	memset(s->iv, 0, AES_BLOCK_SIZE);
+
+	/* and the leftover buffer is empty */
+	memset(s->leftover, 0, AES_BLOCK_SIZE);
+	s->leftover_offset = 0;
+
+	/* Set countdown to max number of calls allowed before re-keying: */
+	s->countdown = MAX_CALLS;
+}
+
+/**
+ * tc_cmac_update - incrementally computes CMAC over the next data segment
+ *
+ * s:		the CMAC state
+ * data:	the next data segment to MAC
+ * dlen:	the length of data in bytes
+ */
+void tc_cmac_update(struct tc_cmac_struct *s, const uint8_t *data, size_t dlen)
+{
+	uint32_t i;
+
+	s->countdown--;
+
+	if (s->leftover_offset > 0) {
+		/* last data added to s didn't end on a AES_BLOCK_SIZE byte
+		 * boundary
+		 */
+		size_t remaining_space = AES_BLOCK_SIZE - s->leftover_offset;
+
+		if (dlen < remaining_space) {
+			/* still not enough data to encrypt this time either */
+			memcpy(&s->leftover[s->leftover_offset], data,
+			       dlen);
+			s->leftover_offset += dlen;
+			return;
+		}
+		/* leftover block is now full; encrypt it first */
+		memcpy(&s->leftover[s->leftover_offset], data, remaining_space);
+		dlen -= remaining_space;
+		data += remaining_space;
+		s->leftover_offset = 0;
+
+		for (i = 0; i < AES_BLOCK_SIZE; ++i)
+			s->iv[i] ^= s->leftover[i];
+
+		aesni_enc(s->ctx, s->iv, s->iv);
+	}
+
+	/* CBC encrypt each (except the last) of the data blocks */
+	while (dlen > AES_BLOCK_SIZE) {
+		for (i = 0; i < AES_BLOCK_SIZE; ++i)
+			s->iv[i] ^= data[i];
+		aesni_enc(s->ctx, s->iv, s->iv);
+		data += AES_BLOCK_SIZE;
+		dlen  -= AES_BLOCK_SIZE;
+	}
+
+	if (dlen > 0) {
+		/* save leftover data for next time */
+		memcpy(s->leftover, data, dlen);
+		s->leftover_offset = dlen;
+	}
+}
+
+/**
+ * tc_cmac_final - generates the tag from the CMAC state
+ *
+ * @tag:	the CMAC tag
+ * @s:		CMAC state
+ */
+void tc_cmac_final(uint8_t *tag, struct tc_cmac_struct *s)
+{
+	uint8_t *k;
+	uint32_t i;
+
+	if (s->leftover_offset == AES_BLOCK_SIZE) {
+		/* the last message block is a full-sized block */
+		k = (uint8_t *) s->K1;
+	} else {
+		/* the final message block is not a full-sized  block */
+		size_t remaining = AES_BLOCK_SIZE - s->leftover_offset;
+
+		memset(&s->leftover[s->leftover_offset], 0, remaining);
+		s->leftover[s->leftover_offset] = TC_CMAC_PADDING;
+		k = (uint8_t *) s->K2;
+	}
+	for (i = 0; i < AES_BLOCK_SIZE; ++i)
+		s->iv[i] ^= s->leftover[i] ^ k[i];
+
+	aesni_enc(s->ctx, tag, s->iv);
+
+	/* erasing state: */
+	tc_cmac_erase(s);
+}
diff --git a/drivers/platform/x86/intel_sgx/le/enclave/cmac_mode.h b/drivers/platform/x86/intel_sgx/le/enclave/cmac_mode.h
new file mode 100644
index 000000000000..18c9223bbd4f
--- /dev/null
+++ b/drivers/platform/x86/intel_sgx/le/enclave/cmac_mode.h
@@ -0,0 +1,54 @@
+// SPDX-License-Identifier: (GPL-2.0 OR BSD-3-Clause)
+// Copyright(c) 2016-17 Intel Corporation.
+//
+// Derived from TinyCrypt CMAC implementation.
+//
+// Authors:
+//
+// Jarkko Sakkinen <jarkko.sakkinen@linux.intel.com>
+
+#ifndef CMAC_MODE_H
+#define CMAC_MODE_H
+
+#include <stddef.h>
+#include <crypto/aes.h>
+
+/* padding for last message block */
+#define TC_CMAC_PADDING 0x80
+
+/* struct tc_cmac_struct represents the state of a CMAC computation */
+struct tc_cmac_struct {
+	/* initialization vector */
+	uint8_t iv[AES_BLOCK_SIZE];
+	/* used if message length is a multiple of block_size bytes */
+	uint8_t K1[AES_BLOCK_SIZE];
+	/* used if message length isn't a multiple block_size bytes */
+	uint8_t K2[AES_BLOCK_SIZE];
+	/* where to put bytes that didn't fill a block */
+	uint8_t leftover[AES_BLOCK_SIZE];
+	/* identifies the encryption key */
+	uint32_t keyid;
+	/* next available leftover location */
+	uint32_t leftover_offset;
+	/* AES key schedule */
+	struct crypto_aes_ctx *ctx;
+	/* calls to tc_cmac_update left before re-key */
+	uint64_t countdown;
+};
+
+void tc_cmac_setup(struct tc_cmac_struct *s, const uint8_t *key,
+		   struct crypto_aes_ctx *ctx);
+
+void tc_cmac_erase(struct tc_cmac_struct *s);
+
+void tc_cmac_init(struct tc_cmac_struct *s);
+
+void tc_cmac_update(struct tc_cmac_struct *s, const uint8_t *data, size_t dlen);
+
+void tc_cmac_final(uint8_t *tag, struct tc_cmac_struct *s);
+
+asmlinkage int aesni_set_key(struct crypto_aes_ctx *ctx, const u8 *in_key,
+			     unsigned int key_len);
+asmlinkage void aesni_enc(struct crypto_aes_ctx *ctx, u8 *out, const u8 *in);
+
+#endif /* CMAC_MODE_H */
diff --git a/drivers/platform/x86/intel_sgx/le/enclave/encl_bootstrap.S b/drivers/platform/x86/intel_sgx/le/enclave/encl_bootstrap.S
new file mode 100644
index 000000000000..603c4e167761
--- /dev/null
+++ b/drivers/platform/x86/intel_sgx/le/enclave/encl_bootstrap.S
@@ -0,0 +1,116 @@
+// SPDX-License-Identifier: (GPL-2.0 OR BSD-3-Clause)
+// Copyright(c) 2016-17 Intel Corporation.
+//
+// Authors:
+//
+// Jarkko Sakkinen <jarkko.sakkinen@linux.intel.com>
+// Haim Cohen <haim.cohen@intel.com>
+// Sean Christopherson <sean.j.christopherson@intel.com>
+
+#include <sgx_asm.h>
+
+	.section ".tcs", "a"
+	.balign	4096
+
+	.fill	1, 8, 0			# STATE (set by CPU)
+	.fill	1, 8, 0			# FLAGS
+	.long	encl_ssa		# OSSA
+	.fill	1, 4, 0
+	.fill	1, 4, 0			# CSSA (set by CPU)
+	.fill	1, 4, 1			# NSSA
+	.long	encl_entry		# OENTRY
+	.fill	1, 4, 0
+	.fill	1, 8, 0			# AEP (set by EENTER and ERESUME)
+	.fill	1, 8, 0			# OFSBASE
+	.fill	1, 8, 0			# OGSBASE
+	.fill	1, 4, 0xFFFFFFFF 	# FSLIMIT
+	.fill	1, 4, 0xFFFFFFFF	# GSLIMIT
+	.fill	503, 8, 0		# Reserved
+
+	.text
+
+encl_entry:
+	# %rbx contains the base address for TCS, which is also the first
+	# address inside the enclave. By adding $le_stack_end to it, we get the
+	# absolute address for the stack.
+	lea	(encl_stack)(%rbx), %rax
+	xchg	%rsp, %rax
+	push	%rax
+
+	push	%rcx # push the address after EENTER
+	push	%rbx # push the enclave base address
+
+	call	encl_body
+
+	pop	%rbx # pop the enclave base address
+
+	# Restore XSAVE registers to a synthetic state.
+	mov     $0xFFFFFFFF, %rax
+	mov     $0xFFFFFFFF, %rdx
+	lea	(xsave_area)(%rbx), %rdi
+	fxrstor	(%rdi)
+
+	# Clear GPRs
+	xor     %rcx, %rcx
+	xor     %rdx, %rdx
+	xor     %rdi, %rdi
+	xor     %rsi, %rsi
+	xor     %r8, %r8
+	xor     %r9, %r9
+	xor     %r10, %r10
+	xor     %r11, %r11
+	xor     %r12, %r12
+	xor     %r13, %r13
+	xor     %r14, %r14
+	xor     %r15, %r15
+
+	# Reset status flags
+	add     %rdx, %rdx # OF = SF = AF = CF = 0; ZF = PF = 1
+
+	pop	%rbx # pop the address after EENTER
+
+	# Restore the caller stack.
+	pop	%rax
+	mov	%rax, %rsp
+
+	# EEXIT
+	mov	$4, %rax
+	enclu
+
+	.global sgx_ereport
+sgx_ereport:
+	push	%rbx
+	xor	%rax, %rax /* EREPORT */
+	mov	%rdi, %rbx  /* TARGETINFO */
+	mov	%rsi, %rcx /* REPORTDATA */
+	ENCLU
+	pop	%rbx
+	ret
+
+	.global sgx_egetkey
+sgx_egetkey:
+	push	%rbx
+	mov	$0x01, %rax /* EGETKEY */
+	mov	%rdi, %rbx /* KEYREQUEST */
+	mov	%rsi, %rcx  /* KEY */
+	ENCLU
+	pop	%rbx
+	ret
+
+	.section ".data", "aw"
+
+encl_ssa:
+	.space 4096
+
+xsave_area:
+	.fill	1, 4, 0x037F		# FCW
+	.fill	5, 4, 0
+	.fill	1, 4, 0x1F80		# MXCSR
+	.fill	1, 4, 0xFFFF		# MXCSR_MASK
+	.fill	123, 4, 0
+	.fill	1, 4, 0x80000000	# XCOMP_BV[63] = 1, compaction mode
+	.fill	12, 4, 0
+
+	.balign 4096
+	.space 8192
+encl_stack:
diff --git a/drivers/platform/x86/intel_sgx/le/enclave/main.c b/drivers/platform/x86/intel_sgx/le/enclave/main.c
new file mode 100644
index 000000000000..f03bdad4d3b6
--- /dev/null
+++ b/drivers/platform/x86/intel_sgx/le/enclave/main.c
@@ -0,0 +1,146 @@
+// SPDX-License-Identifier: (GPL-2.0 OR BSD-3-Clause)
+// Copyright(c) 2016-17 Intel Corporation.
+//
+// Derived from TinyCrypt CMAC implementation.
+//
+// Authors:
+//
+// Jarkko Sakkinen <jarkko.sakkinen@linux.intel.com>
+
+#include <asm/sgx.h>
+#include <asm/sgx_arch.h>
+#include <linux/types.h>
+#include <uapi/asm/sgx.h>
+#include "cmac_mode.h"
+#include "main.h"
+
+static bool rdrand_uint32(uint32_t *value)
+{
+	int i;
+
+	for (i = 0; i < RAND_NR_TRIES; i++) {
+		if (__builtin_ia32_rdrand32_step((unsigned int *)value))
+			return true;
+	}
+
+	return false;
+}
+
+static bool sign_einittoken(struct sgx_einittoken *einittoken)
+{
+	struct sgx_keyrequest keyrequest __aligned(512);
+	uint8_t launch_key[16] __aligned(16);
+	struct tc_cmac_struct cmac_state;
+	struct crypto_aes_ctx ctx;
+	uint32_t *keyid_ptr;
+	int i;
+
+	memset(&ctx, 0, sizeof(ctx));
+
+	/* Despite its misleading name, the only purpose of the keyid field is
+	 * to add entropy to the token so that every token will have an unique
+	 * CMAC.
+	 */
+	keyid_ptr = (uint32_t *)einittoken->keyid;
+
+	for (i = 0; i < sizeof(einittoken->keyid) / 4; i++)
+		if (!rdrand_uint32(&keyid_ptr[i]))
+			return false;
+
+	memset(&keyrequest, 0, sizeof(keyrequest));
+	keyrequest.keyname = 0; /* LICENSE_KEY */
+	memcpy(&keyrequest.keyid, &einittoken->keyid, sizeof(keyrequest.keyid));
+	memcpy(&keyrequest.cpusvn, &(einittoken->cpusvnle),
+	       sizeof(keyrequest.cpusvn));
+	memcpy(&keyrequest.isvsvn, &(einittoken->isvsvnle),
+	       sizeof(keyrequest.isvsvn));
+
+	keyrequest.attributemask = ~SGX_ATTR_MODE64BIT;
+	keyrequest.xfrmmask = 0;
+	keyrequest.miscmask = 0xFFFFFFFF;
+
+	einittoken->maskedmiscselectle &= keyrequest.miscmask;
+	einittoken->maskedattributesle &= keyrequest.attributemask;
+	einittoken->maskedxfrmle &= keyrequest.xfrmmask;
+
+	if (sgx_egetkey(&keyrequest, launch_key))
+		return false;
+
+	tc_cmac_setup(&cmac_state, launch_key, &ctx);
+	tc_cmac_init(&cmac_state);
+	tc_cmac_update(&cmac_state, (const uint8_t *)&einittoken->payload,
+		       sizeof(einittoken->payload));
+	tc_cmac_final(einittoken->mac, &cmac_state);
+
+	memset(launch_key, 0, sizeof(launch_key));
+
+	return true;
+}
+
+static bool create_einittoken(uint8_t *mrenclave,
+			      uint8_t *mrsigner,
+			      uint64_t attributes,
+			      uint64_t xfrm,
+			      struct sgx_einittoken *einittoken)
+{
+
+	struct sgx_targetinfo tginfo __aligned(512);
+	struct sgx_report report __aligned(512);
+	uint8_t reportdata[64] __aligned(128);
+
+	if (attributes & SGX_ATTR_RESERVED_MASK)
+		return false;
+
+	memset(&tginfo, 0, sizeof(tginfo));
+	memset(reportdata, 0, sizeof(reportdata));
+	memset(&report, 0, sizeof(report));
+
+	if (sgx_ereport(&tginfo, reportdata, &report))
+		return false;
+
+	memset(einittoken, 0, sizeof(*einittoken));
+
+	einittoken->payload.valid = 1;
+
+	memcpy(einittoken->payload.mrenclave, mrenclave, 32);
+	memcpy(einittoken->payload.mrsigner, mrsigner, 32);
+	einittoken->payload.attributes = attributes;
+	einittoken->payload.xfrm = xfrm;
+
+	memcpy(&einittoken->cpusvnle, &report.cpusvn,
+		   sizeof(report.cpusvn));
+	einittoken->isvsvnle = report.isvsvn;
+	einittoken->isvprodidle = report.isvprodid;
+
+	einittoken->maskedattributesle = report.attributes;
+	einittoken->maskedxfrmle = report.xfrm;
+	einittoken->maskedmiscselectle = report.miscselect;
+
+	if (!sign_einittoken(einittoken))
+		return false;
+
+	return true;
+}
+
+void encl_body(struct sgx_launch_request *req, struct sgx_einittoken *token)
+{
+	struct sgx_einittoken tmp;
+	uint8_t mrenclave[32];
+	uint8_t mrsigner[32];
+	uint64_t attributes;
+	uint64_t xfrm;
+
+	if (!req)
+		return;
+
+	memcpy(mrenclave, req->mrenclave, sizeof(mrenclave));
+	memcpy(mrsigner, req->mrsigner, sizeof(mrsigner));
+	memcpy(&attributes, &req->attributes, sizeof(uint64_t));
+	memcpy(&xfrm, &req->xfrm, sizeof(uint64_t));
+	memset(&tmp, 0, sizeof(tmp));
+
+	if (!create_einittoken(mrenclave, mrsigner, attributes, xfrm, &tmp))
+		return;
+
+	memcpy(token, &tmp, sizeof(*token));
+}
diff --git a/drivers/platform/x86/intel_sgx/le/enclave/main.h b/drivers/platform/x86/intel_sgx/le/enclave/main.h
new file mode 100644
index 000000000000..2cec72242d0e
--- /dev/null
+++ b/drivers/platform/x86/intel_sgx/le/enclave/main.h
@@ -0,0 +1,19 @@
+// SPDX-License-Identifier: (GPL-2.0 OR BSD-3-Clause)
+// Copyright(c) 2016-17 Intel Corporation.
+//
+// Derived from TinyCrypt CMAC implementation.
+//
+// Authors:
+//
+// Jarkko Sakkinen <jarkko.sakkinen@linux.intel.com>
+
+#ifndef MAIN_H
+#define MAIN_H
+
+#define RAND_NR_TRIES 10
+
+int sgx_ereport(const void *target_info, const void *report_data,
+		void *report);
+int sgx_egetkey(void *key_request, void *key);
+
+#endif /* MAIN_H */
diff --git a/drivers/platform/x86/intel_sgx/le/enclave/sgx_le.lds b/drivers/platform/x86/intel_sgx/le/enclave/sgx_le.lds
new file mode 100644
index 000000000000..4997628813b8
--- /dev/null
+++ b/drivers/platform/x86/intel_sgx/le/enclave/sgx_le.lds
@@ -0,0 +1,33 @@
+OUTPUT_FORMAT(elf64-x86-64)
+
+SECTIONS
+{
+	. = 0;
+	.tcs : {
+		*(.tcs*)
+	}
+
+	. = ALIGN(4096);
+	.text : {
+		*(.text*)
+		*(.rodata*)
+	}
+
+	. = ALIGN(4096);
+	.data : {
+		*(.data*)
+	}
+
+	/DISCARD/ : {
+		*(.data*)
+		*(.comment*)
+		*(.note*)
+		*(.debug*)
+		*(.eh_frame*)
+	}
+}
+
+ASSERT(!DEFINED(.altinstructions), "ALTERNATIVES are not supported in the SGX LE")
+ASSERT(!DEFINED(.altinstr_replacement), "ALTERNATIVES are not supported in the SGX LE")
+ASSERT(!DEFINED(.discard.retpoline_safe), "RETPOLINE ALTERNATIVES are not supported in the SGX LE")
+ASSERT(!DEFINED(.discard.nospec), "RETPOLINE ALTERNATIVES are not supported in the SGX LE")
diff --git a/drivers/platform/x86/intel_sgx/le/enclave/sgxsign.c b/drivers/platform/x86/intel_sgx/le/enclave/sgxsign.c
new file mode 100644
index 000000000000..abbb59195fd3
--- /dev/null
+++ b/drivers/platform/x86/intel_sgx/le/enclave/sgxsign.c
@@ -0,0 +1,551 @@
+/*
+ * This file is provided under a dual BSD/GPLv2 license.  When using or
+ * redistributing this file, you may do so under either license.
+ *
+ * GPL LICENSE SUMMARY
+ *
+ * Copyright(c) 2017 Intel Corporation.
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of version 2 of the GNU General Public License as
+ * published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful, but
+ * WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+ * General Public License for more details.
+ *
+ * Contact Information:
+ * Jarkko Sakkinen <jarkko.sakkinen@linux.intel.com>
+ * Intel Finland Oy - BIC 0357606-4 - Westendinkatu 7, 02160 Espoo
+ *
+ * BSD LICENSE
+ *
+ * Copyright(c) 2017 Intel Corporation.
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions
+ * are met:
+ *
+ *   * Redistributions of source code must retain the above copyright
+ *     notice, this list of conditions and the following disclaimer.
+ *   * Redistributions in binary form must reproduce the above copyright
+ *     notice, this list of conditions and the following disclaimer in
+ *     the documentation and/or other materials provided with the
+ *     distribution.
+ *   * Neither the name of Intel Corporation nor the names of its
+ *     contributors may be used to endorse or promote products derived
+ *     from this software without specific prior written permission.
+ *
+ * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
+ * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
+ * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
+ * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
+ * OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+ * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+ * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+ * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
+ * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+ * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+ * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+ *
+ * Authors:
+ *
+ * Jarkko Sakkinen <jarkko.sakkinen@linux.intel.com>
+ */
+
+#define _GNU_SOURCE
+#include <getopt.h>
+#include <stdbool.h>
+#include <stdint.h>
+#include <stdio.h>
+#include <stdlib.h>
+#include <string.h>
+#include <sys/stat.h>
+#include <sys/types.h>
+#include <unistd.h>
+#include <asm/sgx_arch.h>
+#include <openssl/err.h>
+#include <openssl/pem.h>
+
+static const char *sign_key_pass;
+
+static bool check_crypto_errors(void)
+{
+	int err;
+	bool had_errors = false;
+	const char *filename;
+	int line;
+	char str[256];
+
+	for ( ; ; ) {
+		if (ERR_peek_error() == 0)
+			break;
+
+		had_errors = true;
+		err = ERR_get_error_line(&filename, &line);
+		ERR_error_string_n(err, str, sizeof(str));
+		fprintf(stderr, "crypto: %s: %s:%d\n", str, filename, line);
+	}
+
+	return had_errors;
+}
+
+static void exit_usage(const char *program)
+{
+	fprintf(stderr,
+		"Usage: %s/sign-le <key> <enclave> <sigstruct>\n", program);
+	exit(1);
+}
+
+static int pem_passwd_cb(char *buf, int size, int rwflag, void *u)
+{
+	if (!sign_key_pass)
+		return -1;
+
+	strncpy(buf, sign_key_pass, size);
+	/* no retry */
+	sign_key_pass = NULL;
+
+	return strlen(buf) >= size ? size - 1 : strlen(buf);
+}
+
+static inline const BIGNUM *get_modulus(RSA *key)
+{
+#if OPENSSL_VERSION_NUMBER < 0x10100000L
+	return key->n;
+#else
+	const BIGNUM *n;
+
+	RSA_get0_key(key, &n, NULL, NULL);
+	return n;
+#endif
+}
+
+static RSA *load_sign_key(const char *path)
+{
+	FILE *f;
+	RSA *key;
+
+	f = fopen(path, "rb");
+	if (!f) {
+		fprintf(stderr, "Unable to open %s\n", path);
+		return NULL;
+	}
+	key = RSA_new();
+	if (!PEM_read_RSAPrivateKey(f, &key, pem_passwd_cb, NULL))
+		return NULL;
+	fclose(f);
+
+	if (BN_num_bytes(get_modulus(key)) != SGX_MODULUS_SIZE) {
+		fprintf(stderr, "Invalid key size %d\n",
+			BN_num_bytes(get_modulus(key)));
+		RSA_free(key);
+		return NULL;
+	}
+
+	return key;
+}
+
+static void reverse_bytes(void *data, int length)
+{
+	int i = 0;
+	int j = length - 1;
+	uint8_t temp;
+	uint8_t *ptr = data;
+
+	while (i < j) {
+		temp = ptr[i];
+		ptr[i] = ptr[j];
+		ptr[j] = temp;
+		i++;
+		j--;
+	}
+}
+
+enum mrtags {
+	MRECREATE = 0x0045544145524345,
+	MREADD = 0x0000000044444145,
+	MREEXTEND = 0x00444E4554584545,
+};
+
+static bool mrenclave_update(EVP_MD_CTX *ctx, const void *data)
+{
+	if (!EVP_DigestUpdate(ctx, data, 64)) {
+		fprintf(stderr, "digest update failed\n");
+		return false;
+	}
+
+	return true;
+}
+
+static bool mrenclave_commit(EVP_MD_CTX *ctx, uint8_t *mrenclave)
+{
+	unsigned int size;
+
+	if (!EVP_DigestFinal_ex(ctx, (unsigned char *)mrenclave, &size)) {
+		fprintf(stderr, "digest commit failed\n");
+		return false;
+	}
+
+	if (size != 32) {
+		fprintf(stderr, "invalid digest size = %u\n", size);
+		return false;
+	}
+
+	return true;
+}
+
+struct mrecreate {
+	uint64_t tag;
+	uint32_t ssaframesize;
+	uint64_t size;
+	uint8_t reserved[44];
+} __attribute__((__packed__));
+
+
+static bool mrenclave_ecreate(EVP_MD_CTX *ctx, uint64_t blob_size)
+{
+	struct mrecreate mrecreate;
+	uint64_t encl_size;
+
+	for (encl_size = 0x1000; encl_size < blob_size; )
+		encl_size <<= 1;
+
+	memset(&mrecreate, 0, sizeof(mrecreate));
+	mrecreate.tag = MRECREATE;
+	mrecreate.ssaframesize = 1;
+	mrecreate.size = encl_size;
+
+	if (!EVP_DigestInit_ex(ctx, EVP_sha256(), NULL))
+		return false;
+
+	return mrenclave_update(ctx, &mrecreate);
+}
+
+struct mreadd {
+	uint64_t tag;
+	uint64_t offset;
+	uint64_t flags; /* SECINFO flags */
+	uint8_t reserved[40];
+} __attribute__((__packed__));
+
+static bool mrenclave_eadd(EVP_MD_CTX *ctx, uint64_t offset, uint64_t flags)
+{
+	struct mreadd mreadd;
+
+	memset(&mreadd, 0, sizeof(mreadd));
+	mreadd.tag = MREADD;
+	mreadd.offset = offset;
+	mreadd.flags = flags;
+
+	return mrenclave_update(ctx, &mreadd);
+}
+
+struct mreextend {
+	uint64_t tag;
+	uint64_t offset;
+	uint8_t reserved[48];
+} __attribute__((__packed__));
+
+static bool mrenclave_eextend(EVP_MD_CTX *ctx, uint64_t offset, uint8_t *data)
+{
+	struct mreextend mreextend;
+	int i;
+
+	for (i = 0; i < 0x1000; i += 0x100) {
+		memset(&mreextend, 0, sizeof(mreextend));
+		mreextend.tag = MREEXTEND;
+		mreextend.offset = offset + i;
+
+		if (!mrenclave_update(ctx, &mreextend))
+			return false;
+
+		if (!mrenclave_update(ctx, &data[i + 0x00]))
+			return false;
+
+		if (!mrenclave_update(ctx, &data[i + 0x40]))
+			return false;
+
+		if (!mrenclave_update(ctx, &data[i + 0x80]))
+			return false;
+
+		if (!mrenclave_update(ctx, &data[i + 0xC0]))
+			return false;
+	}
+
+	return true;
+}
+
+/**
+ * measure_encl - measure enclave
+ * @path: path to the enclave
+ * @mrenclave: measurement
+ *
+ * Calculates MRENCLAVE. Assumes that the very first page is a TCS page and
+ * following pages are regular pages. Does not measure the contents of the
+ * enclave as the signing tool is used at the moment only for the launch
+ * enclave, which is pass-through (everything gets a token).
+ */
+static bool measure_encl(const char *path, uint8_t *mrenclave)
+{
+	FILE *file;
+	struct stat sb;
+	EVP_MD_CTX *ctx;
+	uint64_t flags;
+	uint64_t offset;
+	uint8_t data[0x1000];
+	int rc;
+
+	ctx = EVP_MD_CTX_create();
+	if (!ctx)
+		return false;
+
+	file = fopen(path, "rb");
+	if (!file) {
+		perror("fopen");
+		EVP_MD_CTX_destroy(ctx);
+		return false;
+	}
+
+	rc = stat(path, &sb);
+	if (rc) {
+		perror("stat");
+		goto out;
+	}
+
+	if (!sb.st_size || sb.st_size & 0xfff) {
+		fprintf(stderr, "Invalid blob size %lu\n", sb.st_size);
+		goto out;
+	}
+
+	if (!mrenclave_ecreate(ctx, sb.st_size))
+		goto out;
+
+	for (offset = 0; offset < sb.st_size; offset += 0x1000) {
+		if (!offset)
+			flags = SGX_SECINFO_TCS;
+		else
+			flags = SGX_SECINFO_REG | SGX_SECINFO_R |
+				SGX_SECINFO_W | SGX_SECINFO_X;
+
+		if (!mrenclave_eadd(ctx, offset, flags))
+			goto out;
+
+		rc = fread(data, 1, 0x1000, file);
+		if (!rc)
+			break;
+		if (rc < 0x1000)
+			goto out;
+
+		if (!mrenclave_eextend(ctx, offset, data))
+			goto out;
+	}
+
+	if (!mrenclave_commit(ctx, mrenclave))
+		goto out;
+
+	fclose(file);
+	EVP_MD_CTX_destroy(ctx);
+	return true;
+out:
+	fclose(file);
+	EVP_MD_CTX_destroy(ctx);
+	return false;
+}
+
+/**
+ * sign_encl - sign enclave
+ * @sigstruct: pointer to SIGSTRUCT
+ * @key: 3072-bit RSA key
+ * @signature: byte array for the signature
+ *
+ * Calculates EMSA-PKCSv1.5 signature for the given SIGSTRUCT. The result is
+ * stored in big-endian format so that it can be further passed to OpenSSL
+ * libcrypto functions.
+ */
+static bool sign_encl(const struct sgx_sigstruct *sigstruct, RSA *key,
+		      uint8_t *signature)
+{
+	struct sgx_sigstruct_payload payload;
+	unsigned int siglen;
+	uint8_t digest[SHA256_DIGEST_LENGTH];
+	bool ret;
+
+	memcpy(&payload.header, &sigstruct->header, sizeof(sigstruct->header));
+	memcpy(&payload.body, &sigstruct->body, sizeof(sigstruct->body));
+
+	SHA256((unsigned char *)&payload, sizeof(payload), digest);
+
+	ret = RSA_sign(NID_sha256, digest, SHA256_DIGEST_LENGTH, signature,
+		       &siglen, key);
+
+	return ret;
+}
+
+struct q1q2_ctx {
+	BN_CTX *bn_ctx;
+	BIGNUM *m;
+	BIGNUM *s;
+	BIGNUM *q1;
+	BIGNUM *qr;
+	BIGNUM *q2;
+};
+
+static void free_q1q2_ctx(struct q1q2_ctx *ctx)
+{
+	BN_CTX_free(ctx->bn_ctx);
+	BN_free(ctx->m);
+	BN_free(ctx->s);
+	BN_free(ctx->q1);
+	BN_free(ctx->qr);
+	BN_free(ctx->q2);
+}
+
+static bool alloc_q1q2_ctx(const uint8_t *s, const uint8_t *m,
+			   struct q1q2_ctx *ctx)
+{
+	ctx->bn_ctx = BN_CTX_new();
+	ctx->s = BN_bin2bn(s, SGX_MODULUS_SIZE, NULL);
+	ctx->m = BN_bin2bn(m, SGX_MODULUS_SIZE, NULL);
+	ctx->q1 = BN_new();
+	ctx->qr = BN_new();
+	ctx->q2 = BN_new();
+
+	if (!ctx->bn_ctx || !ctx->s || !ctx->m || !ctx->q1 || !ctx->qr ||
+	    !ctx->q2) {
+		free_q1q2_ctx(ctx);
+		return false;
+	}
+
+	return true;
+}
+
+static bool calc_q1q2(const uint8_t *s, const uint8_t *m, uint8_t *q1,
+		      uint8_t *q2)
+{
+	struct q1q2_ctx ctx;
+
+	if (!alloc_q1q2_ctx(s, m, &ctx)) {
+		fprintf(stderr, "Not enough memory for Q1Q2 calculation\n");
+		return false;
+	}
+
+	if (!BN_mul(ctx.q1, ctx.s, ctx.s, ctx.bn_ctx))
+		goto out;
+
+	if (!BN_div(ctx.q1, ctx.qr, ctx.q1, ctx.m, ctx.bn_ctx))
+		goto out;
+
+	if (BN_num_bytes(ctx.q1) > SGX_MODULUS_SIZE) {
+		fprintf(stderr, "Too large Q1 %d bytes\n",
+			BN_num_bytes(ctx.q1));
+		goto out;
+	}
+
+	if (!BN_mul(ctx.q2, ctx.s, ctx.qr, ctx.bn_ctx))
+		goto out;
+
+	if (!BN_div(ctx.q2, NULL, ctx.q2, ctx.m, ctx.bn_ctx))
+		goto out;
+
+	if (BN_num_bytes(ctx.q2) > SGX_MODULUS_SIZE) {
+		fprintf(stderr, "Too large Q2 %d bytes\n",
+			BN_num_bytes(ctx.q2));
+		goto out;
+	}
+
+	BN_bn2bin(ctx.q1, q1);
+	BN_bn2bin(ctx.q2, q2);
+
+	free_q1q2_ctx(&ctx);
+	return true;
+out:
+	free_q1q2_ctx(&ctx);
+	return false;
+}
+
+static bool save_sigstruct(const struct sgx_sigstruct *sigstruct,
+			   const char *path)
+{
+	FILE *f = fopen(path, "wb");
+
+	if (!f) {
+		fprintf(stderr, "Unable to open %s\n", path);
+		return false;
+	}
+
+	fwrite(sigstruct, sizeof(*sigstruct), 1, f);
+	fclose(f);
+	return true;
+}
+
+int main(int argc, char **argv)
+{
+	uint64_t header1[2] = {0x000000E100000006, 0x0000000000010000};
+	uint64_t header2[2] = {0x0000006000000101, 0x0000000100000060};
+	struct sgx_sigstruct ss;
+	const char *program;
+	int opt;
+	RSA *sign_key;
+
+	memset(&ss, 0, sizeof(ss));
+	ss.header.header1[0] = header1[0];
+	ss.header.header1[1] = header1[1];
+	ss.header.header2[0] = header2[0];
+	ss.header.header2[1] = header2[1];
+	ss.exponent = 3;
+	ss.body.attributes = SGX_ATTR_MODE64BIT | SGX_ATTR_EINITTOKENKEY;
+	ss.body.xfrm = 3,
+
+	sign_key_pass = getenv("KBUILD_SGX_SIGN_PIN");
+	program = argv[0];
+
+	do {
+		opt = getopt(argc, argv, "");
+		switch (opt) {
+		case -1:
+			break;
+		default:
+			exit_usage(program);
+		}
+	} while (opt != -1);
+
+	argc -= optind;
+	argv += optind;
+
+	if (argc < 3)
+		exit_usage(program);
+
+	/* sanity check only */
+	if (check_crypto_errors())
+		exit(1);
+
+	sign_key = load_sign_key(argv[0]);
+	if (!sign_key)
+		goto out;
+
+	BN_bn2bin(get_modulus(sign_key), ss.modulus);
+
+	if (!measure_encl(argv[1], ss.body.mrenclave))
+		goto out;
+
+	if (!sign_encl(&ss, sign_key, ss.signature))
+		goto out;
+
+	if (!calc_q1q2(ss.signature, ss.modulus, ss.q1, ss.q2))
+		goto out;
+
+	/* convert to little endian */
+	reverse_bytes(ss.signature, SGX_MODULUS_SIZE);
+	reverse_bytes(ss.modulus, SGX_MODULUS_SIZE);
+	reverse_bytes(ss.q1, SGX_MODULUS_SIZE);
+	reverse_bytes(ss.q2, SGX_MODULUS_SIZE);
+
+	if (!save_sigstruct(&ss, argv[2]))
+		goto out;
+	exit(0);
+out:
+	check_crypto_errors();
+	exit(1);
+}
diff --git a/drivers/platform/x86/intel_sgx/le/enclave/string.c b/drivers/platform/x86/intel_sgx/le/enclave/string.c
new file mode 120000
index 000000000000..5ea3579f0ec6
--- /dev/null
+++ b/drivers/platform/x86/intel_sgx/le/enclave/string.c
@@ -0,0 +1 @@
+../string.c
\ No newline at end of file
diff --git a/drivers/platform/x86/intel_sgx/le/entry.S b/drivers/platform/x86/intel_sgx/le/entry.S
new file mode 100644
index 000000000000..5edab50815a2
--- /dev/null
+++ b/drivers/platform/x86/intel_sgx/le/entry.S
@@ -0,0 +1,70 @@
+// SPDX-License-Identifier: (GPL-2.0 OR BSD-3-Clause)
+// Copyright(c) 2016-17 Intel Corporation.
+//
+// Authors:
+//
+// Jarkko Sakkinen <jarkko.sakkinen@linux.intel.com>
+
+#include <asm/sgx_le.h>
+#include <sgx_asm.h>
+
+	.text
+
+	.global sgx_get_token
+sgx_get_token:
+	push	%rbx
+	mov	$0x02, %rax
+	mov	%rsi, %rbx
+	mov	%rdx, %rsi
+	mov	$sgx_async_exit, %rcx
+sgx_async_exit:
+	ENCLU
+	pop	%rbx
+	ret
+
+	.global sgx_sys_read
+sgx_sys_read:
+	mov	$0, %rax
+	mov	%rsi, %rdx /* buf */
+	mov	%rdi, %rsi /* count */
+	mov	$SGX_LE_PIPE_FD, %rdi
+	syscall
+	ret
+
+	.global sgx_sys_write
+sgx_sys_write:
+	mov	$1, %rax
+	mov	%rsi, %rdx /* buf */
+	mov	%rdi, %rsi /* count */
+	mov	$SGX_LE_PIPE_FD, %rdi
+	syscall
+	ret
+
+	.global sgx_sys_close
+sgx_sys_close:
+	mov	$3, %rax
+	syscall
+	ret
+
+	.global sgx_sys_mmap
+sgx_sys_mmap:
+	mov	$9, %rax
+	mov	%rdi, %r8 /* fd */
+	xor	%rdi, %rdi /* any address */
+
+	mov	$0x07, %rdx /* rwx */
+	mov	$0x01, %r10 /* shared */
+	mov	$0x00, %r9 /* offset */
+	syscall
+	ret
+
+	.global sgx_sys_ioctl
+sgx_sys_ioctl:
+	mov	$16, %rax
+	syscall
+	ret
+
+	.global sgx_sys_exit
+sgx_sys_exit:
+	mov	$60, %rax
+	syscall
diff --git a/drivers/platform/x86/intel_sgx/le/include/sgx_asm.h b/drivers/platform/x86/intel_sgx/le/include/sgx_asm.h
new file mode 100644
index 000000000000..9d32884846fd
--- /dev/null
+++ b/drivers/platform/x86/intel_sgx/le/include/sgx_asm.h
@@ -0,0 +1,15 @@
+// SPDX-License-Identifier: (GPL-2.0 OR BSD-3-Clause)
+// Copyright(c) 2016-17 Intel Corporation.
+//
+// Authors:
+//
+// Jarkko Sakkinen <jarkko.sakkinen@linux.intel.com>
+
+#ifndef SGX_ASM_H
+#define SGX_ASM_H
+
+.macro ENCLU
+.byte 0x0f, 0x01, 0xd7
+.endm
+
+#endif /* SGX_ASM_H */
diff --git a/drivers/platform/x86/intel_sgx/le/main.c b/drivers/platform/x86/intel_sgx/le/main.c
new file mode 100644
index 000000000000..2e2b544b5305
--- /dev/null
+++ b/drivers/platform/x86/intel_sgx/le/main.c
@@ -0,0 +1,140 @@
+// SPDX-License-Identifier: (GPL-2.0 OR BSD-3-Clause)
+// Copyright(c) 2016-17 Intel Corporation.
+//
+// Authors:
+//
+// Jarkko Sakkinen <jarkko.sakkinen@linux.intel.com>
+
+#include <asm/sgx.h>
+#include <asm/sgx_arch.h>
+#include <asm/sgx_le.h>
+#include <linux/string.h>
+#include <linux/types.h>
+#include <uapi/asm/sgx.h>
+#include "main.h"
+
+static void *start_launch_enclave(void)
+{
+	struct sgx_enclave_create create_ioc;
+	struct sgx_enclave_add_page add_ioc;
+	struct sgx_enclave_init init_ioc;
+	struct sgx_secs secs;
+	struct sgx_secinfo secinfo;
+	unsigned long blob_base;
+	unsigned long blob_size;
+	unsigned long offset;
+	int rc;
+
+	memset(&secs, 0, sizeof(secs));
+	memset(&secinfo, 0, sizeof(secinfo));
+
+	secs.ssaframesize = 1;
+	secs.attributes = SGX_ATTR_MODE64BIT | SGX_ATTR_EINITTOKENKEY;
+	secs.xfrm = 3;
+
+	blob_base = (unsigned long)&sgx_le_blob;
+	blob_size = (unsigned long)&sgx_le_blob_end - blob_base;
+
+	for (secs.size = 4096; secs.size < blob_size; )
+		secs.size <<= 1;
+
+	secs.base = (unsigned long)sgx_sys_mmap(SGX_LE_DEV_FD, secs.size);
+	if (secs.base == (unsigned long)MAP_FAILED)
+		goto out;
+
+	create_ioc.src = (unsigned long)&secs;
+	rc = sgx_sys_ioctl(SGX_LE_DEV_FD, SGX_IOC_ENCLAVE_CREATE, &create_ioc);
+	if (rc)
+		goto out;
+
+	add_ioc.secinfo = (unsigned long)&secinfo;
+	add_ioc.mrmask = 0xFFFF;
+
+	for (offset = 0; offset < blob_size; offset += 0x1000) {
+		if (!offset)
+			secinfo.flags = SGX_SECINFO_TCS;
+		else
+			secinfo.flags = SGX_SECINFO_REG | SGX_SECINFO_R |
+					SGX_SECINFO_W | SGX_SECINFO_X;
+
+		add_ioc.addr = secs.base + offset;
+		add_ioc.src = blob_base + offset;
+
+		rc = sgx_sys_ioctl(SGX_LE_DEV_FD, SGX_IOC_ENCLAVE_ADD_PAGE,
+				   &add_ioc);
+		if (rc)
+			goto out;
+	}
+
+	init_ioc.addr = secs.base;
+	init_ioc.sigstruct = (uint64_t)&sgx_le_ss;
+	rc = sgx_sys_ioctl(SGX_LE_DEV_FD, SGX_IOC_ENCLAVE_INIT, &init_ioc);
+	if (rc)
+		goto out;
+
+	return (void *)secs.base;
+out:
+	return NULL;
+}
+
+static int read_input(void *data, unsigned int len)
+{
+	uint8_t *ptr = (uint8_t *)data;
+	long i;
+	long ret;
+
+	for (i = 0; i < len; ) {
+		ret = sgx_sys_read(&ptr[i], len - i);
+		if (ret < 0)
+			return ret;
+
+		i += ret;
+	}
+
+	return 0;
+}
+
+static int write_token(const struct sgx_einittoken *token)
+{
+	const uint8_t *ptr = (const uint8_t *)token;
+	long i;
+	long ret;
+
+	for (i = 0; i < sizeof(*token); ) {
+		ret = sgx_sys_write(&ptr[i], sizeof(*token) - i);
+		if (ret < 0)
+			return ret;
+
+		i += ret;
+	}
+
+	return 0;
+}
+
+void _start(void)
+{
+	struct sgx_launch_request req;
+	struct sgx_einittoken token;
+	void *entry;
+
+	sgx_sys_close(SGX_LE_EXE_FD);
+	entry = start_launch_enclave();
+	sgx_sys_close(SGX_LE_DEV_FD);
+	if (!entry)
+		sgx_sys_exit(1);
+
+	for ( ; ; ) {
+		memset(&req, 0, sizeof(req));
+		memset(&token, 0, sizeof(token));
+
+		if (read_input(&req, sizeof(req)))
+			sgx_sys_exit(1);
+
+		sgx_get_token(&req, entry, &token);
+
+		if (write_token(&token))
+			sgx_sys_exit(1);
+	}
+
+	__builtin_unreachable();
+}
diff --git a/drivers/platform/x86/intel_sgx/le/main.h b/drivers/platform/x86/intel_sgx/le/main.h
new file mode 100644
index 000000000000..7069d416d773
--- /dev/null
+++ b/drivers/platform/x86/intel_sgx/le/main.h
@@ -0,0 +1,30 @@
+// SPDX-License-Identifier: (GPL-2.0 OR BSD-3-Clause)
+// Copyright(c) 2016-17 Intel Corporation.
+//
+// Authors:
+//
+// Jarkko Sakkinen <jarkko.sakkinen@linux.intel.com>
+
+#ifndef MAIN_H
+#define MAIN_H
+
+#ifndef NULL
+#define NULL ((void *)0)
+#endif
+
+#define MAP_FAILED ((void *)-1)
+
+extern unsigned char sgx_le_blob[];
+extern unsigned char sgx_le_blob_end[];
+extern unsigned char sgx_le_ss[];
+
+void sgx_get_token(struct sgx_launch_request *req, void *entry,
+		   struct sgx_einittoken *token);
+long sgx_sys_read(void *buf, unsigned long count);
+long sgx_sys_write(const void *buf, unsigned long count);
+long sgx_sys_close(long fd);
+long sgx_sys_mmap(long fd, unsigned long size);
+long sgx_sys_ioctl(long fd, unsigned long cmd, void *arg);
+long sgx_sys_exit(long status);
+
+#endif /* MAIN_H */
diff --git a/drivers/platform/x86/intel_sgx/le/sgx_le_piggy.S b/drivers/platform/x86/intel_sgx/le/sgx_le_piggy.S
new file mode 100644
index 000000000000..e1881066f089
--- /dev/null
+++ b/drivers/platform/x86/intel_sgx/le/sgx_le_piggy.S
@@ -0,0 +1,22 @@
+// SPDX-License-Identifier: (GPL-2.0 OR BSD-3-Clause)
+// Copyright(c) 2016-17 Intel Corporation.
+//
+// Authors:
+//
+// Jarkko Sakkinen <jarkko.sakkinen@linux.intel.com>
+
+#include <linux/linkage.h>
+#include <asm/page_types.h>
+
+	.section ".rodata","a"
+	.balign PAGE_SIZE
+
+GLOBAL(sgx_le_blob)
+	.incbin	"drivers/platform/x86/intel_sgx/le/enclave/sgx_le.bin"
+END(sgx_le_blob)
+
+GLOBAL(sgx_le_blob_end);
+
+GLOBAL(sgx_le_ss)
+	.incbin	"drivers/platform/x86/intel_sgx/le/enclave/sgx_le.ss"
+END(sgx_le_ss)
diff --git a/drivers/platform/x86/intel_sgx/le/string.c b/drivers/platform/x86/intel_sgx/le/string.c
new file mode 100644
index 000000000000..0a61977f8628
--- /dev/null
+++ b/drivers/platform/x86/intel_sgx/le/string.c
@@ -0,0 +1,39 @@
+// SPDX-License-Identifier: (GPL-2.0 OR BSD-3-Clause)
+// Copyright(c) 2016-17 Intel Corporation.
+//
+// Authors:
+//
+// Jarkko Sakkinen <jarkko.sakkinen@linux.intel.com>
+
+#include <linux/types.h>
+
+/* This might look a bit ugly but is needed because of the way asm/string_64.h
+ * redefines the symbols depending on the CONFIG_KASAN flag.
+ */
+#ifdef CONFIG_KASAN
+void *__memset(void *s, int c, size_t n)
+#else
+void *memset(void *s, int c, size_t n)
+#endif
+{
+	unsigned long i;
+
+	for (i = 0; i < n; i++)
+		((unsigned char *)s)[i] = c;
+
+	return s;
+}
+
+#ifdef CONFIG_KASAN
+void *__memcpy(void *dest, const void *src, size_t n)
+#else
+void *memcpy(void *dest, const void *src, size_t n)
+#endif
+{
+	size_t i;
+
+	for (i = 0; i < n; i++)
+		((char *)dest)[i] = ((char *)src)[i];
+
+	return dest;
+}
diff --git a/drivers/platform/x86/intel_sgx/sgx.h b/drivers/platform/x86/intel_sgx/sgx.h
index 193f5e5b2338..9b9e9d7d09dd 100644
--- a/drivers/platform/x86/intel_sgx/sgx.h
+++ b/drivers/platform/x86/intel_sgx/sgx.h
@@ -103,13 +103,18 @@ struct sgx_encl {
 	struct mmu_notifier mmu_notifier;
 };
 
+extern unsigned char sgx_le_proxy[];
+extern unsigned char sgx_le_proxy_end[];
+extern struct sgx_sigstruct sgx_le_ss;
 extern struct workqueue_struct *sgx_add_page_wq;
 extern u64 sgx_encl_size_max_32;
 extern u64 sgx_encl_size_max_64;
 extern u64 sgx_xfrm_mask;
 extern u32 sgx_misc_reserved;
 extern u32 sgx_xsave_size_tbl[64];
+extern u64 sgx_le_pubkeyhash[4];
 
+extern const struct file_operations sgx_fops;
 extern const struct vm_operations_struct sgx_vm_ops;
 
 int sgx_encl_find(struct mm_struct *mm, unsigned long addr,
@@ -161,4 +166,16 @@ unsigned int sgx_alloc_va_slot(struct sgx_va_page *va_page);
 void sgx_free_va_slot(struct sgx_va_page *va_page, unsigned int offset);
 bool sgx_va_page_full(struct sgx_va_page *va_page);
 
+extern struct sgx_le_ctx sgx_le_ctx;
+
+int sgx_get_key_hash(const void *modulus, void *hash);
+int sgx_le_init(struct sgx_le_ctx *ctx);
+void sgx_le_exit(struct sgx_le_ctx *ctx);
+void sgx_le_stop(struct sgx_le_ctx *ctx, bool update_users);
+int sgx_le_start(struct sgx_le_ctx *ctx);
+int sgx_le_get_token(struct sgx_le_ctx *ctx,
+		     const struct sgx_encl *encl,
+		     const struct sgx_sigstruct *sigstruct,
+		     struct sgx_einittoken *token);
+
 #endif /* __ARCH_X86_INTEL_SGX_H__ */
diff --git a/drivers/platform/x86/intel_sgx/sgx_encl.c b/drivers/platform/x86/intel_sgx/sgx_encl.c
index 35436497530b..043be4bb4829 100644
--- a/drivers/platform/x86/intel_sgx/sgx_encl.c
+++ b/drivers/platform/x86/intel_sgx/sgx_encl.c
@@ -774,20 +774,6 @@ int sgx_encl_add_page(struct sgx_encl *encl, unsigned long addr, void *data,
 	return ret;
 }
 
-static int sgx_einit(struct sgx_encl *encl, struct sgx_sigstruct *sigstruct,
-		     struct sgx_einittoken *token)
-{
-	struct sgx_epc_page *secs_epc = encl->secs.epc_page;
-	void *secs_va;
-	int ret;
-
-	secs_va = sgx_get_page(secs_epc);
-	ret = __einit(sigstruct, token, secs_va);
-	sgx_put_page(secs_va);
-
-	return ret;
-}
-
 /**
  * sgx_encl_init - perform EINIT for the given enclave
  *
@@ -821,8 +807,8 @@ int sgx_encl_init(struct sgx_encl *encl, struct sgx_sigstruct *sigstruct,
 
 	for (i = 0; i < SGX_EINIT_SLEEP_COUNT; i++) {
 		for (j = 0; j < SGX_EINIT_SPIN_COUNT; j++) {
-			ret = sgx_einit(encl, sigstruct, token);
-
+			ret = sgx_einit(sigstruct, token, encl->secs.epc_page,
+					sgx_le_pubkeyhash);
 			if (ret == SGX_UNMASKED_EVENT)
 				continue;
 			else
diff --git a/drivers/platform/x86/intel_sgx/sgx_ioctl.c b/drivers/platform/x86/intel_sgx/sgx_ioctl.c
index 68b04893d66a..44b7bc03c2e7 100644
--- a/drivers/platform/x86/intel_sgx/sgx_ioctl.c
+++ b/drivers/platform/x86/intel_sgx/sgx_ioctl.c
@@ -186,7 +186,9 @@ static long sgx_ioc_enclave_init(struct file *filep, unsigned int cmd,
 	if (ret)
 		goto out;
 
-	ret = sgx_encl_init(encl, sigstruct, einittoken);
+	ret = sgx_le_get_token(&sgx_le_ctx, encl, sigstruct, einittoken);
+	if (!ret)
+		ret = sgx_encl_init(encl, sigstruct, einittoken);
 
 	kref_put(&encl->refcount, sgx_encl_release);
 
diff --git a/drivers/platform/x86/intel_sgx/sgx_le.c b/drivers/platform/x86/intel_sgx/sgx_le.c
new file mode 100644
index 000000000000..ffd0fce07614
--- /dev/null
+++ b/drivers/platform/x86/intel_sgx/sgx_le.c
@@ -0,0 +1,303 @@
+// SPDX-License-Identifier: (GPL-2.0 OR BSD-3-Clause)
+// Copyright(c) 2016-17 Intel Corporation.
+//
+// Authors:
+//
+// Jarkko Sakkinen <jarkko.sakkinen@linux.intel.com>
+
+#include <asm/sgx_le.h>
+#include <linux/anon_inodes.h>
+#include <linux/file.h>
+#include <linux/fs.h>
+#include <linux/kmod.h>
+#include <linux/mutex.h>
+#include <linux/sched/signal.h>
+#include <linux/shmem_fs.h>
+#include <linux/wait.h>
+#include "sgx.h"
+
+struct sgx_le_ctx {
+	struct pid *tgid;
+	char *argv[2];
+	struct crypto_shash *tfm;
+	struct mutex hash_lock;
+	struct mutex launch_lock;
+	struct rw_semaphore users;
+	wait_queue_head_t wq;
+	bool kernel_read;
+	bool user_read;
+	struct file *pipe;
+	struct sgx_launch_request req;
+	struct sgx_einittoken token;
+};
+
+struct sgx_le_ctx sgx_le_ctx;
+
+static int __sgx_get_key_hash(struct crypto_shash *tfm, const void *modulus,
+			      void *hash)
+{
+	SHASH_DESC_ON_STACK(shash, tfm);
+
+	shash->tfm = tfm;
+	shash->flags = CRYPTO_TFM_REQ_MAY_SLEEP;
+
+	return crypto_shash_digest(shash, modulus, SGX_MODULUS_SIZE, hash);
+}
+
+/**
+ * sgx_get_key_hash - calculate SHA256 for a given RSA key
+ * @modulus:	modulus of the key
+ * @hash:	the resulting hash
+ */
+int sgx_get_key_hash(const void *modulus, void *hash)
+{
+	struct crypto_shash *tfm;
+	int ret;
+
+	tfm = crypto_alloc_shash("sha256", 0, CRYPTO_ALG_ASYNC);
+	if (IS_ERR(tfm))
+		return PTR_ERR(tfm);
+
+	ret = __sgx_get_key_hash(tfm, modulus, hash);
+
+	crypto_free_shash(tfm);
+	return ret;
+}
+
+static ssize_t sgx_le_ctx_fops_read(struct file *filp, char __user *buf,
+				    size_t count, loff_t *off)
+{
+	struct sgx_le_ctx *ctx = filp->private_data;
+	int ret;
+
+	if (count != sizeof(ctx->req)) {
+		pr_crit("%s: invalid count %lu\n", __func__, count);
+		return -EIO;
+	}
+
+	ret = wait_event_interruptible(ctx->wq, ctx->user_read);
+	if (ret)
+		return -EINTR;
+
+	ret = copy_to_user(buf, &ctx->req, count);
+	ctx->user_read = false;
+
+	return ret ? ret : count;
+}
+
+static ssize_t sgx_le_ctx_fops_write(struct file *filp, const char __user *buf,
+				     size_t count, loff_t *off)
+{
+	struct sgx_le_ctx *ctx = filp->private_data;
+	int ret;
+
+	if (count != sizeof(ctx->token)) {
+		pr_crit("%s: invalid count %lu\n", __func__, count);
+		return -EIO;
+	}
+
+	ret = copy_from_user(&ctx->token, buf, count);
+	if (!ret)
+		ctx->kernel_read = true;
+	wake_up_interruptible(&ctx->wq);
+
+	return ret ? ret : count;
+}
+
+static const struct file_operations sgx_le_ctx_fops = {
+	.owner = THIS_MODULE,
+	.llseek = no_llseek,
+	.read = sgx_le_ctx_fops_read,
+	.write = sgx_le_ctx_fops_write,
+};
+
+static int sgx_le_task_init(struct subprocess_info *subinfo, struct cred *new)
+{
+	struct sgx_le_ctx *ctx = (struct sgx_le_ctx *)subinfo->data;
+	struct file *tmp_filp;
+	unsigned long len;
+	loff_t pos = 0;
+	int ret;
+
+	len = (unsigned long)&sgx_le_proxy_end - (unsigned long)&sgx_le_proxy;
+
+	tmp_filp = shmem_file_setup("[sgx_le_proxy]", len, 0);
+	if (IS_ERR(tmp_filp)) {
+		ret = PTR_ERR(tmp_filp);
+		return ret;
+	}
+	fd_install(SGX_LE_EXE_FD, tmp_filp);
+
+	ret = kernel_write(tmp_filp, &sgx_le_proxy, len, &pos);
+	if (ret != len && ret >= 0)
+		return -ENOMEM;
+	if (ret < 0)
+		return ret;
+
+	tmp_filp = anon_inode_getfile("[/dev/sgx]", &sgx_fops, NULL, O_RDWR);
+	if (IS_ERR(tmp_filp))
+		return PTR_ERR(tmp_filp);
+	fd_install(SGX_LE_DEV_FD, tmp_filp);
+
+	tmp_filp = anon_inode_getfile("[sgx_le]", &sgx_le_ctx_fops, ctx,
+				      O_RDWR);
+	if (IS_ERR(tmp_filp))
+		return PTR_ERR(tmp_filp);
+	fd_install(SGX_LE_PIPE_FD, tmp_filp);
+
+	ctx->tgid = get_pid(task_tgid(current));
+	ctx->pipe = tmp_filp;
+
+	return 0;
+}
+
+static void __sgx_le_stop(struct sgx_le_ctx *ctx)
+{
+	if (ctx->tgid) {
+		fput(ctx->pipe);
+		kill_pid(ctx->tgid, SIGKILL, 1);
+		put_pid(ctx->tgid);
+		ctx->tgid = NULL;
+	}
+}
+
+void sgx_le_stop(struct sgx_le_ctx *ctx, bool update_users)
+{
+	if (update_users) {
+		up_read(&ctx->users);
+		if (!down_write_trylock(&ctx->users))
+			return;
+	}
+
+	mutex_lock(&ctx->launch_lock);
+	__sgx_le_stop(ctx);
+	mutex_unlock(&ctx->launch_lock);
+
+	if (update_users)
+		up_write(&ctx->users);
+}
+
+static int __sgx_le_start(struct sgx_le_ctx *ctx)
+{
+	struct subprocess_info *subinfo;
+	int ret;
+
+	if (ctx->tgid)
+		return 0;
+
+	ctx->argv[0] = SGX_LE_EXE_PATH;
+	ctx->argv[1] = NULL;
+
+	subinfo = call_usermodehelper_setup(ctx->argv[0], ctx->argv,
+					    NULL, GFP_KERNEL, sgx_le_task_init,
+					    NULL, &sgx_le_ctx);
+	if (!subinfo)
+		return -ENOMEM;
+
+	ret = call_usermodehelper_exec(subinfo, UMH_WAIT_EXEC);
+	if (ret) {
+		__sgx_le_stop(ctx);
+		return ret;
+	}
+
+	return 0;
+}
+
+int sgx_le_start(struct sgx_le_ctx *ctx)
+{
+	int ret;
+
+	down_read(&ctx->users);
+
+	mutex_lock(&ctx->launch_lock);
+	ret = __sgx_le_start(ctx);
+	mutex_unlock(&ctx->launch_lock);
+
+	if (ret)
+		up_read(&ctx->users);
+
+	return ret;
+}
+
+int sgx_le_init(struct sgx_le_ctx *ctx)
+{
+	struct crypto_shash *tfm;
+
+	tfm = crypto_alloc_shash("sha256", 0, CRYPTO_ALG_ASYNC);
+	if (IS_ERR(tfm))
+		return PTR_ERR(tfm);
+
+	ctx->tfm = tfm;
+	mutex_init(&ctx->hash_lock);
+	mutex_init(&ctx->launch_lock);
+	init_rwsem(&ctx->users);
+	init_waitqueue_head(&ctx->wq);
+
+	return 0;
+}
+
+void sgx_le_exit(struct sgx_le_ctx *ctx)
+{
+	mutex_lock(&ctx->launch_lock);
+	crypto_free_shash(ctx->tfm);
+	mutex_unlock(&ctx->launch_lock);
+}
+
+static int __sgx_le_get_token(struct sgx_le_ctx *ctx,
+			      const struct sgx_encl *encl,
+			      struct sgx_einittoken *token)
+{
+	ssize_t ret;
+
+	if (!ctx->tgid)
+		return -EIO;
+
+	ctx->user_read = true;
+	wake_up_interruptible(&ctx->wq);
+
+	ret = wait_event_interruptible(ctx->wq, ctx->kernel_read);
+	if (ret)
+		return -EINTR;
+
+	memcpy(token, &ctx->token, sizeof(*token));
+	ctx->kernel_read = false;
+
+	return 0;
+}
+
+int sgx_le_get_token(struct sgx_le_ctx *ctx,
+		     const struct sgx_encl *encl,
+		     const struct sgx_sigstruct *sigstruct,
+		     struct sgx_einittoken *token)
+{
+	u8 mrsigner[32];
+	int ret;
+
+	mutex_lock(&ctx->hash_lock);
+	ret = __sgx_get_key_hash(ctx->tfm, sigstruct->modulus, mrsigner);
+	if (ret) {
+		mutex_unlock(&ctx->hash_lock);
+		return ret;
+	}
+	if (!memcmp(mrsigner, sgx_le_pubkeyhash, 32)) {
+		token->payload.valid = false;
+		mutex_unlock(&ctx->hash_lock);
+		return 0;
+	}
+	mutex_unlock(&ctx->hash_lock);
+
+	mutex_lock(&ctx->launch_lock);
+	ret = __sgx_le_start(ctx);
+	if (ret) {
+		mutex_unlock(&ctx->launch_lock);
+		return ret;
+	}
+	memcpy(&ctx->req.mrenclave, sigstruct->body.mrenclave, 32);
+	memcpy(&ctx->req.mrsigner, mrsigner, 32);
+	ctx->req.attributes = encl->attributes;
+	ctx->req.xfrm = encl->xfrm;
+	memset(&ctx->token, 0, sizeof(ctx->token));
+	ret = __sgx_le_get_token(ctx, encl, token);
+	mutex_unlock(&ctx->launch_lock);
+	return ret;
+}
diff --git a/drivers/platform/x86/intel_sgx/sgx_le_proxy_piggy.S b/drivers/platform/x86/intel_sgx/sgx_le_proxy_piggy.S
new file mode 100644
index 000000000000..9d2057615d04
--- /dev/null
+++ b/drivers/platform/x86/intel_sgx/sgx_le_proxy_piggy.S
@@ -0,0 +1,22 @@
+// SPDX-License-Identifier: (GPL-2.0 OR BSD-3-Clause)
+// Copyright(c) 2016-17 Intel Corporation.
+//
+// Authors:
+//
+// Jarkko Sakkinen <jarkko.sakkinen@linux.intel.com>
+
+#include <linux/linkage.h>
+#include <asm/export.h>
+#include <asm/page_types.h>
+
+	.section ".rodata","a"
+
+GLOBAL(sgx_le_proxy)
+	.incbin	"drivers/platform/x86/intel_sgx/le/sgx_le_proxy"
+END(sgx_le_proxy)
+
+GLOBAL(sgx_le_proxy_end)
+
+GLOBAL(sgx_le_ss)
+	.incbin	"drivers/platform/x86/intel_sgx/le/enclave/sgx_le.ss"
+END(sgx_le_ss)
diff --git a/drivers/platform/x86/intel_sgx/sgx_main.c b/drivers/platform/x86/intel_sgx/sgx_main.c
index 79d772847026..39560ee94b43 100644
--- a/drivers/platform/x86/intel_sgx/sgx_main.c
+++ b/drivers/platform/x86/intel_sgx/sgx_main.c
@@ -31,6 +31,31 @@ u64 sgx_encl_size_max_64;
 u64 sgx_xfrm_mask = 0x3;
 u32 sgx_misc_reserved;
 u32 sgx_xsave_size_tbl[64];
+u64 sgx_le_pubkeyhash[4];
+
+static DECLARE_RWSEM(sgx_file_sem);
+
+static int sgx_open(struct inode *inode, struct file *file)
+{
+	int ret;
+
+	ret = sgx_le_start(&sgx_le_ctx);
+
+	if (!ret)
+		file->private_data = &sgx_le_ctx;
+
+	return ret;
+}
+
+static int sgx_release(struct inode *inode, struct file *file)
+{
+	if (!file->private_data)
+		return 0;
+
+	sgx_le_stop(file->private_data, true);
+
+	return 0;
+}
 
 #ifdef CONFIG_COMPAT
 long sgx_compat_ioctl(struct file *filep, unsigned int cmd, unsigned long arg)
@@ -84,8 +109,10 @@ static unsigned long sgx_get_unmapped_area(struct file *file,
 	return addr;
 }
 
-static const struct file_operations sgx_fops = {
+const struct file_operations sgx_fops = {
 	.owner			= THIS_MODULE,
+	.open			= sgx_open,
+	.release		= sgx_release,
 	.unlocked_ioctl		= sgx_ioctl,
 #ifdef CONFIG_COMPAT
 	.compat_ioctl		= sgx_compat_ioctl,
@@ -100,6 +127,7 @@ static int sgx_pm_suspend(struct device *dev)
 	struct sgx_epc_page *epc_page;
 	struct sgx_encl *encl;
 
+	sgx_le_stop(&sgx_le_ctx, false);
 	list_for_each_entry(epc_page, &sgx_active_page_list, list) {
 		encl_page = container_of(epc_page->impl, struct sgx_encl_page,
 					 impl);
@@ -176,6 +204,34 @@ static struct sgx_context *sgxm_ctx_alloc(struct device *parent)
 	return ctx;
 }
 
+static int sgx_init_msrs(void)
+{
+	u64 msrs[4];
+	int ret;
+
+	ret = sgx_get_key_hash(sgx_le_ss.modulus, sgx_le_pubkeyhash);
+	if (ret)
+		return ret;
+
+	if (sgx_lc_enabled)
+		return 0;
+
+	rdmsrl(MSR_IA32_SGXLEPUBKEYHASH0, msrs[0]);
+	rdmsrl(MSR_IA32_SGXLEPUBKEYHASH1, msrs[1]);
+	rdmsrl(MSR_IA32_SGXLEPUBKEYHASH2, msrs[2]);
+	rdmsrl(MSR_IA32_SGXLEPUBKEYHASH3, msrs[3]);
+
+	if ((sgx_le_pubkeyhash[0] != msrs[0]) ||
+	    (sgx_le_pubkeyhash[1] != msrs[1]) ||
+	    (sgx_le_pubkeyhash[2] != msrs[2]) ||
+	    (sgx_le_pubkeyhash[3] != msrs[3])) {
+		pr_err("IA32_SGXLEPUBKEYHASHn MSRs do not match to the launch enclave signing key\n");
+		return -ENODEV;
+	}
+
+	return 0;
+}
+
 static int sgx_dev_init(struct device *parent)
 {
 	struct sgx_context *sgx_dev;
@@ -186,6 +242,10 @@ static int sgx_dev_init(struct device *parent)
 	int ret;
 	int i;
 
+	ret = sgx_init_msrs();
+	if (ret)
+		return ret;
+
 	sgx_dev = sgxm_ctx_alloc(parent);
 
 	cpuid_count(SGX_CPUID, SGX_CPUID_CAPABILITIES, &eax, &ebx, &ecx, &edx);
@@ -213,11 +273,17 @@ static int sgx_dev_init(struct device *parent)
 	if (!sgx_add_page_wq)
 		return -ENOMEM;
 
-	ret = cdev_device_add(&sgx_dev->cdev, &sgx_dev->dev);
+	ret = sgx_le_init(&sgx_le_ctx);
 	if (ret)
 		goto out_workqueue;
 
+	ret = cdev_device_add(&sgx_dev->cdev, &sgx_dev->dev);
+	if (ret)
+		goto out_le;
+
 	return 0;
+out_le:
+	sgx_le_exit(&sgx_le_ctx);
 out_workqueue:
 	destroy_workqueue(sgx_add_page_wq);
 	return ret;
@@ -236,6 +302,7 @@ static int sgx_drv_remove(struct platform_device *pdev)
 	struct sgx_context *ctx = dev_get_drvdata(&pdev->dev);
 
 	cdev_device_del(&ctx->cdev, &ctx->dev);
+	sgx_le_exit(&sgx_le_ctx);
 	destroy_workqueue(sgx_add_page_wq);
 
 	return 0;
-- 
2.17.0

Re: [PATCH v11 04/13] x86, sgx: add SGX definitions to msr-index.h

[Dave Hansen]

On 06/08/2018 10:09 AM, Jarkko Sakkinen wrote:
> Signed-off-by: Sean Christopherson <sean.j.christopherson@intel.com>
> Signed-off-by: Haim Cohen <haim.cohen@intel.com>
> Tested-by: Serge Ayoun <serge.ayoun@intel.com>
> Reviewed-by: Jarkko Sakkinen <jarkko.sakkinen@linux.intel.com>
> Tested-by: Jarkko Sakkinen <jarkko.sakkinen@linux.intel.com>
> Signed-off-by: Jarkko Sakkinen <jarkko.sakkinen@linux.intel.com>

These are a mess.  If you're signing off on it, a review of some kind is
pretty implied.

Also, how did Sean write this, send it to Haim to sign off on it, and
then send it to you?  That seems unlikely.

Re: [PATCH v11 06/13] crypto: aesni: add minimal build option for SGX LE

[Dave Hansen]

On 06/08/2018 10:09 AM, Jarkko Sakkinen wrote:
> --- a/arch/x86/crypto/aesni-intel_asm.S
> +++ b/arch/x86/crypto/aesni-intel_asm.S
> @@ -45,6 +45,8 @@
>  #define MOVADQ	movaps
>  #define MOVUDQ	movups
>  
> +#ifndef AESNI_INTEL_MINIMAL
> +
>  #ifdef __x86_64__
>  
>  # constants in mergeable sections, linker can reorder and merge
> @@ -133,6 +135,8 @@ ALL_F:      .octa 0xffffffffffffffffffffffffffffffff
>  #define keysize 2*15*16(%arg1)
>  #endif
>  
> +#endif /* AESNI_INTEL_MINIMAL */
> +

I'd really prefer that these get moved into a separate file rather than
a scattered set of #ifdefs.  This just seem fragile to me.

Can we have a "aesni-intel_asm-minimal.S"?  Or, at least bunch the
minimal set of things *together*?

Re: [PATCH v11 07/13] x86, sgx: detect Intel SGX

[Dave Hansen]

> +config INTEL_SGX_CORE
> +	prompt "Intel SGX core functionality
> +	depends on X86_64 && CPU_SUP_INTEL
> +	help
> +	Intel Software Guard eXtensions (SGX) is a set of CPU instructions
> +	that allows ring 3 applications to create enclaves; private regions
> +	of memory that are protected, by hardware, from unauthorized access
> +	and/or modification.

That semicolon needs to be a colon.  The second half of that sentence is
not a stand-alone statement.

> +	This option enables kernel recognition of SGX, high-level management
> +	of the Enclave Page Cache (EPC), tracking and writing of SGX Launch
> +	Enclave Hash MSRs, and allows for virtualization of SGX via KVM. By
> +	iteslf, this option does not provide SGX support to userspace.
> +
> +	For details, see Documentation/x86/intel_sgx.rst
> +
> +	If unsure, say N.
> +
>  config EFI
>  	bool "EFI runtime service support"
>  	depends on ACPI
> diff --git a/arch/x86/include/asm/sgx.h b/arch/x86/include/asm/sgx.h
> new file mode 100644
> index 000000000000..fa3e6e0eb8af
> --- /dev/null
> +++ b/arch/x86/include/asm/sgx.h
> @@ -0,0 +1,25 @@
> +// SPDX-License-Identifier: (GPL-2.0 OR BSD-3-Clause)
> +// Copyright(c) 2016-18 Intel Corporation.
> +//
> +// Authors:
> +//
> +// Jarkko Sakkinen <jarkko.sakkinen@linux.intel.com>
> +// Suresh Siddha <suresh.b.siddha@intel.com>
> +// Sean Christopherson <sean.j.christopherson@intel.com>
> +
> +#ifndef _ASM_X86_SGX_H
> +#define _ASM_X86_SGX_H
> +
> +#include <linux/types.h>
> +
> +#define SGX_CPUID 0x12

Hey, I just saw 0x12 as a magic, hard-coded number earlier in these
patches.  It seems cruel to hard-code it, and then also have a #define
that isn't used.

> +enum sgx_cpuid {
> +	SGX_CPUID_CAPABILITIES	= 0,
> +	SGX_CPUID_ATTRIBUTES	= 1,
> +	SGX_CPUID_EPC_BANKS	= 2,
> +};

These are cpuid *leaves*, right?  Please make this clear that these are
hardware-defined values and not some kind of software construct.

> +bool sgx_enabled __ro_after_init = false;
> +EXPORT_SYMBOL(sgx_enabled);
> +
> +static __init bool sgx_is_enabled(void)
> +{
> +	unsigned long fc;
> +
> +	if (boot_cpu_data.x86_vendor != X86_VENDOR_INTEL)
> +		return false;

Not necessary. CPUID does this part for you.

> +	if (!boot_cpu_has(X86_FEATURE_SGX))
> +		return false;
> +
> +	if (!boot_cpu_has(X86_FEATURE_SGX1))
> +		return false;
> +
> +	rdmsrl(MSR_IA32_FEATURE_CONTROL, fc);
> +	if (!(fc & FEATURE_CONTROL_LOCKED))
> +		return false;
> +
> +	if (!(fc & FEATURE_CONTROL_SGX_ENABLE))
> +		return false;

Comments, please.  Why would this happen?  What would it mean?

Re: [PATCH v11 08/13] x86, sgx: added ENCLS wrappers

[Dave Hansen]

On 06/08/2018 10:09 AM, Jarkko Sakkinen wrote:
> This commit adds wrappers for Intel(R) SGX ENCLS opcode functionality.

What's ENCLS?  I know what an opcode is, but I don't know what "opcode
functionality" is.  Could you give us more than a single, cryptic
sentence, please?

> +enum sgx_commands {
> +	ECREATE	= 0x0,
> +	EADD	= 0x1,
> +	EINIT	= 0x2,
> +	EREMOVE	= 0x3,
> +	EDGBRD	= 0x4,
> +	EDGBWR	= 0x5,
> +	EEXTEND	= 0x6,
> +	ELDU	= 0x8,
> +	EBLOCK	= 0x9,
> +	EPA	= 0xA,
> +	EWB	= 0xB,
> +	ETRACK	= 0xC,
> +	EAUG	= 0xD,
> +	EMODPR	= 0xE,
> +	EMODT	= 0xF,
> +};

Again, please differentiate hardware-defined values from
software-defines ones.  Also, would it hurt to expand the acronyms a
bit, like:

+	ELDU	= 0x8, /* LoaD Underpants */

> +#define SGX_FN(name, params...)		\
> +{					\
> +	void *epc;			\
> +	int ret;			\
> +	epc = sgx_get_page(epc_page);	\
> +	ret = __##name(params);		\
> +	sgx_put_page(epc);		\
> +	return ret;			\
> +}

Have I seen sgx_*_page() yet in this series?  This seems out of order.

> +#define BUILD_SGX_FN(fn, name)				\
> +static inline int fn(struct sgx_epc_page *epc_page)	\
> +	SGX_FN(name, epc)
> +BUILD_SGX_FN(sgx_eremove, eremove)
> +BUILD_SGX_FN(sgx_eblock, eblock)
> +BUILD_SGX_FN(sgx_etrack, etrack)
> +BUILD_SGX_FN(sgx_epa, epa)
> +
> +static inline int sgx_emodpr(struct sgx_secinfo *secinfo,
> +			     struct sgx_epc_page *epc_page)
> +	SGX_FN(emodpr, secinfo, epc)
> +static inline int sgx_emodt(struct sgx_secinfo *secinfo,
> +			    struct sgx_epc_page *epc_page)
> +	SGX_FN(emodt, secinfo, epc)

Wow, that's hideous.

Can't you just do:

BUILD_SGX_FN(__sgx_emopt, foo)

static inline int sgx_emodt(struct sgx_secinfo *secinfo,
			    struct sgx_epc_page *epc_page)
{
	return __sgx_emopt(secinfo, page);
}

Also, this entire patch seems rather comment-free.  Was that intentional?

Re: [PATCH v11 09/13] x86, sgx: basic routines for enclave page cache

[Jethro Beekman]

[-- Attachment #1: Type: text/plain, Size: 1232 bytes --]

On 2018-06-08 10:09, Jarkko Sakkinen wrote:
> +/*
> + * Writing the LE hash MSRs is extraordinarily expensive, e.g.
> + * 3-4x slower than normal MSRs, so we use a per-cpu cache to
> + * track the last known value of the MSRs to avoid unnecessarily
> + * writing the MSRs with the current value.  Because most Linux
> + * kernels will use an LE that is signed with a non-Intel key,

I don't think you can predict what most Linux kernels will be doing. I 
think not initializing the cache to the CPU's initial value is fine, but 
this particular argument shouldn't appear in the rationale.

> + * i.e. the first EINIT will need to write the MSRs regardless
> + * of the cache, the cache is intentionally left uninitialized
> + * during boot as initializing the cache would be pure overhead
> + * for the majority of systems.  Furthermore, the MSRs are per-cpu
> + * and the boot-time values aren't guaranteed to be identical
> + * across cpus, so we'd have to run code all all cpus to properly
> + * init the cache.  All in all, the complexity and overhead of
> + * initializing the cache is not justified.
> + */
> +static DEFINE_PER_CPU(u64 [4], sgx_le_pubkey_hash_cache);

-- 
Jethro Beekman | Fortanix


[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 3994 bytes --]

Re: [PATCH v11 09/13] x86, sgx: basic routines for enclave page cache

[Dave Hansen]

On 06/08/2018 10:09 AM, Jarkko Sakkinen wrote:
> SGX has a set of data structures to maintain information about the enclaves
> and their security properties. BIOS reserves a fixed size region of
> physical memory for these structures by setting Processor Reserved Memory
> Range Registers (PRMRR). This memory area is called Enclave Page Cache
> (EPC).
> 
> This commit implements the basic routines to allocate and free pages from
> different EPC banks. There is also a swapper thread ksgxswapd for EPC pages
> that gets woken up by sgx_alloc_page() when we run below the low watermark.
> The swapper thread continues swapping pages up until it reaches the high
> watermark.

Yay!  A new memory manager in arch-specific code.

> Each subsystem that uses SGX must provide a set of callbacks for EPC
> pages that are used to reclaim, block and write an EPC page. Kernel
> takes the responsibility of maintaining LRU cache for them.

What does a "subsystem that uses SGX" mean?  Do we have one of those
already?

...
> +struct sgx_secs {
> +	uint64_t size;
> +	uint64_t base;
> +	uint32_t ssaframesize;
> +	uint32_t miscselect;
> +	uint8_t reserved1[SGX_SECS_RESERVED1_SIZE];
> +	uint64_t attributes;
> +	uint64_t xfrm;
> +	uint32_t mrenclave[8];
> +	uint8_t reserved2[SGX_SECS_RESERVED2_SIZE];
> +	uint32_t mrsigner[8];
> +	uint8_t	reserved3[SGX_SECS_RESERVED3_SIZE];
> +	uint16_t isvvprodid;
> +	uint16_t isvsvn;
> +	uint8_t reserved4[SGX_SECS_RESERVED4_SIZE];
> +};

This is a hardware structure, right?  Doesn't it need to be packed?

> +enum sgx_tcs_flags {
> +	SGX_TCS_DBGOPTIN	= 0x01, /* cleared on EADD */
> +};
> +
> +#define SGX_TCS_RESERVED_MASK 0xFFFFFFFFFFFFFFFEL

Would it be possible to separate out the SGX software structures from
SGX hardware?  It's hard to tell them apart.

> +#define SGX_NR_TO_SCAN	16
> +#define SGX_NR_LOW_PAGES 32
> +#define SGX_NR_HIGH_PAGES 64
> +
>  bool sgx_enabled __ro_after_init = false;
>  EXPORT_SYMBOL(sgx_enabled);
> +bool sgx_lc_enabled __ro_after_init;
> +EXPORT_SYMBOL(sgx_lc_enabled);
> +atomic_t sgx_nr_free_pages = ATOMIC_INIT(0);

Hmmm, global atomic.  Doesn't sound very scalable.

> +struct sgx_epc_bank sgx_epc_banks[SGX_MAX_EPC_BANKS];
> +EXPORT_SYMBOL(sgx_epc_banks);
> +int sgx_nr_epc_banks;
> +EXPORT_SYMBOL(sgx_nr_epc_banks);
> +LIST_HEAD(sgx_active_page_list);
> +EXPORT_SYMBOL(sgx_active_page_list);
> +DEFINE_SPINLOCK(sgx_active_page_list_lock);
> +EXPORT_SYMBOL(sgx_active_page_list_lock);

Hmmm, global spinlock protecting a page allocator linked list.  Sounds
even worse than at atomic.

Why is this OK?

> +static struct task_struct *ksgxswapd_tsk;
> +static DECLARE_WAIT_QUEUE_HEAD(ksgxswapd_waitq);
> +
> +/*
> + * Writing the LE hash MSRs is extraordinarily expensive, e.g.
> + * 3-4x slower than normal MSRs, so we use a per-cpu cache to
> + * track the last known value of the MSRs to avoid unnecessarily
> + * writing the MSRs with the current value.  Because most Linux
> + * kernels will use an LE that is signed with a non-Intel key,
> + * i.e. the first EINIT will need to write the MSRs regardless
> + * of the cache, the cache is intentionally left uninitialized
> + * during boot as initializing the cache would be pure overhead
> + * for the majority of systems.  Furthermore, the MSRs are per-cpu
> + * and the boot-time values aren't guaranteed to be identical
> + * across cpus, so we'd have to run code all all cpus to properly
> + * init the cache.  All in all, the complexity and overhead of
> + * initializing the cache is not justified.
> + */
> +static DEFINE_PER_CPU(u64 [4], sgx_le_pubkey_hash_cache);

Justifying the design decisions is great for changelogs, not so great
for comments.  Also, looking at this, I have no idea what this has to do
with the "enclave page cache".

> +static void sgx_swap_cluster(void)
> +{
> +	struct sgx_epc_page *cluster[SGX_NR_TO_SCAN + 1];
> +	struct sgx_epc_page *epc_page;
> +	int i;
> +	int j;

This is rather free of comments or explanation of what this is doing,
how it is related to swapping as everyone else knows it

> +	memset(cluster, 0, sizeof(cluster));
> +
> +	for (i = 0, j = 0; i < SGX_NR_TO_SCAN; i++) {
> +		spin_lock(&sgx_active_page_list_lock);
> +		if (list_empty(&sgx_active_page_list)) {
> +			spin_unlock(&sgx_active_page_list_lock);
> +			break;
> +		}
> +		epc_page = list_first_entry(&sgx_active_page_list,
> +					    struct sgx_epc_page, list);
> +		if (!epc_page->impl->ops->get(epc_page)) {
> +			list_move_tail(&epc_page->list, &sgx_active_page_list);
> +			spin_unlock(&sgx_active_page_list_lock);
> +			continue;
> +		}
> +		list_del(&epc_page->list);
> +		spin_unlock(&sgx_active_page_list_lock);
>  
> -static __init bool sgx_is_enabled(void)
> +		if (epc_page->impl->ops->reclaim(epc_page)) {
> +			cluster[j++] = epc_page;
> +		} else {
> +			spin_lock(&sgx_active_page_list_lock);
> +			list_add_tail(&epc_page->list, &sgx_active_page_list);
> +			spin_unlock(&sgx_active_page_list_lock);
> +			epc_page->impl->ops->put(epc_page);
> +		}
> +	}
> +
> +	for (i = 0; cluster[i]; i++) {
> +		epc_page = cluster[i];
> +		epc_page->impl->ops->block(epc_page);
> +	}
> +
> +	for (i = 0; cluster[i]; i++) {
> +		epc_page = cluster[i];
> +		epc_page->impl->ops->write(epc_page);
> +		epc_page->impl->ops->put(epc_page);
> +		sgx_free_page(epc_page);
> +	}
> +}

This is also gloriously free of any superfluous comments.  Could you fix
that?

> +/**
> + * sgx_try_alloc_page - try to allocate an EPC page
> + * @impl:	implementation for the struct sgx_epc_page
> + *
> + * Try to grab a page from the free EPC page list. If there is a free page
> + * available, it is returned to the caller.
> + *
> + * Return:
> + *   a &struct sgx_epc_page instace,
> + *   NULL otherwise
> + */
> +struct sgx_epc_page *sgx_try_alloc_page(struct sgx_epc_page_impl *impl)
> +{
> +	struct sgx_epc_bank *bank;
> +	struct sgx_epc_page *page = NULL;
> +	int i;
> +
> +	for (i = 0; i < sgx_nr_epc_banks; i++) {
> +		bank = &sgx_epc_banks[i];

What's a bank?  How many banks does a system have?

> +		down_write(&bank->lock);
> +
> +		if (atomic_read(&bank->free_cnt))
> +			page = bank->pages[atomic_dec_return(&bank->free_cnt)];

Why is a semaphore getting used here?  I don't see any sleeping or
anything happening under this lock.

> +		up_write(&bank->lock);
> +
> +		if (page)
> +			break;
> +	}
> +
> +	if (page) {
> +		atomic_dec(&sgx_nr_free_pages);
> +		page->impl = impl;
> +	}
> +
> +	return page;
> +}
> +EXPORT_SYMBOL(sgx_try_alloc_page);
> +
> +/**
> + * sgx_alloc_page - allocate an EPC page
> + * @flags:	allocation flags
> + * @impl:	implementation for the struct sgx_epc_page
> + *
> + * Try to grab a page from the free EPC page list. If there is a free page
> + * available, it is returned to the caller. If called with SGX_ALLOC_ATOMIC,
> + * the function will return immediately if the list is empty. Otherwise, it
> + * will swap pages up until there is a free page available. Upon returning the
> + * low watermark is checked and ksgxswapd is waken up if we are below it.
> + *
> + * Return:
> + *   a &struct sgx_epc_page instace,
> + *   -ENOMEM if all pages are unreclaimable,
> + *   -EBUSY when called with SGX_ALLOC_ATOMIC and out of free pages
> + */
> +struct sgx_epc_page *sgx_alloc_page(struct sgx_epc_page_impl *impl,
> +				    unsigned int flags)
> +{
> +	struct sgx_epc_page *entry;
> +
> +	for ( ; ; ) {
> +		entry = sgx_try_alloc_page(impl);
> +		if (entry)
> +			break;
> +
> +		if (list_empty(&sgx_active_page_list))
> +			return ERR_PTR(-ENOMEM);

"active" pages in the VM are allocated/in-use pages.  This doesn't look
to be using the same terminology.

> +		if (flags & SGX_ALLOC_ATOMIC) {
> +			entry = ERR_PTR(-EBUSY);
> +			break;
> +		}
> +
> +		if (signal_pending(current)) {
> +			entry = ERR_PTR(-ERESTARTSYS);
> +			break;
> +		}
> +
> +		sgx_swap_cluster();
> +		schedule();

What's the schedule trying to do?  Is this the equivalent of "direct
reclaim"?  Why do we need this in addition to the ksgxswapd?

> +	}
> +
> +	if (atomic_read(&sgx_nr_free_pages) < SGX_NR_LOW_PAGES)
> +		wake_up(&ksgxswapd_waitq);
> +
> +	return entry;
> +}
> +EXPORT_SYMBOL(sgx_alloc_page);

Why aren't these _GPL exports?

> +/**
> + * sgx_free_page - free an EPC page
> + *
> + * @page:	any EPC page
> + *
> + * Remove an EPC page and insert it back to the list of free pages.
> + *
> + * Return: SGX error code
> + */
> +int sgx_free_page(struct sgx_epc_page *page)
> +{
> +	struct sgx_epc_bank *bank = SGX_EPC_BANK(page);
> +	int ret;
> +
> +	ret = sgx_eremove(page);
> +	if (ret) {
> +		pr_debug("EREMOVE returned %d\n", ret);
> +		return ret;
> +	}
> +
> +	down_read(&bank->lock);
> +	bank->pages[atomic_inc_return(&bank->free_cnt) - 1] = page;
> +	atomic_inc(&sgx_nr_free_pages);
> +	up_read(&bank->lock);
> +
> +	return 0;
> +}

bank->lock confuses me.  This seems to be writing to a bank, but only
needs a read lock.  Why?

> +/**
> + * sgx_get_page - pin an EPC page
> + * @page:	an EPC page
> + *
> + * Return: a pointer to the pinned EPC page
> + */
> +void *sgx_get_page(struct sgx_epc_page *page)
> +{
> +	struct sgx_epc_bank *bank = SGX_EPC_BANK(page);
> +
> +	if (IS_ENABLED(CONFIG_X86_64))
> +		return (void *)(bank->va + SGX_EPC_ADDR(page) - bank->pa);
> +
> +	return kmap_atomic_pfn(SGX_EPC_PFN(page));
> +}
> +EXPORT_SYMBOL(sgx_get_page);

This is odd.  Do you really want to detect 64-bit, or CONFIG_HIGHMEM?

> +struct page *sgx_get_backing(struct file *file, pgoff_t index)
> +{
> +	struct inode *inode = file->f_path.dentry->d_inode;
> +	struct address_space *mapping = inode->i_mapping;
> +	gfp_t gfpmask = mapping_gfp_mask(mapping);
> +
> +	return shmem_read_mapping_page_gfp(mapping, index, gfpmask);
> +}
> +EXPORT_SYMBOL(sgx_get_backing);

What does shmem have to do with all this?

> +void sgx_put_backing(struct page *backing_page, bool write)
> +{
> +	if (write)
> +		set_page_dirty(backing_page);
> +
> +	put_page(backing_page);
> +}
> +EXPORT_SYMBOL(sgx_put_backing);

I'm not a big fan of stuff getting added with no apparent user and no
explaination of what it is doing.  There's no way for me to assess
whether this is sane or not.

> +static __init int sgx_page_cache_init(void)
> +{
> +	struct task_struct *tsk;
> +	unsigned long size;
> +	unsigned int eax;
> +	unsigned int ebx;
> +	unsigned int ecx;
> +	unsigned int edx;
> +	unsigned long pa;
> +	int i;
> +	int ret;
> +
> +	for (i = 0; i < SGX_MAX_EPC_BANKS; i++) {
> +		cpuid_count(SGX_CPUID, i + SGX_CPUID_EPC_BANKS, &eax, &ebx,
> +			    &ecx, &edx);
> +		if (!(eax & 0xf))
> +			break;
> +
> +		pa   = ((u64)(ebx & 0xfffff) << 32) + (u64)(eax & 0xfffff000);
> +		size = ((u64)(edx & 0xfffff) << 32) + (u64)(ecx & 0xfffff000);

Please align these like I did ^

> +		pr_info("EPC bank 0x%lx-0x%lx\n", pa, pa + size);
> +
> +		ret = sgx_init_epc_bank(pa, size, i, &sgx_epc_banks[i]);
> +		if (ret) {
> +			sgx_page_cache_teardown();
> +			return ret;
> +		}
> +
> +		sgx_nr_epc_banks++;
> +	}

This is also rather sparsely commented.

> +static __init bool sgx_is_enabled(bool *lc_enabled)
>  {
>  	unsigned long fc;
>  
> @@ -41,12 +466,26 @@ static __init bool sgx_is_enabled(void)
>  	if (!(fc & FEATURE_CONTROL_SGX_ENABLE))
>  		return false;
>  
> +	*lc_enabled = !!(fc & FEATURE_CONTROL_SGX_LE_WR);
> +
>  	return true;
>  }

I'm baffled why lc_enabled is connected to the enclave page cache.

>  static __init int sgx_init(void)
>  {
> -	sgx_enabled = sgx_is_enabled();
> +	bool lc_enabled;
> +	int ret;
> +
> +	if (!sgx_is_enabled(&lc_enabled))
> +		return 0;
> +
> +	ret = sgx_page_cache_init();
> +	if (ret)
> +		return ret;
> +
> +	sgx_enabled = true;
> +	sgx_lc_enabled = lc_enabled;
> +
>  	return 0;
>  }
>  
> 

Re: [PATCH v11 12/13] intel_sgx: driver documentation

[Jethro Beekman]

[-- Attachment #1: Type: text/plain, Size: 1511 bytes --]

On 2018-06-08 10:09, Jarkko Sakkinen wrote:
> +Launching enclaves
> +------------------
> +
> +For privileged enclaves the launch is performed simply by submitting the
> +SIGSTRUCT for that enclave to ENCLS(EINIT). For unprivileged enclaves the
> +driver hosts a process in ring-3 that hosts a launch enclave signed with a key
> +supplied for kbuild.
> +
> +The current implementation of the launch enclave generates a token for any
> +enclave. In the future it could be potentially extended to have ways to
> +configure policy what can be lauched.
> +
> +The driver will fail to initialize if it cannot start its own launch enclave.
> +A user space application can submit a SIGSTRUCT instance through the ioctl API.
> +The kernel will take care of the rest.
> +
> +This design assures that the Linux kernel has always full control, which
> +enclaves get to launch and which do not, even if the public key MSRs are

As discussed previously at length, since the kernel needs to execute 
ENCLS[EINIT], it has full control to deny the launching of enclaves 
regardless of any launch enclave implementation. Please change this 
misleading statement.

> +read-only. Having launch intrinsics inside the kernel also enables easy
> +development of enclaves without necessarily needing any heavy weight SDK.
> +Having a low-barrier to implement enclaves could make sense for example for
> +system daemons where amount of dependecies ought to be minimized.

--
Jethro Beekman | Fortanix


[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 3994 bytes --]

Re: [PATCH v11 11/13] intel_sgx: ptrace() support

[Dave Hansen]

On 06/08/2018 10:09 AM, Jarkko Sakkinen wrote:
> +		ret = sgx_edbgrd(encl, entry, align, data);
> +		if (ret)
> +			break;
> +		if (write) {
> +			memcpy(data + offset, buf + i, cnt);
> +			ret = sgx_edbgwr(encl, entry, align, data);
> +			if (ret)
> +				break;
> +		}
> +		else
> +			memcpy(buf + i,data + offset, cnt);
> +	}

The SGX instructions like "edbgrd" be great to put on a license plat,
but we can do better in the kernel.  Can you give these reasonable
english names, please?  sgx_debug_write(), maybe?

Note that we have plenty of incomprehensible instruction names in the
kernel like "wrpkru", but we do our best to keep them as confined as
possible and make sure they don't hurt code readability.

Re: [intel-sgx-kernel-dev] [PATCH v11 13/13] intel_sgx: in-kernel launch enclave

[Andy Lutomirski]

On Fri, Jun 8, 2018 at 10:32 AM Jarkko Sakkinen
<jarkko.sakkinen@linux.intel.com> wrote:
>
> The Launch Enclave (LE) generates cryptographic launch tokens for user
> enclaves. A launch token is used by EINIT to check whether the enclave
> is authorized to launch or not. By having its own launch enclave, Linux
> has full control of the enclave launch process.
>
> LE is wrapped into a user space proxy program that reads enclave
> signatures outputs launch tokens. The kernel-side glue code is
> implemented by using the user space helper framework.  The IPC between
> the LE proxy program and kernel is handled with an anonymous inode.
>
> The commit also adds enclave signing tool that is used by kbuild to
> measure and sign the launch enclave. CONFIG_INTEL_SGX_SIGNING_KEY points
> to a PEM-file for the 3072-bit RSA key that is used as the LE public key
> pair. The default location is:

It might be nice to use the infrastructure that Alexei added for
bpfilter (the umh_blob stuff) here, which is slated for merging in
this merge window.

--Andy

Re: [PATCH v11 10/13] intel_sgx: driver for Intel Software Guard Extensions

[Dave Hansen]

> +#define sgx_pr_ratelimited(level, encl, fmt, ...)			\
> +	pr_ ## level ## _ratelimited("[%d:0x%p] " fmt,			\
> +				     pid_nr((encl)->tgid),		\
> +				     (void *)(encl)->base, ##__VA_ARGS__)
> +
> +#define sgx_dbg(encl, fmt, ...) \
> +	sgx_pr_ratelimited(debug, encl, fmt, ##__VA_ARGS__)
> +#define sgx_info(encl, fmt, ...) \
> +	sgx_pr_ratelimited(info, encl, fmt, ##__VA_ARGS__)
> +#define sgx_warn(encl, fmt, ...) \
> +	sgx_pr_ratelimited(warn, encl, fmt, ##__VA_ARGS__)
> +#define sgx_err(encl, fmt, ...) \
> +	sgx_pr_ratelimited(err, encl, fmt, ##__VA_ARGS__)
> +#define sgx_crit(encl, fmt, ...) \
> +	sgx_pr_ratelimited(crit, encl, fmt, ##__VA_ARGS__)

I thought the pr_* thingies were to keep everyone from having to do this
in each driver.  Why did you need this?

Can you do any better than a 2,000-line patch?  For instance, could you
break out the memory management portion into its own part and have that
reviewed by mm folks?  Or the ioctl()'s by device driver folks?

Re: [PATCH v11 12/13] intel_sgx: driver documentation

[Randy Dunlap]

On 06/08/2018 10:09 AM, Jarkko Sakkinen wrote:
> Documentation of the features of the  Software Guard eXtensions usable
> for the Linux kernel and how the driver internals uses these features.
> In addition, contains documentation for the ioctl API.
> 
> Signed-off-by: Jarkko Sakkinen <jarkko.sakkinen@linux.intel.com>

Hi,

I have a few corrections below...


> ---
>  Documentation/index.rst         |   1 +
>  Documentation/x86/intel_sgx.rst | 195 ++++++++++++++++++++++++++++++++
>  2 files changed, 196 insertions(+)
>  create mode 100644 Documentation/x86/intel_sgx.rst
> 
> diff --git a/Documentation/index.rst b/Documentation/index.rst
> index 3b99ab931d41..b9fb92928e8c 100644
> --- a/Documentation/index.rst
> +++ b/Documentation/index.rst
> @@ -100,6 +100,7 @@ implementation.
>     :maxdepth: 2
>  
>     sh/index
> +   x86/index
>  
>  Korean translations
>  -------------------
> diff --git a/Documentation/x86/intel_sgx.rst b/Documentation/x86/intel_sgx.rst
> new file mode 100644
> index 000000000000..ecbe544eb2cb
> --- /dev/null
> +++ b/Documentation/x86/intel_sgx.rst
> @@ -0,0 +1,195 @@
> +===================
> +Intel(R) SGX driver
> +===================
> +
> +Introduction
> +============
> +
> +Intel(R) SGX is a set of CPU instructions that can be used by applications to
> +set aside private regions of code and data. The code outside the enclave is
> +disallowed to access the memory inside the enclave by the CPU access control.
> +In a way you can think that SGX provides inverted sandbox. It protects the
> +application from a malicious host.
> +
> +You can tell if your CPU supports SGX by looking into ``/proc/cpuinfo``:
> +
> +	``cat /proc/cpuinfo  | grep sgx``
> +
> +Overview of SGX
> +===============
> +
> +SGX has a set of data structures to maintain information about the enclaves and
> +their security properties. BIOS reserves a fixed size region of physical memory
> +for these structures by setting Processor Reserved Memory Range Registers
> +(PRMRR).
> +
> +This memory range is protected from outside access by the CPU and all the data
> +coming in and out of the CPU package is encrypted by a key that is generated for
> +each boot cycle.
> +
> +Enclaves execute in ring-3 in a special enclave submode using pages from the
> +reserved memory range. A fixed logical address range for the enclave is reserved
> +by ENCLS(ECREATE), a leaf instruction used to create enclaves. It is referred in
> +the documentation commonly as the ELRANGE.
> +
> +Every memory access to the ELRANGE is asserted by the CPU. If the CPU is not
> +executing in the enclave mode inside the enclave, #GP is raised. On the other
> +hand enclave code can make memory accesses both inside and outside of the
> +ELRANGE.
> +
> +Enclave can only execute code inside the ELRANGE. Instructions that may cause
> +VMEXIT, IO instructions and instructions that require a privilege change are
> +prohibited inside the enclave. Interrupts and exceptions always cause enclave
> +to exit and jump to an address outside the enclave given when the enclave is
> +entered by using the leaf instruction ENCLS(EENTER).
> +
> +Data types
> +----------
> +
> +The protected memory range contains the following data:
> +
> +* **Enclave Page Cache (EPC):** protected pages
> +* **Enclave Page Cache Map (EPCM):** a database that describes the state of the
> +  pages and link them to an enclave.
> +
> +EPC has a number of different types of pages:
> +
> +* **SGX Enclave Control Structure (SECS)**: describes the global
> +  properties of an enclave.
> +* **Regular (REG):** code and data pages in the ELRANGE.
> +* **Thread Control Structure (TCS):** pages that define entry points inside an
> +  enclave. The enclave can only be entered through these entry points and each
> +  can host a single hardware thread at a time.
> +* **Version Array (VA)**: 64-bit version numbers for pages that have been
> +  swapped outside the enclave. Each page contains 512 version numbers.
> +
> +Launch control
> +--------------
> +
> +To launch an enclave, two structures must be provided for ENCLS(EINIT):
> +
> +1. **SIGSTRUCT:** signed measurement of the enclave binary.
> +2. **EINITTOKEN:** a cryptographic token CMAC-signed with a AES256-key called
> +   *launch key*, which is re-generated for each boot cycle.
> +
> +The CPU holds a SHA256 hash of a 3072-bit RSA public key inside
> +IA32_SGXLEPUBKEYHASHn MSRs. Enclaves with a SIGSTRUCT that is signed with this
> +key do not require a valid EINITTOKEN and can be authorized with special
> +privileges. One of those privileges is ability to acquire the launch key with
> +ENCLS(EGETKEY).
> +
> +**IA32_FEATURE_CONTROL[17]** is used by to BIOS configure whether

                                        by the BIOS to configure whether

> +IA32_SGXLEPUBKEYHASH MSRs are read-only or read-write before locking the
> +feature control register and handing over control to the operating system.
> +
> +Enclave construction
> +--------------------
> +
> +The construction is started by filling out the SECS that contains enclave
> +address range, privileged attributes and measurement of TCS and REG pages (pages
> +that will be mapped to the address range) among the other things. This structure
> +is passed out to the ENCLS(ECREATE) together with a physical address of a page
> +in EPC that will hold the SECS.
> +
> +Then pages are added with ENCLS(EADD) and measured with ENCLS(EEXTEND).  Finally

"measured"?  what does that mean?

> +enclave is initialized with ENCLS(EINIT). ENCLS(INIT) checks that the SIGSTRUCT
> +is signed with the contained public key and that the supplied EINITTOKEN is
> +valid (CMAC'd with the launch key). If these hold, the enclave is successfully
> +initialized.
> +
> +Swapping pages
> +--------------
> +
> +Enclave pages can be swapped out with ENCLS(EWB) to the unprotected memory. In
> +addition to the EPC page, ENCLS(EWB) takes in a VA page and address for PCMD
> +structure (Page Crypto MetaData) as input. The VA page will seal a version
> +number for the page. PCMD is 128 byte structure that contains tracking
> +information for the page, most importantly its MAC. With these structures the
> +enclave is sealed and rollback protected while it resides in the unprotected
> +memory.
> +
> +Before the page can be swapped out it must not have any active TLB references.
> +By using ENCLS(EBLOCK) instructions no new TLB entries can be created to it.
> +After this the a counter called *epoch* associated hardware threads inside the

huh?

> +enclave is increased with ENCLS(ETRACK). After all the threads from the previous
> +epoch have exited the page can be safely swapped out.
> +
> +An enclave memory access to a swapped out pages will cause #PF. #PF handler can
> +fault the page back by using ENCLS(ELDU).
> +
> +Kernel internals
> +================
> +
> +Requirements
> +------------
> +
> +Because SGX has an ever evolving and expanding feature set, it's possible for
> +a BIOS or VMM to configure a system in such a way that not all cpus are equal,

                                                                  CPUs

> +e.g. where Launch Control is only enabled on a subset of cpus.  Linux does

                                                            CPUs.

> +*not* support such a heterogenous system configuration, nor does it even

                        heterogeneous

> +attempt to play nice in the face of a misconfigured system.  With the exception
> +of Launch Control's hash MSRs, which can vary per cpu, Linux assumes that all

                                                     CPU,

> +cpus have a configuration that is identical to the boot cpu.

   CPUs                                                    CPU.

> +
> +
> +Roles and responsibilities
> +--------------------------
> +
> +SGX introduces system resources, e.g. EPC memory, that must be accessible to
> +multiple entities, e.g. the native kernel driver (to expose SGX to userspace)
> +and KVM (to expose SGX to VMs), ideally without introducing any dependencies
> +between each SGX entity.  To that end, the kernel owns and manages the shared
> +system resources, i.e. the EPC and Launch Control MSRs, and defines functions
> +that provide appropriate access to the shared resources.  SGX support for
> +userpace and VMs is left to the SGX platform driver and KVM respectively.

   userspace

> +
> +Launching enclaves
> +------------------
> +
> +For privileged enclaves the launch is performed simply by submitting the
> +SIGSTRUCT for that enclave to ENCLS(EINIT). For unprivileged enclaves the
> +driver hosts a process in ring-3 that hosts a launch enclave signed with a key
> +supplied for kbuild.
> +
> +The current implementation of the launch enclave generates a token for any
> +enclave. In the future it could be potentially extended to have ways to
> +configure policy what can be lauched.

                                launched.

> +
> +The driver will fail to initialize if it cannot start its own launch enclave.
> +A user space application can submit a SIGSTRUCT instance through the ioctl API.
> +The kernel will take care of the rest.
> +
> +This design assures that the Linux kernel has always full control, which
> +enclaves get to launch and which do not, even if the public key MSRs are
> +read-only. Having launch intrinsics inside the kernel also enables easy
> +development of enclaves without necessarily needing any heavy weight SDK.
> +Having a low-barrier to implement enclaves could make sense for example for

            low barrier

> +system daemons where amount of dependecies ought to be minimized.

                                  dependencies

> +
> +EPC management
> +--------------
> +
> +Due to the unique requirements for swapping EPC pages, and because EPC pages
> +(currently) do not have associated page structures, management of the EPC is
> +not handled by the standard Linux swapper.  SGX directly handles swapping
> +of EPC pages, including a kthread to initiate reclaim and a rudimentary LRU
> +mechanism.  Consumsers of EPC pages, e.g. the SGX driver, are required to

               Consumers

> +implement function callbacks that can be invoked by the kernel to age,
> +swap, and/or forcefully reclaim a target EPC page.  In effect, the kernel
> +controls what happens and when, while the consumers (driver, KVM, etc..) do
> +the actual work.
> +
> +SGX uapi
> +========
> +
> +.. kernel-doc:: drivers/platform/x86/intel_sgx/sgx_ioctl.c
> +   :functions: sgx_ioc_enclave_create
> +               sgx_ioc_enclave_add_page
> +               sgx_ioc_enclave_init
> +
> +.. kernel-doc:: arch/x86/include/uapi/asm/sgx.h
> +
> +References
> +==========
> +
> +* System Programming Manual: 39.1.4 Intel® SGX Launch Control Configuration
> 


-- 
~Randy

Re: [PATCH v11 12/13] intel_sgx: driver documentation

[Randy Dunlap]

On 06/08/2018 10:09 AM, Jarkko Sakkinen wrote:
> Documentation of the features of the  Software Guard eXtensions usable
> for the Linux kernel and how the driver internals uses these features.
> In addition, contains documentation for the ioctl API.
> 
> Signed-off-by: Jarkko Sakkinen <jarkko.sakkinen@linux.intel.com>

Hi,

I have a few corrections below...


> ---
>  Documentation/index.rst         |   1 +
>  Documentation/x86/intel_sgx.rst | 195 ++++++++++++++++++++++++++++++++
>  2 files changed, 196 insertions(+)
>  create mode 100644 Documentation/x86/intel_sgx.rst
> 
> diff --git a/Documentation/index.rst b/Documentation/index.rst
> index 3b99ab931d41..b9fb92928e8c 100644
> --- a/Documentation/index.rst
> +++ b/Documentation/index.rst
> @@ -100,6 +100,7 @@ implementation.
>     :maxdepth: 2
>  
>     sh/index
> +   x86/index
>  
>  Korean translations
>  -------------------
> diff --git a/Documentation/x86/intel_sgx.rst b/Documentation/x86/intel_sgx.rst
> new file mode 100644
> index 000000000000..ecbe544eb2cb
> --- /dev/null
> +++ b/Documentation/x86/intel_sgx.rst
> @@ -0,0 +1,195 @@
> +===================
> +Intel(R) SGX driver
> +===================
> +
> +Introduction
> +============
> +
> +Intel(R) SGX is a set of CPU instructions that can be used by applications to
> +set aside private regions of code and data. The code outside the enclave is
> +disallowed to access the memory inside the enclave by the CPU access control.
> +In a way you can think that SGX provides inverted sandbox. It protects the
> +application from a malicious host.
> +
> +You can tell if your CPU supports SGX by looking into ``/proc/cpuinfo``:
> +
> +	``cat /proc/cpuinfo  | grep sgx``
> +
> +Overview of SGX
> +===============
> +
> +SGX has a set of data structures to maintain information about the enclaves and
> +their security properties. BIOS reserves a fixed size region of physical memory
> +for these structures by setting Processor Reserved Memory Range Registers
> +(PRMRR).
> +
> +This memory range is protected from outside access by the CPU and all the data
> +coming in and out of the CPU package is encrypted by a key that is generated for
> +each boot cycle.
> +
> +Enclaves execute in ring-3 in a special enclave submode using pages from the
> +reserved memory range. A fixed logical address range for the enclave is reserved
> +by ENCLS(ECREATE), a leaf instruction used to create enclaves. It is referred in
> +the documentation commonly as the ELRANGE.
> +
> +Every memory access to the ELRANGE is asserted by the CPU. If the CPU is not
> +executing in the enclave mode inside the enclave, #GP is raised. On the other
> +hand enclave code can make memory accesses both inside and outside of the
> +ELRANGE.
> +
> +Enclave can only execute code inside the ELRANGE. Instructions that may cause
> +VMEXIT, IO instructions and instructions that require a privilege change are
> +prohibited inside the enclave. Interrupts and exceptions always cause enclave
> +to exit and jump to an address outside the enclave given when the enclave is
> +entered by using the leaf instruction ENCLS(EENTER).
> +
> +Data types
> +----------
> +
> +The protected memory range contains the following data:
> +
> +* **Enclave Page Cache (EPC):** protected pages
> +* **Enclave Page Cache Map (EPCM):** a database that describes the state of the
> +  pages and link them to an enclave.
> +
> +EPC has a number of different types of pages:
> +
> +* **SGX Enclave Control Structure (SECS)**: describes the global
> +  properties of an enclave.
> +* **Regular (REG):** code and data pages in the ELRANGE.
> +* **Thread Control Structure (TCS):** pages that define entry points inside an
> +  enclave. The enclave can only be entered through these entry points and each
> +  can host a single hardware thread at a time.
> +* **Version Array (VA)**: 64-bit version numbers for pages that have been
> +  swapped outside the enclave. Each page contains 512 version numbers.
> +
> +Launch control
> +--------------
> +
> +To launch an enclave, two structures must be provided for ENCLS(EINIT):
> +
> +1. **SIGSTRUCT:** signed measurement of the enclave binary.
> +2. **EINITTOKEN:** a cryptographic token CMAC-signed with a AES256-key called
> +   *launch key*, which is re-generated for each boot cycle.
> +
> +The CPU holds a SHA256 hash of a 3072-bit RSA public key inside
> +IA32_SGXLEPUBKEYHASHn MSRs. Enclaves with a SIGSTRUCT that is signed with this
> +key do not require a valid EINITTOKEN and can be authorized with special
> +privileges. One of those privileges is ability to acquire the launch key with
> +ENCLS(EGETKEY).
> +
> +**IA32_FEATURE_CONTROL[17]** is used by to BIOS configure whether

                                        by the BIOS to configure whether

> +IA32_SGXLEPUBKEYHASH MSRs are read-only or read-write before locking the
> +feature control register and handing over control to the operating system.
> +
> +Enclave construction
> +--------------------
> +
> +The construction is started by filling out the SECS that contains enclave
> +address range, privileged attributes and measurement of TCS and REG pages (pages
> +that will be mapped to the address range) among the other things. This structure
> +is passed out to the ENCLS(ECREATE) together with a physical address of a page
> +in EPC that will hold the SECS.
> +
> +Then pages are added with ENCLS(EADD) and measured with ENCLS(EEXTEND).  Finally

"measured"?  what does that mean?

> +enclave is initialized with ENCLS(EINIT). ENCLS(INIT) checks that the SIGSTRUCT
> +is signed with the contained public key and that the supplied EINITTOKEN is
> +valid (CMAC'd with the launch key). If these hold, the enclave is successfully
> +initialized.
> +
> +Swapping pages
> +--------------
> +
> +Enclave pages can be swapped out with ENCLS(EWB) to the unprotected memory. In
> +addition to the EPC page, ENCLS(EWB) takes in a VA page and address for PCMD
> +structure (Page Crypto MetaData) as input. The VA page will seal a version
> +number for the page. PCMD is 128 byte structure that contains tracking
> +information for the page, most importantly its MAC. With these structures the
> +enclave is sealed and rollback protected while it resides in the unprotected
> +memory.
> +
> +Before the page can be swapped out it must not have any active TLB references.
> +By using ENCLS(EBLOCK) instructions no new TLB entries can be created to it.
> +After this the a counter called *epoch* associated hardware threads inside the

huh?

> +enclave is increased with ENCLS(ETRACK). After all the threads from the previous
> +epoch have exited the page can be safely swapped out.
> +
> +An enclave memory access to a swapped out pages will cause #PF. #PF handler can
> +fault the page back by using ENCLS(ELDU).
> +
> +Kernel internals
> +================
> +
> +Requirements
> +------------
> +
> +Because SGX has an ever evolving and expanding feature set, it's possible for
> +a BIOS or VMM to configure a system in such a way that not all cpus are equal,

                                                                  CPUs

> +e.g. where Launch Control is only enabled on a subset of cpus.  Linux does

                                                            CPUs.

> +*not* support such a heterogenous system configuration, nor does it even

                        heterogeneous

> +attempt to play nice in the face of a misconfigured system.  With the exception
> +of Launch Control's hash MSRs, which can vary per cpu, Linux assumes that all

                                                     CPU,

> +cpus have a configuration that is identical to the boot cpu.

   CPUs                                                    CPU.

> +
> +
> +Roles and responsibilities
> +--------------------------
> +
> +SGX introduces system resources, e.g. EPC memory, that must be accessible to
> +multiple entities, e.g. the native kernel driver (to expose SGX to userspace)
> +and KVM (to expose SGX to VMs), ideally without introducing any dependencies
> +between each SGX entity.  To that end, the kernel owns and manages the shared
> +system resources, i.e. the EPC and Launch Control MSRs, and defines functions
> +that provide appropriate access to the shared resources.  SGX support for
> +userpace and VMs is left to the SGX platform driver and KVM respectively.

   userspace

> +
> +Launching enclaves
> +------------------
> +
> +For privileged enclaves the launch is performed simply by submitting the
> +SIGSTRUCT for that enclave to ENCLS(EINIT). For unprivileged enclaves the
> +driver hosts a process in ring-3 that hosts a launch enclave signed with a key
> +supplied for kbuild.
> +
> +The current implementation of the launch enclave generates a token for any
> +enclave. In the future it could be potentially extended to have ways to
> +configure policy what can be lauched.

                                launched.

> +
> +The driver will fail to initialize if it cannot start its own launch enclave.
> +A user space application can submit a SIGSTRUCT instance through the ioctl API.
> +The kernel will take care of the rest.
> +
> +This design assures that the Linux kernel has always full control, which
> +enclaves get to launch and which do not, even if the public key MSRs are
> +read-only. Having launch intrinsics inside the kernel also enables easy
> +development of enclaves without necessarily needing any heavy weight SDK.
> +Having a low-barrier to implement enclaves could make sense for example for

            low barrier

> +system daemons where amount of dependecies ought to be minimized.

                                  dependencies

> +
> +EPC management
> +--------------
> +
> +Due to the unique requirements for swapping EPC pages, and because EPC pages
> +(currently) do not have associated page structures, management of the EPC is
> +not handled by the standard Linux swapper.  SGX directly handles swapping
> +of EPC pages, including a kthread to initiate reclaim and a rudimentary LRU
> +mechanism.  Consumsers of EPC pages, e.g. the SGX driver, are required to

               Consumers

> +implement function callbacks that can be invoked by the kernel to age,
> +swap, and/or forcefully reclaim a target EPC page.  In effect, the kernel
> +controls what happens and when, while the consumers (driver, KVM, etc..) do
> +the actual work.
> +
> +SGX uapi
> +========
> +
> +.. kernel-doc:: drivers/platform/x86/intel_sgx/sgx_ioctl.c
> +   :functions: sgx_ioc_enclave_create
> +               sgx_ioc_enclave_add_page
> +               sgx_ioc_enclave_init
> +
> +.. kernel-doc:: arch/x86/include/uapi/asm/sgx.h
> +
> +References
> +==========
> +
> +* System Programming Manual: 39.1.4 Intel® SGX Launch Control Configuration
> 


-- 
~Randy
--
To unsubscribe from this list: send the line "unsubscribe linux-doc" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [intel-sgx-kernel-dev] [PATCH v11 09/13] x86, sgx: basic routines for enclave page cache

[Andy Lutomirski]

On Fri, Jun 8, 2018 at 10:22 AM Jarkko Sakkinen
<jarkko.sakkinen@linux.intel.com> wrote:
>
> SGX has a set of data structures to maintain information about the enclaves
> and their security properties. BIOS reserves a fixed size region of
> physical memory for these structures by setting Processor Reserved Memory
> Range Registers (PRMRR). This memory area is called Enclave Page Cache
> (EPC).
>

> +/**
> + * sgx_einit - EINIT an enclave with the appropriate LE pubkey hash
> + * @sigstruct:         a pointer to the enclave's sigstruct
> + * @token:             a pointer to the enclave's EINIT token
> + * @secs_page:         a pointer to the enclave's SECS EPC page
> + * @le_pubkey_hash:    the desired LE pubkey hash for EINIT
> + */
> +int sgx_einit(struct sgx_sigstruct *sigstruct, struct sgx_einittoken *token,
> +             struct sgx_epc_page *secs_page, u64 le_pubkey_hash[4])
> +{
> +       u64 __percpu *cache;
> +       void *secs;
> +       int i, ret;
> +
> +       secs = sgx_get_page(secs_page);
> +
> +       if (!sgx_lc_enabled) {

I'm confused.  What does this code path do?  It kind of looks like the
driver will load and just malfunction if we don't have write access to
the MSRs.  What is the intended behavior?

> +               ret = __einit(sigstruct, token, secs);
> +               goto out;
> +       }
> +
> +       cache = per_cpu(sgx_le_pubkey_hash_cache, smp_processor_id());
> +
> +       preempt_disable();
> +       for (i = 0; i < 4; i++) {
> +               if (le_pubkey_hash[i] == cache[i])
> +                       continue;
> +
> +               wrmsrl(MSR_IA32_SGXLEPUBKEYHASH0 + i, le_pubkey_hash[i]);
> +               cache[i] = le_pubkey_hash[i];
> +       }
> +       ret = __einit(sigstruct, token, secs);
> +       preempt_enable();
> +
> +out:
> +       sgx_put_page(secs);
> +       return ret;
> +}
> +EXPORT_SYMBOL(sgx_einit);
> +

Re: [intel-sgx-kernel-dev] [PATCH v11 13/13] intel_sgx: in-kernel launch enclave

[Andy Lutomirski]

On Fri, Jun 8, 2018 at 10:32 AM Jarkko Sakkinen
<jarkko.sakkinen@linux.intel.com> wrote:
>
> The Launch Enclave (LE) generates cryptographic launch tokens for user
> enclaves. A launch token is used by EINIT to check whether the enclave
> is authorized to launch or not. By having its own launch enclave, Linux
> has full control of the enclave launch process.
>
> LE is wrapped into a user space proxy program that reads enclave
> signatures outputs launch tokens. The kernel-side glue code is
> implemented by using the user space helper framework.  The IPC between
> the LE proxy program and kernel is handled with an anonymous inode.
>
> The commit also adds enclave signing tool that is used by kbuild to
> measure and sign the launch enclave. CONFIG_INTEL_SGX_SIGNING_KEY points
> to a PEM-file for the 3072-bit RSA key that is used as the LE public key
> pair. The default location is:
>
>   drivers/platform/x86/intel_sgx/sgx_signing_key.pem
>
> If the default key does not exist kbuild will generate a random key and
> place it to this location. KBUILD_SGX_SIGN_PIN can be used to specify
> the passphrase for the LE public key.

It seems to me that it might be more useful to just commit a key pair
into the kernel.  As far as I know, there is no security whatsoever
gained by keeping the private key private, so why not make
reproducible builds easier by simply fixing the key?

Also, this email is so long that gmail won't let me quote the relevant
code, but: what is the intended use case for supporting the mode where
the MSRs are locked but happen to contain the right value?  I could
see the case for bundling a key with the kernel and literally
hard-coding the acceptable MSR values (as in literal values in the
code, not even autogenerated hashes).  The only use case I've thought
of for the code as it stands is that $VENDOR could publish their LE
public key and some daft firmware vendor could get it into their head
that it would be a good idea to lock the MSRs to that value.  This
would add no security at all, but it would add a considerable about of
annoyance and loss of value, so I still tend to think that we
shouldn't support it.

Re: [intel-sgx-kernel-dev] [PATCH v11 13/13] intel_sgx: in-kernel launch enclave

[Andy Lutomirski]

> On Jun 9, 2018, at 10:39 PM, Andy Lutomirski <luto@kernel.org> wrote:
>
> On Fri, Jun 8, 2018 at 10:32 AM Jarkko Sakkinen
> <jarkko.sakkinen@linux.intel.com> wrote:
>>
>> The Launch Enclave (LE) generates cryptographic launch tokens for user
>> enclaves. A launch token is used by EINIT to check whether the enclave
>> is authorized to launch or not. By having its own launch enclave, Linux
>> has full control of the enclave launch process.
>>
>> LE is wrapped into a user space proxy program that reads enclave
>> signatures outputs launch tokens. The kernel-side glue code is
>> implemented by using the user space helper framework.  The IPC between
>> the LE proxy program and kernel is handled with an anonymous inode.
>>
>> The commit also adds enclave signing tool that is used by kbuild to
>> measure and sign the launch enclave. CONFIG_INTEL_SGX_SIGNING_KEY points
>> to a PEM-file for the 3072-bit RSA key that is used as the LE public key
>> pair. The default location is:
>>
>>  drivers/platform/x86/intel_sgx/sgx_signing_key.pem
>>
>> If the default key does not exist kbuild will generate a random key and
>> place it to this location. KBUILD_SGX_SIGN_PIN can be used to specify
>> the passphrase for the LE public key.
>
> It seems to me that it might be more useful to just commit a key pair
> into the kernel.  As far as I know, there is no security whatsoever
> gained by keeping the private key private, so why not make
> reproducible builds easier by simply fixing the key?

Having thought about this some more, I think that you should
completely remove support for specifying a key. Provide a fixed key
pair, hard code the cache, and call it a day. If you make the key
configurable, every vendor that has any vendor keys (Debian, Ubuntu,
Fedora, Red Hat, SuSE, Clear Linux, etc) will see that config option
and set up their own key pair for no gain whatsoever.  Instead, it'll
give some illusion of security and it'll slow down operations in a VM
guest due to swapping out the values of the MSRs.  And, if the code to
support a locked MSR that just happens to have the right value stays
in the kernel, then we'll risk having vendors actually ship one
distro's public key hash, and that will seriously suck.

I'm going to try to get this code working tomorrow.  I'll keep you
posted on how that goes.  Can you point me to the userspace bits (i.e.
something buildable that will run on a kernel with your patches
applied)?

>
> Also, this email is so long that gmail won't let me quote the relevant
> code, but: what is the intended use case for supporting the mode where
> the MSRs are locked but happen to contain the right value?  I could
> see the case for bundling a key with the kernel and literally
> hard-coding the acceptable MSR values (as in literal values in the
> code, not even autogenerated hashes).  The only use case I've thought
> of for the code as it stands is that $VENDOR could publish their LE
> public key and some daft firmware vendor could get it into their head
> that it would be a good idea to lock the MSRs to that value.  This
> would add no security at all, but it would add a considerable about of
> annoyance and loss of value, so I still tend to think that we
> shouldn't support it.

Re: [PATCH v11 07/13] x86, sgx: detect Intel SGX

[Neil Horman]

On Fri, Jun 08, 2018 at 07:09:42PM +0200, Jarkko Sakkinen wrote:
> From: Sean Christopherson <sean.j.christopherson@intel.com>
> 
> Intel(R) SGX is a set of CPU instructions that can be used by applications
> to set aside private regions of code and data. The code outside the enclave
> is disallowed to access the memory inside the enclave by the CPU access
> control.
> 
> This commit adds the check for SGX to arch/x86 and a new config option,
> INTEL_SGX_CORE. Exposes a boolean variable 'sgx_enabled' to query whether
> or not the SGX support is available.
> 
> Signed-off-by: Sean Christopherson <sean.j.christopherson@intel.com>
> Reviewed-by: Jarkko Sakkinen <jarkko.sakkinen@linux.intel.com>
> Tested-by: Jarkko Sakkinen <jarkko.sakkinen@linux.intel.com>
> Signed-off-by: Jarkko Sakkinen <jarkko.sakkinen@linux.intel.com>
> ---
>  arch/x86/Kconfig                | 19 ++++++++++++
>  arch/x86/include/asm/sgx.h      | 25 ++++++++++++++++
>  arch/x86/include/asm/sgx_pr.h   | 20 +++++++++++++
>  arch/x86/kernel/cpu/Makefile    |  1 +
>  arch/x86/kernel/cpu/intel_sgx.c | 53 +++++++++++++++++++++++++++++++++
>  5 files changed, 118 insertions(+)
>  create mode 100644 arch/x86/include/asm/sgx.h
>  create mode 100644 arch/x86/include/asm/sgx_pr.h
>  create mode 100644 arch/x86/kernel/cpu/intel_sgx.c
> 
> diff --git a/arch/x86/Kconfig b/arch/x86/Kconfig
> index c07f492b871a..42015d5366ef 100644
> --- a/arch/x86/Kconfig
> +++ b/arch/x86/Kconfig
> @@ -1925,6 +1925,25 @@ config X86_INTEL_MEMORY_PROTECTION_KEYS
>  
>  	  If unsure, say y.
>  
> +config INTEL_SGX_CORE
> +	prompt "Intel SGX core functionality"
> +	def_bool n
> +	depends on X86_64 && CPU_SUP_INTEL
> +	help
> +	Intel Software Guard eXtensions (SGX) is a set of CPU instructions
> +	that allows ring 3 applications to create enclaves; private regions
> +	of memory that are protected, by hardware, from unauthorized access
> +	and/or modification.
> +
> +	This option enables kernel recognition of SGX, high-level management
> +	of the Enclave Page Cache (EPC), tracking and writing of SGX Launch
> +	Enclave Hash MSRs, and allows for virtualization of SGX via KVM. By
> +	iteslf, this option does not provide SGX support to userspace.
> +
> +	For details, see Documentation/x86/intel_sgx.rst
> +
> +	If unsure, say N.
> +
>  config EFI
>  	bool "EFI runtime service support"
>  	depends on ACPI
> diff --git a/arch/x86/include/asm/sgx.h b/arch/x86/include/asm/sgx.h
> new file mode 100644
> index 000000000000..fa3e6e0eb8af
> --- /dev/null
> +++ b/arch/x86/include/asm/sgx.h
> @@ -0,0 +1,25 @@
> +// SPDX-License-Identifier: (GPL-2.0 OR BSD-3-Clause)
> +// Copyright(c) 2016-18 Intel Corporation.
> +//
> +// Authors:
> +//
> +// Jarkko Sakkinen <jarkko.sakkinen@linux.intel.com>
> +// Suresh Siddha <suresh.b.siddha@intel.com>
> +// Sean Christopherson <sean.j.christopherson@intel.com>
> +
> +#ifndef _ASM_X86_SGX_H
> +#define _ASM_X86_SGX_H
> +
> +#include <linux/types.h>
> +
> +#define SGX_CPUID 0x12
> +
Agree with Dave, this can just be remoed and you can use the feature macro from
cpuid.h instead

Neil

Re: [intel-sgx-kernel-dev] [PATCH v11 13/13] intel_sgx: in-kernel launch enclave

[Neil Horman]

On Sun, Jun 10, 2018 at 10:17:13PM -0700, Andy Lutomirski wrote:
> > On Jun 9, 2018, at 10:39 PM, Andy Lutomirski <luto@kernel.org> wrote:
> >
> > On Fri, Jun 8, 2018 at 10:32 AM Jarkko Sakkinen
> > <jarkko.sakkinen@linux.intel.com> wrote:
> >>
> >> The Launch Enclave (LE) generates cryptographic launch tokens for user
> >> enclaves. A launch token is used by EINIT to check whether the enclave
> >> is authorized to launch or not. By having its own launch enclave, Linux
> >> has full control of the enclave launch process.
> >>
> >> LE is wrapped into a user space proxy program that reads enclave
> >> signatures outputs launch tokens. The kernel-side glue code is
> >> implemented by using the user space helper framework.  The IPC between
> >> the LE proxy program and kernel is handled with an anonymous inode.
> >>
> >> The commit also adds enclave signing tool that is used by kbuild to
> >> measure and sign the launch enclave. CONFIG_INTEL_SGX_SIGNING_KEY points
> >> to a PEM-file for the 3072-bit RSA key that is used as the LE public key
> >> pair. The default location is:
> >>
> >>  drivers/platform/x86/intel_sgx/sgx_signing_key.pem
> >>
> >> If the default key does not exist kbuild will generate a random key and
> >> place it to this location. KBUILD_SGX_SIGN_PIN can be used to specify
> >> the passphrase for the LE public key.
> >
> > It seems to me that it might be more useful to just commit a key pair
> > into the kernel.  As far as I know, there is no security whatsoever
> > gained by keeping the private key private, so why not make
> > reproducible builds easier by simply fixing the key?
> 
> Having thought about this some more, I think that you should
> completely remove support for specifying a key. Provide a fixed key
> pair, hard code the cache, and call it a day. If you make the key
> configurable, every vendor that has any vendor keys (Debian, Ubuntu,
> Fedora, Red Hat, SuSE, Clear Linux, etc) will see that config option
> and set up their own key pair for no gain whatsoever.  Instead, it'll
> give some illusion of security and it'll slow down operations in a VM
> guest due to swapping out the values of the MSRs.  And, if the code to
> support a locked MSR that just happens to have the right value stays
> in the kernel, then we'll risk having vendors actually ship one
> distro's public key hash, and that will seriously suck.
> 
If you hard code the key pair however, doesn't that imply that anyone can sign a
user space binary as a launch enclave, and potentially gain control of the token
granting process?  It was my understanding that the value of the key pair was
that the end user was guaranteed autonomy and security over which processes
could start enclaves.  By publishing a fixed key pair, it seems to remove that
ability.

What would be nicer (I think) would be the abilty to specify both the public and
the private key at run time.  the use case here is not one in which a vendor or
os distribution ships a key pair, but one in which a downstream user doesn't
want a vendor/os distribution to have any cryptographic information installed on
their system

Neil

Re: [PATCH v11 11/13] intel_sgx: ptrace() support

[Sean Christopherson]

On Fri, 2018-06-08 at 11:34 -0700, Dave Hansen wrote:
> On 06/08/2018 10:09 AM, Jarkko Sakkinen wrote:
> > 
> > +		ret = sgx_edbgrd(encl, entry, align, data);
> > +		if (ret)
> > +			break;
> > +		if (write) {
> > +			memcpy(data + offset, buf + i, cnt);
> > +			ret = sgx_edbgwr(encl, entry, align, data);
> > +			if (ret)
> > +				break;
> > +		}
> > +		else
> > +			memcpy(buf + i,data + offset, cnt);
> > +	}
> The SGX instructions like "edbgrd" be great to put on a license plat,
> but we can do better in the kernel.  Can you give these reasonable
> english names, please?  sgx_debug_write(), maybe?

IMO the function names for ENCLS leafs are appropriate.  The real
issue is the lack of documentation of the ENCLS helpers and their
naming conventions.

The sgx_<leaf> functions, e.g. sgx_edbgrd(), are essentially direct
invocations of the specific leaf, i.e. they are dumb wrappers to
the lower level leaf functions, e.g. __edbgrd().  The wrappers exist
primarily to deal with the boilerplate necessary to access a page in
the EPC.  sgx_<leaf> conveys that the function contains the preamble
and/or postamble needed to execute its leaf, but otherwise does not
contain any logic.

Functions with actual logic do have English names, e.g.
sgx_encl_init(), sgx_encl_add_page(), sgx_encl_modify_pages() etc...

> Note that we have plenty of incomprehensible instruction names in the
> kernel like "wrpkru", but we do our best to keep them as confined as
> possible and make sure they don't hurt code readability.

Re: [intel-sgx-kernel-dev] [PATCH v11 09/13] x86, sgx: basic routines for enclave page cache

[Sean Christopherson]

On Sat, 2018-06-09 at 22:32 -0700, Andy Lutomirski wrote:
> On Fri, Jun 8, 2018 at 10:22 AM Jarkko Sakkinen
> <jarkko.sakkinen@linux.intel.com> wrote:
> > 
> > 
> > SGX has a set of data structures to maintain information about the enclaves
> > and their security properties. BIOS reserves a fixed size region of
> > physical memory for these structures by setting Processor Reserved Memory
> > Range Registers (PRMRR). This memory area is called Enclave Page Cache
> > (EPC).
> > 
> > 
> > +/**
> > + * sgx_einit - EINIT an enclave with the appropriate LE pubkey hash
> > + * @sigstruct:         a pointer to the enclave's sigstruct
> > + * @token:             a pointer to the enclave's EINIT token
> > + * @secs_page:         a pointer to the enclave's SECS EPC page
> > + * @le_pubkey_hash:    the desired LE pubkey hash for EINIT
> > + */
> > +int sgx_einit(struct sgx_sigstruct *sigstruct, struct sgx_einittoken *token,
> > +             struct sgx_epc_page *secs_page, u64 le_pubkey_hash[4])
> > +{
> > +       u64 __percpu *cache;
> > +       void *secs;
> > +       int i, ret;
> > +
> > +       secs = sgx_get_page(secs_page);
> > +
> > +       if (!sgx_lc_enabled) {
> I'm confused.  What does this code path do?  It kind of looks like the
> driver will load and just malfunction if we don't have write access to
> the MSRs.  What is the intended behavior?

The driver will also allow itself to load if the MSRs are read-only,
but only if the MSRs' pubkey hash matches that of its launch enclave,
i.e. the system has been pre-configured for the kernel's LE.  Whether
or not that is a valid scenario is probably a different discussion.

> > +               ret = __einit(sigstruct, token, secs);
> > +               goto out;
> > +       }
> > +
> > +       cache = per_cpu(sgx_le_pubkey_hash_cache, smp_processor_id());
> > +
> > +       preempt_disable();
> > +       for (i = 0; i < 4; i++) {
> > +               if (le_pubkey_hash[i] == cache[i])
> > +                       continue;
> > +
> > +               wrmsrl(MSR_IA32_SGXLEPUBKEYHASH0 + i, le_pubkey_hash[i]);
> > +               cache[i] = le_pubkey_hash[i];
> > +       }
> > +       ret = __einit(sigstruct, token, secs);
> > +       preempt_enable();
> > +
> > +out:
> > +       sgx_put_page(secs);
> > +       return ret;
> > +}
> > +EXPORT_SYMBOL(sgx_einit);
> > +

Re: [PATCH v11 06/13] crypto: aesni: add minimal build option for SGX LE

[Sean Christopherson]

On Fri, 2018-06-08 at 10:27 -0700, Dave Hansen wrote:
> On 06/08/2018 10:09 AM, Jarkko Sakkinen wrote:
> > 
> > --- a/arch/x86/crypto/aesni-intel_asm.S
> > +++ b/arch/x86/crypto/aesni-intel_asm.S
> > @@ -45,6 +45,8 @@
> >  #define MOVADQ	movaps
> >  #define MOVUDQ	movups
> >  
> > +#ifndef AESNI_INTEL_MINIMAL
> > +
> >  #ifdef __x86_64__
> >  
> >  # constants in mergeable sections, linker can reorder and merge
> > @@ -133,6 +135,8 @@ ALL_F:      .octa 0xffffffffffffffffffffffffffffffff
> >  #define keysize 2*15*16(%arg1)
> >  #endif
> >  
> > +#endif /* AESNI_INTEL_MINIMAL */
> > +
> I'd really prefer that these get moved into a separate file rather than
> a scattered set of #ifdefs.  This just seem fragile to me.
> 
> Can we have a "aesni-intel_asm-minimal.S"?  Or, at least bunch the
> minimal set of things *together*?

A separate file doesn't seem appropriate because there is no criteria
for including code in the "minimal" build beyond "this code happens to
be needed by SGX".  I considered having SGX somewhere in the define
but opted for AESNI_INTEL_MINIMAL on the off chance that the minimal
build was useful for something other than SGX.

I'm not opposed to bunching the minimal stuff together, my intent was
simply to disturb the code as little as possible.

Re: [intel-sgx-kernel-dev] [PATCH v11 13/13] intel_sgx: in-kernel launch enclave

[Andy Lutomirski]

On Mon, Jun 11, 2018 at 4:52 AM Neil Horman <nhorman@redhat.com> wrote:
>
> On Sun, Jun 10, 2018 at 10:17:13PM -0700, Andy Lutomirski wrote:
> > > On Jun 9, 2018, at 10:39 PM, Andy Lutomirski <luto@kernel.org> wrote:
> > >
> > > On Fri, Jun 8, 2018 at 10:32 AM Jarkko Sakkinen
> > > <jarkko.sakkinen@linux.intel.com> wrote:
> > >>
> > >> The Launch Enclave (LE) generates cryptographic launch tokens for user
> > >> enclaves. A launch token is used by EINIT to check whether the enclave
> > >> is authorized to launch or not. By having its own launch enclave, Linux
> > >> has full control of the enclave launch process.
> > >>
> > >> LE is wrapped into a user space proxy program that reads enclave
> > >> signatures outputs launch tokens. The kernel-side glue code is
> > >> implemented by using the user space helper framework.  The IPC between
> > >> the LE proxy program and kernel is handled with an anonymous inode.
> > >>
> > >> The commit also adds enclave signing tool that is used by kbuild to
> > >> measure and sign the launch enclave. CONFIG_INTEL_SGX_SIGNING_KEY points
> > >> to a PEM-file for the 3072-bit RSA key that is used as the LE public key
> > >> pair. The default location is:
> > >>
> > >>  drivers/platform/x86/intel_sgx/sgx_signing_key.pem
> > >>
> > >> If the default key does not exist kbuild will generate a random key and
> > >> place it to this location. KBUILD_SGX_SIGN_PIN can be used to specify
> > >> the passphrase for the LE public key.
> > >
> > > It seems to me that it might be more useful to just commit a key pair
> > > into the kernel.  As far as I know, there is no security whatsoever
> > > gained by keeping the private key private, so why not make
> > > reproducible builds easier by simply fixing the key?
> >
> > Having thought about this some more, I think that you should
> > completely remove support for specifying a key. Provide a fixed key
> > pair, hard code the cache, and call it a day. If you make the key
> > configurable, every vendor that has any vendor keys (Debian, Ubuntu,
> > Fedora, Red Hat, SuSE, Clear Linux, etc) will see that config option
> > and set up their own key pair for no gain whatsoever.  Instead, it'll
> > give some illusion of security and it'll slow down operations in a VM
> > guest due to swapping out the values of the MSRs.  And, if the code to
> > support a locked MSR that just happens to have the right value stays
> > in the kernel, then we'll risk having vendors actually ship one
> > distro's public key hash, and that will seriously suck.
> >
> If you hard code the key pair however, doesn't that imply that anyone can sign a
> user space binary as a launch enclave, and potentially gain control of the token
> granting process?

Yes and no.

First of all, the kernel driver shouldn't be allowing user code to
launch a launch enclave regardless of signature.  I haven't gotten far
enough in reviewing the code to see whether that's the case, but if
it's not, it should be fixed before it's merged.

But keep in mind that control of the token granting process is not the
same thing as control over the right to launch an enclave.  On systems
without the LE hash MSRs, Intel controls the token granting process
and, barring some attack, an enclave that isn't blessed by Intel can't
be launched.  Support for that model will not be merged into upstream
Linux.  But on systems that have the LE hash MSRs and leave them
unlocked, there is effectively no hardware-enforced launch control.
Instead we have good old kernel policy.  If a user wants to launch an
enclave, they need to get the kernel to launch the enclave, and the
kernel needs to apply its policy.  The patch here (the in-kernel
launch enclave) has a wide-open policy.

So, as a practical matter, if every distro has their own LE key and
keeps it totally safe, then a system that locks the MSRs to one
distro's key makes it quite annoying to run another distro's intel_sgx
driver, but there is no effect on the actual security of the system.

>  It was my understanding that the value of the key pair was
> that the end user was guaranteed autonomy and security over which processes
> could start enclaves.  By publishing a fixed key pair, it seems to remove that
> ability.

If someone comes up with an actual machine that grants the actual end
user (where the end user is the person who bought the thing, not the
OEM) control over the MSRs, *and* the actual end user wants to limit
what enclaves can be launched even if the kernel is compromised, *and*
there is some actual argument for why this is useful (as opposed to
some compliance checkbox), then Linux could reasonably consider adding
support for this use case.  But that would be a separate patch.

>
> What would be nicer (I think) would be the abilty to specify both the public and
> the private key at run time.  the use case here is not one in which a vendor or
> os distribution ships a key pair, but one in which a downstream user doesn't
> want a vendor/os distribution to have any cryptographic information installed on
> their system

For what gain?

Re: [PATCH v11 00/13] Intel SGX1 support

[Pavel Machek]

[-- Attachment #1: Type: text/plain, Size: 769 bytes --]

On Fri 2018-06-08 19:09:35, Jarkko Sakkinen wrote:
> Intel(R) SGX is a set of CPU instructions that can be used by applications
> to set aside private regions of code and data. The code outside the enclave
> is disallowed to access the memory inside the enclave by the CPU access
> control.  In a way you can think that SGX provides inverted sandbox. It
> protects the application from a malicious host.

Do you intend to allow non-root applications to use SGX?

What are non-evil uses for SGX?

...because it is quite useful for some kinds of evil:

https://taesoo.kim/pubs/2017/jang:sgx-bomb.pdf

									Pavel
-- 
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 181 bytes --]

Re: [PATCH v11 00/13] Intel SGX1 support

[Pavel Machek]

[-- Attachment #1: Type: text/plain, Size: 769 bytes --]

On Fri 2018-06-08 19:09:35, Jarkko Sakkinen wrote:
> Intel(R) SGX is a set of CPU instructions that can be used by applications
> to set aside private regions of code and data. The code outside the enclave
> is disallowed to access the memory inside the enclave by the CPU access
> control.  In a way you can think that SGX provides inverted sandbox. It
> protects the application from a malicious host.

Do you intend to allow non-root applications to use SGX?

What are non-evil uses for SGX?

...because it is quite useful for some kinds of evil:

https://taesoo.kim/pubs/2017/jang:sgx-bomb.pdf

									Pavel
-- 
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 181 bytes --]

Re: [intel-sgx-kernel-dev] [PATCH v11 13/13] intel_sgx: in-kernel launch enclave

[Neil Horman]

On Mon, Jun 11, 2018 at 09:55:29PM -0700, Andy Lutomirski wrote:
> On Mon, Jun 11, 2018 at 4:52 AM Neil Horman <nhorman@redhat.com> wrote:
> >
> > On Sun, Jun 10, 2018 at 10:17:13PM -0700, Andy Lutomirski wrote:
> > > > On Jun 9, 2018, at 10:39 PM, Andy Lutomirski <luto@kernel.org> wrote:
> > > >
> > > > On Fri, Jun 8, 2018 at 10:32 AM Jarkko Sakkinen
> > > > <jarkko.sakkinen@linux.intel.com> wrote:
> > > >>
> > > >> The Launch Enclave (LE) generates cryptographic launch tokens for user
> > > >> enclaves. A launch token is used by EINIT to check whether the enclave
> > > >> is authorized to launch or not. By having its own launch enclave, Linux
> > > >> has full control of the enclave launch process.
> > > >>
> > > >> LE is wrapped into a user space proxy program that reads enclave
> > > >> signatures outputs launch tokens. The kernel-side glue code is
> > > >> implemented by using the user space helper framework.  The IPC between
> > > >> the LE proxy program and kernel is handled with an anonymous inode.
> > > >>
> > > >> The commit also adds enclave signing tool that is used by kbuild to
> > > >> measure and sign the launch enclave. CONFIG_INTEL_SGX_SIGNING_KEY points
> > > >> to a PEM-file for the 3072-bit RSA key that is used as the LE public key
> > > >> pair. The default location is:
> > > >>
> > > >>  drivers/platform/x86/intel_sgx/sgx_signing_key.pem
> > > >>
> > > >> If the default key does not exist kbuild will generate a random key and
> > > >> place it to this location. KBUILD_SGX_SIGN_PIN can be used to specify
> > > >> the passphrase for the LE public key.
> > > >
> > > > It seems to me that it might be more useful to just commit a key pair
> > > > into the kernel.  As far as I know, there is no security whatsoever
> > > > gained by keeping the private key private, so why not make
> > > > reproducible builds easier by simply fixing the key?
> > >
> > > Having thought about this some more, I think that you should
> > > completely remove support for specifying a key. Provide a fixed key
> > > pair, hard code the cache, and call it a day. If you make the key
> > > configurable, every vendor that has any vendor keys (Debian, Ubuntu,
> > > Fedora, Red Hat, SuSE, Clear Linux, etc) will see that config option
> > > and set up their own key pair for no gain whatsoever.  Instead, it'll
> > > give some illusion of security and it'll slow down operations in a VM
> > > guest due to swapping out the values of the MSRs.  And, if the code to
> > > support a locked MSR that just happens to have the right value stays
> > > in the kernel, then we'll risk having vendors actually ship one
> > > distro's public key hash, and that will seriously suck.
> > >
> > If you hard code the key pair however, doesn't that imply that anyone can sign a
> > user space binary as a launch enclave, and potentially gain control of the token
> > granting process?
> 
> Yes and no.
> 
> First of all, the kernel driver shouldn't be allowing user code to
> launch a launch enclave regardless of signature.  I haven't gotten far
> enough in reviewing the code to see whether that's the case, but if
> it's not, it should be fixed before it's merged.
> 
Ok, I agree with you here.

> But keep in mind that control of the token granting process is not the
> same thing as control over the right to launch an enclave.  On systems
> without the LE hash MSRs, Intel controls the token granting process
> and, barring some attack, an enclave that isn't blessed by Intel can't
> be launched.  Support for that model will not be merged into upstream
> Linux.  But on systems that have the LE hash MSRs and leave them
> unlocked, there is effectively no hardware-enforced launch control.
> Instead we have good old kernel policy.  If a user wants to launch an
> enclave, they need to get the kernel to launch the enclave, and the
> kernel needs to apply its policy.  The patch here (the in-kernel
> launch enclave) has a wide-open policy.
> 

Right, also agree here.  Systems without Flexible Launch Control are a
non-starter, we're only considering FLC systems here

> So, as a practical matter, if every distro has their own LE key and
> keeps it totally safe, then a system that locks the MSRs to one
> distro's key makes it quite annoying to run another distro's intel_sgx
> driver, but there is no effect on the actual security of the system.
> 
I agree that for systems that firmware-lock the msrs are annoying, but I would
think that IHV's would want to keep those msrs unlocked specifically to allow a
wide range of distributions to use this feature.

As for benefits to security, I think there are some.  Namely, by leaving the
MSRS unlocked, A distribution can, rather than providing their own distirbution
key, pass the root of trust on to the end user.  I can easily envision a
downstream customer that wants to use SGX, and do so in such a way that they are
assured that their OS vendor doesn't have the ability to run an LE on their
system (at least not without the visual cue of specifying a different key hash
at the OS boot).

> >  It was my understanding that the value of the key pair was
> > that the end user was guaranteed autonomy and security over which processes
> > could start enclaves.  By publishing a fixed key pair, it seems to remove that
> > ability.
> 
> If someone comes up with an actual machine that grants the actual end
> user (where the end user is the person who bought the thing, not the
> OEM) control over the MSRs, *and* the actual end user wants to limit
> what enclaves can be launched even if the kernel is compromised, *and*
> there is some actual argument for why this is useful (as opposed to
> some compliance checkbox), then Linux could reasonably consider adding
> support for this use case.  But that would be a separate patch.
> 
> >
> > What would be nicer (I think) would be the abilty to specify both the public and
> > the private key at run time.  the use case here is not one in which a vendor or
> > os distribution ships a key pair, but one in which a downstream user doesn't
> > want a vendor/os distribution to have any cryptographic information installed on
> > their system
> 
> For what gain?
My use case above is the primary one I was thinking of
Neil

Re: [intel-sgx-kernel-dev] [PATCH v11 09/13] x86, sgx: basic routines for enclave page cache

[Andy Lutomirski]

On Mon, Jun 18, 2018 at 8:11 AM Jethro Beekman <jethro@fortanix.com> wrote:
>
> On 2018-06-08 10:09, Jarkko Sakkinen wrote:
> > +/*
> > + * Writing the LE hash MSRs is extraordinarily expensive, e.g.
> > + * 3-4x slower than normal MSRs, so we use a per-cpu cache to
> > + * track the last known value of the MSRs to avoid unnecessarily
> > + * writing the MSRs with the current value.  Because most Linux
> > + * kernels will use an LE that is signed with a non-Intel key,
>
> I don't think you can predict what most Linux kernels will be doing. I
> think not initializing the cache to the CPU's initial value is fine, but
> this particular argument shouldn't appear in the rationale.

No, it's quite predictable.  Upstream Linux will not permit the
Intel-signed LE to be used unless Intel makes some major changes to
the way the launch process works.

>
> > + * i.e. the first EINIT will need to write the MSRs regardless
> > + * of the cache, the cache is intentionally left uninitialized
> > + * during boot as initializing the cache would be pure overhead
> > + * for the majority of systems.

But the comment does need changes.  How about saying that the cache
either contains all zeros or matches the MSRs?  All zeros is used
when, for whatever reason, we choose not to care what the current MSR
values are.  Leaving it genuinely uninitialized would be incorrect.
(And, of course, we're assuming that no one ever needs the MSRs
zeroed.)

When KVM host support goes in, even this won't be good enough if we
want to allow passthrough access to the MSRs because we will no longer
be able to guarantee that all zeros is invalid.  Instead we'd need an
actual flag saying that the cache is invalid.

--Andy

Re: [intel-sgx-kernel-dev] [PATCH v11 07/13] x86, sgx: detect Intel SGX

[Andy Lutomirski]

On Mon, Jun 18, 2018 at 8:11 AM Dave Hansen <dave.hansen@intel.com> wrote:
>
> > +config INTEL_SGX_CORE
> > +     prompt "Intel SGX core functionality
> > +     depends on X86_64 && CPU_SUP_INTEL
> > +     help
> > +     Intel Software Guard eXtensions (SGX) is a set of CPU instructions
> > +     that allows ring 3 applications to create enclaves; private regions
> > +     of memory that are protected, by hardware, from unauthorized access
> > +     and/or modification.
>
> That semicolon needs to be a colon.  The second half of that sentence is
> not a stand-alone statement.
>
> > +     This option enables kernel recognition of SGX, high-level management
> > +     of the Enclave Page Cache (EPC), tracking and writing of SGX Launch
> > +     Enclave Hash MSRs, and allows for virtualization of SGX via KVM. By
> > +     iteslf, this option does not provide SGX support to userspace.
> > +
> > +     For details, see Documentation/x86/intel_sgx.rst
> > +
> > +     If unsure, say N.
> > +
> >  config EFI
> >       bool "EFI runtime service support"
> >       depends on ACPI
> > diff --git a/arch/x86/include/asm/sgx.h b/arch/x86/include/asm/sgx.h
> > new file mode 100644
> > index 000000000000..fa3e6e0eb8af
> > --- /dev/null
> > +++ b/arch/x86/include/asm/sgx.h
> > @@ -0,0 +1,25 @@
> > +// SPDX-License-Identifier: (GPL-2.0 OR BSD-3-Clause)
> > +// Copyright(c) 2016-18 Intel Corporation.
> > +//
> > +// Authors:
> > +//
> > +// Jarkko Sakkinen <jarkko.sakkinen@linux.intel.com>
> > +// Suresh Siddha <suresh.b.siddha@intel.com>
> > +// Sean Christopherson <sean.j.christopherson@intel.com>
> > +
> > +#ifndef _ASM_X86_SGX_H
> > +#define _ASM_X86_SGX_H
> > +
> > +#include <linux/types.h>
> > +
> > +#define SGX_CPUID 0x12
>
> Hey, I just saw 0x12 as a magic, hard-coded number earlier in these
> patches.  It seems cruel to hard-code it, and then also have a #define
> that isn't used.
>
> > +enum sgx_cpuid {
> > +     SGX_CPUID_CAPABILITIES  = 0,
> > +     SGX_CPUID_ATTRIBUTES    = 1,
> > +     SGX_CPUID_EPC_BANKS     = 2,
> > +};
>
> These are cpuid *leaves*, right?  Please make this clear that these are
> hardware-defined values and not some kind of software construct.
>
> > +bool sgx_enabled __ro_after_init = false;
> > +EXPORT_SYMBOL(sgx_enabled);
> > +
> > +static __init bool sgx_is_enabled(void)
> > +{
> > +     unsigned long fc;
> > +
> > +     if (boot_cpu_data.x86_vendor != X86_VENDOR_INTEL)
> > +             return false;
>
> Not necessary. CPUID does this part for you.

More to the point, if a non-Intel vendor chooses to support SGX, then
the driver should allow it.

>
> > +     if (!boot_cpu_has(X86_FEATURE_SGX))
> > +             return false;
> > +
> > +     if (!boot_cpu_has(X86_FEATURE_SGX1))
> > +             return false;
> > +
> > +     rdmsrl(MSR_IA32_FEATURE_CONTROL, fc);
> > +     if (!(fc & FEATURE_CONTROL_LOCKED))
> > +             return false;
> > +
> > +     if (!(fc & FEATURE_CONTROL_SGX_ENABLE))
> > +             return false;
>
> Comments, please.  Why would this happen?  What would it mean?

Let's add actual pr_info() statements to document this, like:

SGX: disabled by firmware
SGX: disabled because CPU does not support flexible launch control
SGX: disabled because firmware does not support flexible launch control
SGX: disabled because the phase of the moon is wrong
SGX: enabled; using SGX1
SGX: enabled, using SGX2

If the CPU doesn't support SGX at all, then I see no reason to print anything.

etc.

(Is the feature actually called flexible launch control?  I may have
made that up.)

Re: [intel-sgx-kernel-dev] [PATCH v11 13/13] intel_sgx: in-kernel launch enclave

[Andy Lutomirski]

On Tue, Jun 12, 2018 at 10:45 AM Neil Horman <nhorman@redhat.com> wrote:
>
> On Mon, Jun 11, 2018 at 09:55:29PM -0700, Andy Lutomirski wrote:
> > On Mon, Jun 11, 2018 at 4:52 AM Neil Horman <nhorman@redhat.com> wrote:
> > >
> > > On Sun, Jun 10, 2018 at 10:17:13PM -0700, Andy Lutomirski wrote:
> > > > > On Jun 9, 2018, at 10:39 PM, Andy Lutomirski <luto@kernel.org> wrote:
> > > > >
> > > > > On Fri, Jun 8, 2018 at 10:32 AM Jarkko Sakkinen
> > > > > <jarkko.sakkinen@linux.intel.com> wrote:
> > > > >>
> > > > >> The Launch Enclave (LE) generates cryptographic launch tokens for user
> > > > >> enclaves. A launch token is used by EINIT to check whether the enclave
> > > > >> is authorized to launch or not. By having its own launch enclave, Linux
> > > > >> has full control of the enclave launch process.
> > > > >>
> > > > >> LE is wrapped into a user space proxy program that reads enclave
> > > > >> signatures outputs launch tokens. The kernel-side glue code is
> > > > >> implemented by using the user space helper framework.  The IPC between
> > > > >> the LE proxy program and kernel is handled with an anonymous inode.
> > > > >>
> > > > >> The commit also adds enclave signing tool that is used by kbuild to
> > > > >> measure and sign the launch enclave. CONFIG_INTEL_SGX_SIGNING_KEY points
> > > > >> to a PEM-file for the 3072-bit RSA key that is used as the LE public key
> > > > >> pair. The default location is:
> > > > >>
> > > > >>  drivers/platform/x86/intel_sgx/sgx_signing_key.pem
> > > > >>
> > > > >> If the default key does not exist kbuild will generate a random key and
> > > > >> place it to this location. KBUILD_SGX_SIGN_PIN can be used to specify
> > > > >> the passphrase for the LE public key.
> > > > >
> > > > > It seems to me that it might be more useful to just commit a key pair
> > > > > into the kernel.  As far as I know, there is no security whatsoever
> > > > > gained by keeping the private key private, so why not make
> > > > > reproducible builds easier by simply fixing the key?
> > > >
> > > > Having thought about this some more, I think that you should
> > > > completely remove support for specifying a key. Provide a fixed key
> > > > pair, hard code the cache, and call it a day. If you make the key
> > > > configurable, every vendor that has any vendor keys (Debian, Ubuntu,
> > > > Fedora, Red Hat, SuSE, Clear Linux, etc) will see that config option
> > > > and set up their own key pair for no gain whatsoever.  Instead, it'll
> > > > give some illusion of security and it'll slow down operations in a VM
> > > > guest due to swapping out the values of the MSRs.  And, if the code to
> > > > support a locked MSR that just happens to have the right value stays
> > > > in the kernel, then we'll risk having vendors actually ship one
> > > > distro's public key hash, and that will seriously suck.
> > > >
> > > If you hard code the key pair however, doesn't that imply that anyone can sign a
> > > user space binary as a launch enclave, and potentially gain control of the token
> > > granting process?
> >
> > Yes and no.
> >
> > First of all, the kernel driver shouldn't be allowing user code to
> > launch a launch enclave regardless of signature.  I haven't gotten far
> > enough in reviewing the code to see whether that's the case, but if
> > it's not, it should be fixed before it's merged.
> >
> Ok, I agree with you here.
>
> > But keep in mind that control of the token granting process is not the
> > same thing as control over the right to launch an enclave.  On systems
> > without the LE hash MSRs, Intel controls the token granting process
> > and, barring some attack, an enclave that isn't blessed by Intel can't
> > be launched.  Support for that model will not be merged into upstream
> > Linux.  But on systems that have the LE hash MSRs and leave them
> > unlocked, there is effectively no hardware-enforced launch control.
> > Instead we have good old kernel policy.  If a user wants to launch an
> > enclave, they need to get the kernel to launch the enclave, and the
> > kernel needs to apply its policy.  The patch here (the in-kernel
> > launch enclave) has a wide-open policy.
> >
>
> Right, also agree here.  Systems without Flexible Launch Control are a
> non-starter, we're only considering FLC systems here
>
> > So, as a practical matter, if every distro has their own LE key and
> > keeps it totally safe, then a system that locks the MSRs to one
> > distro's key makes it quite annoying to run another distro's intel_sgx
> > driver, but there is no effect on the actual security of the system.
> >
> I agree that for systems that firmware-lock the msrs are annoying, but I would
> think that IHV's would want to keep those msrs unlocked specifically to allow a
> wide range of distributions to use this feature.
>
> As for benefits to security, I think there are some.  Namely, by leaving the
> MSRS unlocked, A distribution can, rather than providing their own distirbution
> key, pass the root of trust on to the end user.  I can easily envision a
> downstream customer that wants to use SGX, and do so in such a way that they are
> assured that their OS vendor doesn't have the ability to run an LE on their
> system (at least not without the visual cue of specifying a different key hash
> at the OS boot).

Which achieves what, exactly?  The launch public key hash isn't the
root of trust of anything except for a really awkward mechanism to
limit the enclaves that get run.  If there is actual demand to limit
enclaves that get run, let's do it correctly: add some code in the
kernel that enforces a policy before launching an enclave.

If the MSRs are unlocked, there is no stronger guarantee available
even if you supply your own custom LE.  If the kernel is owned, the
attacker can just change the MSRs.

--Andy

Re: [intel-sgx-kernel-dev] [PATCH v11 13/13] intel_sgx: in-kernel launch enclave

[Neil Horman]

On Mon, Jun 18, 2018 at 02:58:59PM -0700, Andy Lutomirski wrote:
> On Tue, Jun 12, 2018 at 10:45 AM Neil Horman <nhorman@redhat.com> wrote:
> >
> > On Mon, Jun 11, 2018 at 09:55:29PM -0700, Andy Lutomirski wrote:
> > > On Mon, Jun 11, 2018 at 4:52 AM Neil Horman <nhorman@redhat.com> wrote:
> > > >
> > > > On Sun, Jun 10, 2018 at 10:17:13PM -0700, Andy Lutomirski wrote:
> > > > > > On Jun 9, 2018, at 10:39 PM, Andy Lutomirski <luto@kernel.org> wrote:
> > > > > >
> > > > > > On Fri, Jun 8, 2018 at 10:32 AM Jarkko Sakkinen
> > > > > > <jarkko.sakkinen@linux.intel.com> wrote:
> > > > > >>
> > > > > >> The Launch Enclave (LE) generates cryptographic launch tokens for user
> > > > > >> enclaves. A launch token is used by EINIT to check whether the enclave
> > > > > >> is authorized to launch or not. By having its own launch enclave, Linux
> > > > > >> has full control of the enclave launch process.
> > > > > >>
> > > > > >> LE is wrapped into a user space proxy program that reads enclave
> > > > > >> signatures outputs launch tokens. The kernel-side glue code is
> > > > > >> implemented by using the user space helper framework.  The IPC between
> > > > > >> the LE proxy program and kernel is handled with an anonymous inode.
> > > > > >>
> > > > > >> The commit also adds enclave signing tool that is used by kbuild to
> > > > > >> measure and sign the launch enclave. CONFIG_INTEL_SGX_SIGNING_KEY points
> > > > > >> to a PEM-file for the 3072-bit RSA key that is used as the LE public key
> > > > > >> pair. The default location is:
> > > > > >>
> > > > > >>  drivers/platform/x86/intel_sgx/sgx_signing_key.pem
> > > > > >>
> > > > > >> If the default key does not exist kbuild will generate a random key and
> > > > > >> place it to this location. KBUILD_SGX_SIGN_PIN can be used to specify
> > > > > >> the passphrase for the LE public key.
> > > > > >
> > > > > > It seems to me that it might be more useful to just commit a key pair
> > > > > > into the kernel.  As far as I know, there is no security whatsoever
> > > > > > gained by keeping the private key private, so why not make
> > > > > > reproducible builds easier by simply fixing the key?
> > > > >
> > > > > Having thought about this some more, I think that you should
> > > > > completely remove support for specifying a key. Provide a fixed key
> > > > > pair, hard code the cache, and call it a day. If you make the key
> > > > > configurable, every vendor that has any vendor keys (Debian, Ubuntu,
> > > > > Fedora, Red Hat, SuSE, Clear Linux, etc) will see that config option
> > > > > and set up their own key pair for no gain whatsoever.  Instead, it'll
> > > > > give some illusion of security and it'll slow down operations in a VM
> > > > > guest due to swapping out the values of the MSRs.  And, if the code to
> > > > > support a locked MSR that just happens to have the right value stays
> > > > > in the kernel, then we'll risk having vendors actually ship one
> > > > > distro's public key hash, and that will seriously suck.
> > > > >
> > > > If you hard code the key pair however, doesn't that imply that anyone can sign a
> > > > user space binary as a launch enclave, and potentially gain control of the token
> > > > granting process?
> > >
> > > Yes and no.
> > >
> > > First of all, the kernel driver shouldn't be allowing user code to
> > > launch a launch enclave regardless of signature.  I haven't gotten far
> > > enough in reviewing the code to see whether that's the case, but if
> > > it's not, it should be fixed before it's merged.
> > >
> > Ok, I agree with you here.
> >
> > > But keep in mind that control of the token granting process is not the
> > > same thing as control over the right to launch an enclave.  On systems
> > > without the LE hash MSRs, Intel controls the token granting process
> > > and, barring some attack, an enclave that isn't blessed by Intel can't
> > > be launched.  Support for that model will not be merged into upstream
> > > Linux.  But on systems that have the LE hash MSRs and leave them
> > > unlocked, there is effectively no hardware-enforced launch control.
> > > Instead we have good old kernel policy.  If a user wants to launch an
> > > enclave, they need to get the kernel to launch the enclave, and the
> > > kernel needs to apply its policy.  The patch here (the in-kernel
> > > launch enclave) has a wide-open policy.
> > >
> >
> > Right, also agree here.  Systems without Flexible Launch Control are a
> > non-starter, we're only considering FLC systems here
> >
> > > So, as a practical matter, if every distro has their own LE key and
> > > keeps it totally safe, then a system that locks the MSRs to one
> > > distro's key makes it quite annoying to run another distro's intel_sgx
> > > driver, but there is no effect on the actual security of the system.
> > >
> > I agree that for systems that firmware-lock the msrs are annoying, but I would
> > think that IHV's would want to keep those msrs unlocked specifically to allow a
> > wide range of distributions to use this feature.
> >
> > As for benefits to security, I think there are some.  Namely, by leaving the
> > MSRS unlocked, A distribution can, rather than providing their own distirbution
> > key, pass the root of trust on to the end user.  I can easily envision a
> > downstream customer that wants to use SGX, and do so in such a way that they are
> > assured that their OS vendor doesn't have the ability to run an LE on their
> > system (at least not without the visual cue of specifying a different key hash
> > at the OS boot).
> 
> Which achieves what, exactly?  The launch public key hash isn't the
> root of trust of anything except for a really awkward mechanism to
> limit the enclaves that get run.  If there is actual demand to limit
> enclaves that get run, let's do it correctly: add some code in the
> kernel that enforces a policy before launching an enclave.
> 
> If the MSRs are unlocked, there is no stronger guarantee available
> even if you supply your own custom LE.  If the kernel is owned, the
> attacker can just change the MSRs.
> 
So what you're saying is, because the kernel is a more open attack vector,
someone can compromise the kernel, and then because the msrs are unlocked, can
introduce their own private key and bootstrap ownership of any and all enclaves
on the system.  Ok, I'd not thought about it that way, but it makes sense.  As
such, I'd be supportive of a fixed key and kernel enforced policy.

That said, I think we still want the ability to direct what that policy is.  I
wonder if we can load enclave launch policy using an existing mechanism (selinux
perhaps?  Just spitballing)....


Regards
Neil

> --Andy

Re: [PATCH v11 04/13] x86, sgx: add SGX definitions to msr-index.h

[Jarkko Sakkinen]

On Fri, Jun 08, 2018 at 10:25:29AM -0700, Dave Hansen wrote:
> On 06/08/2018 10:09 AM, Jarkko Sakkinen wrote:
> > Signed-off-by: Sean Christopherson <sean.j.christopherson@intel.com>
> > Signed-off-by: Haim Cohen <haim.cohen@intel.com>
> > Tested-by: Serge Ayoun <serge.ayoun@intel.com>
> > Reviewed-by: Jarkko Sakkinen <jarkko.sakkinen@linux.intel.com>
> > Tested-by: Jarkko Sakkinen <jarkko.sakkinen@linux.intel.com>
> > Signed-off-by: Jarkko Sakkinen <jarkko.sakkinen@linux.intel.com>
> 
> These are a mess.  If you're signing off on it, a review of some kind is
> pretty implied.
> 
> Also, how did Sean write this, send it to Haim to sign off on it, and
> then send it to you?  That seems unlikely.

They've both done something to this. How should I change it then?

/Jarkko

Re: [PATCH v11 04/13] x86, sgx: add SGX definitions to msr-index.h

[Jarkko Sakkinen]

On Fri, Jun 08, 2018 at 10:25:29AM -0700, Dave Hansen wrote:
> On 06/08/2018 10:09 AM, Jarkko Sakkinen wrote:
> > Signed-off-by: Sean Christopherson <sean.j.christopherson@intel.com>
> > Signed-off-by: Haim Cohen <haim.cohen@intel.com>
> > Tested-by: Serge Ayoun <serge.ayoun@intel.com>
> > Reviewed-by: Jarkko Sakkinen <jarkko.sakkinen@linux.intel.com>
> > Tested-by: Jarkko Sakkinen <jarkko.sakkinen@linux.intel.com>
> > Signed-off-by: Jarkko Sakkinen <jarkko.sakkinen@linux.intel.com>
> 
> These are a mess.  If you're signing off on it, a review of some kind is
> pretty implied.
> 
> Also, how did Sean write this, send it to Haim to sign off on it, and
> then send it to you?  That seems unlikely.

They've both done something to this. How should I change it then?

/Jarkko

Re: [PATCH v11 08/13] x86, sgx: added ENCLS wrappers

[Jarkko Sakkinen]

On Fri, Jun 08, 2018 at 10:43:50AM -0700, Dave Hansen wrote:
> On 06/08/2018 10:09 AM, Jarkko Sakkinen wrote:
> > This commit adds wrappers for Intel(R) SGX ENCLS opcode functionality.
> 
> What's ENCLS?  I know what an opcode is, but I don't know what "opcode
> functionality" is.  Could you give us more than a single, cryptic
> sentence, please?

Sure.

> > +enum sgx_commands {
> > +	ECREATE	= 0x0,
> > +	EADD	= 0x1,
> > +	EINIT	= 0x2,
> > +	EREMOVE	= 0x3,
> > +	EDGBRD	= 0x4,
> > +	EDGBWR	= 0x5,
> > +	EEXTEND	= 0x6,
> > +	ELDU	= 0x8,
> > +	EBLOCK	= 0x9,
> > +	EPA	= 0xA,
> > +	EWB	= 0xB,
> > +	ETRACK	= 0xC,
> > +	EAUG	= 0xD,
> > +	EMODPR	= 0xE,
> > +	EMODT	= 0xF,
> > +};
> 
> Again, please differentiate hardware-defined values from
> software-defines ones.  Also, would it hurt to expand the acronyms a
> bit, like:
> 
> +	ELDU	= 0x8, /* LoaD Underpants */

Not a bad idea at all.

> > +#define SGX_FN(name, params...)		\
> > +{					\
> > +	void *epc;			\
> > +	int ret;			\
> > +	epc = sgx_get_page(epc_page);	\
> > +	ret = __##name(params);		\
> > +	sgx_put_page(epc);		\
> > +	return ret;			\
> > +}
> 
> Have I seen sgx_*_page() yet in this series?  This seems out of order.

Oops, thanks for spotting this out.

> > +#define BUILD_SGX_FN(fn, name)				\
> > +static inline int fn(struct sgx_epc_page *epc_page)	\
> > +	SGX_FN(name, epc)
> > +BUILD_SGX_FN(sgx_eremove, eremove)
> > +BUILD_SGX_FN(sgx_eblock, eblock)
> > +BUILD_SGX_FN(sgx_etrack, etrack)
> > +BUILD_SGX_FN(sgx_epa, epa)
> > +
> > +static inline int sgx_emodpr(struct sgx_secinfo *secinfo,
> > +			     struct sgx_epc_page *epc_page)
> > +	SGX_FN(emodpr, secinfo, epc)
> > +static inline int sgx_emodt(struct sgx_secinfo *secinfo,
> > +			    struct sgx_epc_page *epc_page)
> > +	SGX_FN(emodt, secinfo, epc)
> 
> Wow, that's hideous.
> 
> Can't you just do:
> 
> BUILD_SGX_FN(__sgx_emopt, foo)
> 
> static inline int sgx_emodt(struct sgx_secinfo *secinfo,
> 			    struct sgx_epc_page *epc_page)
> {
> 	return __sgx_emopt(secinfo, page);
> }
> 
> Also, this entire patch seems rather comment-free.  Was that intentional?

Something that I've ignored (big series) but I'll add comments to
the next version.

/Jarkko

Re: [PATCH v11 08/13] x86, sgx: added ENCLS wrappers

[Jarkko Sakkinen]

On Fri, Jun 08, 2018 at 10:43:50AM -0700, Dave Hansen wrote:
> On 06/08/2018 10:09 AM, Jarkko Sakkinen wrote:
> > This commit adds wrappers for Intel(R) SGX ENCLS opcode functionality.
> 
> What's ENCLS?  I know what an opcode is, but I don't know what "opcode
> functionality" is.  Could you give us more than a single, cryptic
> sentence, please?

Sure.

> > +enum sgx_commands {
> > +	ECREATE	= 0x0,
> > +	EADD	= 0x1,
> > +	EINIT	= 0x2,
> > +	EREMOVE	= 0x3,
> > +	EDGBRD	= 0x4,
> > +	EDGBWR	= 0x5,
> > +	EEXTEND	= 0x6,
> > +	ELDU	= 0x8,
> > +	EBLOCK	= 0x9,
> > +	EPA	= 0xA,
> > +	EWB	= 0xB,
> > +	ETRACK	= 0xC,
> > +	EAUG	= 0xD,
> > +	EMODPR	= 0xE,
> > +	EMODT	= 0xF,
> > +};
> 
> Again, please differentiate hardware-defined values from
> software-defines ones.  Also, would it hurt to expand the acronyms a
> bit, like:
> 
> +	ELDU	= 0x8, /* LoaD Underpants */

Not a bad idea at all.

> > +#define SGX_FN(name, params...)		\
> > +{					\
> > +	void *epc;			\
> > +	int ret;			\
> > +	epc = sgx_get_page(epc_page);	\
> > +	ret = __##name(params);		\
> > +	sgx_put_page(epc);		\
> > +	return ret;			\
> > +}
> 
> Have I seen sgx_*_page() yet in this series?  This seems out of order.

Oops, thanks for spotting this out.

> > +#define BUILD_SGX_FN(fn, name)				\
> > +static inline int fn(struct sgx_epc_page *epc_page)	\
> > +	SGX_FN(name, epc)
> > +BUILD_SGX_FN(sgx_eremove, eremove)
> > +BUILD_SGX_FN(sgx_eblock, eblock)
> > +BUILD_SGX_FN(sgx_etrack, etrack)
> > +BUILD_SGX_FN(sgx_epa, epa)
> > +
> > +static inline int sgx_emodpr(struct sgx_secinfo *secinfo,
> > +			     struct sgx_epc_page *epc_page)
> > +	SGX_FN(emodpr, secinfo, epc)
> > +static inline int sgx_emodt(struct sgx_secinfo *secinfo,
> > +			    struct sgx_epc_page *epc_page)
> > +	SGX_FN(emodt, secinfo, epc)
> 
> Wow, that's hideous.
> 
> Can't you just do:
> 
> BUILD_SGX_FN(__sgx_emopt, foo)
> 
> static inline int sgx_emodt(struct sgx_secinfo *secinfo,
> 			    struct sgx_epc_page *epc_page)
> {
> 	return __sgx_emopt(secinfo, page);
> }
> 
> Also, this entire patch seems rather comment-free.  Was that intentional?

Something that I've ignored (big series) but I'll add comments to
the next version.

/Jarkko

Re: [PATCH v11 10/13] intel_sgx: driver for Intel Software Guard Extensions

[Jarkko Sakkinen]

On Fri, Jun 08, 2018 at 12:35:51PM -0700, Dave Hansen wrote:
> > +#define sgx_pr_ratelimited(level, encl, fmt, ...)			\
> > +	pr_ ## level ## _ratelimited("[%d:0x%p] " fmt,			\
> > +				     pid_nr((encl)->tgid),		\
> > +				     (void *)(encl)->base, ##__VA_ARGS__)
> > +
> > +#define sgx_dbg(encl, fmt, ...) \
> > +	sgx_pr_ratelimited(debug, encl, fmt, ##__VA_ARGS__)
> > +#define sgx_info(encl, fmt, ...) \
> > +	sgx_pr_ratelimited(info, encl, fmt, ##__VA_ARGS__)
> > +#define sgx_warn(encl, fmt, ...) \
> > +	sgx_pr_ratelimited(warn, encl, fmt, ##__VA_ARGS__)
> > +#define sgx_err(encl, fmt, ...) \
> > +	sgx_pr_ratelimited(err, encl, fmt, ##__VA_ARGS__)
> > +#define sgx_crit(encl, fmt, ...) \
> > +	sgx_pr_ratelimited(crit, encl, fmt, ##__VA_ARGS__)
> 
> I thought the pr_* thingies were to keep everyone from having to do this
> in each driver.  Why did you need this?

Prefixing with the enclave information.

> Can you do any better than a 2,000-line patch?  For instance, could you
> break out the memory management portion into its own part and have that
> reviewed by mm folks?  Or the ioctl()'s by device driver folks?

I'll see if this could be broken down into two i.e. driver without
swapping callbacks and one for the callbacks.

/Jarkko

Re: [PATCH v11 10/13] intel_sgx: driver for Intel Software Guard Extensions

[Jarkko Sakkinen]

On Fri, Jun 08, 2018 at 12:35:51PM -0700, Dave Hansen wrote:
> > +#define sgx_pr_ratelimited(level, encl, fmt, ...)			\
> > +	pr_ ## level ## _ratelimited("[%d:0x%p] " fmt,			\
> > +				     pid_nr((encl)->tgid),		\
> > +				     (void *)(encl)->base, ##__VA_ARGS__)
> > +
> > +#define sgx_dbg(encl, fmt, ...) \
> > +	sgx_pr_ratelimited(debug, encl, fmt, ##__VA_ARGS__)
> > +#define sgx_info(encl, fmt, ...) \
> > +	sgx_pr_ratelimited(info, encl, fmt, ##__VA_ARGS__)
> > +#define sgx_warn(encl, fmt, ...) \
> > +	sgx_pr_ratelimited(warn, encl, fmt, ##__VA_ARGS__)
> > +#define sgx_err(encl, fmt, ...) \
> > +	sgx_pr_ratelimited(err, encl, fmt, ##__VA_ARGS__)
> > +#define sgx_crit(encl, fmt, ...) \
> > +	sgx_pr_ratelimited(crit, encl, fmt, ##__VA_ARGS__)
> 
> I thought the pr_* thingies were to keep everyone from having to do this
> in each driver.  Why did you need this?

Prefixing with the enclave information.

> Can you do any better than a 2,000-line patch?  For instance, could you
> break out the memory management portion into its own part and have that
> reviewed by mm folks?  Or the ioctl()'s by device driver folks?

I'll see if this could be broken down into two i.e. driver without
swapping callbacks and one for the callbacks.

/Jarkko

Re: [PATCH v11 12/13] intel_sgx: driver documentation

[Jarkko Sakkinen]

On Fri, Jun 08, 2018 at 11:32:17AM -0700, Jethro Beekman wrote:
> On 2018-06-08 10:09, Jarkko Sakkinen wrote:
> > +Launching enclaves
> > +------------------
> > +
> > +For privileged enclaves the launch is performed simply by submitting the
> > +SIGSTRUCT for that enclave to ENCLS(EINIT). For unprivileged enclaves the
> > +driver hosts a process in ring-3 that hosts a launch enclave signed with a key
> > +supplied for kbuild.
> > +
> > +The current implementation of the launch enclave generates a token for any
> > +enclave. In the future it could be potentially extended to have ways to
> > +configure policy what can be lauched.
> > +
> > +The driver will fail to initialize if it cannot start its own launch enclave.
> > +A user space application can submit a SIGSTRUCT instance through the ioctl API.
> > +The kernel will take care of the rest.
> > +
> > +This design assures that the Linux kernel has always full control, which
> > +enclaves get to launch and which do not, even if the public key MSRs are
> 
> As discussed previously at length, since the kernel needs to execute
> ENCLS[EINIT], it has full control to deny the launching of enclaves
> regardless of any launch enclave implementation. Please change this
> misleading statement.

Remember the discussion, forgot to change it. I'll fix this for the next
version.

/Jarkko

Re: [PATCH v11 12/13] intel_sgx: driver documentation

[Jarkko Sakkinen]

On Fri, Jun 08, 2018 at 11:32:17AM -0700, Jethro Beekman wrote:
> On 2018-06-08 10:09, Jarkko Sakkinen wrote:
> > +Launching enclaves
> > +------------------
> > +
> > +For privileged enclaves the launch is performed simply by submitting the
> > +SIGSTRUCT for that enclave to ENCLS(EINIT). For unprivileged enclaves the
> > +driver hosts a process in ring-3 that hosts a launch enclave signed with a key
> > +supplied for kbuild.
> > +
> > +The current implementation of the launch enclave generates a token for any
> > +enclave. In the future it could be potentially extended to have ways to
> > +configure policy what can be lauched.
> > +
> > +The driver will fail to initialize if it cannot start its own launch enclave.
> > +A user space application can submit a SIGSTRUCT instance through the ioctl API.
> > +The kernel will take care of the rest.
> > +
> > +This design assures that the Linux kernel has always full control, which
> > +enclaves get to launch and which do not, even if the public key MSRs are
> 
> As discussed previously at length, since the kernel needs to execute
> ENCLS[EINIT], it has full control to deny the launching of enclaves
> regardless of any launch enclave implementation. Please change this
> misleading statement.

Remember the discussion, forgot to change it. I'll fix this for the next
version.

/Jarkko
--
To unsubscribe from this list: send the line "unsubscribe linux-doc" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH v11 12/13] intel_sgx: driver documentation

[Jarkko Sakkinen]

On Fri, Jun 08, 2018 at 02:41:07PM -0700, Randy Dunlap wrote:
> On 06/08/2018 10:09 AM, Jarkko Sakkinen wrote:
> > Documentation of the features of the  Software Guard eXtensions usable
> > for the Linux kernel and how the driver internals uses these features.
> > In addition, contains documentation for the ioctl API.
> > 
> > Signed-off-by: Jarkko Sakkinen <jarkko.sakkinen@linux.intel.com>
> 
> Hi,
> 
> I have a few corrections below...
> 
> 
> > ---
> >  Documentation/index.rst         |   1 +
> >  Documentation/x86/intel_sgx.rst | 195 ++++++++++++++++++++++++++++++++
> >  2 files changed, 196 insertions(+)
> >  create mode 100644 Documentation/x86/intel_sgx.rst
> > 
> > diff --git a/Documentation/index.rst b/Documentation/index.rst
> > index 3b99ab931d41..b9fb92928e8c 100644
> > --- a/Documentation/index.rst
> > +++ b/Documentation/index.rst
> > @@ -100,6 +100,7 @@ implementation.
> >     :maxdepth: 2
> >  
> >     sh/index
> > +   x86/index
> >  
> >  Korean translations
> >  -------------------
> > diff --git a/Documentation/x86/intel_sgx.rst b/Documentation/x86/intel_sgx.rst
> > new file mode 100644
> > index 000000000000..ecbe544eb2cb
> > --- /dev/null
> > +++ b/Documentation/x86/intel_sgx.rst
> > @@ -0,0 +1,195 @@
> > +===================
> > +Intel(R) SGX driver
> > +===================
> > +
> > +Introduction
> > +============
> > +
> > +Intel(R) SGX is a set of CPU instructions that can be used by applications to
> > +set aside private regions of code and data. The code outside the enclave is
> > +disallowed to access the memory inside the enclave by the CPU access control.
> > +In a way you can think that SGX provides inverted sandbox. It protects the
> > +application from a malicious host.
> > +
> > +You can tell if your CPU supports SGX by looking into ``/proc/cpuinfo``:
> > +
> > +	``cat /proc/cpuinfo  | grep sgx``
> > +
> > +Overview of SGX
> > +===============
> > +
> > +SGX has a set of data structures to maintain information about the enclaves and
> > +their security properties. BIOS reserves a fixed size region of physical memory
> > +for these structures by setting Processor Reserved Memory Range Registers
> > +(PRMRR).
> > +
> > +This memory range is protected from outside access by the CPU and all the data
> > +coming in and out of the CPU package is encrypted by a key that is generated for
> > +each boot cycle.
> > +
> > +Enclaves execute in ring-3 in a special enclave submode using pages from the
> > +reserved memory range. A fixed logical address range for the enclave is reserved
> > +by ENCLS(ECREATE), a leaf instruction used to create enclaves. It is referred in
> > +the documentation commonly as the ELRANGE.
> > +
> > +Every memory access to the ELRANGE is asserted by the CPU. If the CPU is not
> > +executing in the enclave mode inside the enclave, #GP is raised. On the other
> > +hand enclave code can make memory accesses both inside and outside of the
> > +ELRANGE.
> > +
> > +Enclave can only execute code inside the ELRANGE. Instructions that may cause
> > +VMEXIT, IO instructions and instructions that require a privilege change are
> > +prohibited inside the enclave. Interrupts and exceptions always cause enclave
> > +to exit and jump to an address outside the enclave given when the enclave is
> > +entered by using the leaf instruction ENCLS(EENTER).
> > +
> > +Data types
> > +----------
> > +
> > +The protected memory range contains the following data:
> > +
> > +* **Enclave Page Cache (EPC):** protected pages
> > +* **Enclave Page Cache Map (EPCM):** a database that describes the state of the
> > +  pages and link them to an enclave.
> > +
> > +EPC has a number of different types of pages:
> > +
> > +* **SGX Enclave Control Structure (SECS)**: describes the global
> > +  properties of an enclave.
> > +* **Regular (REG):** code and data pages in the ELRANGE.
> > +* **Thread Control Structure (TCS):** pages that define entry points inside an
> > +  enclave. The enclave can only be entered through these entry points and each
> > +  can host a single hardware thread at a time.
> > +* **Version Array (VA)**: 64-bit version numbers for pages that have been
> > +  swapped outside the enclave. Each page contains 512 version numbers.
> > +
> > +Launch control
> > +--------------
> > +
> > +To launch an enclave, two structures must be provided for ENCLS(EINIT):
> > +
> > +1. **SIGSTRUCT:** signed measurement of the enclave binary.
> > +2. **EINITTOKEN:** a cryptographic token CMAC-signed with a AES256-key called
> > +   *launch key*, which is re-generated for each boot cycle.
> > +
> > +The CPU holds a SHA256 hash of a 3072-bit RSA public key inside
> > +IA32_SGXLEPUBKEYHASHn MSRs. Enclaves with a SIGSTRUCT that is signed with this
> > +key do not require a valid EINITTOKEN and can be authorized with special
> > +privileges. One of those privileges is ability to acquire the launch key with
> > +ENCLS(EGETKEY).
> > +
> > +**IA32_FEATURE_CONTROL[17]** is used by to BIOS configure whether
> 
>                                         by the BIOS to configure whether
> 
> > +IA32_SGXLEPUBKEYHASH MSRs are read-only or read-write before locking the
> > +feature control register and handing over control to the operating system.
> > +
> > +Enclave construction
> > +--------------------
> > +
> > +The construction is started by filling out the SECS that contains enclave
> > +address range, privileged attributes and measurement of TCS and REG pages (pages
> > +that will be mapped to the address range) among the other things. This structure
> > +is passed out to the ENCLS(ECREATE) together with a physical address of a page
> > +in EPC that will hold the SECS.
> > +
> > +Then pages are added with ENCLS(EADD) and measured with ENCLS(EEXTEND).  Finally
> 
> "measured"?  what does that mean?
> 
> > +enclave is initialized with ENCLS(EINIT). ENCLS(INIT) checks that the SIGSTRUCT
> > +is signed with the contained public key and that the supplied EINITTOKEN is
> > +valid (CMAC'd with the launch key). If these hold, the enclave is successfully
> > +initialized.
> > +
> > +Swapping pages
> > +--------------
> > +
> > +Enclave pages can be swapped out with ENCLS(EWB) to the unprotected memory. In
> > +addition to the EPC page, ENCLS(EWB) takes in a VA page and address for PCMD
> > +structure (Page Crypto MetaData) as input. The VA page will seal a version
> > +number for the page. PCMD is 128 byte structure that contains tracking
> > +information for the page, most importantly its MAC. With these structures the
> > +enclave is sealed and rollback protected while it resides in the unprotected
> > +memory.
> > +
> > +Before the page can be swapped out it must not have any active TLB references.
> > +By using ENCLS(EBLOCK) instructions no new TLB entries can be created to it.
> > +After this the a counter called *epoch* associated hardware threads inside the
> 
> huh?
> 
> > +enclave is increased with ENCLS(ETRACK). After all the threads from the previous
> > +epoch have exited the page can be safely swapped out.
> > +
> > +An enclave memory access to a swapped out pages will cause #PF. #PF handler can
> > +fault the page back by using ENCLS(ELDU).
> > +
> > +Kernel internals
> > +================
> > +
> > +Requirements
> > +------------
> > +
> > +Because SGX has an ever evolving and expanding feature set, it's possible for
> > +a BIOS or VMM to configure a system in such a way that not all cpus are equal,
> 
>                                                                   CPUs
> 
> > +e.g. where Launch Control is only enabled on a subset of cpus.  Linux does
> 
>                                                             CPUs.
> 
> > +*not* support such a heterogenous system configuration, nor does it even
> 
>                         heterogeneous
> 
> > +attempt to play nice in the face of a misconfigured system.  With the exception
> > +of Launch Control's hash MSRs, which can vary per cpu, Linux assumes that all
> 
>                                                      CPU,
> 
> > +cpus have a configuration that is identical to the boot cpu.
> 
>    CPUs                                                    CPU.
> 
> > +
> > +
> > +Roles and responsibilities
> > +--------------------------
> > +
> > +SGX introduces system resources, e.g. EPC memory, that must be accessible to
> > +multiple entities, e.g. the native kernel driver (to expose SGX to userspace)
> > +and KVM (to expose SGX to VMs), ideally without introducing any dependencies
> > +between each SGX entity.  To that end, the kernel owns and manages the shared
> > +system resources, i.e. the EPC and Launch Control MSRs, and defines functions
> > +that provide appropriate access to the shared resources.  SGX support for
> > +userpace and VMs is left to the SGX platform driver and KVM respectively.
> 
>    userspace
> 
> > +
> > +Launching enclaves
> > +------------------
> > +
> > +For privileged enclaves the launch is performed simply by submitting the
> > +SIGSTRUCT for that enclave to ENCLS(EINIT). For unprivileged enclaves the
> > +driver hosts a process in ring-3 that hosts a launch enclave signed with a key
> > +supplied for kbuild.
> > +
> > +The current implementation of the launch enclave generates a token for any
> > +enclave. In the future it could be potentially extended to have ways to
> > +configure policy what can be lauched.
> 
>                                 launched.
> 
> > +
> > +The driver will fail to initialize if it cannot start its own launch enclave.
> > +A user space application can submit a SIGSTRUCT instance through the ioctl API.
> > +The kernel will take care of the rest.
> > +
> > +This design assures that the Linux kernel has always full control, which
> > +enclaves get to launch and which do not, even if the public key MSRs are
> > +read-only. Having launch intrinsics inside the kernel also enables easy
> > +development of enclaves without necessarily needing any heavy weight SDK.
> > +Having a low-barrier to implement enclaves could make sense for example for
> 
>             low barrier
> 
> > +system daemons where amount of dependecies ought to be minimized.
> 
>                                   dependencies
> 
> > +
> > +EPC management
> > +--------------
> > +
> > +Due to the unique requirements for swapping EPC pages, and because EPC pages
> > +(currently) do not have associated page structures, management of the EPC is
> > +not handled by the standard Linux swapper.  SGX directly handles swapping
> > +of EPC pages, including a kthread to initiate reclaim and a rudimentary LRU
> > +mechanism.  Consumsers of EPC pages, e.g. the SGX driver, are required to
> 
>                Consumers
> 
> > +implement function callbacks that can be invoked by the kernel to age,
> > +swap, and/or forcefully reclaim a target EPC page.  In effect, the kernel
> > +controls what happens and when, while the consumers (driver, KVM, etc..) do
> > +the actual work.
> > +
> > +SGX uapi
> > +========
> > +
> > +.. kernel-doc:: drivers/platform/x86/intel_sgx/sgx_ioctl.c
> > +   :functions: sgx_ioc_enclave_create
> > +               sgx_ioc_enclave_add_page
> > +               sgx_ioc_enclave_init
> > +
> > +.. kernel-doc:: arch/x86/include/uapi/asm/sgx.h
> > +
> > +References
> > +==========
> > +
> > +* System Programming Manual: 39.1.4 Intel® SGX Launch Control Configuration
> > 
> 
> 
> -- 
> ~Randy

Thank you, I'll refine the parts that you pointed out for the next
version.

/Jarkko

Re: [PATCH v11 12/13] intel_sgx: driver documentation

[Jarkko Sakkinen]

On Fri, Jun 08, 2018 at 02:41:07PM -0700, Randy Dunlap wrote:
> On 06/08/2018 10:09 AM, Jarkko Sakkinen wrote:
> > Documentation of the features of the  Software Guard eXtensions usable
> > for the Linux kernel and how the driver internals uses these features.
> > In addition, contains documentation for the ioctl API.
> > 
> > Signed-off-by: Jarkko Sakkinen <jarkko.sakkinen@linux.intel.com>
> 
> Hi,
> 
> I have a few corrections below...
> 
> 
> > ---
> >  Documentation/index.rst         |   1 +
> >  Documentation/x86/intel_sgx.rst | 195 ++++++++++++++++++++++++++++++++
> >  2 files changed, 196 insertions(+)
> >  create mode 100644 Documentation/x86/intel_sgx.rst
> > 
> > diff --git a/Documentation/index.rst b/Documentation/index.rst
> > index 3b99ab931d41..b9fb92928e8c 100644
> > --- a/Documentation/index.rst
> > +++ b/Documentation/index.rst
> > @@ -100,6 +100,7 @@ implementation.
> >     :maxdepth: 2
> >  
> >     sh/index
> > +   x86/index
> >  
> >  Korean translations
> >  -------------------
> > diff --git a/Documentation/x86/intel_sgx.rst b/Documentation/x86/intel_sgx.rst
> > new file mode 100644
> > index 000000000000..ecbe544eb2cb
> > --- /dev/null
> > +++ b/Documentation/x86/intel_sgx.rst
> > @@ -0,0 +1,195 @@
> > +===================
> > +Intel(R) SGX driver
> > +===================
> > +
> > +Introduction
> > +============
> > +
> > +Intel(R) SGX is a set of CPU instructions that can be used by applications to
> > +set aside private regions of code and data. The code outside the enclave is
> > +disallowed to access the memory inside the enclave by the CPU access control.
> > +In a way you can think that SGX provides inverted sandbox. It protects the
> > +application from a malicious host.
> > +
> > +You can tell if your CPU supports SGX by looking into ``/proc/cpuinfo``:
> > +
> > +	``cat /proc/cpuinfo  | grep sgx``
> > +
> > +Overview of SGX
> > +===============
> > +
> > +SGX has a set of data structures to maintain information about the enclaves and
> > +their security properties. BIOS reserves a fixed size region of physical memory
> > +for these structures by setting Processor Reserved Memory Range Registers
> > +(PRMRR).
> > +
> > +This memory range is protected from outside access by the CPU and all the data
> > +coming in and out of the CPU package is encrypted by a key that is generated for
> > +each boot cycle.
> > +
> > +Enclaves execute in ring-3 in a special enclave submode using pages from the
> > +reserved memory range. A fixed logical address range for the enclave is reserved
> > +by ENCLS(ECREATE), a leaf instruction used to create enclaves. It is referred in
> > +the documentation commonly as the ELRANGE.
> > +
> > +Every memory access to the ELRANGE is asserted by the CPU. If the CPU is not
> > +executing in the enclave mode inside the enclave, #GP is raised. On the other
> > +hand enclave code can make memory accesses both inside and outside of the
> > +ELRANGE.
> > +
> > +Enclave can only execute code inside the ELRANGE. Instructions that may cause
> > +VMEXIT, IO instructions and instructions that require a privilege change are
> > +prohibited inside the enclave. Interrupts and exceptions always cause enclave
> > +to exit and jump to an address outside the enclave given when the enclave is
> > +entered by using the leaf instruction ENCLS(EENTER).
> > +
> > +Data types
> > +----------
> > +
> > +The protected memory range contains the following data:
> > +
> > +* **Enclave Page Cache (EPC):** protected pages
> > +* **Enclave Page Cache Map (EPCM):** a database that describes the state of the
> > +  pages and link them to an enclave.
> > +
> > +EPC has a number of different types of pages:
> > +
> > +* **SGX Enclave Control Structure (SECS)**: describes the global
> > +  properties of an enclave.
> > +* **Regular (REG):** code and data pages in the ELRANGE.
> > +* **Thread Control Structure (TCS):** pages that define entry points inside an
> > +  enclave. The enclave can only be entered through these entry points and each
> > +  can host a single hardware thread at a time.
> > +* **Version Array (VA)**: 64-bit version numbers for pages that have been
> > +  swapped outside the enclave. Each page contains 512 version numbers.
> > +
> > +Launch control
> > +--------------
> > +
> > +To launch an enclave, two structures must be provided for ENCLS(EINIT):
> > +
> > +1. **SIGSTRUCT:** signed measurement of the enclave binary.
> > +2. **EINITTOKEN:** a cryptographic token CMAC-signed with a AES256-key called
> > +   *launch key*, which is re-generated for each boot cycle.
> > +
> > +The CPU holds a SHA256 hash of a 3072-bit RSA public key inside
> > +IA32_SGXLEPUBKEYHASHn MSRs. Enclaves with a SIGSTRUCT that is signed with this
> > +key do not require a valid EINITTOKEN and can be authorized with special
> > +privileges. One of those privileges is ability to acquire the launch key with
> > +ENCLS(EGETKEY).
> > +
> > +**IA32_FEATURE_CONTROL[17]** is used by to BIOS configure whether
> 
>                                         by the BIOS to configure whether
> 
> > +IA32_SGXLEPUBKEYHASH MSRs are read-only or read-write before locking the
> > +feature control register and handing over control to the operating system.
> > +
> > +Enclave construction
> > +--------------------
> > +
> > +The construction is started by filling out the SECS that contains enclave
> > +address range, privileged attributes and measurement of TCS and REG pages (pages
> > +that will be mapped to the address range) among the other things. This structure
> > +is passed out to the ENCLS(ECREATE) together with a physical address of a page
> > +in EPC that will hold the SECS.
> > +
> > +Then pages are added with ENCLS(EADD) and measured with ENCLS(EEXTEND).  Finally
> 
> "measured"?  what does that mean?
> 
> > +enclave is initialized with ENCLS(EINIT). ENCLS(INIT) checks that the SIGSTRUCT
> > +is signed with the contained public key and that the supplied EINITTOKEN is
> > +valid (CMAC'd with the launch key). If these hold, the enclave is successfully
> > +initialized.
> > +
> > +Swapping pages
> > +--------------
> > +
> > +Enclave pages can be swapped out with ENCLS(EWB) to the unprotected memory. In
> > +addition to the EPC page, ENCLS(EWB) takes in a VA page and address for PCMD
> > +structure (Page Crypto MetaData) as input. The VA page will seal a version
> > +number for the page. PCMD is 128 byte structure that contains tracking
> > +information for the page, most importantly its MAC. With these structures the
> > +enclave is sealed and rollback protected while it resides in the unprotected
> > +memory.
> > +
> > +Before the page can be swapped out it must not have any active TLB references.
> > +By using ENCLS(EBLOCK) instructions no new TLB entries can be created to it.
> > +After this the a counter called *epoch* associated hardware threads inside the
> 
> huh?
> 
> > +enclave is increased with ENCLS(ETRACK). After all the threads from the previous
> > +epoch have exited the page can be safely swapped out.
> > +
> > +An enclave memory access to a swapped out pages will cause #PF. #PF handler can
> > +fault the page back by using ENCLS(ELDU).
> > +
> > +Kernel internals
> > +================
> > +
> > +Requirements
> > +------------
> > +
> > +Because SGX has an ever evolving and expanding feature set, it's possible for
> > +a BIOS or VMM to configure a system in such a way that not all cpus are equal,
> 
>                                                                   CPUs
> 
> > +e.g. where Launch Control is only enabled on a subset of cpus.  Linux does
> 
>                                                             CPUs.
> 
> > +*not* support such a heterogenous system configuration, nor does it even
> 
>                         heterogeneous
> 
> > +attempt to play nice in the face of a misconfigured system.  With the exception
> > +of Launch Control's hash MSRs, which can vary per cpu, Linux assumes that all
> 
>                                                      CPU,
> 
> > +cpus have a configuration that is identical to the boot cpu.
> 
>    CPUs                                                    CPU.
> 
> > +
> > +
> > +Roles and responsibilities
> > +--------------------------
> > +
> > +SGX introduces system resources, e.g. EPC memory, that must be accessible to
> > +multiple entities, e.g. the native kernel driver (to expose SGX to userspace)
> > +and KVM (to expose SGX to VMs), ideally without introducing any dependencies
> > +between each SGX entity.  To that end, the kernel owns and manages the shared
> > +system resources, i.e. the EPC and Launch Control MSRs, and defines functions
> > +that provide appropriate access to the shared resources.  SGX support for
> > +userpace and VMs is left to the SGX platform driver and KVM respectively.
> 
>    userspace
> 
> > +
> > +Launching enclaves
> > +------------------
> > +
> > +For privileged enclaves the launch is performed simply by submitting the
> > +SIGSTRUCT for that enclave to ENCLS(EINIT). For unprivileged enclaves the
> > +driver hosts a process in ring-3 that hosts a launch enclave signed with a key
> > +supplied for kbuild.
> > +
> > +The current implementation of the launch enclave generates a token for any
> > +enclave. In the future it could be potentially extended to have ways to
> > +configure policy what can be lauched.
> 
>                                 launched.
> 
> > +
> > +The driver will fail to initialize if it cannot start its own launch enclave.
> > +A user space application can submit a SIGSTRUCT instance through the ioctl API.
> > +The kernel will take care of the rest.
> > +
> > +This design assures that the Linux kernel has always full control, which
> > +enclaves get to launch and which do not, even if the public key MSRs are
> > +read-only. Having launch intrinsics inside the kernel also enables easy
> > +development of enclaves without necessarily needing any heavy weight SDK.
> > +Having a low-barrier to implement enclaves could make sense for example for
> 
>             low barrier
> 
> > +system daemons where amount of dependecies ought to be minimized.
> 
>                                   dependencies
> 
> > +
> > +EPC management
> > +--------------
> > +
> > +Due to the unique requirements for swapping EPC pages, and because EPC pages
> > +(currently) do not have associated page structures, management of the EPC is
> > +not handled by the standard Linux swapper.  SGX directly handles swapping
> > +of EPC pages, including a kthread to initiate reclaim and a rudimentary LRU
> > +mechanism.  Consumsers of EPC pages, e.g. the SGX driver, are required to
> 
>                Consumers
> 
> > +implement function callbacks that can be invoked by the kernel to age,
> > +swap, and/or forcefully reclaim a target EPC page.  In effect, the kernel
> > +controls what happens and when, while the consumers (driver, KVM, etc..) do
> > +the actual work.
> > +
> > +SGX uapi
> > +========
> > +
> > +.. kernel-doc:: drivers/platform/x86/intel_sgx/sgx_ioctl.c
> > +   :functions: sgx_ioc_enclave_create
> > +               sgx_ioc_enclave_add_page
> > +               sgx_ioc_enclave_init
> > +
> > +.. kernel-doc:: arch/x86/include/uapi/asm/sgx.h
> > +
> > +References
> > +==========
> > +
> > +* System Programming Manual: 39.1.4 Intel® SGX Launch Control Configuration
> > 
> 
> 
> -- 
> ~Randy

Thank you, I'll refine the parts that you pointed out for the next
version.

/Jarkko
--
To unsubscribe from this list: send the line "unsubscribe linux-doc" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH v11 07/13] x86, sgx: detect Intel SGX

[Jarkko Sakkinen]

On Fri, Jun 08, 2018 at 10:36:14AM -0700, Dave Hansen wrote:
> > +config INTEL_SGX_CORE
> > +	prompt "Intel SGX core functionality
> > +	depends on X86_64 && CPU_SUP_INTEL
> > +	help
> > +	Intel Software Guard eXtensions (SGX) is a set of CPU instructions
> > +	that allows ring 3 applications to create enclaves; private regions
> > +	of memory that are protected, by hardware, from unauthorized access
> > +	and/or modification.
> 
> That semicolon needs to be a colon.  The second half of that sentence is
> not a stand-alone statement.
> 
> > +	This option enables kernel recognition of SGX, high-level management
> > +	of the Enclave Page Cache (EPC), tracking and writing of SGX Launch
> > +	Enclave Hash MSRs, and allows for virtualization of SGX via KVM. By
> > +	iteslf, this option does not provide SGX support to userspace.
> > +
> > +	For details, see Documentation/x86/intel_sgx.rst
> > +
> > +	If unsure, say N.
> > +
> >  config EFI
> >  	bool "EFI runtime service support"
> >  	depends on ACPI
> > diff --git a/arch/x86/include/asm/sgx.h b/arch/x86/include/asm/sgx.h
> > new file mode 100644
> > index 000000000000..fa3e6e0eb8af
> > --- /dev/null
> > +++ b/arch/x86/include/asm/sgx.h
> > @@ -0,0 +1,25 @@
> > +// SPDX-License-Identifier: (GPL-2.0 OR BSD-3-Clause)
> > +// Copyright(c) 2016-18 Intel Corporation.
> > +//
> > +// Authors:
> > +//
> > +// Jarkko Sakkinen <jarkko.sakkinen@linux.intel.com>
> > +// Suresh Siddha <suresh.b.siddha@intel.com>
> > +// Sean Christopherson <sean.j.christopherson@intel.com>
> > +
> > +#ifndef _ASM_X86_SGX_H
> > +#define _ASM_X86_SGX_H
> > +
> > +#include <linux/types.h>
> > +
> > +#define SGX_CPUID 0x12
> 
> Hey, I just saw 0x12 as a magic, hard-coded number earlier in these
> patches.  It seems cruel to hard-code it, and then also have a #define
> that isn't used.

It i then is a regression in the series. I'll fix it.

> > +enum sgx_cpuid {
> > +	SGX_CPUID_CAPABILITIES	= 0,
> > +	SGX_CPUID_ATTRIBUTES	= 1,
> > +	SGX_CPUID_EPC_BANKS	= 2,
> > +};
> 
> These are cpuid *leaves*, right?  Please make this clear that these are
> hardware-defined values and not some kind of software construct.
> 
> > +bool sgx_enabled __ro_after_init = false;
> > +EXPORT_SYMBOL(sgx_enabled);
> > +
> > +static __init bool sgx_is_enabled(void)
> > +{
> > +	unsigned long fc;
> > +
> > +	if (boot_cpu_data.x86_vendor != X86_VENDOR_INTEL)
> > +		return false;
> 
> Not necessary. CPUID does this part for you.
> 
> > +	if (!boot_cpu_has(X86_FEATURE_SGX))
> > +		return false;
> > +
> > +	if (!boot_cpu_has(X86_FEATURE_SGX1))
> > +		return false;
> > +
> > +	rdmsrl(MSR_IA32_FEATURE_CONTROL, fc);
> > +	if (!(fc & FEATURE_CONTROL_LOCKED))
> > +		return false;
> > +
> > +	if (!(fc & FEATURE_CONTROL_SGX_ENABLE))
> > +		return false;
> 
> Comments, please.  Why would this happen?  What would it mean?

Thanks for the feedback! I'll refine the parts that you pointed out.

/Jarkko

Re: [PATCH v11 07/13] x86, sgx: detect Intel SGX

[Jarkko Sakkinen]

On Fri, Jun 08, 2018 at 10:36:14AM -0700, Dave Hansen wrote:
> > +config INTEL_SGX_CORE
> > +	prompt "Intel SGX core functionality
> > +	depends on X86_64 && CPU_SUP_INTEL
> > +	help
> > +	Intel Software Guard eXtensions (SGX) is a set of CPU instructions
> > +	that allows ring 3 applications to create enclaves; private regions
> > +	of memory that are protected, by hardware, from unauthorized access
> > +	and/or modification.
> 
> That semicolon needs to be a colon.  The second half of that sentence is
> not a stand-alone statement.
> 
> > +	This option enables kernel recognition of SGX, high-level management
> > +	of the Enclave Page Cache (EPC), tracking and writing of SGX Launch
> > +	Enclave Hash MSRs, and allows for virtualization of SGX via KVM. By
> > +	iteslf, this option does not provide SGX support to userspace.
> > +
> > +	For details, see Documentation/x86/intel_sgx.rst
> > +
> > +	If unsure, say N.
> > +
> >  config EFI
> >  	bool "EFI runtime service support"
> >  	depends on ACPI
> > diff --git a/arch/x86/include/asm/sgx.h b/arch/x86/include/asm/sgx.h
> > new file mode 100644
> > index 000000000000..fa3e6e0eb8af
> > --- /dev/null
> > +++ b/arch/x86/include/asm/sgx.h
> > @@ -0,0 +1,25 @@
> > +// SPDX-License-Identifier: (GPL-2.0 OR BSD-3-Clause)
> > +// Copyright(c) 2016-18 Intel Corporation.
> > +//
> > +// Authors:
> > +//
> > +// Jarkko Sakkinen <jarkko.sakkinen@linux.intel.com>
> > +// Suresh Siddha <suresh.b.siddha@intel.com>
> > +// Sean Christopherson <sean.j.christopherson@intel.com>
> > +
> > +#ifndef _ASM_X86_SGX_H
> > +#define _ASM_X86_SGX_H
> > +
> > +#include <linux/types.h>
> > +
> > +#define SGX_CPUID 0x12
> 
> Hey, I just saw 0x12 as a magic, hard-coded number earlier in these
> patches.  It seems cruel to hard-code it, and then also have a #define
> that isn't used.

It i then is a regression in the series. I'll fix it.

> > +enum sgx_cpuid {
> > +	SGX_CPUID_CAPABILITIES	= 0,
> > +	SGX_CPUID_ATTRIBUTES	= 1,
> > +	SGX_CPUID_EPC_BANKS	= 2,
> > +};
> 
> These are cpuid *leaves*, right?  Please make this clear that these are
> hardware-defined values and not some kind of software construct.
> 
> > +bool sgx_enabled __ro_after_init = false;
> > +EXPORT_SYMBOL(sgx_enabled);
> > +
> > +static __init bool sgx_is_enabled(void)
> > +{
> > +	unsigned long fc;
> > +
> > +	if (boot_cpu_data.x86_vendor != X86_VENDOR_INTEL)
> > +		return false;
> 
> Not necessary. CPUID does this part for you.
> 
> > +	if (!boot_cpu_has(X86_FEATURE_SGX))
> > +		return false;
> > +
> > +	if (!boot_cpu_has(X86_FEATURE_SGX1))
> > +		return false;
> > +
> > +	rdmsrl(MSR_IA32_FEATURE_CONTROL, fc);
> > +	if (!(fc & FEATURE_CONTROL_LOCKED))
> > +		return false;
> > +
> > +	if (!(fc & FEATURE_CONTROL_SGX_ENABLE))
> > +		return false;
> 
> Comments, please.  Why would this happen?  What would it mean?

Thanks for the feedback! I'll refine the parts that you pointed out.

/Jarkko

Re: [PATCH v11 07/13] x86, sgx: detect Intel SGX

[Jarkko Sakkinen]

On Mon, Jun 11, 2018 at 07:35:21AM -0400, Neil Horman wrote:
> On Fri, Jun 08, 2018 at 07:09:42PM +0200, Jarkko Sakkinen wrote:
> > From: Sean Christopherson <sean.j.christopherson@intel.com>
> > 
> > Intel(R) SGX is a set of CPU instructions that can be used by applications
> > to set aside private regions of code and data. The code outside the enclave
> > is disallowed to access the memory inside the enclave by the CPU access
> > control.
> > 
> > This commit adds the check for SGX to arch/x86 and a new config option,
> > INTEL_SGX_CORE. Exposes a boolean variable 'sgx_enabled' to query whether
> > or not the SGX support is available.
> > 
> > Signed-off-by: Sean Christopherson <sean.j.christopherson@intel.com>
> > Reviewed-by: Jarkko Sakkinen <jarkko.sakkinen@linux.intel.com>
> > Tested-by: Jarkko Sakkinen <jarkko.sakkinen@linux.intel.com>
> > Signed-off-by: Jarkko Sakkinen <jarkko.sakkinen@linux.intel.com>
> > ---
> >  arch/x86/Kconfig                | 19 ++++++++++++
> >  arch/x86/include/asm/sgx.h      | 25 ++++++++++++++++
> >  arch/x86/include/asm/sgx_pr.h   | 20 +++++++++++++
> >  arch/x86/kernel/cpu/Makefile    |  1 +
> >  arch/x86/kernel/cpu/intel_sgx.c | 53 +++++++++++++++++++++++++++++++++
> >  5 files changed, 118 insertions(+)
> >  create mode 100644 arch/x86/include/asm/sgx.h
> >  create mode 100644 arch/x86/include/asm/sgx_pr.h
> >  create mode 100644 arch/x86/kernel/cpu/intel_sgx.c
> > 
> > diff --git a/arch/x86/Kconfig b/arch/x86/Kconfig
> > index c07f492b871a..42015d5366ef 100644
> > --- a/arch/x86/Kconfig
> > +++ b/arch/x86/Kconfig
> > @@ -1925,6 +1925,25 @@ config X86_INTEL_MEMORY_PROTECTION_KEYS
> >  
> >  	  If unsure, say y.
> >  
> > +config INTEL_SGX_CORE
> > +	prompt "Intel SGX core functionality"
> > +	def_bool n
> > +	depends on X86_64 && CPU_SUP_INTEL
> > +	help
> > +	Intel Software Guard eXtensions (SGX) is a set of CPU instructions
> > +	that allows ring 3 applications to create enclaves; private regions
> > +	of memory that are protected, by hardware, from unauthorized access
> > +	and/or modification.
> > +
> > +	This option enables kernel recognition of SGX, high-level management
> > +	of the Enclave Page Cache (EPC), tracking and writing of SGX Launch
> > +	Enclave Hash MSRs, and allows for virtualization of SGX via KVM. By
> > +	iteslf, this option does not provide SGX support to userspace.
> > +
> > +	For details, see Documentation/x86/intel_sgx.rst
> > +
> > +	If unsure, say N.
> > +
> >  config EFI
> >  	bool "EFI runtime service support"
> >  	depends on ACPI
> > diff --git a/arch/x86/include/asm/sgx.h b/arch/x86/include/asm/sgx.h
> > new file mode 100644
> > index 000000000000..fa3e6e0eb8af
> > --- /dev/null
> > +++ b/arch/x86/include/asm/sgx.h
> > @@ -0,0 +1,25 @@
> > +// SPDX-License-Identifier: (GPL-2.0 OR BSD-3-Clause)
> > +// Copyright(c) 2016-18 Intel Corporation.
> > +//
> > +// Authors:
> > +//
> > +// Jarkko Sakkinen <jarkko.sakkinen@linux.intel.com>
> > +// Suresh Siddha <suresh.b.siddha@intel.com>
> > +// Sean Christopherson <sean.j.christopherson@intel.com>
> > +
> > +#ifndef _ASM_X86_SGX_H
> > +#define _ASM_X86_SGX_H
> > +
> > +#include <linux/types.h>
> > +
> > +#define SGX_CPUID 0x12
> > +
> Agree with Dave, this can just be remoed and you can use the feature macro from
> cpuid.h instead
> 
> Neil

Will do, thanks.

/Jarkko

Re: [PATCH v11 07/13] x86, sgx: detect Intel SGX

[Jarkko Sakkinen]

On Mon, Jun 11, 2018 at 07:35:21AM -0400, Neil Horman wrote:
> On Fri, Jun 08, 2018 at 07:09:42PM +0200, Jarkko Sakkinen wrote:
> > From: Sean Christopherson <sean.j.christopherson@intel.com>
> > 
> > Intel(R) SGX is a set of CPU instructions that can be used by applications
> > to set aside private regions of code and data. The code outside the enclave
> > is disallowed to access the memory inside the enclave by the CPU access
> > control.
> > 
> > This commit adds the check for SGX to arch/x86 and a new config option,
> > INTEL_SGX_CORE. Exposes a boolean variable 'sgx_enabled' to query whether
> > or not the SGX support is available.
> > 
> > Signed-off-by: Sean Christopherson <sean.j.christopherson@intel.com>
> > Reviewed-by: Jarkko Sakkinen <jarkko.sakkinen@linux.intel.com>
> > Tested-by: Jarkko Sakkinen <jarkko.sakkinen@linux.intel.com>
> > Signed-off-by: Jarkko Sakkinen <jarkko.sakkinen@linux.intel.com>
> > ---
> >  arch/x86/Kconfig                | 19 ++++++++++++
> >  arch/x86/include/asm/sgx.h      | 25 ++++++++++++++++
> >  arch/x86/include/asm/sgx_pr.h   | 20 +++++++++++++
> >  arch/x86/kernel/cpu/Makefile    |  1 +
> >  arch/x86/kernel/cpu/intel_sgx.c | 53 +++++++++++++++++++++++++++++++++
> >  5 files changed, 118 insertions(+)
> >  create mode 100644 arch/x86/include/asm/sgx.h
> >  create mode 100644 arch/x86/include/asm/sgx_pr.h
> >  create mode 100644 arch/x86/kernel/cpu/intel_sgx.c
> > 
> > diff --git a/arch/x86/Kconfig b/arch/x86/Kconfig
> > index c07f492b871a..42015d5366ef 100644
> > --- a/arch/x86/Kconfig
> > +++ b/arch/x86/Kconfig
> > @@ -1925,6 +1925,25 @@ config X86_INTEL_MEMORY_PROTECTION_KEYS
> >  
> >  	  If unsure, say y.
> >  
> > +config INTEL_SGX_CORE
> > +	prompt "Intel SGX core functionality"
> > +	def_bool n
> > +	depends on X86_64 && CPU_SUP_INTEL
> > +	help
> > +	Intel Software Guard eXtensions (SGX) is a set of CPU instructions
> > +	that allows ring 3 applications to create enclaves; private regions
> > +	of memory that are protected, by hardware, from unauthorized access
> > +	and/or modification.
> > +
> > +	This option enables kernel recognition of SGX, high-level management
> > +	of the Enclave Page Cache (EPC), tracking and writing of SGX Launch
> > +	Enclave Hash MSRs, and allows for virtualization of SGX via KVM. By
> > +	iteslf, this option does not provide SGX support to userspace.
> > +
> > +	For details, see Documentation/x86/intel_sgx.rst
> > +
> > +	If unsure, say N.
> > +
> >  config EFI
> >  	bool "EFI runtime service support"
> >  	depends on ACPI
> > diff --git a/arch/x86/include/asm/sgx.h b/arch/x86/include/asm/sgx.h
> > new file mode 100644
> > index 000000000000..fa3e6e0eb8af
> > --- /dev/null
> > +++ b/arch/x86/include/asm/sgx.h
> > @@ -0,0 +1,25 @@
> > +// SPDX-License-Identifier: (GPL-2.0 OR BSD-3-Clause)
> > +// Copyright(c) 2016-18 Intel Corporation.
> > +//
> > +// Authors:
> > +//
> > +// Jarkko Sakkinen <jarkko.sakkinen@linux.intel.com>
> > +// Suresh Siddha <suresh.b.siddha@intel.com>
> > +// Sean Christopherson <sean.j.christopherson@intel.com>
> > +
> > +#ifndef _ASM_X86_SGX_H
> > +#define _ASM_X86_SGX_H
> > +
> > +#include <linux/types.h>
> > +
> > +#define SGX_CPUID 0x12
> > +
> Agree with Dave, this can just be remoed and you can use the feature macro from
> cpuid.h instead
> 
> Neil

Will do, thanks.

/Jarkko

Re: [PATCH v11 11/13] intel_sgx: ptrace() support

[Jarkko Sakkinen]

On Mon, Jun 11, 2018 at 08:02:09AM -0700, Sean Christopherson wrote:
> On Fri, 2018-06-08 at 11:34 -0700, Dave Hansen wrote:
> > On 06/08/2018 10:09 AM, Jarkko Sakkinen wrote:
> > > 
> > > +		ret = sgx_edbgrd(encl, entry, align, data);
> > > +		if (ret)
> > > +			break;
> > > +		if (write) {
> > > +			memcpy(data + offset, buf + i, cnt);
> > > +			ret = sgx_edbgwr(encl, entry, align, data);
> > > +			if (ret)
> > > +				break;
> > > +		}
> > > +		else
> > > +			memcpy(buf + i,data + offset, cnt);
> > > +	}
> > The SGX instructions like "edbgrd" be great to put on a license plat,
> > but we can do better in the kernel.  Can you give these reasonable
> > english names, please?  sgx_debug_write(), maybe?
> 
> IMO the function names for ENCLS leafs are appropriate.  The real
> issue is the lack of documentation of the ENCLS helpers and their
> naming conventions.
> 
> The sgx_<leaf> functions, e.g. sgx_edbgrd(), are essentially direct
> invocations of the specific leaf, i.e. they are dumb wrappers to
> the lower level leaf functions, e.g. __edbgrd().  The wrappers exist
> primarily to deal with the boilerplate necessary to access a page in
> the EPC.  sgx_<leaf> conveys that the function contains the preamble
> and/or postamble needed to execute its leaf, but otherwise does not
> contain any logic.
> 
> Functions with actual logic do have English names, e.g.
> sgx_encl_init(), sgx_encl_add_page(), sgx_encl_modify_pages() etc...

I agree with Sean on the naming and agree with Dave on his comments in
earlier patch in the series needs a better documentation.

/Jarkko

Re: [PATCH v11 04/13] x86, sgx: add SGX definitions to msr-index.h

[Dave Hansen]

On 06/19/2018 06:18 AM, Jarkko Sakkinen wrote:
> On Fri, Jun 08, 2018 at 10:25:29AM -0700, Dave Hansen wrote:
>> On 06/08/2018 10:09 AM, Jarkko Sakkinen wrote:
>>> Signed-off-by: Sean Christopherson <sean.j.christopherson@intel.com>
>>> Signed-off-by: Haim Cohen <haim.cohen@intel.com>
>>> Tested-by: Serge Ayoun <serge.ayoun@intel.com>
>>> Reviewed-by: Jarkko Sakkinen <jarkko.sakkinen@linux.intel.com>
>>> Tested-by: Jarkko Sakkinen <jarkko.sakkinen@linux.intel.com>
>>> Signed-off-by: Jarkko Sakkinen <jarkko.sakkinen@linux.intel.com>
>>
>> These are a mess.  If you're signing off on it, a review of some kind is
>> pretty implied.
>>
>> Also, how did Sean write this, send it to Haim to sign off on it, and
>> then send it to you?  That seems unlikely.
> 
> They've both done something to this. How should I change it then?

Who added their Signed-off-by?

I would probably add a single "Co-developed-by:" for each of them and a
single Signed-off-by, like commit 12c89130a56ae:

    Co-developed-by: Tony Luck <tony.luck@intel.com>
    Reported-by: Mika Penttilä <mika.penttila@nextfour.com>
    Signed-off-by: Dan Williams <dan.j.williams@intel.com>
    Cc: Al Viro <viro@zeniv.linux.org.uk>
    Cc: Andrew Morton <akpm@linux-foundation.org>
    Cc: Andy Lutomirski <luto@amacapital.net>
    ...

Re: [PATCH v11 04/13] x86, sgx: add SGX definitions to msr-index.h

[Dave Hansen]

On 06/19/2018 06:18 AM, Jarkko Sakkinen wrote:
> On Fri, Jun 08, 2018 at 10:25:29AM -0700, Dave Hansen wrote:
>> On 06/08/2018 10:09 AM, Jarkko Sakkinen wrote:
>>> Signed-off-by: Sean Christopherson <sean.j.christopherson@intel.com>
>>> Signed-off-by: Haim Cohen <haim.cohen@intel.com>
>>> Tested-by: Serge Ayoun <serge.ayoun@intel.com>
>>> Reviewed-by: Jarkko Sakkinen <jarkko.sakkinen@linux.intel.com>
>>> Tested-by: Jarkko Sakkinen <jarkko.sakkinen@linux.intel.com>
>>> Signed-off-by: Jarkko Sakkinen <jarkko.sakkinen@linux.intel.com>
>>
>> These are a mess.  If you're signing off on it, a review of some kind is
>> pretty implied.
>>
>> Also, how did Sean write this, send it to Haim to sign off on it, and
>> then send it to you?  That seems unlikely.
> 
> They've both done something to this. How should I change it then?

Who added their Signed-off-by?

I would probably add a single "Co-developed-by:" for each of them and a
single Signed-off-by, like commit 12c89130a56ae:

    Co-developed-by: Tony Luck <tony.luck@intel.com>
    Reported-by: Mika Penttilä <mika.penttila@nextfour.com>
    Signed-off-by: Dan Williams <dan.j.williams@intel.com>
    Cc: Al Viro <viro@zeniv.linux.org.uk>
    Cc: Andrew Morton <akpm@linux-foundation.org>
    Cc: Andy Lutomirski <luto@amacapital.net>
    ...

Re: [PATCH v11 09/13] x86, sgx: basic routines for enclave page cache

[Jarkko Sakkinen]

On Fri, Jun 08, 2018 at 11:21:48AM -0700, Jethro Beekman wrote:
> On 2018-06-08 10:09, Jarkko Sakkinen wrote:
> > +/*
> > + * Writing the LE hash MSRs is extraordinarily expensive, e.g.
> > + * 3-4x slower than normal MSRs, so we use a per-cpu cache to
> > + * track the last known value of the MSRs to avoid unnecessarily
> > + * writing the MSRs with the current value.  Because most Linux
> > + * kernels will use an LE that is signed with a non-Intel key,
> 
> I don't think you can predict what most Linux kernels will be doing. I think
> not initializing the cache to the CPU's initial value is fine, but this
> particular argument shouldn't appear in the rationale.

Are you just referring to the last sentence or the whole paragraph?

/Jarkko

Re: [PATCH v11 09/13] x86, sgx: basic routines for enclave page cache

[Jarkko Sakkinen]

On Fri, Jun 08, 2018 at 11:21:48AM -0700, Jethro Beekman wrote:
> On 2018-06-08 10:09, Jarkko Sakkinen wrote:
> > +/*
> > + * Writing the LE hash MSRs is extraordinarily expensive, e.g.
> > + * 3-4x slower than normal MSRs, so we use a per-cpu cache to
> > + * track the last known value of the MSRs to avoid unnecessarily
> > + * writing the MSRs with the current value.  Because most Linux
> > + * kernels will use an LE that is signed with a non-Intel key,
> 
> I don't think you can predict what most Linux kernels will be doing. I think
> not initializing the cache to the CPU's initial value is fine, but this
> particular argument shouldn't appear in the rationale.

Are you just referring to the last sentence or the whole paragraph?

/Jarkko

Re: [PATCH v11 09/13] x86, sgx: basic routines for enclave page cache

[Jarkko Sakkinen]

On Fri, Jun 08, 2018 at 11:24:12AM -0700, Dave Hansen wrote:
> On 06/08/2018 10:09 AM, Jarkko Sakkinen wrote:
> > SGX has a set of data structures to maintain information about the enclaves
> > and their security properties. BIOS reserves a fixed size region of
> > physical memory for these structures by setting Processor Reserved Memory
> > Range Registers (PRMRR). This memory area is called Enclave Page Cache
> > (EPC).
> > 
> > This commit implements the basic routines to allocate and free pages from
> > different EPC banks. There is also a swapper thread ksgxswapd for EPC pages
> > that gets woken up by sgx_alloc_page() when we run below the low watermark.
> > The swapper thread continues swapping pages up until it reaches the high
> > watermark.
> 
> Yay!  A new memory manager in arch-specific code.
> 
> > Each subsystem that uses SGX must provide a set of callbacks for EPC
> > pages that are used to reclaim, block and write an EPC page. Kernel
> > takes the responsibility of maintaining LRU cache for them.
> 
> What does a "subsystem that uses SGX" mean?  Do we have one of those
> already?

Driver and KVM.

> > +struct sgx_secs {
> > +	uint64_t size;
> > +	uint64_t base;
> > +	uint32_t ssaframesize;
> > +	uint32_t miscselect;
> > +	uint8_t reserved1[SGX_SECS_RESERVED1_SIZE];
> > +	uint64_t attributes;
> > +	uint64_t xfrm;
> > +	uint32_t mrenclave[8];
> > +	uint8_t reserved2[SGX_SECS_RESERVED2_SIZE];
> > +	uint32_t mrsigner[8];
> > +	uint8_t	reserved3[SGX_SECS_RESERVED3_SIZE];
> > +	uint16_t isvvprodid;
> > +	uint16_t isvsvn;
> > +	uint8_t reserved4[SGX_SECS_RESERVED4_SIZE];
> > +};
> 
> This is a hardware structure, right?  Doesn't it need to be packed?

Everything is aligned properly in this struct.

> > +enum sgx_tcs_flags {
> > +	SGX_TCS_DBGOPTIN	= 0x01, /* cleared on EADD */
> > +};
> > +
> > +#define SGX_TCS_RESERVED_MASK 0xFFFFFFFFFFFFFFFEL
> 
> Would it be possible to separate out the SGX software structures from
> SGX hardware?  It's hard to tell them apart.

How do you draw the line in the architectural structures?

> > +#define SGX_NR_TO_SCAN	16
> > +#define SGX_NR_LOW_PAGES 32
> > +#define SGX_NR_HIGH_PAGES 64
> > +
> >  bool sgx_enabled __ro_after_init = false;
> >  EXPORT_SYMBOL(sgx_enabled);
> > +bool sgx_lc_enabled __ro_after_init;
> > +EXPORT_SYMBOL(sgx_lc_enabled);
> > +atomic_t sgx_nr_free_pages = ATOMIC_INIT(0);
> 
> Hmmm, global atomic.  Doesn't sound very scalable.

We could potentially remove this completely as banks have 'free_cnt'
field and use the sum when needed as the value.

> > +struct sgx_epc_bank sgx_epc_banks[SGX_MAX_EPC_BANKS];
> > +EXPORT_SYMBOL(sgx_epc_banks);
> > +int sgx_nr_epc_banks;
> > +EXPORT_SYMBOL(sgx_nr_epc_banks);
> > +LIST_HEAD(sgx_active_page_list);
> > +EXPORT_SYMBOL(sgx_active_page_list);
> > +DEFINE_SPINLOCK(sgx_active_page_list_lock);
> > +EXPORT_SYMBOL(sgx_active_page_list_lock);
> 
> Hmmm, global spinlock protecting a page allocator linked list.  Sounds
> even worse than at atomic.
> 
> Why is this OK?

Any suggestions what would be a better place in order to make a
fine grained granularity?

> > +static struct task_struct *ksgxswapd_tsk;
> > +static DECLARE_WAIT_QUEUE_HEAD(ksgxswapd_waitq);
> > +
> > +/*
> > + * Writing the LE hash MSRs is extraordinarily expensive, e.g.
> > + * 3-4x slower than normal MSRs, so we use a per-cpu cache to
> > + * track the last known value of the MSRs to avoid unnecessarily
> > + * writing the MSRs with the current value.  Because most Linux
> > + * kernels will use an LE that is signed with a non-Intel key,
> > + * i.e. the first EINIT will need to write the MSRs regardless
> > + * of the cache, the cache is intentionally left uninitialized
> > + * during boot as initializing the cache would be pure overhead
> > + * for the majority of systems.  Furthermore, the MSRs are per-cpu
> > + * and the boot-time values aren't guaranteed to be identical
> > + * across cpus, so we'd have to run code all all cpus to properly
> > + * init the cache.  All in all, the complexity and overhead of
> > + * initializing the cache is not justified.
> > + */
> > +static DEFINE_PER_CPU(u64 [4], sgx_le_pubkey_hash_cache);
> 
> Justifying the design decisions is great for changelogs, not so great
> for comments.  Also, looking at this, I have no idea what this has to do
> with the "enclave page cache".

We'll have to revisit comment, I see your point.

> > +static void sgx_swap_cluster(void)
> > +{
> > +	struct sgx_epc_page *cluster[SGX_NR_TO_SCAN + 1];
> > +	struct sgx_epc_page *epc_page;
> > +	int i;
> > +	int j;
> 
> This is rather free of comments or explanation of what this is doing,
> how it is related to swapping as everyone else knows it

I can document this function properly.

> > +	memset(cluster, 0, sizeof(cluster));
> > +
> > +	for (i = 0, j = 0; i < SGX_NR_TO_SCAN; i++) {
> > +		spin_lock(&sgx_active_page_list_lock);
> > +		if (list_empty(&sgx_active_page_list)) {
> > +			spin_unlock(&sgx_active_page_list_lock);
> > +			break;
> > +		}
> > +		epc_page = list_first_entry(&sgx_active_page_list,
> > +					    struct sgx_epc_page, list);
> > +		if (!epc_page->impl->ops->get(epc_page)) {
> > +			list_move_tail(&epc_page->list, &sgx_active_page_list);
> > +			spin_unlock(&sgx_active_page_list_lock);
> > +			continue;
> > +		}
> > +		list_del(&epc_page->list);
> > +		spin_unlock(&sgx_active_page_list_lock);
> >  
> > -static __init bool sgx_is_enabled(void)
> > +		if (epc_page->impl->ops->reclaim(epc_page)) {
> > +			cluster[j++] = epc_page;
> > +		} else {
> > +			spin_lock(&sgx_active_page_list_lock);
> > +			list_add_tail(&epc_page->list, &sgx_active_page_list);
> > +			spin_unlock(&sgx_active_page_list_lock);
> > +			epc_page->impl->ops->put(epc_page);
> > +		}
> > +	}
> > +
> > +	for (i = 0; cluster[i]; i++) {
> > +		epc_page = cluster[i];
> > +		epc_page->impl->ops->block(epc_page);
> > +	}
> > +
> > +	for (i = 0; cluster[i]; i++) {
> > +		epc_page = cluster[i];
> > +		epc_page->impl->ops->write(epc_page);
> > +		epc_page->impl->ops->put(epc_page);
> > +		sgx_free_page(epc_page);
> > +	}
> > +}
> 
> This is also gloriously free of any superfluous comments.  Could you fix
> that?

Yes.

> > +/**
> > + * sgx_try_alloc_page - try to allocate an EPC page
> > + * @impl:	implementation for the struct sgx_epc_page
> > + *
> > + * Try to grab a page from the free EPC page list. If there is a free page
> > + * available, it is returned to the caller.
> > + *
> > + * Return:
> > + *   a &struct sgx_epc_page instace,
> > + *   NULL otherwise
> > + */
> > +struct sgx_epc_page *sgx_try_alloc_page(struct sgx_epc_page_impl *impl)
> > +{
> > +	struct sgx_epc_bank *bank;
> > +	struct sgx_epc_page *page = NULL;
> > +	int i;
> > +
> > +	for (i = 0; i < sgx_nr_epc_banks; i++) {
> > +		bank = &sgx_epc_banks[i];
> 
> What's a bank?  How many banks does a system have?

AFAIK, UMA systems have one bank. NUMA have multiple. It is a physical
memory region reserved for enclave pages.

> > +		down_write(&bank->lock);
> > +
> > +		if (atomic_read(&bank->free_cnt))
> > +			page = bank->pages[atomic_dec_return(&bank->free_cnt)];
> 
> Why is a semaphore getting used here?  I don't see any sleeping or
> anything happening under this lock.

Should be changed to reader-writer spinlock, thanks.

> > +		up_write(&bank->lock);
> > +
> > +		if (page)
> > +			break;
> > +	}
> > +
> > +	if (page) {
> > +		atomic_dec(&sgx_nr_free_pages);
> > +		page->impl = impl;
> > +	}
> > +
> > +	return page;
> > +}
> > +EXPORT_SYMBOL(sgx_try_alloc_page);
> > +
> > +/**
> > + * sgx_alloc_page - allocate an EPC page
> > + * @flags:	allocation flags
> > + * @impl:	implementation for the struct sgx_epc_page
> > + *
> > + * Try to grab a page from the free EPC page list. If there is a free page
> > + * available, it is returned to the caller. If called with SGX_ALLOC_ATOMIC,
> > + * the function will return immediately if the list is empty. Otherwise, it
> > + * will swap pages up until there is a free page available. Upon returning the
> > + * low watermark is checked and ksgxswapd is waken up if we are below it.
> > + *
> > + * Return:
> > + *   a &struct sgx_epc_page instace,
> > + *   -ENOMEM if all pages are unreclaimable,
> > + *   -EBUSY when called with SGX_ALLOC_ATOMIC and out of free pages
> > + */
> > +struct sgx_epc_page *sgx_alloc_page(struct sgx_epc_page_impl *impl,
> > +				    unsigned int flags)
> > +{
> > +	struct sgx_epc_page *entry;
> > +
> > +	for ( ; ; ) {
> > +		entry = sgx_try_alloc_page(impl);
> > +		if (entry)
> > +			break;
> > +
> > +		if (list_empty(&sgx_active_page_list))
> > +			return ERR_PTR(-ENOMEM);
> 
> "active" pages in the VM are allocated/in-use pages.  This doesn't look
> to be using the same terminology.
> 
> > +		if (flags & SGX_ALLOC_ATOMIC) {
> > +			entry = ERR_PTR(-EBUSY);
> > +			break;
> > +		}
> > +
> > +		if (signal_pending(current)) {
> > +			entry = ERR_PTR(-ERESTARTSYS);
> > +			break;
> > +		}
> > +
> > +		sgx_swap_cluster();
> > +		schedule();
> 
> What's the schedule trying to do?  Is this the equivalent of "direct
> reclaim"?  Why do we need this in addition to the ksgxswapd?

It tries to direct reclaim. Ugh, that schedule() call does not make
much sense though...

> > +	}
> > +
> > +	if (atomic_read(&sgx_nr_free_pages) < SGX_NR_LOW_PAGES)
> > +		wake_up(&ksgxswapd_waitq);
> > +
> > +	return entry;
> > +}
> > +EXPORT_SYMBOL(sgx_alloc_page);
> 
> Why aren't these _GPL exports?

Source files a dual licensed.

> > +/**
> > + * sgx_free_page - free an EPC page
> > + *
> > + * @page:	any EPC page
> > + *
> > + * Remove an EPC page and insert it back to the list of free pages.
> > + *
> > + * Return: SGX error code
> > + */
> > +int sgx_free_page(struct sgx_epc_page *page)
> > +{
> > +	struct sgx_epc_bank *bank = SGX_EPC_BANK(page);
> > +	int ret;
> > +
> > +	ret = sgx_eremove(page);
> > +	if (ret) {
> > +		pr_debug("EREMOVE returned %d\n", ret);
> > +		return ret;
> > +	}
> > +
> > +	down_read(&bank->lock);
> > +	bank->pages[atomic_inc_return(&bank->free_cnt) - 1] = page;
> > +	atomic_inc(&sgx_nr_free_pages);
> > +	up_read(&bank->lock);
> > +
> > +	return 0;
> > +}
> 
> bank->lock confuses me.  This seems to be writing to a bank, but only
> needs a read lock.  Why?

It could be either way around:

1. Allow multiple threads that free a page to access the array.
2. Allow multiple threads that alloc a page to access the array.

> > +/**
> > + * sgx_get_page - pin an EPC page
> > + * @page:	an EPC page
> > + *
> > + * Return: a pointer to the pinned EPC page
> > + */
> > +void *sgx_get_page(struct sgx_epc_page *page)
> > +{
> > +	struct sgx_epc_bank *bank = SGX_EPC_BANK(page);
> > +
> > +	if (IS_ENABLED(CONFIG_X86_64))
> > +		return (void *)(bank->va + SGX_EPC_ADDR(page) - bank->pa);
> > +
> > +	return kmap_atomic_pfn(SGX_EPC_PFN(page));
> > +}
> > +EXPORT_SYMBOL(sgx_get_page);
> 
> This is odd.  Do you really want to detect 64-bit, or CONFIG_HIGHMEM?

For 32-bit (albeit not supported at this point) it makes sense to always
use kmap_atomic_pfn() as the virtua address area is very limited.

> > +struct page *sgx_get_backing(struct file *file, pgoff_t index)
> > +{
> > +	struct inode *inode = file->f_path.dentry->d_inode;
> > +	struct address_space *mapping = inode->i_mapping;
> > +	gfp_t gfpmask = mapping_gfp_mask(mapping);
> > +
> > +	return shmem_read_mapping_page_gfp(mapping, index, gfpmask);
> > +}
> > +EXPORT_SYMBOL(sgx_get_backing);
> 
> What does shmem have to do with all this?

Backing storage is an shmem file similarly is in drm.

> > +void sgx_put_backing(struct page *backing_page, bool write)
> > +{
> > +	if (write)
> > +		set_page_dirty(backing_page);
> > +
> > +	put_page(backing_page);
> > +}
> > +EXPORT_SYMBOL(sgx_put_backing);
> 
> I'm not a big fan of stuff getting added with no apparent user and no
> explaination of what it is doing.  There's no way for me to assess
> whether this is sane or not.

I'll add the documetation.

> > +static __init int sgx_page_cache_init(void)
> > +{
> > +	struct task_struct *tsk;
> > +	unsigned long size;
> > +	unsigned int eax;
> > +	unsigned int ebx;
> > +	unsigned int ecx;
> > +	unsigned int edx;
> > +	unsigned long pa;
> > +	int i;
> > +	int ret;
> > +
> > +	for (i = 0; i < SGX_MAX_EPC_BANKS; i++) {
> > +		cpuid_count(SGX_CPUID, i + SGX_CPUID_EPC_BANKS, &eax, &ebx,
> > +			    &ecx, &edx);
> > +		if (!(eax & 0xf))
> > +			break;
> > +
> > +		pa   = ((u64)(ebx & 0xfffff) << 32) + (u64)(eax & 0xfffff000);
> > +		size = ((u64)(edx & 0xfffff) << 32) + (u64)(ecx & 0xfffff000);
> 
> Please align these like I did ^
> 
> > +		pr_info("EPC bank 0x%lx-0x%lx\n", pa, pa + size);
> > +
> > +		ret = sgx_init_epc_bank(pa, size, i, &sgx_epc_banks[i]);
> > +		if (ret) {
> > +			sgx_page_cache_teardown();
> > +			return ret;
> > +		}
> > +
> > +		sgx_nr_epc_banks++;
> > +	}
> 
> This is also rather sparsely commented.
> 
> > +static __init bool sgx_is_enabled(bool *lc_enabled)
> >  {
> >  	unsigned long fc;
> >  
> > @@ -41,12 +466,26 @@ static __init bool sgx_is_enabled(void)
> >  	if (!(fc & FEATURE_CONTROL_SGX_ENABLE))
> >  		return false;
> >  
> > +	*lc_enabled = !!(fc & FEATURE_CONTROL_SGX_LE_WR);
> > +
> >  	return true;
> >  }
> 
> I'm baffled why lc_enabled is connected to the enclave page cache.

KVM works only with writable MSRs. Driver works both with writable
and read-only MSRs.

Thanks, I'll try my best to deal with all this :-)

/Jarkko

Re: [PATCH v11 09/13] x86, sgx: basic routines for enclave page cache

[Jarkko Sakkinen]

On Fri, Jun 08, 2018 at 11:24:12AM -0700, Dave Hansen wrote:
> On 06/08/2018 10:09 AM, Jarkko Sakkinen wrote:
> > SGX has a set of data structures to maintain information about the enclaves
> > and their security properties. BIOS reserves a fixed size region of
> > physical memory for these structures by setting Processor Reserved Memory
> > Range Registers (PRMRR). This memory area is called Enclave Page Cache
> > (EPC).
> > 
> > This commit implements the basic routines to allocate and free pages from
> > different EPC banks. There is also a swapper thread ksgxswapd for EPC pages
> > that gets woken up by sgx_alloc_page() when we run below the low watermark.
> > The swapper thread continues swapping pages up until it reaches the high
> > watermark.
> 
> Yay!  A new memory manager in arch-specific code.
> 
> > Each subsystem that uses SGX must provide a set of callbacks for EPC
> > pages that are used to reclaim, block and write an EPC page. Kernel
> > takes the responsibility of maintaining LRU cache for them.
> 
> What does a "subsystem that uses SGX" mean?  Do we have one of those
> already?

Driver and KVM.

> > +struct sgx_secs {
> > +	uint64_t size;
> > +	uint64_t base;
> > +	uint32_t ssaframesize;
> > +	uint32_t miscselect;
> > +	uint8_t reserved1[SGX_SECS_RESERVED1_SIZE];
> > +	uint64_t attributes;
> > +	uint64_t xfrm;
> > +	uint32_t mrenclave[8];
> > +	uint8_t reserved2[SGX_SECS_RESERVED2_SIZE];
> > +	uint32_t mrsigner[8];
> > +	uint8_t	reserved3[SGX_SECS_RESERVED3_SIZE];
> > +	uint16_t isvvprodid;
> > +	uint16_t isvsvn;
> > +	uint8_t reserved4[SGX_SECS_RESERVED4_SIZE];
> > +};
> 
> This is a hardware structure, right?  Doesn't it need to be packed?

Everything is aligned properly in this struct.

> > +enum sgx_tcs_flags {
> > +	SGX_TCS_DBGOPTIN	= 0x01, /* cleared on EADD */
> > +};
> > +
> > +#define SGX_TCS_RESERVED_MASK 0xFFFFFFFFFFFFFFFEL
> 
> Would it be possible to separate out the SGX software structures from
> SGX hardware?  It's hard to tell them apart.

How do you draw the line in the architectural structures?

> > +#define SGX_NR_TO_SCAN	16
> > +#define SGX_NR_LOW_PAGES 32
> > +#define SGX_NR_HIGH_PAGES 64
> > +
> >  bool sgx_enabled __ro_after_init = false;
> >  EXPORT_SYMBOL(sgx_enabled);
> > +bool sgx_lc_enabled __ro_after_init;
> > +EXPORT_SYMBOL(sgx_lc_enabled);
> > +atomic_t sgx_nr_free_pages = ATOMIC_INIT(0);
> 
> Hmmm, global atomic.  Doesn't sound very scalable.

We could potentially remove this completely as banks have 'free_cnt'
field and use the sum when needed as the value.

> > +struct sgx_epc_bank sgx_epc_banks[SGX_MAX_EPC_BANKS];
> > +EXPORT_SYMBOL(sgx_epc_banks);
> > +int sgx_nr_epc_banks;
> > +EXPORT_SYMBOL(sgx_nr_epc_banks);
> > +LIST_HEAD(sgx_active_page_list);
> > +EXPORT_SYMBOL(sgx_active_page_list);
> > +DEFINE_SPINLOCK(sgx_active_page_list_lock);
> > +EXPORT_SYMBOL(sgx_active_page_list_lock);
> 
> Hmmm, global spinlock protecting a page allocator linked list.  Sounds
> even worse than at atomic.
> 
> Why is this OK?

Any suggestions what would be a better place in order to make a
fine grained granularity?

> > +static struct task_struct *ksgxswapd_tsk;
> > +static DECLARE_WAIT_QUEUE_HEAD(ksgxswapd_waitq);
> > +
> > +/*
> > + * Writing the LE hash MSRs is extraordinarily expensive, e.g.
> > + * 3-4x slower than normal MSRs, so we use a per-cpu cache to
> > + * track the last known value of the MSRs to avoid unnecessarily
> > + * writing the MSRs with the current value.  Because most Linux
> > + * kernels will use an LE that is signed with a non-Intel key,
> > + * i.e. the first EINIT will need to write the MSRs regardless
> > + * of the cache, the cache is intentionally left uninitialized
> > + * during boot as initializing the cache would be pure overhead
> > + * for the majority of systems.  Furthermore, the MSRs are per-cpu
> > + * and the boot-time values aren't guaranteed to be identical
> > + * across cpus, so we'd have to run code all all cpus to properly
> > + * init the cache.  All in all, the complexity and overhead of
> > + * initializing the cache is not justified.
> > + */
> > +static DEFINE_PER_CPU(u64 [4], sgx_le_pubkey_hash_cache);
> 
> Justifying the design decisions is great for changelogs, not so great
> for comments.  Also, looking at this, I have no idea what this has to do
> with the "enclave page cache".

We'll have to revisit comment, I see your point.

> > +static void sgx_swap_cluster(void)
> > +{
> > +	struct sgx_epc_page *cluster[SGX_NR_TO_SCAN + 1];
> > +	struct sgx_epc_page *epc_page;
> > +	int i;
> > +	int j;
> 
> This is rather free of comments or explanation of what this is doing,
> how it is related to swapping as everyone else knows it

I can document this function properly.

> > +	memset(cluster, 0, sizeof(cluster));
> > +
> > +	for (i = 0, j = 0; i < SGX_NR_TO_SCAN; i++) {
> > +		spin_lock(&sgx_active_page_list_lock);
> > +		if (list_empty(&sgx_active_page_list)) {
> > +			spin_unlock(&sgx_active_page_list_lock);
> > +			break;
> > +		}
> > +		epc_page = list_first_entry(&sgx_active_page_list,
> > +					    struct sgx_epc_page, list);
> > +		if (!epc_page->impl->ops->get(epc_page)) {
> > +			list_move_tail(&epc_page->list, &sgx_active_page_list);
> > +			spin_unlock(&sgx_active_page_list_lock);
> > +			continue;
> > +		}
> > +		list_del(&epc_page->list);
> > +		spin_unlock(&sgx_active_page_list_lock);
> >  
> > -static __init bool sgx_is_enabled(void)
> > +		if (epc_page->impl->ops->reclaim(epc_page)) {
> > +			cluster[j++] = epc_page;
> > +		} else {
> > +			spin_lock(&sgx_active_page_list_lock);
> > +			list_add_tail(&epc_page->list, &sgx_active_page_list);
> > +			spin_unlock(&sgx_active_page_list_lock);
> > +			epc_page->impl->ops->put(epc_page);
> > +		}
> > +	}
> > +
> > +	for (i = 0; cluster[i]; i++) {
> > +		epc_page = cluster[i];
> > +		epc_page->impl->ops->block(epc_page);
> > +	}
> > +
> > +	for (i = 0; cluster[i]; i++) {
> > +		epc_page = cluster[i];
> > +		epc_page->impl->ops->write(epc_page);
> > +		epc_page->impl->ops->put(epc_page);
> > +		sgx_free_page(epc_page);
> > +	}
> > +}
> 
> This is also gloriously free of any superfluous comments.  Could you fix
> that?

Yes.

> > +/**
> > + * sgx_try_alloc_page - try to allocate an EPC page
> > + * @impl:	implementation for the struct sgx_epc_page
> > + *
> > + * Try to grab a page from the free EPC page list. If there is a free page
> > + * available, it is returned to the caller.
> > + *
> > + * Return:
> > + *   a &struct sgx_epc_page instace,
> > + *   NULL otherwise
> > + */
> > +struct sgx_epc_page *sgx_try_alloc_page(struct sgx_epc_page_impl *impl)
> > +{
> > +	struct sgx_epc_bank *bank;
> > +	struct sgx_epc_page *page = NULL;
> > +	int i;
> > +
> > +	for (i = 0; i < sgx_nr_epc_banks; i++) {
> > +		bank = &sgx_epc_banks[i];
> 
> What's a bank?  How many banks does a system have?

AFAIK, UMA systems have one bank. NUMA have multiple. It is a physical
memory region reserved for enclave pages.

> > +		down_write(&bank->lock);
> > +
> > +		if (atomic_read(&bank->free_cnt))
> > +			page = bank->pages[atomic_dec_return(&bank->free_cnt)];
> 
> Why is a semaphore getting used here?  I don't see any sleeping or
> anything happening under this lock.

Should be changed to reader-writer spinlock, thanks.

> > +		up_write(&bank->lock);
> > +
> > +		if (page)
> > +			break;
> > +	}
> > +
> > +	if (page) {
> > +		atomic_dec(&sgx_nr_free_pages);
> > +		page->impl = impl;
> > +	}
> > +
> > +	return page;
> > +}
> > +EXPORT_SYMBOL(sgx_try_alloc_page);
> > +
> > +/**
> > + * sgx_alloc_page - allocate an EPC page
> > + * @flags:	allocation flags
> > + * @impl:	implementation for the struct sgx_epc_page
> > + *
> > + * Try to grab a page from the free EPC page list. If there is a free page
> > + * available, it is returned to the caller. If called with SGX_ALLOC_ATOMIC,
> > + * the function will return immediately if the list is empty. Otherwise, it
> > + * will swap pages up until there is a free page available. Upon returning the
> > + * low watermark is checked and ksgxswapd is waken up if we are below it.
> > + *
> > + * Return:
> > + *   a &struct sgx_epc_page instace,
> > + *   -ENOMEM if all pages are unreclaimable,
> > + *   -EBUSY when called with SGX_ALLOC_ATOMIC and out of free pages
> > + */
> > +struct sgx_epc_page *sgx_alloc_page(struct sgx_epc_page_impl *impl,
> > +				    unsigned int flags)
> > +{
> > +	struct sgx_epc_page *entry;
> > +
> > +	for ( ; ; ) {
> > +		entry = sgx_try_alloc_page(impl);
> > +		if (entry)
> > +			break;
> > +
> > +		if (list_empty(&sgx_active_page_list))
> > +			return ERR_PTR(-ENOMEM);
> 
> "active" pages in the VM are allocated/in-use pages.  This doesn't look
> to be using the same terminology.
> 
> > +		if (flags & SGX_ALLOC_ATOMIC) {
> > +			entry = ERR_PTR(-EBUSY);
> > +			break;
> > +		}
> > +
> > +		if (signal_pending(current)) {
> > +			entry = ERR_PTR(-ERESTARTSYS);
> > +			break;
> > +		}
> > +
> > +		sgx_swap_cluster();
> > +		schedule();
> 
> What's the schedule trying to do?  Is this the equivalent of "direct
> reclaim"?  Why do we need this in addition to the ksgxswapd?

It tries to direct reclaim. Ugh, that schedule() call does not make
much sense though...

> > +	}
> > +
> > +	if (atomic_read(&sgx_nr_free_pages) < SGX_NR_LOW_PAGES)
> > +		wake_up(&ksgxswapd_waitq);
> > +
> > +	return entry;
> > +}
> > +EXPORT_SYMBOL(sgx_alloc_page);
> 
> Why aren't these _GPL exports?

Source files a dual licensed.

> > +/**
> > + * sgx_free_page - free an EPC page
> > + *
> > + * @page:	any EPC page
> > + *
> > + * Remove an EPC page and insert it back to the list of free pages.
> > + *
> > + * Return: SGX error code
> > + */
> > +int sgx_free_page(struct sgx_epc_page *page)
> > +{
> > +	struct sgx_epc_bank *bank = SGX_EPC_BANK(page);
> > +	int ret;
> > +
> > +	ret = sgx_eremove(page);
> > +	if (ret) {
> > +		pr_debug("EREMOVE returned %d\n", ret);
> > +		return ret;
> > +	}
> > +
> > +	down_read(&bank->lock);
> > +	bank->pages[atomic_inc_return(&bank->free_cnt) - 1] = page;
> > +	atomic_inc(&sgx_nr_free_pages);
> > +	up_read(&bank->lock);
> > +
> > +	return 0;
> > +}
> 
> bank->lock confuses me.  This seems to be writing to a bank, but only
> needs a read lock.  Why?

It could be either way around:

1. Allow multiple threads that free a page to access the array.
2. Allow multiple threads that alloc a page to access the array.

> > +/**
> > + * sgx_get_page - pin an EPC page
> > + * @page:	an EPC page
> > + *
> > + * Return: a pointer to the pinned EPC page
> > + */
> > +void *sgx_get_page(struct sgx_epc_page *page)
> > +{
> > +	struct sgx_epc_bank *bank = SGX_EPC_BANK(page);
> > +
> > +	if (IS_ENABLED(CONFIG_X86_64))
> > +		return (void *)(bank->va + SGX_EPC_ADDR(page) - bank->pa);
> > +
> > +	return kmap_atomic_pfn(SGX_EPC_PFN(page));
> > +}
> > +EXPORT_SYMBOL(sgx_get_page);
> 
> This is odd.  Do you really want to detect 64-bit, or CONFIG_HIGHMEM?

For 32-bit (albeit not supported at this point) it makes sense to always
use kmap_atomic_pfn() as the virtua address area is very limited.

> > +struct page *sgx_get_backing(struct file *file, pgoff_t index)
> > +{
> > +	struct inode *inode = file->f_path.dentry->d_inode;
> > +	struct address_space *mapping = inode->i_mapping;
> > +	gfp_t gfpmask = mapping_gfp_mask(mapping);
> > +
> > +	return shmem_read_mapping_page_gfp(mapping, index, gfpmask);
> > +}
> > +EXPORT_SYMBOL(sgx_get_backing);
> 
> What does shmem have to do with all this?

Backing storage is an shmem file similarly is in drm.

> > +void sgx_put_backing(struct page *backing_page, bool write)
> > +{
> > +	if (write)
> > +		set_page_dirty(backing_page);
> > +
> > +	put_page(backing_page);
> > +}
> > +EXPORT_SYMBOL(sgx_put_backing);
> 
> I'm not a big fan of stuff getting added with no apparent user and no
> explaination of what it is doing.  There's no way for me to assess
> whether this is sane or not.

I'll add the documetation.

> > +static __init int sgx_page_cache_init(void)
> > +{
> > +	struct task_struct *tsk;
> > +	unsigned long size;
> > +	unsigned int eax;
> > +	unsigned int ebx;
> > +	unsigned int ecx;
> > +	unsigned int edx;
> > +	unsigned long pa;
> > +	int i;
> > +	int ret;
> > +
> > +	for (i = 0; i < SGX_MAX_EPC_BANKS; i++) {
> > +		cpuid_count(SGX_CPUID, i + SGX_CPUID_EPC_BANKS, &eax, &ebx,
> > +			    &ecx, &edx);
> > +		if (!(eax & 0xf))
> > +			break;
> > +
> > +		pa   = ((u64)(ebx & 0xfffff) << 32) + (u64)(eax & 0xfffff000);
> > +		size = ((u64)(edx & 0xfffff) << 32) + (u64)(ecx & 0xfffff000);
> 
> Please align these like I did ^
> 
> > +		pr_info("EPC bank 0x%lx-0x%lx\n", pa, pa + size);
> > +
> > +		ret = sgx_init_epc_bank(pa, size, i, &sgx_epc_banks[i]);
> > +		if (ret) {
> > +			sgx_page_cache_teardown();
> > +			return ret;
> > +		}
> > +
> > +		sgx_nr_epc_banks++;
> > +	}
> 
> This is also rather sparsely commented.
> 
> > +static __init bool sgx_is_enabled(bool *lc_enabled)
> >  {
> >  	unsigned long fc;
> >  
> > @@ -41,12 +466,26 @@ static __init bool sgx_is_enabled(void)
> >  	if (!(fc & FEATURE_CONTROL_SGX_ENABLE))
> >  		return false;
> >  
> > +	*lc_enabled = !!(fc & FEATURE_CONTROL_SGX_LE_WR);
> > +
> >  	return true;
> >  }
> 
> I'm baffled why lc_enabled is connected to the enclave page cache.

KVM works only with writable MSRs. Driver works both with writable
and read-only MSRs.

Thanks, I'll try my best to deal with all this :-)

/Jarkko

Re: [PATCH v11 00/13] Intel SGX1 support

[Jarkko Sakkinen]

On Tue, Jun 12, 2018 at 12:50:12PM +0200, Pavel Machek wrote:
> On Fri 2018-06-08 19:09:35, Jarkko Sakkinen wrote:
> > Intel(R) SGX is a set of CPU instructions that can be used by applications
> > to set aside private regions of code and data. The code outside the enclave
> > is disallowed to access the memory inside the enclave by the CPU access
> > control.  In a way you can think that SGX provides inverted sandbox. It
> > protects the application from a malicious host.
> 
> Do you intend to allow non-root applications to use SGX?
> 
> What are non-evil uses for SGX?
> 
> ...because it is quite useful for some kinds of evil:

The default permissions for the device are 600.

/Jarkko

Re: [PATCH v11 00/13] Intel SGX1 support

[Jarkko Sakkinen]

On Tue, Jun 12, 2018 at 12:50:12PM +0200, Pavel Machek wrote:
> On Fri 2018-06-08 19:09:35, Jarkko Sakkinen wrote:
> > Intel(R) SGX is a set of CPU instructions that can be used by applications
> > to set aside private regions of code and data. The code outside the enclave
> > is disallowed to access the memory inside the enclave by the CPU access
> > control.  In a way you can think that SGX provides inverted sandbox. It
> > protects the application from a malicious host.
> 
> Do you intend to allow non-root applications to use SGX?
> 
> What are non-evil uses for SGX?
> 
> ...because it is quite useful for some kinds of evil:

The default permissions for the device are 600.

/Jarkko

Re: [PATCH v11 00/13] Intel SGX1 support

[Jarkko Sakkinen]

On Tue, Jun 12, 2018 at 12:50:12PM +0200, Pavel Machek wrote:
> On Fri 2018-06-08 19:09:35, Jarkko Sakkinen wrote:
> > Intel(R) SGX is a set of CPU instructions that can be used by applications
> > to set aside private regions of code and data. The code outside the enclave
> > is disallowed to access the memory inside the enclave by the CPU access
> > control.  In a way you can think that SGX provides inverted sandbox. It
> > protects the application from a malicious host.
> 
> Do you intend to allow non-root applications to use SGX?
> 
> What are non-evil uses for SGX?
> 
> ...because it is quite useful for some kinds of evil:

The default permissions for the device are 600.

/Jarkko
--
To unsubscribe from this list: send the line "unsubscribe linux-doc" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [intel-sgx-kernel-dev] [PATCH v11 13/13] intel_sgx: in-kernel launch enclave

[Jarkko Sakkinen]

On Fri, Jun 08, 2018 at 11:50:14AM -0700, Andy Lutomirski wrote:
> On Fri, Jun 8, 2018 at 10:32 AM Jarkko Sakkinen
> <jarkko.sakkinen@linux.intel.com> wrote:
> >
> > The Launch Enclave (LE) generates cryptographic launch tokens for user
> > enclaves. A launch token is used by EINIT to check whether the enclave
> > is authorized to launch or not. By having its own launch enclave, Linux
> > has full control of the enclave launch process.
> >
> > LE is wrapped into a user space proxy program that reads enclave
> > signatures outputs launch tokens. The kernel-side glue code is
> > implemented by using the user space helper framework.  The IPC between
> > the LE proxy program and kernel is handled with an anonymous inode.
> >
> > The commit also adds enclave signing tool that is used by kbuild to
> > measure and sign the launch enclave. CONFIG_INTEL_SGX_SIGNING_KEY points
> > to a PEM-file for the 3072-bit RSA key that is used as the LE public key
> > pair. The default location is:
> 
> It might be nice to use the infrastructure that Alexei added for
> bpfilter (the umh_blob stuff) here, which is slated for merging in
> this merge window.
> 
> --Andy

Thanks, not familiar with this work. Is there any documentation for
it available?

/Jarkko

Re: [PATCH v11 09/13] x86, sgx: basic routines for enclave page cache

[Neil Horman]

On Tue, Jun 19, 2018 at 05:57:53PM +0300, Jarkko Sakkinen wrote:
> On Fri, Jun 08, 2018 at 11:24:12AM -0700, Dave Hansen wrote:
> > On 06/08/2018 10:09 AM, Jarkko Sakkinen wrote:
> > > SGX has a set of data structures to maintain information about the enclaves
> > > and their security properties. BIOS reserves a fixed size region of
> > > physical memory for these structures by setting Processor Reserved Memory
> > > Range Registers (PRMRR). This memory area is called Enclave Page Cache
> > > (EPC).
> > > 
> > > This commit implements the basic routines to allocate and free pages from
> > > different EPC banks. There is also a swapper thread ksgxswapd for EPC pages
> > > that gets woken up by sgx_alloc_page() when we run below the low watermark.
> > > The swapper thread continues swapping pages up until it reaches the high
> > > watermark.
> > 
> > Yay!  A new memory manager in arch-specific code.
> > 
> > > Each subsystem that uses SGX must provide a set of callbacks for EPC
> > > pages that are used to reclaim, block and write an EPC page. Kernel
> > > takes the responsibility of maintaining LRU cache for them.
> > 
> > What does a "subsystem that uses SGX" mean?  Do we have one of those
> > already?
> 
> Driver and KVM.
> 
> > > +struct sgx_secs {
> > > +	uint64_t size;
> > > +	uint64_t base;
> > > +	uint32_t ssaframesize;
> > > +	uint32_t miscselect;
> > > +	uint8_t reserved1[SGX_SECS_RESERVED1_SIZE];
> > > +	uint64_t attributes;
> > > +	uint64_t xfrm;
> > > +	uint32_t mrenclave[8];
> > > +	uint8_t reserved2[SGX_SECS_RESERVED2_SIZE];
> > > +	uint32_t mrsigner[8];
> > > +	uint8_t	reserved3[SGX_SECS_RESERVED3_SIZE];
> > > +	uint16_t isvvprodid;
> > > +	uint16_t isvsvn;
> > > +	uint8_t reserved4[SGX_SECS_RESERVED4_SIZE];
> > > +};
> > 
> > This is a hardware structure, right?  Doesn't it need to be packed?
> 
> Everything is aligned properly in this struct.
> 
I don't think you can guarantee that.  I understand that the reserved size is
likely computed to turn those u8's into 32/64 byte regions, but the uint16t
isvvprodid and isvsvn might get padded to 32 or 64 bytes in software dependent
on the processor you build for.

And even so, its suceptible to being
misaligned if a new version of the hardware adds or removes elements.  Adding a
packed attribute seems like a safe approach (or at least a no-op in the current
state)

Neil

Re: [PATCH v11 09/13] x86, sgx: basic routines for enclave page cache

[Neil Horman]

On Tue, Jun 19, 2018 at 05:57:53PM +0300, Jarkko Sakkinen wrote:
> On Fri, Jun 08, 2018 at 11:24:12AM -0700, Dave Hansen wrote:
> > On 06/08/2018 10:09 AM, Jarkko Sakkinen wrote:
> > > SGX has a set of data structures to maintain information about the enclaves
> > > and their security properties. BIOS reserves a fixed size region of
> > > physical memory for these structures by setting Processor Reserved Memory
> > > Range Registers (PRMRR). This memory area is called Enclave Page Cache
> > > (EPC).
> > > 
> > > This commit implements the basic routines to allocate and free pages from
> > > different EPC banks. There is also a swapper thread ksgxswapd for EPC pages
> > > that gets woken up by sgx_alloc_page() when we run below the low watermark.
> > > The swapper thread continues swapping pages up until it reaches the high
> > > watermark.
> > 
> > Yay!  A new memory manager in arch-specific code.
> > 
> > > Each subsystem that uses SGX must provide a set of callbacks for EPC
> > > pages that are used to reclaim, block and write an EPC page. Kernel
> > > takes the responsibility of maintaining LRU cache for them.
> > 
> > What does a "subsystem that uses SGX" mean?  Do we have one of those
> > already?
> 
> Driver and KVM.
> 
> > > +struct sgx_secs {
> > > +	uint64_t size;
> > > +	uint64_t base;
> > > +	uint32_t ssaframesize;
> > > +	uint32_t miscselect;
> > > +	uint8_t reserved1[SGX_SECS_RESERVED1_SIZE];
> > > +	uint64_t attributes;
> > > +	uint64_t xfrm;
> > > +	uint32_t mrenclave[8];
> > > +	uint8_t reserved2[SGX_SECS_RESERVED2_SIZE];
> > > +	uint32_t mrsigner[8];
> > > +	uint8_t	reserved3[SGX_SECS_RESERVED3_SIZE];
> > > +	uint16_t isvvprodid;
> > > +	uint16_t isvsvn;
> > > +	uint8_t reserved4[SGX_SECS_RESERVED4_SIZE];
> > > +};
> > 
> > This is a hardware structure, right?  Doesn't it need to be packed?
> 
> Everything is aligned properly in this struct.
> 
I don't think you can guarantee that.  I understand that the reserved size is
likely computed to turn those u8's into 32/64 byte regions, but the uint16t
isvvprodid and isvsvn might get padded to 32 or 64 bytes in software dependent
on the processor you build for.

And even so, its suceptible to being
misaligned if a new version of the hardware adds or removes elements.  Adding a
packed attribute seems like a safe approach (or at least a no-op in the current
state)

Neil

Re: [PATCH v11 09/13] x86, sgx: basic routines for enclave page cache

[Dave Hansen]

On 06/19/2018 07:57 AM, Jarkko Sakkinen wrote:
> On Fri, Jun 08, 2018 at 11:24:12AM -0700, Dave Hansen wrote:
>>> Each subsystem that uses SGX must provide a set of callbacks for EPC
>>> pages that are used to reclaim, block and write an EPC page. Kernel
>>> takes the responsibility of maintaining LRU cache for them.
>>
>> What does a "subsystem that uses SGX" mean?  Do we have one of those
>> already?
> 
> Driver and KVM.

Could you just say "the SGX and driver both provide a set of callbacks"?

>>> +struct sgx_secs {
>>> +	uint64_t size;
>>> +	uint64_t base;
>>> +	uint32_t ssaframesize;
>>> +	uint32_t miscselect;
>>> +	uint8_t reserved1[SGX_SECS_RESERVED1_SIZE];
>>> +	uint64_t attributes;
>>> +	uint64_t xfrm;
>>> +	uint32_t mrenclave[8];
>>> +	uint8_t reserved2[SGX_SECS_RESERVED2_SIZE];
>>> +	uint32_t mrsigner[8];
>>> +	uint8_t	reserved3[SGX_SECS_RESERVED3_SIZE];
>>> +	uint16_t isvvprodid;
>>> +	uint16_t isvsvn;
>>> +	uint8_t reserved4[SGX_SECS_RESERVED4_SIZE];
>>> +};
>>
>> This is a hardware structure, right?  Doesn't it need to be packed?
> 
> Everything is aligned properly in this struct.

The compiler doesn't guarantee the way you have it laid out.  It might
work today, but it's subject to being changed.

>>> +enum sgx_tcs_flags {
>>> +	SGX_TCS_DBGOPTIN	= 0x01, /* cleared on EADD */
>>> +};
>>> +
>>> +#define SGX_TCS_RESERVED_MASK 0xFFFFFFFFFFFFFFFEL
>>
>> Would it be possible to separate out the SGX software structures from
>> SGX hardware?  It's hard to tell them apart.
> 
> How do you draw the line in the architectural structures?

I know then when I see them.

"SGX_TCS_DBGOPTIN" - Hardware
"SGX_NR_TO_SCAN" - Software

Please at least make an effort to do this.

>>> +#define SGX_NR_TO_SCAN	16
>>> +#define SGX_NR_LOW_PAGES 32
>>> +#define SGX_NR_HIGH_PAGES 64
>>> +
>>>  bool sgx_enabled __ro_after_init = false;
>>>  EXPORT_SYMBOL(sgx_enabled);
>>> +bool sgx_lc_enabled __ro_after_init;
>>> +EXPORT_SYMBOL(sgx_lc_enabled);
>>> +atomic_t sgx_nr_free_pages = ATOMIC_INIT(0);
>>
>> Hmmm, global atomic.  Doesn't sound very scalable.
> 
> We could potentially remove this completely as banks have 'free_cnt'
> field and use the sum when needed as the value.

That seems prudent.

>>> +struct sgx_epc_bank sgx_epc_banks[SGX_MAX_EPC_BANKS];
>>> +EXPORT_SYMBOL(sgx_epc_banks);
>>> +int sgx_nr_epc_banks;
>>> +EXPORT_SYMBOL(sgx_nr_epc_banks);
>>> +LIST_HEAD(sgx_active_page_list);
>>> +EXPORT_SYMBOL(sgx_active_page_list);
>>> +DEFINE_SPINLOCK(sgx_active_page_list_lock);
>>> +EXPORT_SYMBOL(sgx_active_page_list_lock);
>>
>> Hmmm, global spinlock protecting a page allocator linked list.  Sounds
>> even worse than at atomic.
>>
>> Why is this OK?
> 
> Any suggestions what would be a better place in order to make a
> fine grained granularity?

The bank seems a logical place.  Or, create a structure that actually
hangs off NUMA nodes.

BTW, do we *have* locality information for SGX banks?
>>> +/**
>>> + * sgx_try_alloc_page - try to allocate an EPC page
>>> + * @impl:	implementation for the struct sgx_epc_page
>>> + *
>>> + * Try to grab a page from the free EPC page list. If there is a free page
>>> + * available, it is returned to the caller.
>>> + *
>>> + * Return:
>>> + *   a &struct sgx_epc_page instace,
>>> + *   NULL otherwise
>>> + */
>>> +struct sgx_epc_page *sgx_try_alloc_page(struct sgx_epc_page_impl *impl)
>>> +{
>>> +	struct sgx_epc_bank *bank;
>>> +	struct sgx_epc_page *page = NULL;
>>> +	int i;
>>> +
>>> +	for (i = 0; i < sgx_nr_epc_banks; i++) {
>>> +		bank = &sgx_epc_banks[i];
>>
>> What's a bank?  How many banks does a system have?
> 
> AFAIK, UMA systems have one bank. NUMA have multiple. It is a physical
> memory region reserved for enclave pages.

That's great text to include near the structure definition for
sgx_epc_bank.

>>> +		down_write(&bank->lock);
>>> +
>>> +		if (atomic_read(&bank->free_cnt))
>>> +			page = bank->pages[atomic_dec_return(&bank->free_cnt)];
>>
>> Why is a semaphore getting used here?  I don't see any sleeping or
>> anything happening under this lock.
> 
> Should be changed to reader-writer spinlock, thanks.

Which also reminds me...  It would be nice to explicitly call out why
you need an atomic_t inside a lock-protected structure.

>>> +	}
>>> +
>>> +	if (atomic_read(&sgx_nr_free_pages) < SGX_NR_LOW_PAGES)
>>> +		wake_up(&ksgxswapd_waitq);
>>> +
>>> +	return entry;
>>> +}
>>> +EXPORT_SYMBOL(sgx_alloc_page);
>>
>> Why aren't these _GPL exports?
> 
> Source files a dual licensed.

Sounds like a great thing to ask your licensing or legal team about.

>>> +/**
>>> + * sgx_free_page - free an EPC page
>>> + *
>>> + * @page:	any EPC page
>>> + *
>>> + * Remove an EPC page and insert it back to the list of free pages.
>>> + *
>>> + * Return: SGX error code
>>> + */
>>> +int sgx_free_page(struct sgx_epc_page *page)
>>> +{
>>> +	struct sgx_epc_bank *bank = SGX_EPC_BANK(page);
>>> +	int ret;
>>> +
>>> +	ret = sgx_eremove(page);
>>> +	if (ret) {
>>> +		pr_debug("EREMOVE returned %d\n", ret);
>>> +		return ret;
>>> +	}
>>> +
>>> +	down_read(&bank->lock);
>>> +	bank->pages[atomic_inc_return(&bank->free_cnt) - 1] = page;
>>> +	atomic_inc(&sgx_nr_free_pages);
>>> +	up_read(&bank->lock);
>>> +
>>> +	return 0;
>>> +}
>>
>> bank->lock confuses me.  This seems to be writing to a bank, but only
>> needs a read lock.  Why?
> 
> It could be either way around:
> 
> 1. Allow multiple threads that free a page to access the array.
> 2. Allow multiple threads that alloc a page to access the array.

Whatever way you choose, please document the locking scheme.

>>> +/**
>>> + * sgx_get_page - pin an EPC page
>>> + * @page:	an EPC page
>>> + *
>>> + * Return: a pointer to the pinned EPC page
>>> + */
>>> +void *sgx_get_page(struct sgx_epc_page *page)
>>> +{
>>> +	struct sgx_epc_bank *bank = SGX_EPC_BANK(page);
>>> +
>>> +	if (IS_ENABLED(CONFIG_X86_64))
>>> +		return (void *)(bank->va + SGX_EPC_ADDR(page) - bank->pa);
>>> +
>>> +	return kmap_atomic_pfn(SGX_EPC_PFN(page));
>>> +}
>>> +EXPORT_SYMBOL(sgx_get_page);
>>
>> This is odd.  Do you really want to detect 64-bit, or CONFIG_HIGHMEM?
> 
> For 32-bit (albeit not supported at this point) it makes sense to always
> use kmap_atomic_pfn() as the virtua address area is very limited.

That makes no sense.  32-bit kernels have plenty of virtual address
space if not using highmem.

>>> +struct page *sgx_get_backing(struct file *file, pgoff_t index)
>>> +{
>>> +	struct inode *inode = file->f_path.dentry->d_inode;
>>> +	struct address_space *mapping = inode->i_mapping;
>>> +	gfp_t gfpmask = mapping_gfp_mask(mapping);
>>> +
>>> +	return shmem_read_mapping_page_gfp(mapping, index, gfpmask);
>>> +}
>>> +EXPORT_SYMBOL(sgx_get_backing);
>>
>> What does shmem have to do with all this?
> 
> Backing storage is an shmem file similarly is in drm.

That's something good to call out in the changelog: how shmem gets used
here.

>>> +static __init bool sgx_is_enabled(bool *lc_enabled)
>>>  {
>>>  	unsigned long fc;
>>>  
>>> @@ -41,12 +466,26 @@ static __init bool sgx_is_enabled(void)
>>>  	if (!(fc & FEATURE_CONTROL_SGX_ENABLE))
>>>  		return false;
>>>  
>>> +	*lc_enabled = !!(fc & FEATURE_CONTROL_SGX_LE_WR);
>>> +
>>>  	return true;
>>>  }
>>
>> I'm baffled why lc_enabled is connected to the enclave page cache.
> 
> KVM works only with writable MSRs. Driver works both with writable
> and read-only MSRs.

Could you help with my confusion by documenting this a bit?

Re: [PATCH v11 09/13] x86, sgx: basic routines for enclave page cache

[Dave Hansen]

On 06/19/2018 07:57 AM, Jarkko Sakkinen wrote:
> On Fri, Jun 08, 2018 at 11:24:12AM -0700, Dave Hansen wrote:
>>> Each subsystem that uses SGX must provide a set of callbacks for EPC
>>> pages that are used to reclaim, block and write an EPC page. Kernel
>>> takes the responsibility of maintaining LRU cache for them.
>>
>> What does a "subsystem that uses SGX" mean?  Do we have one of those
>> already?
> 
> Driver and KVM.

Could you just say "the SGX and driver both provide a set of callbacks"?

>>> +struct sgx_secs {
>>> +	uint64_t size;
>>> +	uint64_t base;
>>> +	uint32_t ssaframesize;
>>> +	uint32_t miscselect;
>>> +	uint8_t reserved1[SGX_SECS_RESERVED1_SIZE];
>>> +	uint64_t attributes;
>>> +	uint64_t xfrm;
>>> +	uint32_t mrenclave[8];
>>> +	uint8_t reserved2[SGX_SECS_RESERVED2_SIZE];
>>> +	uint32_t mrsigner[8];
>>> +	uint8_t	reserved3[SGX_SECS_RESERVED3_SIZE];
>>> +	uint16_t isvvprodid;
>>> +	uint16_t isvsvn;
>>> +	uint8_t reserved4[SGX_SECS_RESERVED4_SIZE];
>>> +};
>>
>> This is a hardware structure, right?  Doesn't it need to be packed?
> 
> Everything is aligned properly in this struct.

The compiler doesn't guarantee the way you have it laid out.  It might
work today, but it's subject to being changed.

>>> +enum sgx_tcs_flags {
>>> +	SGX_TCS_DBGOPTIN	= 0x01, /* cleared on EADD */
>>> +};
>>> +
>>> +#define SGX_TCS_RESERVED_MASK 0xFFFFFFFFFFFFFFFEL
>>
>> Would it be possible to separate out the SGX software structures from
>> SGX hardware?  It's hard to tell them apart.
> 
> How do you draw the line in the architectural structures?

I know then when I see them.

"SGX_TCS_DBGOPTIN" - Hardware
"SGX_NR_TO_SCAN" - Software

Please at least make an effort to do this.

>>> +#define SGX_NR_TO_SCAN	16
>>> +#define SGX_NR_LOW_PAGES 32
>>> +#define SGX_NR_HIGH_PAGES 64
>>> +
>>>  bool sgx_enabled __ro_after_init = false;
>>>  EXPORT_SYMBOL(sgx_enabled);
>>> +bool sgx_lc_enabled __ro_after_init;
>>> +EXPORT_SYMBOL(sgx_lc_enabled);
>>> +atomic_t sgx_nr_free_pages = ATOMIC_INIT(0);
>>
>> Hmmm, global atomic.  Doesn't sound very scalable.
> 
> We could potentially remove this completely as banks have 'free_cnt'
> field and use the sum when needed as the value.

That seems prudent.

>>> +struct sgx_epc_bank sgx_epc_banks[SGX_MAX_EPC_BANKS];
>>> +EXPORT_SYMBOL(sgx_epc_banks);
>>> +int sgx_nr_epc_banks;
>>> +EXPORT_SYMBOL(sgx_nr_epc_banks);
>>> +LIST_HEAD(sgx_active_page_list);
>>> +EXPORT_SYMBOL(sgx_active_page_list);
>>> +DEFINE_SPINLOCK(sgx_active_page_list_lock);
>>> +EXPORT_SYMBOL(sgx_active_page_list_lock);
>>
>> Hmmm, global spinlock protecting a page allocator linked list.  Sounds
>> even worse than at atomic.
>>
>> Why is this OK?
> 
> Any suggestions what would be a better place in order to make a
> fine grained granularity?

The bank seems a logical place.  Or, create a structure that actually
hangs off NUMA nodes.

BTW, do we *have* locality information for SGX banks?
>>> +/**
>>> + * sgx_try_alloc_page - try to allocate an EPC page
>>> + * @impl:	implementation for the struct sgx_epc_page
>>> + *
>>> + * Try to grab a page from the free EPC page list. If there is a free page
>>> + * available, it is returned to the caller.
>>> + *
>>> + * Return:
>>> + *   a &struct sgx_epc_page instace,
>>> + *   NULL otherwise
>>> + */
>>> +struct sgx_epc_page *sgx_try_alloc_page(struct sgx_epc_page_impl *impl)
>>> +{
>>> +	struct sgx_epc_bank *bank;
>>> +	struct sgx_epc_page *page = NULL;
>>> +	int i;
>>> +
>>> +	for (i = 0; i < sgx_nr_epc_banks; i++) {
>>> +		bank = &sgx_epc_banks[i];
>>
>> What's a bank?  How many banks does a system have?
> 
> AFAIK, UMA systems have one bank. NUMA have multiple. It is a physical
> memory region reserved for enclave pages.

That's great text to include near the structure definition for
sgx_epc_bank.

>>> +		down_write(&bank->lock);
>>> +
>>> +		if (atomic_read(&bank->free_cnt))
>>> +			page = bank->pages[atomic_dec_return(&bank->free_cnt)];
>>
>> Why is a semaphore getting used here?  I don't see any sleeping or
>> anything happening under this lock.
> 
> Should be changed to reader-writer spinlock, thanks.

Which also reminds me...  It would be nice to explicitly call out why
you need an atomic_t inside a lock-protected structure.

>>> +	}
>>> +
>>> +	if (atomic_read(&sgx_nr_free_pages) < SGX_NR_LOW_PAGES)
>>> +		wake_up(&ksgxswapd_waitq);
>>> +
>>> +	return entry;
>>> +}
>>> +EXPORT_SYMBOL(sgx_alloc_page);
>>
>> Why aren't these _GPL exports?
> 
> Source files a dual licensed.

Sounds like a great thing to ask your licensing or legal team about.

>>> +/**
>>> + * sgx_free_page - free an EPC page
>>> + *
>>> + * @page:	any EPC page
>>> + *
>>> + * Remove an EPC page and insert it back to the list of free pages.
>>> + *
>>> + * Return: SGX error code
>>> + */
>>> +int sgx_free_page(struct sgx_epc_page *page)
>>> +{
>>> +	struct sgx_epc_bank *bank = SGX_EPC_BANK(page);
>>> +	int ret;
>>> +
>>> +	ret = sgx_eremove(page);
>>> +	if (ret) {
>>> +		pr_debug("EREMOVE returned %d\n", ret);
>>> +		return ret;
>>> +	}
>>> +
>>> +	down_read(&bank->lock);
>>> +	bank->pages[atomic_inc_return(&bank->free_cnt) - 1] = page;
>>> +	atomic_inc(&sgx_nr_free_pages);
>>> +	up_read(&bank->lock);
>>> +
>>> +	return 0;
>>> +}
>>
>> bank->lock confuses me.  This seems to be writing to a bank, but only
>> needs a read lock.  Why?
> 
> It could be either way around:
> 
> 1. Allow multiple threads that free a page to access the array.
> 2. Allow multiple threads that alloc a page to access the array.

Whatever way you choose, please document the locking scheme.

>>> +/**
>>> + * sgx_get_page - pin an EPC page
>>> + * @page:	an EPC page
>>> + *
>>> + * Return: a pointer to the pinned EPC page
>>> + */
>>> +void *sgx_get_page(struct sgx_epc_page *page)
>>> +{
>>> +	struct sgx_epc_bank *bank = SGX_EPC_BANK(page);
>>> +
>>> +	if (IS_ENABLED(CONFIG_X86_64))
>>> +		return (void *)(bank->va + SGX_EPC_ADDR(page) - bank->pa);
>>> +
>>> +	return kmap_atomic_pfn(SGX_EPC_PFN(page));
>>> +}
>>> +EXPORT_SYMBOL(sgx_get_page);
>>
>> This is odd.  Do you really want to detect 64-bit, or CONFIG_HIGHMEM?
> 
> For 32-bit (albeit not supported at this point) it makes sense to always
> use kmap_atomic_pfn() as the virtua address area is very limited.

That makes no sense.  32-bit kernels have plenty of virtual address
space if not using highmem.

>>> +struct page *sgx_get_backing(struct file *file, pgoff_t index)
>>> +{
>>> +	struct inode *inode = file->f_path.dentry->d_inode;
>>> +	struct address_space *mapping = inode->i_mapping;
>>> +	gfp_t gfpmask = mapping_gfp_mask(mapping);
>>> +
>>> +	return shmem_read_mapping_page_gfp(mapping, index, gfpmask);
>>> +}
>>> +EXPORT_SYMBOL(sgx_get_backing);
>>
>> What does shmem have to do with all this?
> 
> Backing storage is an shmem file similarly is in drm.

That's something good to call out in the changelog: how shmem gets used
here.

>>> +static __init bool sgx_is_enabled(bool *lc_enabled)
>>>  {
>>>  	unsigned long fc;
>>>  
>>> @@ -41,12 +466,26 @@ static __init bool sgx_is_enabled(void)
>>>  	if (!(fc & FEATURE_CONTROL_SGX_ENABLE))
>>>  		return false;
>>>  
>>> +	*lc_enabled = !!(fc & FEATURE_CONTROL_SGX_LE_WR);
>>> +
>>>  	return true;
>>>  }
>>
>> I'm baffled why lc_enabled is connected to the enclave page cache.
> 
> KVM works only with writable MSRs. Driver works both with writable
> and read-only MSRs.

Could you help with my confusion by documenting this a bit?

Re: [PATCH v11 09/13] x86, sgx: basic routines for enclave page cache

[Jethro Beekman]

[-- Attachment #1: Type: text/plain, Size: 861 bytes --]

On 2018-06-19 07:08, Jarkko Sakkinen wrote:
> On Fri, Jun 08, 2018 at 11:21:48AM -0700, Jethro Beekman wrote:
>> On 2018-06-08 10:09, Jarkko Sakkinen wrote:
>>> +/*
>>> + * Writing the LE hash MSRs is extraordinarily expensive, e.g.
>>> + * 3-4x slower than normal MSRs, so we use a per-cpu cache to
>>> + * track the last known value of the MSRs to avoid unnecessarily
>>> + * writing the MSRs with the current value.  Because most Linux
>>> + * kernels will use an LE that is signed with a non-Intel key,
>>
>> I don't think you can predict what most Linux kernels will be doing. I think
>> not initializing the cache to the CPU's initial value is fine, but this
>> particular argument shouldn't appear in the rationale.
> 
> Are you just referring to the last sentence or the whole paragraph?

Just the last sentence.

> /Jarkko
Jethro


[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 3994 bytes --]

Re: [PATCH v11 09/13] x86, sgx: basic routines for enclave page cache

[Jethro Beekman]

[-- Attachment #1: Type: text/plain, Size: 861 bytes --]

On 2018-06-19 07:08, Jarkko Sakkinen wrote:
> On Fri, Jun 08, 2018 at 11:21:48AM -0700, Jethro Beekman wrote:
>> On 2018-06-08 10:09, Jarkko Sakkinen wrote:
>>> +/*
>>> + * Writing the LE hash MSRs is extraordinarily expensive, e.g.
>>> + * 3-4x slower than normal MSRs, so we use a per-cpu cache to
>>> + * track the last known value of the MSRs to avoid unnecessarily
>>> + * writing the MSRs with the current value.  Because most Linux
>>> + * kernels will use an LE that is signed with a non-Intel key,
>>
>> I don't think you can predict what most Linux kernels will be doing. I think
>> not initializing the cache to the CPU's initial value is fine, but this
>> particular argument shouldn't appear in the rationale.
> 
> Are you just referring to the last sentence or the whole paragraph?

Just the last sentence.

> /Jarkko
Jethro


[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 3994 bytes --]

Re: [PATCH v11 09/13] x86, sgx: basic routines for enclave page cache

[Sean Christopherson]

On Tue, Jun 19, 2018 at 05:57:53PM +0300, Jarkko Sakkinen wrote:
> On Fri, Jun 08, 2018 at 11:24:12AM -0700, Dave Hansen wrote:
> > On 06/08/2018 10:09 AM, Jarkko Sakkinen wrote:
> > > +static __init bool sgx_is_enabled(bool *lc_enabled)
> > >  {
> > >  	unsigned long fc;
> > >  
> > > @@ -41,12 +466,26 @@ static __init bool sgx_is_enabled(void)
> > >  	if (!(fc & FEATURE_CONTROL_SGX_ENABLE))
> > >  		return false;
> > >  
> > > +	*lc_enabled = !!(fc & FEATURE_CONTROL_SGX_LE_WR);
> > > +
> > >  	return true;
> > >  }
> > 
> > I'm baffled why lc_enabled is connected to the enclave page cache.
> 
> KVM works only with writable MSRs. Driver works both with writable
> and read-only MSRs.

That's not true, KVM can/will support SGX regardless of whether or not
Launch Control (LC) is available and/or enabled.  KVM does need to
know whether or not LC is enabled.

Back to Dave's question, LC isn't connected to the EPC management,
the LC code should be split into a separate patch.
 
> Thanks, I'll try my best to deal with all this :-)
> 
> /Jarkko

Re: [PATCH v11 09/13] x86, sgx: basic routines for enclave page cache

[Sean Christopherson]

On Tue, Jun 19, 2018 at 05:57:53PM +0300, Jarkko Sakkinen wrote:
> On Fri, Jun 08, 2018 at 11:24:12AM -0700, Dave Hansen wrote:
> > On 06/08/2018 10:09 AM, Jarkko Sakkinen wrote:
> > > +static __init bool sgx_is_enabled(bool *lc_enabled)
> > >  {
> > >  	unsigned long fc;
> > >  
> > > @@ -41,12 +466,26 @@ static __init bool sgx_is_enabled(void)
> > >  	if (!(fc & FEATURE_CONTROL_SGX_ENABLE))
> > >  		return false;
> > >  
> > > +	*lc_enabled = !!(fc & FEATURE_CONTROL_SGX_LE_WR);
> > > +
> > >  	return true;
> > >  }
> > 
> > I'm baffled why lc_enabled is connected to the enclave page cache.
> 
> KVM works only with writable MSRs. Driver works both with writable
> and read-only MSRs.

That's not true, KVM can/will support SGX regardless of whether or not
Launch Control (LC) is available and/or enabled.  KVM does need to
know whether or not LC is enabled.

Back to Dave's question, LC isn't connected to the EPC management,
the LC code should be split into a separate patch.
 
> Thanks, I'll try my best to deal with all this :-)
> 
> /Jarkko

Re: [PATCH v11 00/13] Intel SGX1 support

[Pavel Machek]

[-- Attachment #1: Type: text/plain, Size: 1050 bytes --]

On Tue 2018-06-19 17:59:43, Jarkko Sakkinen wrote:
> On Tue, Jun 12, 2018 at 12:50:12PM +0200, Pavel Machek wrote:
> > On Fri 2018-06-08 19:09:35, Jarkko Sakkinen wrote:
> > > Intel(R) SGX is a set of CPU instructions that can be used by applications
> > > to set aside private regions of code and data. The code outside the enclave
> > > is disallowed to access the memory inside the enclave by the CPU access
> > > control.  In a way you can think that SGX provides inverted sandbox. It
> > > protects the application from a malicious host.
> > 
> > Do you intend to allow non-root applications to use SGX?
> > 
> > What are non-evil uses for SGX?
> > 
> > ...because it is quite useful for some kinds of evil:
> 
> The default permissions for the device are 600.

Good. This does not belong to non-root. But question still remains:

What are some non-evil uses for SGX?

									Pavel
-- 
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 181 bytes --]

Re: [PATCH v11 00/13] Intel SGX1 support

[Pavel Machek]

[-- Attachment #1: Type: text/plain, Size: 1050 bytes --]

On Tue 2018-06-19 17:59:43, Jarkko Sakkinen wrote:
> On Tue, Jun 12, 2018 at 12:50:12PM +0200, Pavel Machek wrote:
> > On Fri 2018-06-08 19:09:35, Jarkko Sakkinen wrote:
> > > Intel(R) SGX is a set of CPU instructions that can be used by applications
> > > to set aside private regions of code and data. The code outside the enclave
> > > is disallowed to access the memory inside the enclave by the CPU access
> > > control.  In a way you can think that SGX provides inverted sandbox. It
> > > protects the application from a malicious host.
> > 
> > Do you intend to allow non-root applications to use SGX?
> > 
> > What are non-evil uses for SGX?
> > 
> > ...because it is quite useful for some kinds of evil:
> 
> The default permissions for the device are 600.

Good. This does not belong to non-root. But question still remains:

What are some non-evil uses for SGX?

									Pavel
-- 
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 181 bytes --]

Re: [PATCH v11 00/13] Intel SGX1 support

[Peter Zijlstra]

On Tue, Jun 19, 2018 at 10:04:15PM +0200, Pavel Machek wrote:
> What are some non-evil uses for SGX?

Soft-TPM is one that was proposed at some time.

Re: [PATCH v11 00/13] Intel SGX1 support

[Peter Zijlstra]

On Tue, Jun 19, 2018 at 10:04:15PM +0200, Pavel Machek wrote:
> What are some non-evil uses for SGX?

Soft-TPM is one that was proposed at some time.

Re: [PATCH v11 00/13] Intel SGX1 support

[Peter Zijlstra]

On Tue, Jun 19, 2018 at 10:04:15PM +0200, Pavel Machek wrote:
> What are some non-evil uses for SGX?

Soft-TPM is one that was proposed at some time.

Re: [PATCH v11 00/13] Intel SGX1 support

[Peter Zijlstra]

On Tue, Jun 19, 2018 at 10:04:15PM +0200, Pavel Machek wrote:
> What are some non-evil uses for SGX?

Soft-TPM is one that was proposed at some time.
--
To unsubscribe from this list: send the line "unsubscribe linux-doc" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH v11 00/13] Intel SGX1 support

[Peter Zijlstra]

*sigh*, could you not cross-post with closed/moderated lists, that's rude.

Re: [PATCH v11 00/13] Intel SGX1 support

[Peter Zijlstra]

*sigh*, could you not cross-post with closed/moderated lists, that's rude.

Re: [PATCH v11 00/13] Intel SGX1 support

[Peter Zijlstra]

*sigh*, could you not cross-post with closed/moderated lists, that's rude.

Re: [PATCH v11 00/13] Intel SGX1 support

[Peter Zijlstra]

*sigh*, could you not cross-post with closed/moderated lists, that's rude.
--
To unsubscribe from this list: send the line "unsubscribe linux-doc" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH v11 00/13] Intel SGX1 support

[Josh Triplett]

On Tue, Jun 19, 2018 at 10:04:15PM +0200, Pavel Machek wrote:
> On Tue 2018-06-19 17:59:43, Jarkko Sakkinen wrote:
> > On Tue, Jun 12, 2018 at 12:50:12PM +0200, Pavel Machek wrote:
> > > On Fri 2018-06-08 19:09:35, Jarkko Sakkinen wrote:
> > > > Intel(R) SGX is a set of CPU instructions that can be used by applications
> > > > to set aside private regions of code and data. The code outside the enclave
> > > > is disallowed to access the memory inside the enclave by the CPU access
> > > > control.  In a way you can think that SGX provides inverted sandbox. It
> > > > protects the application from a malicious host.
> > > 
> > > Do you intend to allow non-root applications to use SGX?
> > > 
> > > What are non-evil uses for SGX?
> > > 
> > > ...because it is quite useful for some kinds of evil:
> > 
> > The default permissions for the device are 600.
> 
> Good. This does not belong to non-root.

There are entirely legitimate use cases for using this as an
unprivileged user. However, that'll be up to system and distribution
policy, which can evolve over time, and it makes sense for the *initial*
kernel permission to start out root-only and then adjust permissions via
udev.

> What are some non-evil uses for SGX?

Building a software certificate store. Hardening key-agent software like
ssh-agent or gpg-agent. Building a challenge-response authentication
system. Providing more assurance that your server infrastructure is
uncompromised. Offloading computation to a system without having to
fully trust that system.

As one of many possibilities, imagine a distcc that didn't have to trust
the compile nodes. The compile nodes could fail to return results at
all, but they couldn't alter the results.

Re: [PATCH v11 00/13] Intel SGX1 support

[Josh Triplett]

On Tue, Jun 19, 2018 at 10:04:15PM +0200, Pavel Machek wrote:
> On Tue 2018-06-19 17:59:43, Jarkko Sakkinen wrote:
> > On Tue, Jun 12, 2018 at 12:50:12PM +0200, Pavel Machek wrote:
> > > On Fri 2018-06-08 19:09:35, Jarkko Sakkinen wrote:
> > > > Intel(R) SGX is a set of CPU instructions that can be used by applications
> > > > to set aside private regions of code and data. The code outside the enclave
> > > > is disallowed to access the memory inside the enclave by the CPU access
> > > > control.  In a way you can think that SGX provides inverted sandbox. It
> > > > protects the application from a malicious host.
> > > 
> > > Do you intend to allow non-root applications to use SGX?
> > > 
> > > What are non-evil uses for SGX?
> > > 
> > > ...because it is quite useful for some kinds of evil:
> > 
> > The default permissions for the device are 600.
> 
> Good. This does not belong to non-root.

There are entirely legitimate use cases for using this as an
unprivileged user. However, that'll be up to system and distribution
policy, which can evolve over time, and it makes sense for the *initial*
kernel permission to start out root-only and then adjust permissions via
udev.

> What are some non-evil uses for SGX?

Building a software certificate store. Hardening key-agent software like
ssh-agent or gpg-agent. Building a challenge-response authentication
system. Providing more assurance that your server infrastructure is
uncompromised. Offloading computation to a system without having to
fully trust that system.

As one of many possibilities, imagine a distcc that didn't have to trust
the compile nodes. The compile nodes could fail to return results at
all, but they couldn't alter the results.

Re: [PATCH v11 00/13] Intel SGX1 support

[Josh Triplett]

On Tue, Jun 19, 2018 at 10:04:15PM +0200, Pavel Machek wrote:
> On Tue 2018-06-19 17:59:43, Jarkko Sakkinen wrote:
> > On Tue, Jun 12, 2018 at 12:50:12PM +0200, Pavel Machek wrote:
> > > On Fri 2018-06-08 19:09:35, Jarkko Sakkinen wrote:
> > > > Intel(R) SGX is a set of CPU instructions that can be used by applications
> > > > to set aside private regions of code and data. The code outside the enclave
> > > > is disallowed to access the memory inside the enclave by the CPU access
> > > > control.  In a way you can think that SGX provides inverted sandbox. It
> > > > protects the application from a malicious host.
> > > 
> > > Do you intend to allow non-root applications to use SGX?
> > > 
> > > What are non-evil uses for SGX?
> > > 
> > > ...because it is quite useful for some kinds of evil:
> > 
> > The default permissions for the device are 600.
> 
> Good. This does not belong to non-root.

There are entirely legitimate use cases for using this as an
unprivileged user. However, that'll be up to system and distribution
policy, which can evolve over time, and it makes sense for the *initial*
kernel permission to start out root-only and then adjust permissions via
udev.

> What are some non-evil uses for SGX?

Building a software certificate store. Hardening key-agent software like
ssh-agent or gpg-agent. Building a challenge-response authentication
system. Providing more assurance that your server infrastructure is
uncompromised. Offloading computation to a system without having to
fully trust that system.

As one of many possibilities, imagine a distcc that didn't have to trust
the compile nodes. The compile nodes could fail to return results at
all, but they couldn't alter the results.

Re: [PATCH v11 00/13] Intel SGX1 support

[Josh Triplett]

On Tue, Jun 19, 2018 at 10:04:15PM +0200, Pavel Machek wrote:
> On Tue 2018-06-19 17:59:43, Jarkko Sakkinen wrote:
> > On Tue, Jun 12, 2018 at 12:50:12PM +0200, Pavel Machek wrote:
> > > On Fri 2018-06-08 19:09:35, Jarkko Sakkinen wrote:
> > > > Intel(R) SGX is a set of CPU instructions that can be used by applications
> > > > to set aside private regions of code and data. The code outside the enclave
> > > > is disallowed to access the memory inside the enclave by the CPU access
> > > > control.  In a way you can think that SGX provides inverted sandbox. It
> > > > protects the application from a malicious host.
> > > 
> > > Do you intend to allow non-root applications to use SGX?
> > > 
> > > What are non-evil uses for SGX?
> > > 
> > > ...because it is quite useful for some kinds of evil:
> > 
> > The default permissions for the device are 600.
> 
> Good. This does not belong to non-root.

There are entirely legitimate use cases for using this as an
unprivileged user. However, that'll be up to system and distribution
policy, which can evolve over time, and it makes sense for the *initial*
kernel permission to start out root-only and then adjust permissions via
udev.

> What are some non-evil uses for SGX?

Building a software certificate store. Hardening key-agent software like
ssh-agent or gpg-agent. Building a challenge-response authentication
system. Providing more assurance that your server infrastructure is
uncompromised. Offloading computation to a system without having to
fully trust that system.

As one of many possibilities, imagine a distcc that didn't have to trust
the compile nodes. The compile nodes could fail to return results at
all, but they couldn't alter the results.
--
To unsubscribe from this list: send the line "unsubscribe linux-doc" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [intel-sgx-kernel-dev] [PATCH v11 13/13] intel_sgx: in-kernel launch enclave

[Jarkko Sakkinen]

On Sun, Jun 10, 2018 at 10:17:13PM -0700, Andy Lutomirski wrote:
> > On Jun 9, 2018, at 10:39 PM, Andy Lutomirski <luto@kernel.org> wrote:
> >
> > On Fri, Jun 8, 2018 at 10:32 AM Jarkko Sakkinen
> > <jarkko.sakkinen@linux.intel.com> wrote:
> >>
> >> The Launch Enclave (LE) generates cryptographic launch tokens for user
> >> enclaves. A launch token is used by EINIT to check whether the enclave
> >> is authorized to launch or not. By having its own launch enclave, Linux
> >> has full control of the enclave launch process.
> >>
> >> LE is wrapped into a user space proxy program that reads enclave
> >> signatures outputs launch tokens. The kernel-side glue code is
> >> implemented by using the user space helper framework.  The IPC between
> >> the LE proxy program and kernel is handled with an anonymous inode.
> >>
> >> The commit also adds enclave signing tool that is used by kbuild to
> >> measure and sign the launch enclave. CONFIG_INTEL_SGX_SIGNING_KEY points
> >> to a PEM-file for the 3072-bit RSA key that is used as the LE public key
> >> pair. The default location is:
> >>
> >>  drivers/platform/x86/intel_sgx/sgx_signing_key.pem
> >>
> >> If the default key does not exist kbuild will generate a random key and
> >> place it to this location. KBUILD_SGX_SIGN_PIN can be used to specify
> >> the passphrase for the LE public key.
> >
> > It seems to me that it might be more useful to just commit a key pair
> > into the kernel.  As far as I know, there is no security whatsoever
> > gained by keeping the private key private, so why not make
> > reproducible builds easier by simply fixing the key?
> 
> Having thought about this some more, I think that you should
> completely remove support for specifying a key. Provide a fixed key
> pair, hard code the cache, and call it a day. If you make the key
> configurable, every vendor that has any vendor keys (Debian, Ubuntu,
> Fedora, Red Hat, SuSE, Clear Linux, etc) will see that config option
> and set up their own key pair for no gain whatsoever.  Instead, it'll
> give some illusion of security and it'll slow down operations in a VM
> guest due to swapping out the values of the MSRs.  And, if the code to
> support a locked MSR that just happens to have the right value stays
> in the kernel, then we'll risk having vendors actually ship one
> distro's public key hash, and that will seriously suck.
> 
> I'm going to try to get this code working tomorrow.  I'll keep you
> posted on how that goes.  Can you point me to the userspace bits (i.e.
> something buildable that will run on a kernel with your patches
> applied)?

Sorry for some delay. I was on leave last week.

The SDK supports my driver starting from 2.1 release:

  https://github.com/intel/linux-sgx

SampleCode folder contains some trivial test code to run.

/Jarkko

Re: [PATCH v11 08/13] x86, sgx: added ENCLS wrappers

[Sean Christopherson]

On Fri, 2018-06-08 at 19:09 +0200, Jarkko Sakkinen wrote:
> This commit adds wrappers for Intel(R) SGX ENCLS opcode functionality.
> 
> Signed-off-by: Jarkko Sakkinen <jarkko.sakkinen@linux.intel.com>
> ---
>  arch/x86/include/asm/sgx.h | 198 +++++++++++++++++++++++++++++++++++++
>  1 file changed, 198 insertions(+)
> 
> diff --git a/arch/x86/include/asm/sgx.h b/arch/x86/include/asm/sgx.h
> index fa3e6e0eb8af..a2f727f85b91 100644
> --- a/arch/x86/include/asm/sgx.h
> +++ b/arch/x86/include/asm/sgx.h
> @@ -10,6 +10,10 @@
>  #ifndef _ASM_X86_SGX_H
>  #define _ASM_X86_SGX_H
>  
> +#include <asm/sgx_arch.h>
> +#include <asm/asm.h>
> +#include <linux/bitops.h>
> +#include <linux/err.h>
>  #include <linux/types.h>
>  
>  #define SGX_CPUID 0x12
> @@ -20,6 +24,200 @@ enum sgx_cpuid {
>  	SGX_CPUID_EPC_BANKS	= 2,
>  };
>  
> +enum sgx_commands {

This should be something like "sgx_encls_leafs" and probably moved
to sgx_arch.h (as Dave alluded to, these are architectural values).
"sgx_commands" is not accurate (they're only the cpl0 "commands")
and will have collision issues in the future, e.g. with the ENCLV
instruction and its leafs.

> +	ECREATE	= 0x0,
> +	EADD	= 0x1,
> +	EINIT	= 0x2,
> +	EREMOVE	= 0x3,
> +	EDGBRD	= 0x4,
> +	EDGBWR	= 0x5,
> +	EEXTEND	= 0x6,

Even though it's not used in the code (yet...), I think ELDB,
leaf 0x7, should be defined here for completeness.

> +	ELDU	= 0x8,
> +	EBLOCK	= 0x9,
> +	EPA	= 0xA,
> +	EWB	= 0xB,
> +	ETRACK	= 0xC,
> +	EAUG	= 0xD,
> +	EMODPR	= 0xE,
> +	EMODT	= 0xF,
> +};

Re: [PATCH v11 08/13] x86, sgx: added ENCLS wrappers

[Sean Christopherson]

On Fri, 2018-06-08 at 19:09 +0200, Jarkko Sakkinen wrote:
> This commit adds wrappers for Intel(R) SGX ENCLS opcode functionality.
> 
> Signed-off-by: Jarkko Sakkinen <jarkko.sakkinen@linux.intel.com>
> ---
>  arch/x86/include/asm/sgx.h | 198 +++++++++++++++++++++++++++++++++++++
>  1 file changed, 198 insertions(+)
> 
> diff --git a/arch/x86/include/asm/sgx.h b/arch/x86/include/asm/sgx.h
> index fa3e6e0eb8af..a2f727f85b91 100644
> --- a/arch/x86/include/asm/sgx.h
> +++ b/arch/x86/include/asm/sgx.h
> @@ -10,6 +10,10 @@
>  #ifndef _ASM_X86_SGX_H
>  #define _ASM_X86_SGX_H
>  
> +#include <asm/sgx_arch.h>
> +#include <asm/asm.h>
> +#include <linux/bitops.h>
> +#include <linux/err.h>
>  #include <linux/types.h>
>  
>  #define SGX_CPUID 0x12
> @@ -20,6 +24,200 @@ enum sgx_cpuid {
>  	SGX_CPUID_EPC_BANKS	= 2,
>  };
>  
> +enum sgx_commands {

This should be something like "sgx_encls_leafs" and probably moved
to sgx_arch.h (as Dave alluded to, these are architectural values).
"sgx_commands" is not accurate (they're only the cpl0 "commands")
and will have collision issues in the future, e.g. with the ENCLV
instruction and its leafs.

> +	ECREATE	= 0x0,
> +	EADD	= 0x1,
> +	EINIT	= 0x2,
> +	EREMOVE	= 0x3,
> +	EDGBRD	= 0x4,
> +	EDGBWR	= 0x5,
> +	EEXTEND	= 0x6,

Even though it's not used in the code (yet...), I think ELDB,
leaf 0x7, should be defined here for completeness.

> +	ELDU	= 0x8,
> +	EBLOCK	= 0x9,
> +	EPA	= 0xA,
> +	EWB	= 0xB,
> +	ETRACK	= 0xC,
> +	EAUG	= 0xD,
> +	EMODPR	= 0xE,
> +	EMODT	= 0xF,
> +};

Re: [PATCH v11 09/13] x86, sgx: basic routines for enclave page cache

[Sean Christopherson]

On Fri, 2018-06-08 at 19:09 +0200, Jarkko Sakkinen wrote:
> SGX has a set of data structures to maintain information about the enclaves
> and their security properties. BIOS reserves a fixed size region of
> physical memory for these structures by setting Processor Reserved Memory
> Range Registers (PRMRR). This memory area is called Enclave Page Cache
> (EPC).
> 
> This commit implements the basic routines to allocate and free pages from
> different EPC banks. There is also a swapper thread ksgxswapd for EPC pages
> that gets woken up by sgx_alloc_page() when we run below the low watermark.
> The swapper thread continues swapping pages up until it reaches the high
> watermark.
> 
> Each subsystem that uses SGX must provide a set of callbacks for EPC
> pages that are used to reclaim, block and write an EPC page. Kernel
> takes the responsibility of maintaining LRU cache for them.
> 
> Signed-off-by: Jarkko Sakkinen <jarkko.sakkinen@linux.intel.com>
> ---
>  arch/x86/include/asm/sgx.h      |  67 +++++
>  arch/x86/include/asm/sgx_arch.h | 224 ++++++++++++++++
>  arch/x86/kernel/cpu/intel_sgx.c | 443 +++++++++++++++++++++++++++++++-
>  3 files changed, 732 insertions(+), 2 deletions(-)
>  create mode 100644 arch/x86/include/asm/sgx_arch.h

...

> +struct sgx_pcmd {
> +	struct sgx_secinfo secinfo;
> +	uint64_t enclave_id;
> +	uint8_t reserved[40];
> +	uint8_t mac[16];
> +};

sgx_pcmd has a 128-byte alignment requirement.  I think it's
worth specifying here as sgx_pcmd is small enough that it could
be put on the stack, e.g. by KVM when trapping and executing
ELD* on behalf of a guest VM.

In fact, it probably makes sense to add alightment attributes
to all SGX structs for self-documentation purposes, even though
many of them will never be allocated statically or on the stack.

> +
> +#define SGX_MODULUS_SIZE 384
> +
> +struct sgx_sigstruct_header {
> +	uint64_t header1[2];
> +	uint32_t vendor;
> +	uint32_t date;
> +	uint64_t header2[2];
> +	uint32_t swdefined;
> +	uint8_t reserved1[84];
> +};
> +
> +struct sgx_sigstruct_body {
> +	uint32_t miscselect;
> +	uint32_t miscmask;
> +	uint8_t reserved2[20];
> +	uint64_t attributes;
> +	uint64_t xfrm;
> +	uint8_t attributemask[16];
> +	uint8_t mrenclave[32];
> +	uint8_t reserved3[32];
> +	uint16_t isvprodid;
> +	uint16_t isvsvn;
> +} __attribute__((__packed__));
> +
> +struct sgx_sigstruct {
> +	struct sgx_sigstruct_header header;
> +	uint8_t modulus[SGX_MODULUS_SIZE];
> +	uint32_t exponent;
> +	uint8_t signature[SGX_MODULUS_SIZE];
> +	struct sgx_sigstruct_body body;
> +	uint8_t reserved4[12];
> +	uint8_t q1[SGX_MODULUS_SIZE];
> +	uint8_t q2[SGX_MODULUS_SIZE];
> +};
> +
> +struct sgx_sigstruct_payload {
> +	struct sgx_sigstruct_header header;
> +	struct sgx_sigstruct_body body;
> +};
> +
> +struct sgx_einittoken_payload {
> +	uint32_t valid;
> +	uint32_t reserved1[11];
> +	uint64_t attributes;
> +	uint64_t xfrm;
> +	uint8_t mrenclave[32];
> +	uint8_t reserved2[32];
> +	uint8_t mrsigner[32];
> +	uint8_t reserved3[32];
> +};
> +
> +struct sgx_einittoken {
> +	struct sgx_einittoken_payload payload;
> +	uint8_t cpusvnle[16];
> +	uint16_t isvprodidle;
> +	uint16_t isvsvnle;
> +	uint8_t reserved2[24];
> +	uint32_t maskedmiscselectle;
> +	uint64_t maskedattributesle;
> +	uint64_t maskedxfrmle;
> +	uint8_t keyid[32];
> +	uint8_t mac[16];
> +};
> +
> +struct sgx_report {
> +	uint8_t cpusvn[16];
> +	uint32_t miscselect;
> +	uint8_t reserved1[28];
> +	uint64_t attributes;
> +	uint64_t xfrm;
> +	uint8_t mrenclave[32];
> +	uint8_t reserved2[32];
> +	uint8_t mrsigner[32];
> +	uint8_t reserved3[96];
> +	uint16_t isvprodid;
> +	uint16_t isvsvn;
> +	uint8_t reserved4[60];
> +	uint8_t reportdata[64];
> +	uint8_t keyid[32];
> +	uint8_t mac[16];
> +};
> +
> +struct sgx_targetinfo {
> +	uint8_t mrenclave[32];
> +	uint64_t attributes;
> +	uint64_t xfrm;
> +	uint8_t reserved1[4];
> +	uint32_t miscselect;
> +	uint8_t reserved2[456];
> +};
> +
> +struct sgx_keyrequest {
> +	uint16_t keyname;
> +	uint16_t keypolicy;
> +	uint16_t isvsvn;
> +	uint16_t reserved1;
> +	uint8_t cpusvn[16];
> +	uint64_t attributemask;
> +	uint64_t xfrmmask;
> +	uint8_t keyid[32];
> +	uint32_t miscmask;
> +	uint8_t reserved2[436];
> +};

Re: [PATCH v11 09/13] x86, sgx: basic routines for enclave page cache

[Sean Christopherson]

On Fri, 2018-06-08 at 19:09 +0200, Jarkko Sakkinen wrote:
> SGX has a set of data structures to maintain information about the enclaves
> and their security properties. BIOS reserves a fixed size region of
> physical memory for these structures by setting Processor Reserved Memory
> Range Registers (PRMRR). This memory area is called Enclave Page Cache
> (EPC).
> 
> This commit implements the basic routines to allocate and free pages from
> different EPC banks. There is also a swapper thread ksgxswapd for EPC pages
> that gets woken up by sgx_alloc_page() when we run below the low watermark.
> The swapper thread continues swapping pages up until it reaches the high
> watermark.
> 
> Each subsystem that uses SGX must provide a set of callbacks for EPC
> pages that are used to reclaim, block and write an EPC page. Kernel
> takes the responsibility of maintaining LRU cache for them.
> 
> Signed-off-by: Jarkko Sakkinen <jarkko.sakkinen@linux.intel.com>
> ---
>  arch/x86/include/asm/sgx.h      |  67 +++++
>  arch/x86/include/asm/sgx_arch.h | 224 ++++++++++++++++
>  arch/x86/kernel/cpu/intel_sgx.c | 443 +++++++++++++++++++++++++++++++-
>  3 files changed, 732 insertions(+), 2 deletions(-)
>  create mode 100644 arch/x86/include/asm/sgx_arch.h

...

> +struct sgx_pcmd {
> +	struct sgx_secinfo secinfo;
> +	uint64_t enclave_id;
> +	uint8_t reserved[40];
> +	uint8_t mac[16];
> +};

sgx_pcmd has a 128-byte alignment requirement.  I think it's
worth specifying here as sgx_pcmd is small enough that it could
be put on the stack, e.g. by KVM when trapping and executing
ELD* on behalf of a guest VM.

In fact, it probably makes sense to add alightment attributes
to all SGX structs for self-documentation purposes, even though
many of them will never be allocated statically or on the stack.

> +
> +#define SGX_MODULUS_SIZE 384
> +
> +struct sgx_sigstruct_header {
> +	uint64_t header1[2];
> +	uint32_t vendor;
> +	uint32_t date;
> +	uint64_t header2[2];
> +	uint32_t swdefined;
> +	uint8_t reserved1[84];
> +};
> +
> +struct sgx_sigstruct_body {
> +	uint32_t miscselect;
> +	uint32_t miscmask;
> +	uint8_t reserved2[20];
> +	uint64_t attributes;
> +	uint64_t xfrm;
> +	uint8_t attributemask[16];
> +	uint8_t mrenclave[32];
> +	uint8_t reserved3[32];
> +	uint16_t isvprodid;
> +	uint16_t isvsvn;
> +} __attribute__((__packed__));
> +
> +struct sgx_sigstruct {
> +	struct sgx_sigstruct_header header;
> +	uint8_t modulus[SGX_MODULUS_SIZE];
> +	uint32_t exponent;
> +	uint8_t signature[SGX_MODULUS_SIZE];
> +	struct sgx_sigstruct_body body;
> +	uint8_t reserved4[12];
> +	uint8_t q1[SGX_MODULUS_SIZE];
> +	uint8_t q2[SGX_MODULUS_SIZE];
> +};
> +
> +struct sgx_sigstruct_payload {
> +	struct sgx_sigstruct_header header;
> +	struct sgx_sigstruct_body body;
> +};
> +
> +struct sgx_einittoken_payload {
> +	uint32_t valid;
> +	uint32_t reserved1[11];
> +	uint64_t attributes;
> +	uint64_t xfrm;
> +	uint8_t mrenclave[32];
> +	uint8_t reserved2[32];
> +	uint8_t mrsigner[32];
> +	uint8_t reserved3[32];
> +};
> +
> +struct sgx_einittoken {
> +	struct sgx_einittoken_payload payload;
> +	uint8_t cpusvnle[16];
> +	uint16_t isvprodidle;
> +	uint16_t isvsvnle;
> +	uint8_t reserved2[24];
> +	uint32_t maskedmiscselectle;
> +	uint64_t maskedattributesle;
> +	uint64_t maskedxfrmle;
> +	uint8_t keyid[32];
> +	uint8_t mac[16];
> +};
> +
> +struct sgx_report {
> +	uint8_t cpusvn[16];
> +	uint32_t miscselect;
> +	uint8_t reserved1[28];
> +	uint64_t attributes;
> +	uint64_t xfrm;
> +	uint8_t mrenclave[32];
> +	uint8_t reserved2[32];
> +	uint8_t mrsigner[32];
> +	uint8_t reserved3[96];
> +	uint16_t isvprodid;
> +	uint16_t isvsvn;
> +	uint8_t reserved4[60];
> +	uint8_t reportdata[64];
> +	uint8_t keyid[32];
> +	uint8_t mac[16];
> +};
> +
> +struct sgx_targetinfo {
> +	uint8_t mrenclave[32];
> +	uint64_t attributes;
> +	uint64_t xfrm;
> +	uint8_t reserved1[4];
> +	uint32_t miscselect;
> +	uint8_t reserved2[456];
> +};
> +
> +struct sgx_keyrequest {
> +	uint16_t keyname;
> +	uint16_t keypolicy;
> +	uint16_t isvsvn;
> +	uint16_t reserved1;
> +	uint8_t cpusvn[16];
> +	uint64_t attributemask;
> +	uint64_t xfrmmask;
> +	uint8_t keyid[32];
> +	uint32_t miscmask;
> +	uint8_t reserved2[436];
> +};

Re: [PATCH v11 09/13] x86, sgx: basic routines for enclave page cache

[Sean Christopherson]

On Fri, 2018-06-08 at 19:09 +0200, Jarkko Sakkinen wrote:
> SGX has a set of data structures to maintain information about the enclaves
> and their security properties. BIOS reserves a fixed size region of
> physical memory for these structures by setting Processor Reserved Memory
> Range Registers (PRMRR). This memory area is called Enclave Page Cache
> (EPC).
> 
> This commit implements the basic routines to allocate and free pages from
> different EPC banks. There is also a swapper thread ksgxswapd for EPC pages
> that gets woken up by sgx_alloc_page() when we run below the low watermark.
> The swapper thread continues swapping pages up until it reaches the high
> watermark.
> 
> Each subsystem that uses SGX must provide a set of callbacks for EPC
> pages that are used to reclaim, block and write an EPC page. Kernel
> takes the responsibility of maintaining LRU cache for them.
> 
> Signed-off-by: Jarkko Sakkinen <jarkko.sakkinen@linux.intel.com>
> ---
>  arch/x86/include/asm/sgx.h      |  67 +++++
>  arch/x86/include/asm/sgx_arch.h | 224 ++++++++++++++++
>  arch/x86/kernel/cpu/intel_sgx.c | 443 +++++++++++++++++++++++++++++++-
>  3 files changed, 732 insertions(+), 2 deletions(-)
>  create mode 100644 arch/x86/include/asm/sgx_arch.h

...

> diff --git a/arch/x86/kernel/cpu/intel_sgx.c b/arch/x86/kernel/cpu/intel_sgx.c
> index db6b315334f4..ae2b5c5b455f 100644
> --- a/arch/x86/kernel/cpu/intel_sgx.c
> +++ b/arch/x86/kernel/cpu/intel_sgx.c
> @@ -14,14 +14,439 @@
>  #include <linux/freezer.h>
>  #include <linux/highmem.h>
>  #include <linux/kthread.h>
> +#include <linux/pagemap.h>
>  #include <linux/ratelimit.h>
>  #include <linux/sched/signal.h>
> +#include <linux/shmem_fs.h>
>  #include <linux/slab.h>
>  
> +#define SGX_NR_TO_SCAN	16
> +#define SGX_NR_LOW_PAGES 32
> +#define SGX_NR_HIGH_PAGES 64
> +
>  bool sgx_enabled __ro_after_init = false;
>  EXPORT_SYMBOL(sgx_enabled);
> +bool sgx_lc_enabled __ro_after_init;
> +EXPORT_SYMBOL(sgx_lc_enabled);
> +atomic_t sgx_nr_free_pages = ATOMIC_INIT(0);
> +EXPORT_SYMBOL(sgx_nr_free_pages);
> +struct sgx_epc_bank sgx_epc_banks[SGX_MAX_EPC_BANKS];
> +EXPORT_SYMBOL(sgx_epc_banks);
> +int sgx_nr_epc_banks;
> +EXPORT_SYMBOL(sgx_nr_epc_banks);
> +LIST_HEAD(sgx_active_page_list);
> +EXPORT_SYMBOL(sgx_active_page_list);
> +DEFINE_SPINLOCK(sgx_active_page_list_lock);
> +EXPORT_SYMBOL(sgx_active_page_list_lock);

I don't think we should be exporting anything other than sgx_enabled
and sgx_lc_enabled.  The only external use of a symbol that can't be
trivially (re)moved is in the driver's sgx_pm_suspend() in sgx_main.c,
which uses the sgx_active_page_list to invalidate enclaves.  And that
behavior seems unsafe, e.g. an enclave could theoretically have zero
pages on the active list and so could be missed in the suspend flow.

> +static struct task_struct *ksgxswapd_tsk;
> +static DECLARE_WAIT_QUEUE_HEAD(ksgxswapd_waitq);

Re: [PATCH v11 09/13] x86, sgx: basic routines for enclave page cache

[Sean Christopherson]

On Fri, 2018-06-08 at 19:09 +0200, Jarkko Sakkinen wrote:
> SGX has a set of data structures to maintain information about the enclaves
> and their security properties. BIOS reserves a fixed size region of
> physical memory for these structures by setting Processor Reserved Memory
> Range Registers (PRMRR). This memory area is called Enclave Page Cache
> (EPC).
> 
> This commit implements the basic routines to allocate and free pages from
> different EPC banks. There is also a swapper thread ksgxswapd for EPC pages
> that gets woken up by sgx_alloc_page() when we run below the low watermark.
> The swapper thread continues swapping pages up until it reaches the high
> watermark.
> 
> Each subsystem that uses SGX must provide a set of callbacks for EPC
> pages that are used to reclaim, block and write an EPC page. Kernel
> takes the responsibility of maintaining LRU cache for them.
> 
> Signed-off-by: Jarkko Sakkinen <jarkko.sakkinen@linux.intel.com>
> ---
>  arch/x86/include/asm/sgx.h      |  67 +++++
>  arch/x86/include/asm/sgx_arch.h | 224 ++++++++++++++++
>  arch/x86/kernel/cpu/intel_sgx.c | 443 +++++++++++++++++++++++++++++++-
>  3 files changed, 732 insertions(+), 2 deletions(-)
>  create mode 100644 arch/x86/include/asm/sgx_arch.h

...

> diff --git a/arch/x86/kernel/cpu/intel_sgx.c b/arch/x86/kernel/cpu/intel_sgx.c
> index db6b315334f4..ae2b5c5b455f 100644
> --- a/arch/x86/kernel/cpu/intel_sgx.c
> +++ b/arch/x86/kernel/cpu/intel_sgx.c
> @@ -14,14 +14,439 @@
>  #include <linux/freezer.h>
>  #include <linux/highmem.h>
>  #include <linux/kthread.h>
> +#include <linux/pagemap.h>
>  #include <linux/ratelimit.h>
>  #include <linux/sched/signal.h>
> +#include <linux/shmem_fs.h>
>  #include <linux/slab.h>
>  
> +#define SGX_NR_TO_SCAN	16
> +#define SGX_NR_LOW_PAGES 32
> +#define SGX_NR_HIGH_PAGES 64
> +
>  bool sgx_enabled __ro_after_init = false;
>  EXPORT_SYMBOL(sgx_enabled);
> +bool sgx_lc_enabled __ro_after_init;
> +EXPORT_SYMBOL(sgx_lc_enabled);
> +atomic_t sgx_nr_free_pages = ATOMIC_INIT(0);
> +EXPORT_SYMBOL(sgx_nr_free_pages);
> +struct sgx_epc_bank sgx_epc_banks[SGX_MAX_EPC_BANKS];
> +EXPORT_SYMBOL(sgx_epc_banks);
> +int sgx_nr_epc_banks;
> +EXPORT_SYMBOL(sgx_nr_epc_banks);
> +LIST_HEAD(sgx_active_page_list);
> +EXPORT_SYMBOL(sgx_active_page_list);
> +DEFINE_SPINLOCK(sgx_active_page_list_lock);
> +EXPORT_SYMBOL(sgx_active_page_list_lock);

I don't think we should be exporting anything other than sgx_enabled
and sgx_lc_enabled.  The only external use of a symbol that can't be
trivially (re)moved is in the driver's sgx_pm_suspend() in sgx_main.c,
which uses the sgx_active_page_list to invalidate enclaves.  And that
behavior seems unsafe, e.g. an enclave could theoretically have zero
pages on the active list and so could be missed in the suspend flow.

> +static struct task_struct *ksgxswapd_tsk;
> +static DECLARE_WAIT_QUEUE_HEAD(ksgxswapd_waitq);

Re: [intel-sgx-kernel-dev] [PATCH v11 13/13] intel_sgx: in-kernel launch enclave

[Nathaniel McCallum]

As I understand it, the current policy models under discussion look like this:

1. SGX w/o FLC (not being merged) looks like this:
  Intel CPU => (Intel signed) launch enclave => enclaves

2. SGX w/ FLC, looks like this:
  Intel CPU => kernel => launch enclave => enclaves

3. Andy is proposing this:
  Intel CPU => kernel => enclaves

This proposal is based on the fact that if the kernel can write to the
MSRs then a kernel compromise allows an attacker to run their own
launch enclave and therefore having an independent launch enclave adds
only complexity but not security.

Is it possible to restrict the ability of the kernel to change the
MSRs? For example, could a UEFI module manage the MSRs? Could the
launch enclave live entirely within that UEFI module?

4. I am suggesting this:
  Intel CPU => UEFI module => enclaves

Under this architecture, the kernel isn't involved in policy at all
and users get exactly the same freedoms they already have with Secure
Boot. Further, the launch enclave can be independently updated and
could be distributed in linux-firmware. The UEFI module can also be
shared across operating systems. If I want to have my own enclave
policy, then I can build the UEFI module myself, with my
modifications, and I can disable Secure Boot. Alternatively,
distributions that want to set their own policies can build their own
UEFI module and sign it with their vendor key.
On Mon, Jun 18, 2018 at 5:59 PM Andy Lutomirski <luto@kernel.org> wrote:
>
> On Tue, Jun 12, 2018 at 10:45 AM Neil Horman <nhorman@redhat.com> wrote:
> >
> > On Mon, Jun 11, 2018 at 09:55:29PM -0700, Andy Lutomirski wrote:
> > > On Mon, Jun 11, 2018 at 4:52 AM Neil Horman <nhorman@redhat.com> wrote:
> > > >
> > > > On Sun, Jun 10, 2018 at 10:17:13PM -0700, Andy Lutomirski wrote:
> > > > > > On Jun 9, 2018, at 10:39 PM, Andy Lutomirski <luto@kernel.org> wrote:
> > > > > >
> > > > > > On Fri, Jun 8, 2018 at 10:32 AM Jarkko Sakkinen
> > > > > > <jarkko.sakkinen@linux.intel.com> wrote:
> > > > > >>
> > > > > >> The Launch Enclave (LE) generates cryptographic launch tokens for user
> > > > > >> enclaves. A launch token is used by EINIT to check whether the enclave
> > > > > >> is authorized to launch or not. By having its own launch enclave, Linux
> > > > > >> has full control of the enclave launch process.
> > > > > >>
> > > > > >> LE is wrapped into a user space proxy program that reads enclave
> > > > > >> signatures outputs launch tokens. The kernel-side glue code is
> > > > > >> implemented by using the user space helper framework.  The IPC between
> > > > > >> the LE proxy program and kernel is handled with an anonymous inode.
> > > > > >>
> > > > > >> The commit also adds enclave signing tool that is used by kbuild to
> > > > > >> measure and sign the launch enclave. CONFIG_INTEL_SGX_SIGNING_KEY points
> > > > > >> to a PEM-file for the 3072-bit RSA key that is used as the LE public key
> > > > > >> pair. The default location is:
> > > > > >>
> > > > > >>  drivers/platform/x86/intel_sgx/sgx_signing_key.pem
> > > > > >>
> > > > > >> If the default key does not exist kbuild will generate a random key and
> > > > > >> place it to this location. KBUILD_SGX_SIGN_PIN can be used to specify
> > > > > >> the passphrase for the LE public key.
> > > > > >
> > > > > > It seems to me that it might be more useful to just commit a key pair
> > > > > > into the kernel.  As far as I know, there is no security whatsoever
> > > > > > gained by keeping the private key private, so why not make
> > > > > > reproducible builds easier by simply fixing the key?
> > > > >
> > > > > Having thought about this some more, I think that you should
> > > > > completely remove support for specifying a key. Provide a fixed key
> > > > > pair, hard code the cache, and call it a day. If you make the key
> > > > > configurable, every vendor that has any vendor keys (Debian, Ubuntu,
> > > > > Fedora, Red Hat, SuSE, Clear Linux, etc) will see that config option
> > > > > and set up their own key pair for no gain whatsoever.  Instead, it'll
> > > > > give some illusion of security and it'll slow down operations in a VM
> > > > > guest due to swapping out the values of the MSRs.  And, if the code to
> > > > > support a locked MSR that just happens to have the right value stays
> > > > > in the kernel, then we'll risk having vendors actually ship one
> > > > > distro's public key hash, and that will seriously suck.
> > > > >
> > > > If you hard code the key pair however, doesn't that imply that anyone can sign a
> > > > user space binary as a launch enclave, and potentially gain control of the token
> > > > granting process?
> > >
> > > Yes and no.
> > >
> > > First of all, the kernel driver shouldn't be allowing user code to
> > > launch a launch enclave regardless of signature.  I haven't gotten far
> > > enough in reviewing the code to see whether that's the case, but if
> > > it's not, it should be fixed before it's merged.
> > >
> > Ok, I agree with you here.
> >
> > > But keep in mind that control of the token granting process is not the
> > > same thing as control over the right to launch an enclave.  On systems
> > > without the LE hash MSRs, Intel controls the token granting process
> > > and, barring some attack, an enclave that isn't blessed by Intel can't
> > > be launched.  Support for that model will not be merged into upstream
> > > Linux.  But on systems that have the LE hash MSRs and leave them
> > > unlocked, there is effectively no hardware-enforced launch control.
> > > Instead we have good old kernel policy.  If a user wants to launch an
> > > enclave, they need to get the kernel to launch the enclave, and the
> > > kernel needs to apply its policy.  The patch here (the in-kernel
> > > launch enclave) has a wide-open policy.
> > >
> >
> > Right, also agree here.  Systems without Flexible Launch Control are a
> > non-starter, we're only considering FLC systems here
> >
> > > So, as a practical matter, if every distro has their own LE key and
> > > keeps it totally safe, then a system that locks the MSRs to one
> > > distro's key makes it quite annoying to run another distro's intel_sgx
> > > driver, but there is no effect on the actual security of the system.
> > >
> > I agree that for systems that firmware-lock the msrs are annoying, but I would
> > think that IHV's would want to keep those msrs unlocked specifically to allow a
> > wide range of distributions to use this feature.
> >
> > As for benefits to security, I think there are some.  Namely, by leaving the
> > MSRS unlocked, A distribution can, rather than providing their own distirbution
> > key, pass the root of trust on to the end user.  I can easily envision a
> > downstream customer that wants to use SGX, and do so in such a way that they are
> > assured that their OS vendor doesn't have the ability to run an LE on their
> > system (at least not without the visual cue of specifying a different key hash
> > at the OS boot).
>
> Which achieves what, exactly?  The launch public key hash isn't the
> root of trust of anything except for a really awkward mechanism to
> limit the enclaves that get run.  If there is actual demand to limit
> enclaves that get run, let's do it correctly: add some code in the
> kernel that enforces a policy before launching an enclave.
>
> If the MSRs are unlocked, there is no stronger guarantee available
> even if you supply your own custom LE.  If the kernel is owned, the
> attacker can just change the MSRs.
>
> --Andy

Re: [intel-sgx-kernel-dev] [PATCH v11 13/13] intel_sgx: in-kernel launch enclave

[Jethro Beekman]

[-- Attachment #1: Type: text/plain, Size: 2710 bytes --]

On 2018-06-20 09:28, Nathaniel McCallum wrote:
> As I understand it, the current policy models under discussion look like this:
> 
> 1. SGX w/o FLC (not being merged) looks like this:
>    Intel CPU => (Intel signed) launch enclave => enclaves

I think you mean:

      Intel CPU => kernel => (Intel signed) launch enclave => enclaves

> 
> 2. SGX w/ FLC, looks like this:
>    Intel CPU => kernel => launch enclave => enclaves
> 
> 3. Andy is proposing this:
>    Intel CPU => kernel => enclaves
> 
> This proposal is based on the fact that if the kernel can write to the
> MSRs then a kernel compromise allows an attacker to run their own
> launch enclave and therefore having an independent launch enclave adds
> only complexity but not security.
> 
> Is it possible to restrict the ability of the kernel to change the
> MSRs? For example, could a UEFI module manage the MSRs? Could the
> launch enclave live entirely within that UEFI module?

On 2017-03-17 09:15, Jethro Beekman wrote:
 > While collecting my thoughts about the issue I read through the
 > documentation again and it seems that it will not be possible for a
 > platform to lock in a non-Intel hash at all. From Section 39.1.4 of the
 > latest Intel SDM:
 >
 >  > IA32_SGXLEPUBKEYHASH defaults to digest of Intel’s launch enclave
 >  > signing key after reset.
 >  >
 >  > IA32_FEATURE_CONTROL bit 17 controls the permissions on the
 >  > IA32_SGXLEPUBKEYHASH MSRs when CPUID.(EAX=12H, ECX=00H):EAX[0] = 1.
 >  > If IA32_FEATURE_CONTROL is locked with bit 17 set,
 >  > IA32_SGXLEPUBKEYHASH MSRs are reconfigurable (writeable). If either
 >  > IA32_FEATURE_CONTROL is not locked or bit 17 is clear, the MSRs are
 >  > read only.
 >
 > This last bit is also repeated in different words in Table 35-2 and
 > Section 42.2.2. The MSRs are *not writable* before the write-lock bit
 > itself is locked. Meaning the MSRs are either locked with Intel's key
 > hash, or not locked at all.

> 
> 4. I am suggesting this:
>    Intel CPU => UEFI module => enclaves
> 
> Under this architecture, the kernel isn't involved in policy at all
> and users get exactly the same freedoms they already have with Secure
> Boot. Further, the launch enclave can be independently updated and
> could be distributed in linux-firmware. The UEFI module can also be
> shared across operating systems. If I want to have my own enclave
> policy, then I can build the UEFI module myself, with my
> modifications, and I can disable Secure Boot. Alternatively,
> distributions that want to set their own policies can build their own
> UEFI module and sign it with their vendor key.

Jethro Beekman | Fortanix


[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 3994 bytes --]

Re: [intel-sgx-kernel-dev] [PATCH v11 13/13] intel_sgx: in-kernel launch enclave

[Jethro Beekman]

[-- Attachment #1: Type: text/plain, Size: 928 bytes --]

On 2018-06-20 11:16, Jethro Beekman wrote:
>  > This last bit is also repeated in different words in Table 35-2 and
>  > Section 42.2.2. The MSRs are *not writable* before the write-lock bit
>  > itself is locked. Meaning the MSRs are either locked with Intel's key
>  > hash, or not locked at all.

Actually, this might be a documentation bug. I have some test hardware 
and I was able to configure the MSRs in the BIOS and then read the MSRs 
after boot like this:

MSR 0x3a 0x0000000000040005
MSR 0x8c 0x20180620aaaaaaaa
MSR 0x8d 0x20180620bbbbbbbb
MSR 0x8e 0x20180620cccccccc
MSR 0x8f 0x20180620dddddddd

Since this is not production hardware, it could also be a CPU bug of course.

If it is indeed possible to configure AND lock the MSR values to 
non-Intel values, I'm very much in favor of Nathaniels proposal to treat 
the launch enclave like any other firmware blob.

Jethro Beekman | Fortanix


[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 3994 bytes --]

Re: [intel-sgx-kernel-dev] [PATCH v11 13/13] intel_sgx: in-kernel launch enclave

[Sean Christopherson]

On Wed, Jun 20, 2018 at 11:39:00AM -0700, Jethro Beekman wrote:
> On 2018-06-20 11:16, Jethro Beekman wrote:
> > > This last bit is also repeated in different words in Table 35-2 and
> > > Section 42.2.2. The MSRs are *not writable* before the write-lock bit
> > > itself is locked. Meaning the MSRs are either locked with Intel's key
> > > hash, or not locked at all.
> 
> Actually, this might be a documentation bug. I have some test hardware and I
> was able to configure the MSRs in the BIOS and then read the MSRs after boot
> like this:
> 
> MSR 0x3a 0x0000000000040005
> MSR 0x8c 0x20180620aaaaaaaa
> MSR 0x8d 0x20180620bbbbbbbb
> MSR 0x8e 0x20180620cccccccc
> MSR 0x8f 0x20180620dddddddd
> 
> Since this is not production hardware, it could also be a CPU bug of course.
> 
> If it is indeed possible to configure AND lock the MSR values to non-Intel
> values, I'm very much in favor of Nathaniels proposal to treat the launch
> enclave like any other firmware blob.

It's not a CPU or documentation bug (though the latter is arguable).
SGX has an activation step that is triggered by doing a WRMSR(0x7a)
with bit 0 set.  Until SGX is activated, the SGX related bits in
IA32_FEATURE_CONTROL cannot be set, i.e. SGX can't be enabled.  But,
the LE hash MSRs are fully writable prior to activation, e.g. to
allow firmware to lock down the LE key with a non-Intel value.

So yes, it's possible to lock the MSRs to a non-Intel value.  The
obvious caveat is that whatever blob is used to write the MSRs would
need be executed prior to activation.

As for the SDM, it's a documentation... omission?  SGX activation
is intentionally omitted from the SDM.  The intended usage model is
that firmware will always do the activation (if it wants SGX enabled),
i.e. post-firmware software will only ever "see" SGX as disabled or
in the fully activated state, and so the SDM doesn't describe SGX
behavior prior to activation.  I believe the activation process, or
at least what is required from firmware, is documented in the BIOS
writer's guide.
 
> Jethro Beekman | Fortanix
> 

Re: [intel-sgx-kernel-dev] [PATCH v11 13/13] intel_sgx: in-kernel launch enclave

[Nathaniel McCallum]

On Wed, Jun 20, 2018 at 2:16 PM Jethro Beekman <jethro@fortanix.com> wrote:
>
> On 2018-06-20 09:28, Nathaniel McCallum wrote:
> > As I understand it, the current policy models under discussion look like this:
> >
> > 1. SGX w/o FLC (not being merged) looks like this:
> >    Intel CPU => (Intel signed) launch enclave => enclaves
>
> I think you mean:
>
>       Intel CPU => kernel => (Intel signed) launch enclave => enclaves

I get what you mean. But it wasn't what I intended. I'm referring to
cryptographic policy. While it is true that the kernel would provide
hardware support and would still enforce other non-cryptographic
policy under this model (such as filesystem permissions to /dev/sgx),
the kernel doesn't verify signatures or pick the key used to verify
signatures. The Intel CPU validates the signature of the launch
enclave using a hard-coded key. Since the kernel doesn't get to pick
the key, it has no say in the cryptographic policy.

> > 2. SGX w/ FLC, looks like this:
> >    Intel CPU => kernel => launch enclave => enclaves

In this case, the kernel picks the key used to verify the signature of
the LE and writes it to the appropriate MSRs. So it has a say in the
policy chain.

> > 3. Andy is proposing this:
> >    Intel CPU => kernel => enclaves

In this case, the kernel basically throws away the concept of LEs
based on the fact that proposal #2 doesn't actually reduce the attack
surface and therefore adds complexity without security. So the kernel
takes over the evaluation of the cryptographic policy.

> > This proposal is based on the fact that if the kernel can write to the
> > MSRs then a kernel compromise allows an attacker to run their own
> > launch enclave and therefore having an independent launch enclave adds
> > only complexity but not security.
> >
> > Is it possible to restrict the ability of the kernel to change the
> > MSRs? For example, could a UEFI module manage the MSRs? Could the
> > launch enclave live entirely within that UEFI module?
>
> On 2017-03-17 09:15, Jethro Beekman wrote:
>  > While collecting my thoughts about the issue I read through the
>  > documentation again and it seems that it will not be possible for a
>  > platform to lock in a non-Intel hash at all. From Section 39.1.4 of the
>  > latest Intel SDM:
>  >
>  >  > IA32_SGXLEPUBKEYHASH defaults to digest of Intel’s launch enclave
>  >  > signing key after reset.
>  >  >
>  >  > IA32_FEATURE_CONTROL bit 17 controls the permissions on the
>  >  > IA32_SGXLEPUBKEYHASH MSRs when CPUID.(EAX=12H, ECX=00H):EAX[0] = 1.
>  >  > If IA32_FEATURE_CONTROL is locked with bit 17 set,
>  >  > IA32_SGXLEPUBKEYHASH MSRs are reconfigurable (writeable). If either
>  >  > IA32_FEATURE_CONTROL is not locked or bit 17 is clear, the MSRs are
>  >  > read only.
>  >
>  > This last bit is also repeated in different words in Table 35-2 and
>  > Section 42.2.2. The MSRs are *not writable* before the write-lock bit
>  > itself is locked. Meaning the MSRs are either locked with Intel's key
>  > hash, or not locked at all.
>
> >
> > 4. I am suggesting this:
> >    Intel CPU => UEFI module => enclaves
> >
> > Under this architecture, the kernel isn't involved in policy at all
> > and users get exactly the same freedoms they already have with Secure
> > Boot. Further, the launch enclave can be independently updated and
> > could be distributed in linux-firmware. The UEFI module can also be
> > shared across operating systems. If I want to have my own enclave
> > policy, then I can build the UEFI module myself, with my
> > modifications, and I can disable Secure Boot. Alternatively,
> > distributions that want to set their own policies can build their own
> > UEFI module and sign it with their vendor key.
>
> Jethro Beekman | Fortanix
>

Re: [intel-sgx-kernel-dev] [PATCH v11 13/13] intel_sgx: in-kernel launch enclave

[Nathaniel McCallum]

On Wed, Jun 20, 2018 at 5:02 PM Sean Christopherson
<sean.j.christopherson@intel.com> wrote:
>
> On Wed, Jun 20, 2018 at 11:39:00AM -0700, Jethro Beekman wrote:
> > On 2018-06-20 11:16, Jethro Beekman wrote:
> > > > This last bit is also repeated in different words in Table 35-2 and
> > > > Section 42.2.2. The MSRs are *not writable* before the write-lock bit
> > > > itself is locked. Meaning the MSRs are either locked with Intel's key
> > > > hash, or not locked at all.
> >
> > Actually, this might be a documentation bug. I have some test hardware and I
> > was able to configure the MSRs in the BIOS and then read the MSRs after boot
> > like this:
> >
> > MSR 0x3a 0x0000000000040005
> > MSR 0x8c 0x20180620aaaaaaaa
> > MSR 0x8d 0x20180620bbbbbbbb
> > MSR 0x8e 0x20180620cccccccc
> > MSR 0x8f 0x20180620dddddddd
> >
> > Since this is not production hardware, it could also be a CPU bug of course.
> >
> > If it is indeed possible to configure AND lock the MSR values to non-Intel
> > values, I'm very much in favor of Nathaniels proposal to treat the launch
> > enclave like any other firmware blob.
>
> It's not a CPU or documentation bug (though the latter is arguable).
> SGX has an activation step that is triggered by doing a WRMSR(0x7a)
> with bit 0 set.  Until SGX is activated, the SGX related bits in
> IA32_FEATURE_CONTROL cannot be set, i.e. SGX can't be enabled.  But,
> the LE hash MSRs are fully writable prior to activation, e.g. to
> allow firmware to lock down the LE key with a non-Intel value.
>
> So yes, it's possible to lock the MSRs to a non-Intel value.  The
> obvious caveat is that whatever blob is used to write the MSRs would
> need be executed prior to activation.

This implies that it should be possible to create MSR activation (and
an embedded launch enclave?) entirely as a UEFI module. The kernel
would still get to manage who has access to /dev/sgx and other
important non-cryptographic policy details. Users would still be able
to control the cryptographic policy details (via BIOS Secure Boot
configuration that exists today). Distributions could still control
cryptographic policy details via signing of the UEFI module with their
own Secure Boot key (or using something like shim). The UEFI module
(and possibly the external launch enclave) could be distributed via
linux-firmware.

Andy/Neil, does this work for you?

> As for the SDM, it's a documentation... omission?  SGX activation
> is intentionally omitted from the SDM.  The intended usage model is
> that firmware will always do the activation (if it wants SGX enabled),
> i.e. post-firmware software will only ever "see" SGX as disabled or
> in the fully activated state, and so the SDM doesn't describe SGX
> behavior prior to activation.  I believe the activation process, or
> at least what is required from firmware, is documented in the BIOS
> writer's guide.
>
> > Jethro Beekman | Fortanix
> >
>
>

Re: [PATCH v11 00/13] Intel SGX1 support

[Ingo Molnar]

* Jarkko Sakkinen <jarkko.sakkinen@linux.intel.com> wrote:

>  create mode 100644 drivers/platform/x86/intel_sgx/Kconfig
>  create mode 100644 drivers/platform/x86/intel_sgx/Makefile
>  create mode 100644 drivers/platform/x86/intel_sgx/le/Makefile
>  create mode 100644 drivers/platform/x86/intel_sgx/le/enclave/Makefile
>  create mode 120000 drivers/platform/x86/intel_sgx/le/enclave/aesni-intel_asm.S
>  create mode 100644 drivers/platform/x86/intel_sgx/le/enclave/cmac_mode.c
>  create mode 100644 drivers/platform/x86/intel_sgx/le/enclave/cmac_mode.h
>  create mode 100644 drivers/platform/x86/intel_sgx/le/enclave/encl_bootstrap.S
>  create mode 100644 drivers/platform/x86/intel_sgx/le/enclave/main.c
>  create mode 100644 drivers/platform/x86/intel_sgx/le/enclave/main.h
>  create mode 100644 drivers/platform/x86/intel_sgx/le/enclave/sgx_le.lds
>  create mode 100644 drivers/platform/x86/intel_sgx/le/enclave/sgxsign.c
>  create mode 120000 drivers/platform/x86/intel_sgx/le/enclave/string.c
>  create mode 100644 drivers/platform/x86/intel_sgx/le/entry.S
>  create mode 100644 drivers/platform/x86/intel_sgx/le/include/sgx_asm.h
>  create mode 100644 drivers/platform/x86/intel_sgx/le/main.c
>  create mode 100644 drivers/platform/x86/intel_sgx/le/main.h
>  create mode 100644 drivers/platform/x86/intel_sgx/le/sgx_le_piggy.S
>  create mode 100644 drivers/platform/x86/intel_sgx/le/string.c
>  create mode 100644 drivers/platform/x86/intel_sgx/sgx.h
>  create mode 100644 drivers/platform/x86/intel_sgx/sgx_encl.c
>  create mode 100644 drivers/platform/x86/intel_sgx/sgx_encl_page.c
>  create mode 100644 drivers/platform/x86/intel_sgx/sgx_fault.c
>  create mode 100644 drivers/platform/x86/intel_sgx/sgx_ioctl.c
>  create mode 100644 drivers/platform/x86/intel_sgx/sgx_le.c
>  create mode 100644 drivers/platform/x86/intel_sgx/sgx_le_proxy_piggy.S
>  create mode 100644 drivers/platform/x86/intel_sgx/sgx_main.c
>  create mode 100644 drivers/platform/x86/intel_sgx/sgx_vma.c

Just some quick feedback: these are core kernel functionality and should be in 
arch/x86/, not in drivers/platform/.

Thanks,

	Ingo

Re: [PATCH v11 00/13] Intel SGX1 support

[Ingo Molnar]

* Jarkko Sakkinen <jarkko.sakkinen@linux.intel.com> wrote:

>  create mode 100644 drivers/platform/x86/intel_sgx/Kconfig
>  create mode 100644 drivers/platform/x86/intel_sgx/Makefile
>  create mode 100644 drivers/platform/x86/intel_sgx/le/Makefile
>  create mode 100644 drivers/platform/x86/intel_sgx/le/enclave/Makefile
>  create mode 120000 drivers/platform/x86/intel_sgx/le/enclave/aesni-intel_asm.S
>  create mode 100644 drivers/platform/x86/intel_sgx/le/enclave/cmac_mode.c
>  create mode 100644 drivers/platform/x86/intel_sgx/le/enclave/cmac_mode.h
>  create mode 100644 drivers/platform/x86/intel_sgx/le/enclave/encl_bootstrap.S
>  create mode 100644 drivers/platform/x86/intel_sgx/le/enclave/main.c
>  create mode 100644 drivers/platform/x86/intel_sgx/le/enclave/main.h
>  create mode 100644 drivers/platform/x86/intel_sgx/le/enclave/sgx_le.lds
>  create mode 100644 drivers/platform/x86/intel_sgx/le/enclave/sgxsign.c
>  create mode 120000 drivers/platform/x86/intel_sgx/le/enclave/string.c
>  create mode 100644 drivers/platform/x86/intel_sgx/le/entry.S
>  create mode 100644 drivers/platform/x86/intel_sgx/le/include/sgx_asm.h
>  create mode 100644 drivers/platform/x86/intel_sgx/le/main.c
>  create mode 100644 drivers/platform/x86/intel_sgx/le/main.h
>  create mode 100644 drivers/platform/x86/intel_sgx/le/sgx_le_piggy.S
>  create mode 100644 drivers/platform/x86/intel_sgx/le/string.c
>  create mode 100644 drivers/platform/x86/intel_sgx/sgx.h
>  create mode 100644 drivers/platform/x86/intel_sgx/sgx_encl.c
>  create mode 100644 drivers/platform/x86/intel_sgx/sgx_encl_page.c
>  create mode 100644 drivers/platform/x86/intel_sgx/sgx_fault.c
>  create mode 100644 drivers/platform/x86/intel_sgx/sgx_ioctl.c
>  create mode 100644 drivers/platform/x86/intel_sgx/sgx_le.c
>  create mode 100644 drivers/platform/x86/intel_sgx/sgx_le_proxy_piggy.S
>  create mode 100644 drivers/platform/x86/intel_sgx/sgx_main.c
>  create mode 100644 drivers/platform/x86/intel_sgx/sgx_vma.c

Just some quick feedback: these are core kernel functionality and should be in 
arch/x86/, not in drivers/platform/.

Thanks,

	Ingo

Re: [PATCH v11 00/13] Intel SGX1 support

[Ingo Molnar]

* Jarkko Sakkinen <jarkko.sakkinen@linux.intel.com> wrote:

>  create mode 100644 drivers/platform/x86/intel_sgx/Kconfig
>  create mode 100644 drivers/platform/x86/intel_sgx/Makefile
>  create mode 100644 drivers/platform/x86/intel_sgx/le/Makefile
>  create mode 100644 drivers/platform/x86/intel_sgx/le/enclave/Makefile
>  create mode 120000 drivers/platform/x86/intel_sgx/le/enclave/aesni-intel_asm.S
>  create mode 100644 drivers/platform/x86/intel_sgx/le/enclave/cmac_mode.c
>  create mode 100644 drivers/platform/x86/intel_sgx/le/enclave/cmac_mode.h
>  create mode 100644 drivers/platform/x86/intel_sgx/le/enclave/encl_bootstrap.S
>  create mode 100644 drivers/platform/x86/intel_sgx/le/enclave/main.c
>  create mode 100644 drivers/platform/x86/intel_sgx/le/enclave/main.h
>  create mode 100644 drivers/platform/x86/intel_sgx/le/enclave/sgx_le.lds
>  create mode 100644 drivers/platform/x86/intel_sgx/le/enclave/sgxsign.c
>  create mode 120000 drivers/platform/x86/intel_sgx/le/enclave/string.c
>  create mode 100644 drivers/platform/x86/intel_sgx/le/entry.S
>  create mode 100644 drivers/platform/x86/intel_sgx/le/include/sgx_asm.h
>  create mode 100644 drivers/platform/x86/intel_sgx/le/main.c
>  create mode 100644 drivers/platform/x86/intel_sgx/le/main.h
>  create mode 100644 drivers/platform/x86/intel_sgx/le/sgx_le_piggy.S
>  create mode 100644 drivers/platform/x86/intel_sgx/le/string.c
>  create mode 100644 drivers/platform/x86/intel_sgx/sgx.h
>  create mode 100644 drivers/platform/x86/intel_sgx/sgx_encl.c
>  create mode 100644 drivers/platform/x86/intel_sgx/sgx_encl_page.c
>  create mode 100644 drivers/platform/x86/intel_sgx/sgx_fault.c
>  create mode 100644 drivers/platform/x86/intel_sgx/sgx_ioctl.c
>  create mode 100644 drivers/platform/x86/intel_sgx/sgx_le.c
>  create mode 100644 drivers/platform/x86/intel_sgx/sgx_le_proxy_piggy.S
>  create mode 100644 drivers/platform/x86/intel_sgx/sgx_main.c
>  create mode 100644 drivers/platform/x86/intel_sgx/sgx_vma.c

Just some quick feedback: these are core kernel functionality and should be in 
arch/x86/, not in drivers/platform/.

Thanks,

	Ingo
--
To unsubscribe from this list: send the line "unsubscribe linux-doc" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [intel-sgx-kernel-dev] [PATCH v11 13/13] intel_sgx: in-kernel launch enclave

[Neil Horman]

On Thu, Jun 21, 2018 at 08:32:25AM -0400, Nathaniel McCallum wrote:
> On Wed, Jun 20, 2018 at 5:02 PM Sean Christopherson
> <sean.j.christopherson@intel.com> wrote:
> >
> > On Wed, Jun 20, 2018 at 11:39:00AM -0700, Jethro Beekman wrote:
> > > On 2018-06-20 11:16, Jethro Beekman wrote:
> > > > > This last bit is also repeated in different words in Table 35-2 and
> > > > > Section 42.2.2. The MSRs are *not writable* before the write-lock bit
> > > > > itself is locked. Meaning the MSRs are either locked with Intel's key
> > > > > hash, or not locked at all.
> > >
> > > Actually, this might be a documentation bug. I have some test hardware and I
> > > was able to configure the MSRs in the BIOS and then read the MSRs after boot
> > > like this:
> > >
> > > MSR 0x3a 0x0000000000040005
> > > MSR 0x8c 0x20180620aaaaaaaa
> > > MSR 0x8d 0x20180620bbbbbbbb
> > > MSR 0x8e 0x20180620cccccccc
> > > MSR 0x8f 0x20180620dddddddd
> > >
> > > Since this is not production hardware, it could also be a CPU bug of course.
> > >
> > > If it is indeed possible to configure AND lock the MSR values to non-Intel
> > > values, I'm very much in favor of Nathaniels proposal to treat the launch
> > > enclave like any other firmware blob.
> >
> > It's not a CPU or documentation bug (though the latter is arguable).
> > SGX has an activation step that is triggered by doing a WRMSR(0x7a)
> > with bit 0 set.  Until SGX is activated, the SGX related bits in
> > IA32_FEATURE_CONTROL cannot be set, i.e. SGX can't be enabled.  But,
> > the LE hash MSRs are fully writable prior to activation, e.g. to
> > allow firmware to lock down the LE key with a non-Intel value.
> >
> > So yes, it's possible to lock the MSRs to a non-Intel value.  The
> > obvious caveat is that whatever blob is used to write the MSRs would
> > need be executed prior to activation.
> 
> This implies that it should be possible to create MSR activation (and
> an embedded launch enclave?) entirely as a UEFI module. The kernel
> would still get to manage who has access to /dev/sgx and other
> important non-cryptographic policy details. Users would still be able
> to control the cryptographic policy details (via BIOS Secure Boot
> configuration that exists today). Distributions could still control
> cryptographic policy details via signing of the UEFI module with their
> own Secure Boot key (or using something like shim). The UEFI module
> (and possibly the external launch enclave) could be distributed via
> linux-firmware.
> 
> Andy/Neil, does this work for you?
> 
I need some time to digest it.  Who in your mind is writing the UEFI module.  Is
that the firmware vendor or IHV?

Neil

> > As for the SDM, it's a documentation... omission?  SGX activation
> > is intentionally omitted from the SDM.  The intended usage model is
> > that firmware will always do the activation (if it wants SGX enabled),
> > i.e. post-firmware software will only ever "see" SGX as disabled or
> > in the fully activated state, and so the SDM doesn't describe SGX
> > behavior prior to activation.  I believe the activation process, or
> > at least what is required from firmware, is documented in the BIOS
> > writer's guide.
> >
> > > Jethro Beekman | Fortanix
> > >
> >
> >

Re: [PATCH v11 04/13] x86, sgx: add SGX definitions to msr-index.h

[Jarkko Sakkinen]

On Tue, Jun 19, 2018 at 07:01:57AM -0700, Dave Hansen wrote:
> On 06/19/2018 06:18 AM, Jarkko Sakkinen wrote:
> > On Fri, Jun 08, 2018 at 10:25:29AM -0700, Dave Hansen wrote:
> >> On 06/08/2018 10:09 AM, Jarkko Sakkinen wrote:
> >>> Signed-off-by: Sean Christopherson <sean.j.christopherson@intel.com>
> >>> Signed-off-by: Haim Cohen <haim.cohen@intel.com>
> >>> Tested-by: Serge Ayoun <serge.ayoun@intel.com>
> >>> Reviewed-by: Jarkko Sakkinen <jarkko.sakkinen@linux.intel.com>
> >>> Tested-by: Jarkko Sakkinen <jarkko.sakkinen@linux.intel.com>
> >>> Signed-off-by: Jarkko Sakkinen <jarkko.sakkinen@linux.intel.com>
> >>
> >> These are a mess.  If you're signing off on it, a review of some kind is
> >> pretty implied.
> >>
> >> Also, how did Sean write this, send it to Haim to sign off on it, and
> >> then send it to you?  That seems unlikely.
> > 
> > They've both done something to this. How should I change it then?
> 
> Who added their Signed-off-by?
> 
> I would probably add a single "Co-developed-by:" for each of them and a
> single Signed-off-by, like commit 12c89130a56ae:
> 
>     Co-developed-by: Tony Luck <tony.luck@intel.com>
>     Reported-by: Mika Penttilä <mika.penttila@nextfour.com>
>     Signed-off-by: Dan Williams <dan.j.williams@intel.com>
>     Cc: Al Viro <viro@zeniv.linux.org.uk>
>     Cc: Andrew Morton <akpm@linux-foundation.org>
>     Cc: Andy Lutomirski <luto@amacapital.net>
>     ...

Thanks, I was not aware of that tag.

/Jarkko

Re: [PATCH v11 04/13] x86, sgx: add SGX definitions to msr-index.h

[Jarkko Sakkinen]

On Tue, Jun 19, 2018 at 07:01:57AM -0700, Dave Hansen wrote:
> On 06/19/2018 06:18 AM, Jarkko Sakkinen wrote:
> > On Fri, Jun 08, 2018 at 10:25:29AM -0700, Dave Hansen wrote:
> >> On 06/08/2018 10:09 AM, Jarkko Sakkinen wrote:
> >>> Signed-off-by: Sean Christopherson <sean.j.christopherson@intel.com>
> >>> Signed-off-by: Haim Cohen <haim.cohen@intel.com>
> >>> Tested-by: Serge Ayoun <serge.ayoun@intel.com>
> >>> Reviewed-by: Jarkko Sakkinen <jarkko.sakkinen@linux.intel.com>
> >>> Tested-by: Jarkko Sakkinen <jarkko.sakkinen@linux.intel.com>
> >>> Signed-off-by: Jarkko Sakkinen <jarkko.sakkinen@linux.intel.com>
> >>
> >> These are a mess.  If you're signing off on it, a review of some kind is
> >> pretty implied.
> >>
> >> Also, how did Sean write this, send it to Haim to sign off on it, and
> >> then send it to you?  That seems unlikely.
> > 
> > They've both done something to this. How should I change it then?
> 
> Who added their Signed-off-by?
> 
> I would probably add a single "Co-developed-by:" for each of them and a
> single Signed-off-by, like commit 12c89130a56ae:
> 
>     Co-developed-by: Tony Luck <tony.luck@intel.com>
>     Reported-by: Mika Penttilä <mika.penttila@nextfour.com>
>     Signed-off-by: Dan Williams <dan.j.williams@intel.com>
>     Cc: Al Viro <viro@zeniv.linux.org.uk>
>     Cc: Andrew Morton <akpm@linux-foundation.org>
>     Cc: Andy Lutomirski <luto@amacapital.net>
>     ...

Thanks, I was not aware of that tag.

/Jarkko

Re: [intel-sgx-kernel-dev] [PATCH v11 13/13] intel_sgx: in-kernel launch enclave

[Nathaniel McCallum]

If this is acceptable for everyone, my hope is the following:

1. Intel would split the existing code into one of the following
schemas (I don't care which):
  A. three parts: UEFI module, FLC-only kernel driver and user-space
launch enclave
  B. two parts: UEFI module (including launch enclave) and FLC-only
kernel driver

2. Intel would release a reproducible build of the GPL UEFI module
sources signed with a SecureBoot trusted key and provide an
acceptable[0] binary redistribution license.

3. The kernel community would agree to merge the kernel driver given
the above criteria (and, obviously, acceptable kernel code).

The question of how to distribute the UEFI module and possible launch
enclave remains open. I see two options: independent distribution and
bundling it in linux-firmware. The former may be a better
technological fit since the UEFI module will likely need to be run
before the kernel (and the boot loader; and shim). However, the latter
has the benefit of already being a well-known entity to our downstream
distributors. I could go either way on this.

I know this plan is more work for everyone involved, but I think it
manages to actually maximize both security and freedom.

[0]: details here -
https://git.kernel.org/pub/scm/linux/kernel/git/firmware/linux-firmware.git/tree/README#n19
On Thu, Jun 21, 2018 at 11:29 AM Neil Horman <nhorman@redhat.com> wrote:
>
> On Thu, Jun 21, 2018 at 08:32:25AM -0400, Nathaniel McCallum wrote:
> > On Wed, Jun 20, 2018 at 5:02 PM Sean Christopherson
> > <sean.j.christopherson@intel.com> wrote:
> > >
> > > On Wed, Jun 20, 2018 at 11:39:00AM -0700, Jethro Beekman wrote:
> > > > On 2018-06-20 11:16, Jethro Beekman wrote:
> > > > > > This last bit is also repeated in different words in Table 35-2 and
> > > > > > Section 42.2.2. The MSRs are *not writable* before the write-lock bit
> > > > > > itself is locked. Meaning the MSRs are either locked with Intel's key
> > > > > > hash, or not locked at all.
> > > >
> > > > Actually, this might be a documentation bug. I have some test hardware and I
> > > > was able to configure the MSRs in the BIOS and then read the MSRs after boot
> > > > like this:
> > > >
> > > > MSR 0x3a 0x0000000000040005
> > > > MSR 0x8c 0x20180620aaaaaaaa
> > > > MSR 0x8d 0x20180620bbbbbbbb
> > > > MSR 0x8e 0x20180620cccccccc
> > > > MSR 0x8f 0x20180620dddddddd
> > > >
> > > > Since this is not production hardware, it could also be a CPU bug of course.
> > > >
> > > > If it is indeed possible to configure AND lock the MSR values to non-Intel
> > > > values, I'm very much in favor of Nathaniels proposal to treat the launch
> > > > enclave like any other firmware blob.
> > >
> > > It's not a CPU or documentation bug (though the latter is arguable).
> > > SGX has an activation step that is triggered by doing a WRMSR(0x7a)
> > > with bit 0 set.  Until SGX is activated, the SGX related bits in
> > > IA32_FEATURE_CONTROL cannot be set, i.e. SGX can't be enabled.  But,
> > > the LE hash MSRs are fully writable prior to activation, e.g. to
> > > allow firmware to lock down the LE key with a non-Intel value.
> > >
> > > So yes, it's possible to lock the MSRs to a non-Intel value.  The
> > > obvious caveat is that whatever blob is used to write the MSRs would
> > > need be executed prior to activation.
> >
> > This implies that it should be possible to create MSR activation (and
> > an embedded launch enclave?) entirely as a UEFI module. The kernel
> > would still get to manage who has access to /dev/sgx and other
> > important non-cryptographic policy details. Users would still be able
> > to control the cryptographic policy details (via BIOS Secure Boot
> > configuration that exists today). Distributions could still control
> > cryptographic policy details via signing of the UEFI module with their
> > own Secure Boot key (or using something like shim). The UEFI module
> > (and possibly the external launch enclave) could be distributed via
> > linux-firmware.
> >
> > Andy/Neil, does this work for you?
> >
> I need some time to digest it.  Who in your mind is writing the UEFI module.  Is
> that the firmware vendor or IHV?
>
> Neil
>
> > > As for the SDM, it's a documentation... omission?  SGX activation
> > > is intentionally omitted from the SDM.  The intended usage model is
> > > that firmware will always do the activation (if it wants SGX enabled),
> > > i.e. post-firmware software will only ever "see" SGX as disabled or
> > > in the fully activated state, and so the SDM doesn't describe SGX
> > > behavior prior to activation.  I believe the activation process, or
> > > at least what is required from firmware, is documented in the BIOS
> > > writer's guide.
> > >
> > > > Jethro Beekman | Fortanix
> > > >
> > >
> > >

Re: [intel-sgx-kernel-dev] [PATCH v11 13/13] intel_sgx: in-kernel launch enclave

[Sean Christopherson]

On Thu, Jun 21, 2018 at 03:11:18PM -0400, Nathaniel McCallum wrote:
> If this is acceptable for everyone, my hope is the following:
> 
> 1. Intel would split the existing code into one of the following
> schemas (I don't care which):
>   A. three parts: UEFI module, FLC-only kernel driver and user-space
> launch enclave
>   B. two parts: UEFI module (including launch enclave) and FLC-only
> kernel driver

To make sure I understand correctly...

The UEFI module would lock the LE MSRs with a public key hardcoded
into both the UEFI module and the kernel at build time?

And for the kernel, it would only load its SGX driver if FLC is
supported and the MSRs are locked to the expected key?

IIUC, this approach will cause problems for virtualization.  Running
VMs with different LE keys would require the bare metal firmware to
configure the LE MSRs to be unlocked, which would effectively make
using SGX in the host OS mutually exlusive with exposing SGX to KVM
guests.  Theoretically it would be possible for KVM to emulate the
guest's LE and use the host's LE to generate EINIT tokens, but
emulating an enclave would likely require a massive amount of code
and/or complexity.

> 2. Intel would release a reproducible build of the GPL UEFI module
> sources signed with a SecureBoot trusted key and provide an
> acceptable[0] binary redistribution license.
> 
> 3. The kernel community would agree to merge the kernel driver given
> the above criteria (and, obviously, acceptable kernel code).
> 
> The question of how to distribute the UEFI module and possible launch
> enclave remains open. I see two options: independent distribution and
> bundling it in linux-firmware. The former may be a better
> technological fit since the UEFI module will likely need to be run
> before the kernel (and the boot loader; and shim). However, the latter
> has the benefit of already being a well-known entity to our downstream
> distributors. I could go either way on this.

Writing and locks the LE MSRs effectively needs to be done before
running the bootloader/kernel/etc...  Delaying activation would
require, at a minimum, leaving IA32_FEATURE_CONTROL unlocked since
IA32_FEATURE_CONTROL's SGX bits can't be set until SGX is activated.

> I know this plan is more work for everyone involved, but I think it
> manages to actually maximize both security and freedom.
> 
> [0]: details here -
> https://git.kernel.org/pub/scm/linux/kernel/git/firmware/linux-firmware.git/tree/README#n19
> On Thu, Jun 21, 2018 at 11:29 AM Neil Horman <nhorman@redhat.com> wrote:
> >
> > On Thu, Jun 21, 2018 at 08:32:25AM -0400, Nathaniel McCallum wrote:
> > > On Wed, Jun 20, 2018 at 5:02 PM Sean Christopherson
> > > <sean.j.christopherson@intel.com> wrote:
> > > >
> > > > On Wed, Jun 20, 2018 at 11:39:00AM -0700, Jethro Beekman wrote:
> > > > > On 2018-06-20 11:16, Jethro Beekman wrote:
> > > > > > > This last bit is also repeated in different words in Table 35-2 and
> > > > > > > Section 42.2.2. The MSRs are *not writable* before the write-lock bit
> > > > > > > itself is locked. Meaning the MSRs are either locked with Intel's key
> > > > > > > hash, or not locked at all.
> > > > >
> > > > > Actually, this might be a documentation bug. I have some test hardware and I
> > > > > was able to configure the MSRs in the BIOS and then read the MSRs after boot
> > > > > like this:
> > > > >
> > > > > MSR 0x3a 0x0000000000040005
> > > > > MSR 0x8c 0x20180620aaaaaaaa
> > > > > MSR 0x8d 0x20180620bbbbbbbb
> > > > > MSR 0x8e 0x20180620cccccccc
> > > > > MSR 0x8f 0x20180620dddddddd
> > > > >
> > > > > Since this is not production hardware, it could also be a CPU bug of course.
> > > > >
> > > > > If it is indeed possible to configure AND lock the MSR values to non-Intel
> > > > > values, I'm very much in favor of Nathaniels proposal to treat the launch
> > > > > enclave like any other firmware blob.
> > > >
> > > > It's not a CPU or documentation bug (though the latter is arguable).
> > > > SGX has an activation step that is triggered by doing a WRMSR(0x7a)
> > > > with bit 0 set.  Until SGX is activated, the SGX related bits in
> > > > IA32_FEATURE_CONTROL cannot be set, i.e. SGX can't be enabled.  But,
> > > > the LE hash MSRs are fully writable prior to activation, e.g. to
> > > > allow firmware to lock down the LE key with a non-Intel value.
> > > >
> > > > So yes, it's possible to lock the MSRs to a non-Intel value.  The
> > > > obvious caveat is that whatever blob is used to write the MSRs would
> > > > need be executed prior to activation.
> > >
> > > This implies that it should be possible to create MSR activation (and
> > > an embedded launch enclave?) entirely as a UEFI module. The kernel
> > > would still get to manage who has access to /dev/sgx and other
> > > important non-cryptographic policy details. Users would still be able
> > > to control the cryptographic policy details (via BIOS Secure Boot
> > > configuration that exists today). Distributions could still control
> > > cryptographic policy details via signing of the UEFI module with their
> > > own Secure Boot key (or using something like shim). The UEFI module
> > > (and possibly the external launch enclave) could be distributed via
> > > linux-firmware.
> > >
> > > Andy/Neil, does this work for you?
> > >
> > I need some time to digest it.  Who in your mind is writing the UEFI module.  Is
> > that the firmware vendor or IHV?
> >
> > Neil
> >
> > > > As for the SDM, it's a documentation... omission?  SGX activation
> > > > is intentionally omitted from the SDM.  The intended usage model is
> > > > that firmware will always do the activation (if it wants SGX enabled),
> > > > i.e. post-firmware software will only ever "see" SGX as disabled or
> > > > in the fully activated state, and so the SDM doesn't describe SGX
> > > > behavior prior to activation.  I believe the activation process, or
> > > > at least what is required from firmware, is documented in the BIOS
> > > > writer's guide.
> > > >
> > > > > Jethro Beekman | Fortanix
> > > > >
> > > >
> > > >

Re: [intel-sgx-kernel-dev] [PATCH v11 13/13] intel_sgx: in-kernel launch enclave

[Andy Lutomirski]

On Thu, Jun 21, 2018 at 12:11 PM Nathaniel McCallum
<npmccallum@redhat.com> wrote:
>
> If this is acceptable for everyone, my hope is the following:
>
> 1. Intel would split the existing code into one of the following
> schemas (I don't care which):
>   A. three parts: UEFI module, FLC-only kernel driver and user-space
> launch enclave
>   B. two parts: UEFI module (including launch enclave) and FLC-only
> kernel driver
>
> 2. Intel would release a reproducible build of the GPL UEFI module
> sources signed with a SecureBoot trusted key and provide an
> acceptable[0] binary redistribution license.
>
> 3. The kernel community would agree to merge the kernel driver given
> the above criteria (and, obviously, acceptable kernel code).
>
> The question of how to distribute the UEFI module and possible launch
> enclave remains open. I see two options: independent distribution and
> bundling it in linux-firmware. The former may be a better
> technological fit since the UEFI module will likely need to be run
> before the kernel (and the boot loader; and shim). However, the latter
> has the benefit of already being a well-known entity to our downstream
> distributors. I could go either way on this.

This is a lot of complication and effort for a gain that is not
entirely clear.  I really really really do *not* want to see Intel or
anyone else start enforcing policy on which programs can and cannot
run using this mechanism.  (This is exactly why non-FLC systems aren't
about to be supported upstream.)  So my preference is to not merge
anything that supports this type of use case unless there is
compelling evidence that it is (a) genuinely useful, (b) will be used
to improve security and (c) won't be abused for, say, revenue
purposes.

--Andy

Re: [intel-sgx-kernel-dev] [PATCH v11 09/13] x86, sgx: basic routines for enclave page cache

[Jarkko Sakkinen]

On Mon, 2018-06-18 at 14:33 -0700, Andy Lutomirski wrote:
> When KVM host support goes in, even this won't be good enough if we
> want to allow passthrough access to the MSRs because we will no longer
> be able to guarantee that all zeros is invalid.  Instead we'd need an
> actual flag saying that the cache is invalid.

I'm not sure if I understood this part. If it was pass-through, and
there was a flag, how that flag in the host would get updated?

/Jarkko

Re: [intel-sgx-kernel-dev] [PATCH v11 07/13] x86, sgx: detect Intel SGX

[Jarkko Sakkinen]

On Mon, 2018-06-18 at 14:36 -0700, Andy Lutomirski wrote:
> SGX: disabled by firmware
> SGX: disabled because CPU does not support flexible launch control
> SGX: disabled because firmware does not support flexible launch control
> SGX: disabled because the phase of the moon is wrong
> SGX: enabled; using SGX1
> SGX: enabled, using SGX2

Makes sense for trouble-shooting (and in that case the check for
SGX_ENABLE flag is more appropriate).

/Jarkko

Re: [PATCH v11 09/13] x86, sgx: basic routines for enclave page cache

[Jarkko Sakkinen]

On Tue, 2018-06-19 at 08:32 -0700, Dave Hansen wrote:
> > For 32-bit (albeit not supported at this point) it makes sense to always
> > use kmap_atomic_pfn() as the virtua address area is very limited.
> 
> That makes no sense.  32-bit kernels have plenty of virtual address
> space if not using highmem.

AFAIK the vmalloc area is located in the last 128MB that and ioremap()
of EPC would consume a big portion of that.

> > KVM works only with writable MSRs. Driver works both with writable
> > and read-only MSRs.
> 
> Could you help with my confusion by documenting this a bit?

Sure.

/Jarkko

Re: [PATCH v11 09/13] x86, sgx: basic routines for enclave page cache

[Jarkko Sakkinen]

On Tue, 2018-06-19 at 08:32 -0700, Dave Hansen wrote:
> > For 32-bit (albeit not supported at this point) it makes sense to always
> > use kmap_atomic_pfn() as the virtua address area is very limited.
> 
> That makes no sense.  32-bit kernels have plenty of virtual address
> space if not using highmem.

AFAIK the vmalloc area is located in the last 128MB that and ioremap()
of EPC would consume a big portion of that.

> > KVM works only with writable MSRs. Driver works both with writable
> > and read-only MSRs.
> 
> Could you help with my confusion by documenting this a bit?

Sure.

/Jarkko

Re: [PATCH v11 09/13] x86, sgx: basic routines for enclave page cache

[Jarkko Sakkinen]

On Tue, 2018-06-19 at 08:59 -0700, Sean Christopherson wrote:
> On Tue, Jun 19, 2018 at 05:57:53PM +0300, Jarkko Sakkinen wrote:
> > On Fri, Jun 08, 2018 at 11:24:12AM -0700, Dave Hansen wrote:
> > > On 06/08/2018 10:09 AM, Jarkko Sakkinen wrote:
> > > > +static __init bool sgx_is_enabled(bool *lc_enabled)
> > > >  {
> > > >  	unsigned long fc;
> > > >  
> > > > @@ -41,12 +466,26 @@ static __init bool sgx_is_enabled(void)
> > > >  	if (!(fc & FEATURE_CONTROL_SGX_ENABLE))
> > > >  		return false;
> > > >  
> > > > +	*lc_enabled = !!(fc & FEATURE_CONTROL_SGX_LE_WR);
> > > > +
> > > >  	return true;
> > > >  }
> > > 
> > > I'm baffled why lc_enabled is connected to the enclave page cache.
> > 
> > KVM works only with writable MSRs. Driver works both with writable
> > and read-only MSRs.
> 
> That's not true, KVM can/will support SGX regardless of whether or not
> Launch Control (LC) is available and/or enabled.  KVM does need to
> know whether or not LC is enabled.

I understood from earlier discussion with talks that this was
additional requirement for KVM but please shed some light.

Could this be dropped altogether from this patch set and then brought up
in the KVM patch set? Right now it is unused.

/Jarkko

Re: [PATCH v11 09/13] x86, sgx: basic routines for enclave page cache

[Jarkko Sakkinen]

On Tue, 2018-06-19 at 08:59 -0700, Sean Christopherson wrote:
> On Tue, Jun 19, 2018 at 05:57:53PM +0300, Jarkko Sakkinen wrote:
> > On Fri, Jun 08, 2018 at 11:24:12AM -0700, Dave Hansen wrote:
> > > On 06/08/2018 10:09 AM, Jarkko Sakkinen wrote:
> > > > +static __init bool sgx_is_enabled(bool *lc_enabled)
> > > >  {
> > > >  	unsigned long fc;
> > > >  
> > > > @@ -41,12 +466,26 @@ static __init bool sgx_is_enabled(void)
> > > >  	if (!(fc & FEATURE_CONTROL_SGX_ENABLE))
> > > >  		return false;
> > > >  
> > > > +	*lc_enabled = !!(fc & FEATURE_CONTROL_SGX_LE_WR);
> > > > +
> > > >  	return true;
> > > >  }
> > > 
> > > I'm baffled why lc_enabled is connected to the enclave page cache.
> > 
> > KVM works only with writable MSRs. Driver works both with writable
> > and read-only MSRs.
> 
> That's not true, KVM can/will support SGX regardless of whether or not
> Launch Control (LC) is available and/or enabled.  KVM does need to
> know whether or not LC is enabled.

I understood from earlier discussion with talks that this was
additional requirement for KVM but please shed some light.

Could this be dropped altogether from this patch set and then brought up
in the KVM patch set? Right now it is unused.

/Jarkko

Re: [PATCH v11 08/13] x86, sgx: added ENCLS wrappers

[Jarkko Sakkinen]

On Wed, 2018-06-20 at 06:12 -0700, Sean Christopherson wrote:
> On Fri, 2018-06-08 at 19:09 +0200, Jarkko Sakkinen wrote:
> > This commit adds wrappers for Intel(R) SGX ENCLS opcode functionality.
> > 
> > Signed-off-by: Jarkko Sakkinen <jarkko.sakkinen@linux.intel.com>
> > ---
> >  arch/x86/include/asm/sgx.h | 198 +++++++++++++++++++++++++++++++++++++
> >  1 file changed, 198 insertions(+)
> > 
> > diff --git a/arch/x86/include/asm/sgx.h b/arch/x86/include/asm/sgx.h
> > index fa3e6e0eb8af..a2f727f85b91 100644
> > --- a/arch/x86/include/asm/sgx.h
> > +++ b/arch/x86/include/asm/sgx.h
> > @@ -10,6 +10,10 @@
> >  #ifndef _ASM_X86_SGX_H
> >  #define _ASM_X86_SGX_H
> >  
> > +#include <asm/sgx_arch.h>
> > +#include <asm/asm.h>
> > +#include <linux/bitops.h>
> > +#include <linux/err.h>
> >  #include <linux/types.h>
> >  
> >  #define SGX_CPUID 0x12
> > @@ -20,6 +24,200 @@ enum sgx_cpuid {
> >  	SGX_CPUID_EPC_BANKS	= 2,
> >  };
> >  
> > +enum sgx_commands {
> 
> This should be something like "sgx_encls_leafs" and probably moved
> to sgx_arch.h (as Dave alluded to, these are architectural values).
> "sgx_commands" is not accurate (they're only the cpl0 "commands")
> and will have collision issues in the future, e.g. with the ENCLV
> instruction and its leafs.
> 
> > +	ECREATE	= 0x0,
> > +	EADD	= 0x1,
> > +	EINIT	= 0x2,
> > +	EREMOVE	= 0x3,
> > +	EDGBRD	= 0x4,
> > +	EDGBWR	= 0x5,
> > +	EEXTEND	= 0x6,
> 
> Even though it's not used in the code (yet...), I think ELDB,
> leaf 0x7, should be defined here for completeness.

Sure.

/Jarkko

Re: [PATCH v11 08/13] x86, sgx: added ENCLS wrappers

[Jarkko Sakkinen]

On Wed, 2018-06-20 at 06:12 -0700, Sean Christopherson wrote:
> On Fri, 2018-06-08 at 19:09 +0200, Jarkko Sakkinen wrote:
> > This commit adds wrappers for Intel(R) SGX ENCLS opcode functionality.
> > 
> > Signed-off-by: Jarkko Sakkinen <jarkko.sakkinen@linux.intel.com>
> > ---
> >  arch/x86/include/asm/sgx.h | 198 +++++++++++++++++++++++++++++++++++++
> >  1 file changed, 198 insertions(+)
> > 
> > diff --git a/arch/x86/include/asm/sgx.h b/arch/x86/include/asm/sgx.h
> > index fa3e6e0eb8af..a2f727f85b91 100644
> > --- a/arch/x86/include/asm/sgx.h
> > +++ b/arch/x86/include/asm/sgx.h
> > @@ -10,6 +10,10 @@
> >  #ifndef _ASM_X86_SGX_H
> >  #define _ASM_X86_SGX_H
> >  
> > +#include <asm/sgx_arch.h>
> > +#include <asm/asm.h>
> > +#include <linux/bitops.h>
> > +#include <linux/err.h>
> >  #include <linux/types.h>
> >  
> >  #define SGX_CPUID 0x12
> > @@ -20,6 +24,200 @@ enum sgx_cpuid {
> >  	SGX_CPUID_EPC_BANKS	= 2,
> >  };
> >  
> > +enum sgx_commands {
> 
> This should be something like "sgx_encls_leafs" and probably moved
> to sgx_arch.h (as Dave alluded to, these are architectural values).
> "sgx_commands" is not accurate (they're only the cpl0 "commands")
> and will have collision issues in the future, e.g. with the ENCLV
> instruction and its leafs.
> 
> > +	ECREATE	= 0x0,
> > +	EADD	= 0x1,
> > +	EINIT	= 0x2,
> > +	EREMOVE	= 0x3,
> > +	EDGBRD	= 0x4,
> > +	EDGBWR	= 0x5,
> > +	EEXTEND	= 0x6,
> 
> Even though it's not used in the code (yet...), I think ELDB,
> leaf 0x7, should be defined here for completeness.

Sure.

/Jarkko

Re: [PATCH v11 09/13] x86, sgx: basic routines for enclave page cache

[Jarkko Sakkinen]

On Wed, 2018-06-20 at 06:21 -0700, Sean Christopherson wrote:
> On Fri, 2018-06-08 at 19:09 +0200, Jarkko Sakkinen wrote:
> > SGX has a set of data structures to maintain information about the enclaves
> > and their security properties. BIOS reserves a fixed size region of
> > physical memory for these structures by setting Processor Reserved Memory
> > Range Registers (PRMRR). This memory area is called Enclave Page Cache
> > (EPC).
> > 
> > This commit implements the basic routines to allocate and free pages from
> > different EPC banks. There is also a swapper thread ksgxswapd for EPC pages
> > that gets woken up by sgx_alloc_page() when we run below the low watermark.
> > The swapper thread continues swapping pages up until it reaches the high
> > watermark.
> > 
> > Each subsystem that uses SGX must provide a set of callbacks for EPC
> > pages that are used to reclaim, block and write an EPC page. Kernel
> > takes the responsibility of maintaining LRU cache for them.
> > 
> > Signed-off-by: Jarkko Sakkinen <jarkko.sakkinen@linux.intel.com>
> > ---
> >  arch/x86/include/asm/sgx.h      |  67 +++++
> >  arch/x86/include/asm/sgx_arch.h | 224 ++++++++++++++++
> >  arch/x86/kernel/cpu/intel_sgx.c | 443 +++++++++++++++++++++++++++++++-
> >  3 files changed, 732 insertions(+), 2 deletions(-)
> >  create mode 100644 arch/x86/include/asm/sgx_arch.h
> 
> ...
> 
> > +struct sgx_pcmd {
> > +	struct sgx_secinfo secinfo;
> > +	uint64_t enclave_id;
> > +	uint8_t reserved[40];
> > +	uint8_t mac[16];
> > +};
> 
> sgx_pcmd has a 128-byte alignment requirement.  I think it's
> worth specifying here as sgx_pcmd is small enough that it could
> be put on the stack, e.g. by KVM when trapping and executing
> ELD* on behalf of a guest VM.
> 
> In fact, it probably makes sense to add alightment attributes
> to all SGX structs for self-documentation purposes, even though
> many of them will never be allocated statically or on the stack.

I agree with this. It also documents stuff so that you don't have
to look it up from the SDM.

Neil: this should also clear your concerns.

/Jarkko

Re: [PATCH v11 09/13] x86, sgx: basic routines for enclave page cache

[Jarkko Sakkinen]

On Wed, 2018-06-20 at 06:21 -0700, Sean Christopherson wrote:
> On Fri, 2018-06-08 at 19:09 +0200, Jarkko Sakkinen wrote:
> > SGX has a set of data structures to maintain information about the enclaves
> > and their security properties. BIOS reserves a fixed size region of
> > physical memory for these structures by setting Processor Reserved Memory
> > Range Registers (PRMRR). This memory area is called Enclave Page Cache
> > (EPC).
> > 
> > This commit implements the basic routines to allocate and free pages from
> > different EPC banks. There is also a swapper thread ksgxswapd for EPC pages
> > that gets woken up by sgx_alloc_page() when we run below the low watermark.
> > The swapper thread continues swapping pages up until it reaches the high
> > watermark.
> > 
> > Each subsystem that uses SGX must provide a set of callbacks for EPC
> > pages that are used to reclaim, block and write an EPC page. Kernel
> > takes the responsibility of maintaining LRU cache for them.
> > 
> > Signed-off-by: Jarkko Sakkinen <jarkko.sakkinen@linux.intel.com>
> > ---
> >  arch/x86/include/asm/sgx.h      |  67 +++++
> >  arch/x86/include/asm/sgx_arch.h | 224 ++++++++++++++++
> >  arch/x86/kernel/cpu/intel_sgx.c | 443 +++++++++++++++++++++++++++++++-
> >  3 files changed, 732 insertions(+), 2 deletions(-)
> >  create mode 100644 arch/x86/include/asm/sgx_arch.h
> 
> ...
> 
> > +struct sgx_pcmd {
> > +	struct sgx_secinfo secinfo;
> > +	uint64_t enclave_id;
> > +	uint8_t reserved[40];
> > +	uint8_t mac[16];
> > +};
> 
> sgx_pcmd has a 128-byte alignment requirement.  I think it's
> worth specifying here as sgx_pcmd is small enough that it could
> be put on the stack, e.g. by KVM when trapping and executing
> ELD* on behalf of a guest VM.
> 
> In fact, it probably makes sense to add alightment attributes
> to all SGX structs for self-documentation purposes, even though
> many of them will never be allocated statically or on the stack.

I agree with this. It also documents stuff so that you don't have
to look it up from the SDM.

Neil: this should also clear your concerns.

/Jarkko

Re: [PATCH v11 09/13] x86, sgx: basic routines for enclave page cache

[Jarkko Sakkinen]

On Wed, 2018-06-20 at 08:26 -0700, Sean Christopherson wrote:
> >  bool sgx_enabled __ro_after_init = false;
> >  EXPORT_SYMBOL(sgx_enabled);
> > +bool sgx_lc_enabled __ro_after_init;
> > +EXPORT_SYMBOL(sgx_lc_enabled);
> > +atomic_t sgx_nr_free_pages = ATOMIC_INIT(0);
> > +EXPORT_SYMBOL(sgx_nr_free_pages);
> > +struct sgx_epc_bank sgx_epc_banks[SGX_MAX_EPC_BANKS];
> > +EXPORT_SYMBOL(sgx_epc_banks);
> > +int sgx_nr_epc_banks;
> > +EXPORT_SYMBOL(sgx_nr_epc_banks);
> > +LIST_HEAD(sgx_active_page_list);
> > +EXPORT_SYMBOL(sgx_active_page_list);
> > +DEFINE_SPINLOCK(sgx_active_page_list_lock);
> > +EXPORT_SYMBOL(sgx_active_page_list_lock);
> 
> I don't think we should be exporting anything other than sgx_enabled
> and sgx_lc_enabled.  The only external use of a symbol that can't be
> trivially (re)moved is in the driver's sgx_pm_suspend() in sgx_main.c,
> which uses the sgx_active_page_list to invalidate enclaves.  And that
> behavior seems unsafe, e.g. an enclave could theoretically have zero
> pages on the active list and so could be missed in the suspend flow.

Fully agreed. Just forgotten cruft...

/Jarkko

Re: [PATCH v11 09/13] x86, sgx: basic routines for enclave page cache

[Jarkko Sakkinen]

On Wed, 2018-06-20 at 08:26 -0700, Sean Christopherson wrote:
> >  bool sgx_enabled __ro_after_init = false;
> >  EXPORT_SYMBOL(sgx_enabled);
> > +bool sgx_lc_enabled __ro_after_init;
> > +EXPORT_SYMBOL(sgx_lc_enabled);
> > +atomic_t sgx_nr_free_pages = ATOMIC_INIT(0);
> > +EXPORT_SYMBOL(sgx_nr_free_pages);
> > +struct sgx_epc_bank sgx_epc_banks[SGX_MAX_EPC_BANKS];
> > +EXPORT_SYMBOL(sgx_epc_banks);
> > +int sgx_nr_epc_banks;
> > +EXPORT_SYMBOL(sgx_nr_epc_banks);
> > +LIST_HEAD(sgx_active_page_list);
> > +EXPORT_SYMBOL(sgx_active_page_list);
> > +DEFINE_SPINLOCK(sgx_active_page_list_lock);
> > +EXPORT_SYMBOL(sgx_active_page_list_lock);
> 
> I don't think we should be exporting anything other than sgx_enabled
> and sgx_lc_enabled.  The only external use of a symbol that can't be
> trivially (re)moved is in the driver's sgx_pm_suspend() in sgx_main.c,
> which uses the sgx_active_page_list to invalidate enclaves.  And that
> behavior seems unsafe, e.g. an enclave could theoretically have zero
> pages on the active list and so could be missed in the suspend flow.

Fully agreed. Just forgotten cruft...

/Jarkko

Re: [intel-sgx-kernel-dev] [PATCH v11 13/13] intel_sgx: in-kernel launch enclave

[Jarkko Sakkinen]

On Wed, 2018-06-20 at 12:28 -0400, Nathaniel McCallum wrote:
> As I understand it, the current policy models under discussion look like this:
> 
> 1. SGX w/o FLC (not being merged) looks like this:
>   Intel CPU => (Intel signed) launch enclave => enclaves
> 
> 2. SGX w/ FLC, looks like this:
>   Intel CPU => kernel => launch enclave => enclaves
> 
> 3. Andy is proposing this:
>   Intel CPU => kernel => enclaves

What if MSRs are not writable after hand over to the OS? It is a legit
configuration at least according to the SDM.

/Jarkko

Re: [intel-sgx-kernel-dev] [PATCH v11 13/13] intel_sgx: in-kernel launch enclave

[Jarkko Sakkinen]

On Thu, 2018-06-21 at 08:32 -0400, Nathaniel McCallum wrote:
> This implies that it should be possible to create MSR activation (and
> an embedded launch enclave?) entirely as a UEFI module. The kernel
> would still get to manage who has access to /dev/sgx and other
> important non-cryptographic policy details. Users would still be able
> to control the cryptographic policy details (via BIOS Secure Boot
> configuration that exists today). Distributions could still control
> cryptographic policy details via signing of the UEFI module with their
> own Secure Boot key (or using something like shim). The UEFI module
> (and possibly the external launch enclave) could be distributed via
> linux-firmware.
> 
> Andy/Neil, does this work for you?

Nothing against having UEFI module for MSR activation step.

And we would move the existing in-kernel LE to firmware so that it is
avaible for locked-in-to-non-Intel-values case? 

/Jarkko

Re: [PATCH v11 00/13] Intel SGX1 support

[Jarkko Sakkinen]

On Thu, 2018-06-21 at 14:55 +0200, Ingo Molnar wrote:
> Just some quick feedback: these are core kernel functionality and should be
> in 
> arch/x86/, not in drivers/platform/.

We have in core what is used by both KVM and the driver right now (EPC
management, SGX detection and some other bits). The driver uses this
functionality to launch enclaves.

> Thanks,
> 
> 	Ingo

/Jarkko

Re: [PATCH v11 00/13] Intel SGX1 support

[Jarkko Sakkinen]

On Thu, 2018-06-21 at 14:55 +0200, Ingo Molnar wrote:
> Just some quick feedback: these are core kernel functionality and should be
> in 
> arch/x86/, not in drivers/platform/.

We have in core what is used by both KVM and the driver right now (EPC
management, SGX detection and some other bits). The driver uses this
functionality to launch enclaves.

> Thanks,
> 
> 	Ingo

/Jarkko

Re: [PATCH v11 00/13] Intel SGX1 support

[Jarkko Sakkinen]

On Thu, 2018-06-21 at 14:55 +0200, Ingo Molnar wrote:
> Just some quick feedback: these are core kernel functionality and should be
> in 
> arch/x86/, not in drivers/platform/.

We have in core what is used by both KVM and the driver right now (EPC
management, SGX detection and some other bits). The driver uses this
functionality to launch enclaves.

> Thanks,
> 
> 	Ingo

/Jarkko
--
To unsubscribe from this list: send the line "unsubscribe linux-doc" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [intel-sgx-kernel-dev] [PATCH v11 13/13] intel_sgx: in-kernel launch enclave

[Andy Lutomirski]

On Mon, Jun 25, 2018 at 2:41 AM Jarkko Sakkinen
<jarkko.sakkinen@linux.intel.com> wrote:
>
> On Thu, 2018-06-21 at 08:32 -0400, Nathaniel McCallum wrote:
> > This implies that it should be possible to create MSR activation (and
> > an embedded launch enclave?) entirely as a UEFI module. The kernel
> > would still get to manage who has access to /dev/sgx and other
> > important non-cryptographic policy details. Users would still be able
> > to control the cryptographic policy details (via BIOS Secure Boot
> > configuration that exists today). Distributions could still control
> > cryptographic policy details via signing of the UEFI module with their
> > own Secure Boot key (or using something like shim). The UEFI module
> > (and possibly the external launch enclave) could be distributed via
> > linux-firmware.
> >
> > Andy/Neil, does this work for you?
>
> Nothing against having UEFI module for MSR activation step.
>
> And we would move the existing in-kernel LE to firmware so that it is
> avaible for locked-in-to-non-Intel-values case?
>

This is a hell of a lot of complexity.  To get it right we'd need an
actual formal spec of what firmware is supposed to do and how it
integrates with the kernel, and we'd need a reason why it's useful.

I'm personally rather strongly in favor of the vastly simpler model in
which we first merge SGX without LE support at all.  Instead we use
the approach where we just twiddle the MSRs to launch normal enclaves
without an init token at all, which is probably considerably faster
and will remove several thousand lines of code.  If and when a bona
fide use case for LE support shows up, we can work out the details and
merge it.

Right now, we're talking about a lot of design considerations, a lot
of interoperability considerations, and a lot of code to support a use
case that doesn't clearly exist.

--Andy

Re: [PATCH v11 09/13] x86, sgx: basic routines for enclave page cache

[Neil Horman]

On Mon, Jun 25, 2018 at 12:21:22PM +0300, Jarkko Sakkinen wrote:
> On Wed, 2018-06-20 at 06:21 -0700, Sean Christopherson wrote:
> > On Fri, 2018-06-08 at 19:09 +0200, Jarkko Sakkinen wrote:
> > > SGX has a set of data structures to maintain information about the enclaves
> > > and their security properties. BIOS reserves a fixed size region of
> > > physical memory for these structures by setting Processor Reserved Memory
> > > Range Registers (PRMRR). This memory area is called Enclave Page Cache
> > > (EPC).
> > > 
> > > This commit implements the basic routines to allocate and free pages from
> > > different EPC banks. There is also a swapper thread ksgxswapd for EPC pages
> > > that gets woken up by sgx_alloc_page() when we run below the low watermark.
> > > The swapper thread continues swapping pages up until it reaches the high
> > > watermark.
> > > 
> > > Each subsystem that uses SGX must provide a set of callbacks for EPC
> > > pages that are used to reclaim, block and write an EPC page. Kernel
> > > takes the responsibility of maintaining LRU cache for them.
> > > 
> > > Signed-off-by: Jarkko Sakkinen <jarkko.sakkinen@linux.intel.com>
> > > ---
> > >  arch/x86/include/asm/sgx.h      |  67 +++++
> > >  arch/x86/include/asm/sgx_arch.h | 224 ++++++++++++++++
> > >  arch/x86/kernel/cpu/intel_sgx.c | 443 +++++++++++++++++++++++++++++++-
> > >  3 files changed, 732 insertions(+), 2 deletions(-)
> > >  create mode 100644 arch/x86/include/asm/sgx_arch.h
> > 
> > ...
> > 
> > > +struct sgx_pcmd {
> > > +	struct sgx_secinfo secinfo;
> > > +	uint64_t enclave_id;
> > > +	uint8_t reserved[40];
> > > +	uint8_t mac[16];
> > > +};
> > 
> > sgx_pcmd has a 128-byte alignment requirement.  I think it's
> > worth specifying here as sgx_pcmd is small enough that it could
> > be put on the stack, e.g. by KVM when trapping and executing
> > ELD* on behalf of a guest VM.
> > 
> > In fact, it probably makes sense to add alightment attributes
> > to all SGX structs for self-documentation purposes, even though
> > many of them will never be allocated statically or on the stack.
> 
> I agree with this. It also documents stuff so that you don't have
> to look it up from the SDM.
> 
> Neil: this should also clear your concerns.
> 
Agreed
Neil

> /Jarkko

Re: [PATCH v11 09/13] x86, sgx: basic routines for enclave page cache

[Neil Horman]

On Mon, Jun 25, 2018 at 12:21:22PM +0300, Jarkko Sakkinen wrote:
> On Wed, 2018-06-20 at 06:21 -0700, Sean Christopherson wrote:
> > On Fri, 2018-06-08 at 19:09 +0200, Jarkko Sakkinen wrote:
> > > SGX has a set of data structures to maintain information about the enclaves
> > > and their security properties. BIOS reserves a fixed size region of
> > > physical memory for these structures by setting Processor Reserved Memory
> > > Range Registers (PRMRR). This memory area is called Enclave Page Cache
> > > (EPC).
> > > 
> > > This commit implements the basic routines to allocate and free pages from
> > > different EPC banks. There is also a swapper thread ksgxswapd for EPC pages
> > > that gets woken up by sgx_alloc_page() when we run below the low watermark.
> > > The swapper thread continues swapping pages up until it reaches the high
> > > watermark.
> > > 
> > > Each subsystem that uses SGX must provide a set of callbacks for EPC
> > > pages that are used to reclaim, block and write an EPC page. Kernel
> > > takes the responsibility of maintaining LRU cache for them.
> > > 
> > > Signed-off-by: Jarkko Sakkinen <jarkko.sakkinen@linux.intel.com>
> > > ---
> > >  arch/x86/include/asm/sgx.h      |  67 +++++
> > >  arch/x86/include/asm/sgx_arch.h | 224 ++++++++++++++++
> > >  arch/x86/kernel/cpu/intel_sgx.c | 443 +++++++++++++++++++++++++++++++-
> > >  3 files changed, 732 insertions(+), 2 deletions(-)
> > >  create mode 100644 arch/x86/include/asm/sgx_arch.h
> > 
> > ...
> > 
> > > +struct sgx_pcmd {
> > > +	struct sgx_secinfo secinfo;
> > > +	uint64_t enclave_id;
> > > +	uint8_t reserved[40];
> > > +	uint8_t mac[16];
> > > +};
> > 
> > sgx_pcmd has a 128-byte alignment requirement.  I think it's
> > worth specifying here as sgx_pcmd is small enough that it could
> > be put on the stack, e.g. by KVM when trapping and executing
> > ELD* on behalf of a guest VM.
> > 
> > In fact, it probably makes sense to add alightment attributes
> > to all SGX structs for self-documentation purposes, even though
> > many of them will never be allocated statically or on the stack.
> 
> I agree with this. It also documents stuff so that you don't have
> to look it up from the SDM.
> 
> Neil: this should also clear your concerns.
> 
Agreed
Neil

> /Jarkko

Re: [intel-sgx-kernel-dev] [PATCH v11 13/13] intel_sgx: in-kernel launch enclave

[Nathaniel McCallum]

On Thu, Jun 21, 2018 at 5:21 PM Sean Christopherson
<sean.j.christopherson@intel.com> wrote:
>
> On Thu, Jun 21, 2018 at 03:11:18PM -0400, Nathaniel McCallum wrote:
> > If this is acceptable for everyone, my hope is the following:
> >
> > 1. Intel would split the existing code into one of the following
> > schemas (I don't care which):
> >   A. three parts: UEFI module, FLC-only kernel driver and user-space
> > launch enclave
> >   B. two parts: UEFI module (including launch enclave) and FLC-only
> > kernel driver
>
> To make sure I understand correctly...
>
> The UEFI module would lock the LE MSRs with a public key hardcoded
> into both the UEFI module and the kernel at build time?
>
> And for the kernel, it would only load its SGX driver if FLC is
> supported and the MSRs are locked to the expected key?
>
> IIUC, this approach will cause problems for virtualization.  Running
> VMs with different LE keys would require the bare metal firmware to
> configure the LE MSRs to be unlocked, which would effectively make
> using SGX in the host OS mutually exlusive with exposing SGX to KVM
> guests.  Theoretically it would be possible for KVM to emulate the
> guest's LE and use the host's LE to generate EINIT tokens, but
> emulating an enclave would likely require a massive amount of code
> and/or complexity.

How is this different from any other scenario where you lock the LE
MSRs? Unless Intel provides hardware support between the LE MSRs and
the VMX instructions, I don't see any way around this besides letting
any launch enclave run.

> > 2. Intel would release a reproducible build of the GPL UEFI module
> > sources signed with a SecureBoot trusted key and provide an
> > acceptable[0] binary redistribution license.
> >
> > 3. The kernel community would agree to merge the kernel driver given
> > the above criteria (and, obviously, acceptable kernel code).
> >
> > The question of how to distribute the UEFI module and possible launch
> > enclave remains open. I see two options: independent distribution and
> > bundling it in linux-firmware. The former may be a better
> > technological fit since the UEFI module will likely need to be run
> > before the kernel (and the boot loader; and shim). However, the latter
> > has the benefit of already being a well-known entity to our downstream
> > distributors. I could go either way on this.
>
> Writing and locks the LE MSRs effectively needs to be done before
> running the bootloader/kernel/etc...  Delaying activation would
> require, at a minimum, leaving IA32_FEATURE_CONTROL unlocked since
> IA32_FEATURE_CONTROL's SGX bits can't be set until SGX is activated.
>
> > I know this plan is more work for everyone involved, but I think it
> > manages to actually maximize both security and freedom.
> >
> > [0]: details here -
> > https://git.kernel.org/pub/scm/linux/kernel/git/firmware/linux-firmware.git/tree/README#n19
> > On Thu, Jun 21, 2018 at 11:29 AM Neil Horman <nhorman@redhat.com> wrote:
> > >
> > > On Thu, Jun 21, 2018 at 08:32:25AM -0400, Nathaniel McCallum wrote:
> > > > On Wed, Jun 20, 2018 at 5:02 PM Sean Christopherson
> > > > <sean.j.christopherson@intel.com> wrote:
> > > > >
> > > > > On Wed, Jun 20, 2018 at 11:39:00AM -0700, Jethro Beekman wrote:
> > > > > > On 2018-06-20 11:16, Jethro Beekman wrote:
> > > > > > > > This last bit is also repeated in different words in Table 35-2 and
> > > > > > > > Section 42.2.2. The MSRs are *not writable* before the write-lock bit
> > > > > > > > itself is locked. Meaning the MSRs are either locked with Intel's key
> > > > > > > > hash, or not locked at all.
> > > > > >
> > > > > > Actually, this might be a documentation bug. I have some test hardware and I
> > > > > > was able to configure the MSRs in the BIOS and then read the MSRs after boot
> > > > > > like this:
> > > > > >
> > > > > > MSR 0x3a 0x0000000000040005
> > > > > > MSR 0x8c 0x20180620aaaaaaaa
> > > > > > MSR 0x8d 0x20180620bbbbbbbb
> > > > > > MSR 0x8e 0x20180620cccccccc
> > > > > > MSR 0x8f 0x20180620dddddddd
> > > > > >
> > > > > > Since this is not production hardware, it could also be a CPU bug of course.
> > > > > >
> > > > > > If it is indeed possible to configure AND lock the MSR values to non-Intel
> > > > > > values, I'm very much in favor of Nathaniels proposal to treat the launch
> > > > > > enclave like any other firmware blob.
> > > > >
> > > > > It's not a CPU or documentation bug (though the latter is arguable).
> > > > > SGX has an activation step that is triggered by doing a WRMSR(0x7a)
> > > > > with bit 0 set.  Until SGX is activated, the SGX related bits in
> > > > > IA32_FEATURE_CONTROL cannot be set, i.e. SGX can't be enabled.  But,
> > > > > the LE hash MSRs are fully writable prior to activation, e.g. to
> > > > > allow firmware to lock down the LE key with a non-Intel value.
> > > > >
> > > > > So yes, it's possible to lock the MSRs to a non-Intel value.  The
> > > > > obvious caveat is that whatever blob is used to write the MSRs would
> > > > > need be executed prior to activation.
> > > >
> > > > This implies that it should be possible to create MSR activation (and
> > > > an embedded launch enclave?) entirely as a UEFI module. The kernel
> > > > would still get to manage who has access to /dev/sgx and other
> > > > important non-cryptographic policy details. Users would still be able
> > > > to control the cryptographic policy details (via BIOS Secure Boot
> > > > configuration that exists today). Distributions could still control
> > > > cryptographic policy details via signing of the UEFI module with their
> > > > own Secure Boot key (or using something like shim). The UEFI module
> > > > (and possibly the external launch enclave) could be distributed via
> > > > linux-firmware.
> > > >
> > > > Andy/Neil, does this work for you?
> > > >
> > > I need some time to digest it.  Who in your mind is writing the UEFI module.  Is
> > > that the firmware vendor or IHV?
> > >
> > > Neil
> > >
> > > > > As for the SDM, it's a documentation... omission?  SGX activation
> > > > > is intentionally omitted from the SDM.  The intended usage model is
> > > > > that firmware will always do the activation (if it wants SGX enabled),
> > > > > i.e. post-firmware software will only ever "see" SGX as disabled or
> > > > > in the fully activated state, and so the SDM doesn't describe SGX
> > > > > behavior prior to activation.  I believe the activation process, or
> > > > > at least what is required from firmware, is documented in the BIOS
> > > > > writer's guide.
> > > > >
> > > > > > Jethro Beekman | Fortanix
> > > > > >
> > > > >
> > > > >

Re: [intel-sgx-kernel-dev] [PATCH v11 13/13] intel_sgx: in-kernel launch enclave

[Nathaniel McCallum]

On Thu, Jun 21, 2018 at 6:49 PM Andy Lutomirski <luto@kernel.org> wrote:
>
> On Thu, Jun 21, 2018 at 12:11 PM Nathaniel McCallum
> <npmccallum@redhat.com> wrote:
> >
> > If this is acceptable for everyone, my hope is the following:
> >
> > 1. Intel would split the existing code into one of the following
> > schemas (I don't care which):
> >   A. three parts: UEFI module, FLC-only kernel driver and user-space
> > launch enclave
> >   B. two parts: UEFI module (including launch enclave) and FLC-only
> > kernel driver
> >
> > 2. Intel would release a reproducible build of the GPL UEFI module
> > sources signed with a SecureBoot trusted key and provide an
> > acceptable[0] binary redistribution license.
> >
> > 3. The kernel community would agree to merge the kernel driver given
> > the above criteria (and, obviously, acceptable kernel code).
> >
> > The question of how to distribute the UEFI module and possible launch
> > enclave remains open. I see two options: independent distribution and
> > bundling it in linux-firmware. The former may be a better
> > technological fit since the UEFI module will likely need to be run
> > before the kernel (and the boot loader; and shim). However, the latter
> > has the benefit of already being a well-known entity to our downstream
> > distributors. I could go either way on this.
>
> This is a lot of complication and effort for a gain that is not
> entirely clear.

Root kits and evil maid attacks are two worth considering.

> I really really really do *not* want to see Intel or
> anyone else start enforcing policy on which programs can and cannot
> run using this mechanism.

We already do this. It is called SecureBoot.

> (This is exactly why non-FLC systems aren't
> about to be supported upstream.)  So my preference is to not merge
> anything that supports this type of use case unless there is
> compelling evidence that it is (a) genuinely useful, (b) will be used
> to improve security and (c) won't be abused for, say, revenue
> purposes.

I think there are benefits for (a) and (b). I agree with you about
(c). But, again, we already have SecureBoot.

Re: [intel-sgx-kernel-dev] [PATCH v11 13/13] intel_sgx: in-kernel launch enclave

[Nathaniel McCallum]

On Mon, Jun 25, 2018 at 5:28 AM Jarkko Sakkinen
<jarkko.sakkinen@linux.intel.com> wrote:
>
> On Wed, 2018-06-20 at 12:28 -0400, Nathaniel McCallum wrote:
> > As I understand it, the current policy models under discussion look like this:
> >
> > 1. SGX w/o FLC (not being merged) looks like this:
> >   Intel CPU => (Intel signed) launch enclave => enclaves
> >
> > 2. SGX w/ FLC, looks like this:
> >   Intel CPU => kernel => launch enclave => enclaves
> >
> > 3. Andy is proposing this:
> >   Intel CPU => kernel => enclaves
>
> What if MSRs are not writable after hand over to the OS? It is a legit
> configuration at least according to the SDM.

It seems to me that "set the MSRs in the BIOS" and "set the MSRs in a
UEFI module" are functionally equivalent.

Re: [intel-sgx-kernel-dev] [PATCH v11 13/13] intel_sgx: in-kernel launch enclave

[Nathaniel McCallum]

On Mon, Jun 25, 2018 at 11:45 AM Andy Lutomirski <luto@kernel.org> wrote:
>
> On Mon, Jun 25, 2018 at 2:41 AM Jarkko Sakkinen
> <jarkko.sakkinen@linux.intel.com> wrote:
> >
> > On Thu, 2018-06-21 at 08:32 -0400, Nathaniel McCallum wrote:
> > > This implies that it should be possible to create MSR activation (and
> > > an embedded launch enclave?) entirely as a UEFI module. The kernel
> > > would still get to manage who has access to /dev/sgx and other
> > > important non-cryptographic policy details. Users would still be able
> > > to control the cryptographic policy details (via BIOS Secure Boot
> > > configuration that exists today). Distributions could still control
> > > cryptographic policy details via signing of the UEFI module with their
> > > own Secure Boot key (or using something like shim). The UEFI module
> > > (and possibly the external launch enclave) could be distributed via
> > > linux-firmware.
> > >
> > > Andy/Neil, does this work for you?
> >
> > Nothing against having UEFI module for MSR activation step.
> >
> > And we would move the existing in-kernel LE to firmware so that it is
> > avaible for locked-in-to-non-Intel-values case?
> >
>
> This is a hell of a lot of complexity.  To get it right we'd need an
> actual formal spec of what firmware is supposed to do and how it
> integrates with the kernel, and we'd need a reason why it's useful.

What do you want the kernel's behavior to be in the case where an FLC
CPU is present, but the MSRs have been locked by the BIOS? Because I
think the workflow for the UEFI module idea is the same.

> I'm personally rather strongly in favor of the vastly simpler model in
> which we first merge SGX without LE support at all.  Instead we use
> the approach where we just twiddle the MSRs to launch normal enclaves
> without an init token at all, which is probably considerably faster
> and will remove several thousand lines of code.  If and when a bona
> fide use case for LE support shows up, we can work out the details and
> merge it.

I'm also okay with an incremental approach, BTW. I just want us to
think through the issues. And I do think that SGX root kits are a
serious threat. But I'm willing to move in stages. In fact, if we can
merge something without any LE support faster, I'm in favor of doing
so.

> Right now, we're talking about a lot of design considerations, a lot
> of interoperability considerations, and a lot of code to support a use
> case that doesn't clearly exist.

I agree.

Re: [intel-sgx-kernel-dev] [PATCH v11 13/13] intel_sgx: in-kernel launch enclave

[Sean Christopherson]

On Mon, Jun 25, 2018 at 05:00:05PM -0400, Nathaniel McCallum wrote:
> On Thu, Jun 21, 2018 at 5:21 PM Sean Christopherson
> <sean.j.christopherson@intel.com> wrote:
> >
> > On Thu, Jun 21, 2018 at 03:11:18PM -0400, Nathaniel McCallum wrote:
> > > If this is acceptable for everyone, my hope is the following:
> > >
> > > 1. Intel would split the existing code into one of the following
> > > schemas (I don't care which):
> > >   A. three parts: UEFI module, FLC-only kernel driver and user-space
> > > launch enclave
> > >   B. two parts: UEFI module (including launch enclave) and FLC-only
> > > kernel driver
> >
> > To make sure I understand correctly...
> >
> > The UEFI module would lock the LE MSRs with a public key hardcoded
> > into both the UEFI module and the kernel at build time?
> >
> > And for the kernel, it would only load its SGX driver if FLC is
> > supported and the MSRs are locked to the expected key?
> >
> > IIUC, this approach will cause problems for virtualization.  Running
> > VMs with different LE keys would require the bare metal firmware to
> > configure the LE MSRs to be unlocked, which would effectively make
> > using SGX in the host OS mutually exlusive with exposing SGX to KVM
> > guests.  Theoretically it would be possible for KVM to emulate the
> > guest's LE and use the host's LE to generate EINIT tokens, but
> > emulating an enclave would likely require a massive amount of code
> > and/or complexity.
> 
> How is this different from any other scenario where you lock the LE
> MSRs? Unless Intel provides hardware support between the LE MSRs and
> the VMX instructions, I don't see any way around this besides letting
> any launch enclave run.

It's not.  All prior discussions have effectively required unlocked
MSRs, so that's my baseline.

> > > 2. Intel would release a reproducible build of the GPL UEFI module
> > > sources signed with a SecureBoot trusted key and provide an
> > > acceptable[0] binary redistribution license.
> > >
> > > 3. The kernel community would agree to merge the kernel driver given
> > > the above criteria (and, obviously, acceptable kernel code).
> > >
> > > The question of how to distribute the UEFI module and possible launch
> > > enclave remains open. I see two options: independent distribution and
> > > bundling it in linux-firmware. The former may be a better
> > > technological fit since the UEFI module will likely need to be run
> > > before the kernel (and the boot loader; and shim). However, the latter
> > > has the benefit of already being a well-known entity to our downstream
> > > distributors. I could go either way on this.
> >
> > Writing and locks the LE MSRs effectively needs to be done before
> > running the bootloader/kernel/etc...  Delaying activation would
> > require, at a minimum, leaving IA32_FEATURE_CONTROL unlocked since
> > IA32_FEATURE_CONTROL's SGX bits can't be set until SGX is activated.
> >
> > > I know this plan is more work for everyone involved, but I think it
> > > manages to actually maximize both security and freedom.
> > >
> > > [0]: details here -
> > > https://git.kernel.org/pub/scm/linux/kernel/git/firmware/linux-firmware.git/tree/README#n19
> > > On Thu, Jun 21, 2018 at 11:29 AM Neil Horman <nhorman@redhat.com> wrote:
> > > >
> > > > On Thu, Jun 21, 2018 at 08:32:25AM -0400, Nathaniel McCallum wrote:
> > > > > On Wed, Jun 20, 2018 at 5:02 PM Sean Christopherson
> > > > > <sean.j.christopherson@intel.com> wrote:
> > > > > >
> > > > > > On Wed, Jun 20, 2018 at 11:39:00AM -0700, Jethro Beekman wrote:
> > > > > > > On 2018-06-20 11:16, Jethro Beekman wrote:
> > > > > > > > > This last bit is also repeated in different words in Table 35-2 and
> > > > > > > > > Section 42.2.2. The MSRs are *not writable* before the write-lock bit
> > > > > > > > > itself is locked. Meaning the MSRs are either locked with Intel's key
> > > > > > > > > hash, or not locked at all.
> > > > > > >
> > > > > > > Actually, this might be a documentation bug. I have some test hardware and I
> > > > > > > was able to configure the MSRs in the BIOS and then read the MSRs after boot
> > > > > > > like this:
> > > > > > >
> > > > > > > MSR 0x3a 0x0000000000040005
> > > > > > > MSR 0x8c 0x20180620aaaaaaaa
> > > > > > > MSR 0x8d 0x20180620bbbbbbbb
> > > > > > > MSR 0x8e 0x20180620cccccccc
> > > > > > > MSR 0x8f 0x20180620dddddddd
> > > > > > >
> > > > > > > Since this is not production hardware, it could also be a CPU bug of course.
> > > > > > >
> > > > > > > If it is indeed possible to configure AND lock the MSR values to non-Intel
> > > > > > > values, I'm very much in favor of Nathaniels proposal to treat the launch
> > > > > > > enclave like any other firmware blob.
> > > > > >
> > > > > > It's not a CPU or documentation bug (though the latter is arguable).
> > > > > > SGX has an activation step that is triggered by doing a WRMSR(0x7a)
> > > > > > with bit 0 set.  Until SGX is activated, the SGX related bits in
> > > > > > IA32_FEATURE_CONTROL cannot be set, i.e. SGX can't be enabled.  But,
> > > > > > the LE hash MSRs are fully writable prior to activation, e.g. to
> > > > > > allow firmware to lock down the LE key with a non-Intel value.
> > > > > >
> > > > > > So yes, it's possible to lock the MSRs to a non-Intel value.  The
> > > > > > obvious caveat is that whatever blob is used to write the MSRs would
> > > > > > need be executed prior to activation.
> > > > >
> > > > > This implies that it should be possible to create MSR activation (and
> > > > > an embedded launch enclave?) entirely as a UEFI module. The kernel
> > > > > would still get to manage who has access to /dev/sgx and other
> > > > > important non-cryptographic policy details. Users would still be able
> > > > > to control the cryptographic policy details (via BIOS Secure Boot
> > > > > configuration that exists today). Distributions could still control
> > > > > cryptographic policy details via signing of the UEFI module with their
> > > > > own Secure Boot key (or using something like shim). The UEFI module
> > > > > (and possibly the external launch enclave) could be distributed via
> > > > > linux-firmware.
> > > > >
> > > > > Andy/Neil, does this work for you?
> > > > >
> > > > I need some time to digest it.  Who in your mind is writing the UEFI module.  Is
> > > > that the firmware vendor or IHV?
> > > >
> > > > Neil
> > > >
> > > > > > As for the SDM, it's a documentation... omission?  SGX activation
> > > > > > is intentionally omitted from the SDM.  The intended usage model is
> > > > > > that firmware will always do the activation (if it wants SGX enabled),
> > > > > > i.e. post-firmware software will only ever "see" SGX as disabled or
> > > > > > in the fully activated state, and so the SDM doesn't describe SGX
> > > > > > behavior prior to activation.  I believe the activation process, or
> > > > > > at least what is required from firmware, is documented in the BIOS
> > > > > > writer's guide.
> > > > > >
> > > > > > > Jethro Beekman | Fortanix
> > > > > > >
> > > > > >
> > > > > >

Re: [intel-sgx-kernel-dev] [PATCH v11 13/13] intel_sgx: in-kernel launch enclave

[Andy Lutomirski]

On Mon, Jun 25, 2018 at 2:06 PM Nathaniel McCallum
<npmccallum@redhat.com> wrote:
>
> On Thu, Jun 21, 2018 at 6:49 PM Andy Lutomirski <luto@kernel.org> wrote:
> >
> > On Thu, Jun 21, 2018 at 12:11 PM Nathaniel McCallum
> > <npmccallum@redhat.com> wrote:
> > >
> > > If this is acceptable for everyone, my hope is the following:
> > >
> > > 1. Intel would split the existing code into one of the following
> > > schemas (I don't care which):
> > >   A. three parts: UEFI module, FLC-only kernel driver and user-space
> > > launch enclave
> > >   B. two parts: UEFI module (including launch enclave) and FLC-only
> > > kernel driver
> > >
> > > 2. Intel would release a reproducible build of the GPL UEFI module
> > > sources signed with a SecureBoot trusted key and provide an
> > > acceptable[0] binary redistribution license.
> > >
> > > 3. The kernel community would agree to merge the kernel driver given
> > > the above criteria (and, obviously, acceptable kernel code).
> > >
> > > The question of how to distribute the UEFI module and possible launch
> > > enclave remains open. I see two options: independent distribution and
> > > bundling it in linux-firmware. The former may be a better
> > > technological fit since the UEFI module will likely need to be run
> > > before the kernel (and the boot loader; and shim). However, the latter
> > > has the benefit of already being a well-known entity to our downstream
> > > distributors. I could go either way on this.
> >
> > This is a lot of complication and effort for a gain that is not
> > entirely clear.
>
> Root kits and evil maid attacks are two worth considering.
>

I think that SGX malware is a real issue.  I'm less convinced that SGX
root kits and SGX evil maid attacks are particularly interesting,
except insofar as SGX can be used to make a root kit's behavior harder
to reverse engineer.  Can you explain exactly what type of attack you
have in mind and exactly how all this complexity helps?

(Keep in mind that SGX, by itself, cannot actually obfuscate malware.
SGX plus a command-and-control system that uses remote attestation
*can* obfuscate malware, but Intel has tight [0], online controls to
protect against *that*, and they have nothing to do with launch
control [1].)

> > I really really really do *not* want to see Intel or
> > anyone else start enforcing policy on which programs can and cannot
> > run using this mechanism.
>
> We already do this. It is called SecureBoot.

And we have a mechanism for letting people run whatever OS they want
on a SecureBoot system, and if you can get your favorite Linux to boot
on a Secure Boot machine, it's fully functional.  SGX, not so much.

>
> > (This is exactly why non-FLC systems aren't
> > about to be supported upstream.)  So my preference is to not merge
> > anything that supports this type of use case unless there is
> > compelling evidence that it is (a) genuinely useful, (b) will be used
> > to improve security and (c) won't be abused for, say, revenue
> > purposes.
>
> I think there are benefits for (a) and (b). I agree with you about
> (c). But, again, we already have SecureBoot.

And Secure Boot is great (aside from being overcomplicated, using SMM
in ridiculous ways, and having some misguided OEMs providing buggy
implementations).  And Secure Boot, applied correctly, is decent
protection against root kits independently of SGX.

[0] Well, maybe they're tight.  I don't know whether Intel pays
adequate attention.  Also, IIRC, you need an NDA to even learn the
rules about Intel's attestation service.

[1] I'd need to reread the SDM, but it's possible that a buggy
signed-by-Intel launch enclave would break attestation.  But a
not-signed-by-Intel enclave can't have any particular effect on
attestation, because the *attestation* root of trust involves Intel
knowing the provisioning keys of all the genuine SGX CPUs in the
world, and Intel is the only party with that information, so a
third-party provisioning enclave signed by a third party can't
actually root its trust anywhere.  This situation is somewhat
analogous to how TPM-based DRM used to be impossible but is not
sort-of-possible even though TPMs have never had any equivalent of
launch control.

Re: [intel-sgx-kernel-dev] [PATCH v11 13/13] intel_sgx: in-kernel launch enclave

[Jarkko Sakkinen]

On Mon, 2018-06-25 at 08:45 -0700, Andy Lutomirski wrote:
> I'm personally rather strongly in favor of the vastly simpler model in
> which we first merge SGX without LE support at all.  Instead we use
> the approach where we just twiddle the MSRs to launch normal enclaves
> without an init token at all, which is probably considerably faster
> and will remove several thousand lines of code.  If and when a bona
> fide use case for LE support shows up, we can work out the details and
> merge it.

Andy, I was going to propose exactly the same :-)

We can upstream SGX that supports only unlocked MSRs and that does
not preventing to upstream support for locked MSRs later. Even if
we had a consensus for locked MSRs, making two milestones for the
mainline would make perfect sense.

I came into this conclusion last night because all the other review
comments not concerning the launch control are easily sorted out.

/Jarkko

Re: [intel-sgx-kernel-dev] [PATCH v11 13/13] intel_sgx: in-kernel launch enclave

[Nathaniel McCallum]

On Tue, Jun 26, 2018 at 4:44 AM Jarkko Sakkinen
<jarkko.sakkinen@linux.intel.com> wrote:
>
> On Mon, 2018-06-25 at 08:45 -0700, Andy Lutomirski wrote:
> > I'm personally rather strongly in favor of the vastly simpler model in
> > which we first merge SGX without LE support at all.  Instead we use
> > the approach where we just twiddle the MSRs to launch normal enclaves
> > without an init token at all, which is probably considerably faster
> > and will remove several thousand lines of code.  If and when a bona
> > fide use case for LE support shows up, we can work out the details and
> > merge it.
>
> Andy, I was going to propose exactly the same :-)
>
> We can upstream SGX that supports only unlocked MSRs and that does
> not preventing to upstream support for locked MSRs later. Even if
> we had a consensus for locked MSRs, making two milestones for the
> mainline would make perfect sense.
>
> I came into this conclusion last night because all the other review
> comments not concerning the launch control are easily sorted out.

+1. Let's do this and get it merged without launch enclave support
lockdown now. We can revisit this once we have hands on experience
with the technology.

Re: [intel-sgx-kernel-dev] [PATCH v11 13/13] intel_sgx: in-kernel launch enclave

[Jarkko Sakkinen]

On Tue, 2018-06-26 at 11:01 -0400, Nathaniel McCallum wrote:
> On Tue, Jun 26, 2018 at 4:44 AM Jarkko Sakkinen
> <jarkko.sakkinen@linux.intel.com> wrote:
> > 
> > On Mon, 2018-06-25 at 08:45 -0700, Andy Lutomirski wrote:
> > > I'm personally rather strongly in favor of the vastly simpler model in
> > > which we first merge SGX without LE support at all.  Instead we use
> > > the approach where we just twiddle the MSRs to launch normal enclaves
> > > without an init token at all, which is probably considerably faster
> > > and will remove several thousand lines of code.  If and when a bona
> > > fide use case for LE support shows up, we can work out the details and
> > > merge it.
> > 
> > Andy, I was going to propose exactly the same :-)
> > 
> > We can upstream SGX that supports only unlocked MSRs and that does
> > not preventing to upstream support for locked MSRs later. Even if
> > we had a consensus for locked MSRs, making two milestones for the
> > mainline would make perfect sense.
> > 
> > I came into this conclusion last night because all the other review
> > comments not concerning the launch control are easily sorted out.
> 
> +1. Let's do this and get it merged without launch enclave support
> lockdown now. We can revisit this once we have hands on experience
> with the technology.

I'm proceeding with this ATM.

I'm going to send v12 next week.

I'll do my best to address all of the review comments but expect to miss some
of them.

/Jarkko

Re: [PATCH v11 00/13] Intel SGX1 support

[Pavel Machek]

[-- Attachment #1: Type: text/plain, Size: 2362 bytes --]

Hi!

(sorry to bring up old thread).

> > > > > Intel(R) SGX is a set of CPU instructions that can be used by applications
> > > > > to set aside private regions of code and data. The code outside the enclave
> > > > > is disallowed to access the memory inside the enclave by the CPU access
> > > > > control.  In a way you can think that SGX provides inverted sandbox. It
> > > > > protects the application from a malicious host.
> > > > 
> > > > Do you intend to allow non-root applications to use SGX?
> > > > 
> > > > What are non-evil uses for SGX?
> > > > 
> > > > ...because it is quite useful for some kinds of evil:
> > > 
> > > The default permissions for the device are 600.
> > 
> > Good. This does not belong to non-root.
> 
> There are entirely legitimate use cases for using this as an
> unprivileged user. However, that'll be up to system and distribution
> policy, which can evolve over time, and it makes sense for the *initial*
> kernel permission to start out root-only and then adjust permissions via
> udev.

Agreed.

> > What are some non-evil uses for SGX?
> 
> Building a software certificate store. Hardening key-agent software like
> ssh-agent or gpg-agent. Building a challenge-response authentication
> system. Providing more assurance that your server infrastructure is
> uncompromised. Offloading computation to a system without having to
> fully trust that system.

I think you can do the crypto stuff... as crypto already verifies the
results. But I don't think you can do the computation offload.

> As one of many possibilities, imagine a distcc that didn't have to trust
> the compile nodes. The compile nodes could fail to return results at
> all, but they couldn't alter the results.

distcc on untrusted nodes ... oh yes, that would be great.

Except that you can't do it, right? :-).

First, AFAICT it would be quite hard to get gcc to run under SGX. But
maybe you have spare month or three and can do it.

But ... you really can't guarantee getting right results. Evil owner
of the machine might intentionaly overheat the CPU, glitch the power,
induce single-bit errors using gamma source, ... You can't do it.

									Pavel
-- 
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 181 bytes --]

Re: [PATCH v11 00/13] Intel SGX1 support

[Pavel Machek]

[-- Attachment #1: Type: text/plain, Size: 2362 bytes --]

Hi!

(sorry to bring up old thread).

> > > > > Intel(R) SGX is a set of CPU instructions that can be used by applications
> > > > > to set aside private regions of code and data. The code outside the enclave
> > > > > is disallowed to access the memory inside the enclave by the CPU access
> > > > > control.  In a way you can think that SGX provides inverted sandbox. It
> > > > > protects the application from a malicious host.
> > > > 
> > > > Do you intend to allow non-root applications to use SGX?
> > > > 
> > > > What are non-evil uses for SGX?
> > > > 
> > > > ...because it is quite useful for some kinds of evil:
> > > 
> > > The default permissions for the device are 600.
> > 
> > Good. This does not belong to non-root.
> 
> There are entirely legitimate use cases for using this as an
> unprivileged user. However, that'll be up to system and distribution
> policy, which can evolve over time, and it makes sense for the *initial*
> kernel permission to start out root-only and then adjust permissions via
> udev.

Agreed.

> > What are some non-evil uses for SGX?
> 
> Building a software certificate store. Hardening key-agent software like
> ssh-agent or gpg-agent. Building a challenge-response authentication
> system. Providing more assurance that your server infrastructure is
> uncompromised. Offloading computation to a system without having to
> fully trust that system.

I think you can do the crypto stuff... as crypto already verifies the
results. But I don't think you can do the computation offload.

> As one of many possibilities, imagine a distcc that didn't have to trust
> the compile nodes. The compile nodes could fail to return results at
> all, but they couldn't alter the results.

distcc on untrusted nodes ... oh yes, that would be great.

Except that you can't do it, right? :-).

First, AFAICT it would be quite hard to get gcc to run under SGX. But
maybe you have spare month or three and can do it.

But ... you really can't guarantee getting right results. Evil owner
of the machine might intentionaly overheat the CPU, glitch the power,
induce single-bit errors using gamma source, ... You can't do it.

									Pavel
-- 
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 181 bytes --]

Re: [PATCH v11 00/13] Intel SGX1 support

[Pavel Machek]

[-- Attachment #1: Type: text/plain, Size: 2362 bytes --]

Hi!

(sorry to bring up old thread).

> > > > > Intel(R) SGX is a set of CPU instructions that can be used by applications
> > > > > to set aside private regions of code and data. The code outside the enclave
> > > > > is disallowed to access the memory inside the enclave by the CPU access
> > > > > control.  In a way you can think that SGX provides inverted sandbox. It
> > > > > protects the application from a malicious host.
> > > > 
> > > > Do you intend to allow non-root applications to use SGX?
> > > > 
> > > > What are non-evil uses for SGX?
> > > > 
> > > > ...because it is quite useful for some kinds of evil:
> > > 
> > > The default permissions for the device are 600.
> > 
> > Good. This does not belong to non-root.
> 
> There are entirely legitimate use cases for using this as an
> unprivileged user. However, that'll be up to system and distribution
> policy, which can evolve over time, and it makes sense for the *initial*
> kernel permission to start out root-only and then adjust permissions via
> udev.

Agreed.

> > What are some non-evil uses for SGX?
> 
> Building a software certificate store. Hardening key-agent software like
> ssh-agent or gpg-agent. Building a challenge-response authentication
> system. Providing more assurance that your server infrastructure is
> uncompromised. Offloading computation to a system without having to
> fully trust that system.

I think you can do the crypto stuff... as crypto already verifies the
results. But I don't think you can do the computation offload.

> As one of many possibilities, imagine a distcc that didn't have to trust
> the compile nodes. The compile nodes could fail to return results at
> all, but they couldn't alter the results.

distcc on untrusted nodes ... oh yes, that would be great.

Except that you can't do it, right? :-).

First, AFAICT it would be quite hard to get gcc to run under SGX. But
maybe you have spare month or three and can do it.

But ... you really can't guarantee getting right results. Evil owner
of the machine might intentionaly overheat the CPU, glitch the power,
induce single-bit errors using gamma source, ... You can't do it.

									Pavel
-- 
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 181 bytes --]

Re: [PATCH v11 00/13] Intel SGX1 support

[Pavel Machek]

[-- Attachment #1: Type: text/plain, Size: 2362 bytes --]

Hi!

(sorry to bring up old thread).

> > > > > Intel(R) SGX is a set of CPU instructions that can be used by applications
> > > > > to set aside private regions of code and data. The code outside the enclave
> > > > > is disallowed to access the memory inside the enclave by the CPU access
> > > > > control.  In a way you can think that SGX provides inverted sandbox. It
> > > > > protects the application from a malicious host.
> > > > 
> > > > Do you intend to allow non-root applications to use SGX?
> > > > 
> > > > What are non-evil uses for SGX?
> > > > 
> > > > ...because it is quite useful for some kinds of evil:
> > > 
> > > The default permissions for the device are 600.
> > 
> > Good. This does not belong to non-root.
> 
> There are entirely legitimate use cases for using this as an
> unprivileged user. However, that'll be up to system and distribution
> policy, which can evolve over time, and it makes sense for the *initial*
> kernel permission to start out root-only and then adjust permissions via
> udev.

Agreed.

> > What are some non-evil uses for SGX?
> 
> Building a software certificate store. Hardening key-agent software like
> ssh-agent or gpg-agent. Building a challenge-response authentication
> system. Providing more assurance that your server infrastructure is
> uncompromised. Offloading computation to a system without having to
> fully trust that system.

I think you can do the crypto stuff... as crypto already verifies the
results. But I don't think you can do the computation offload.

> As one of many possibilities, imagine a distcc that didn't have to trust
> the compile nodes. The compile nodes could fail to return results at
> all, but they couldn't alter the results.

distcc on untrusted nodes ... oh yes, that would be great.

Except that you can't do it, right? :-).

First, AFAICT it would be quite hard to get gcc to run under SGX. But
maybe you have spare month or three and can do it.

But ... you really can't guarantee getting right results. Evil owner
of the machine might intentionaly overheat the CPU, glitch the power,
induce single-bit errors using gamma source, ... You can't do it.

									Pavel
-- 
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 181 bytes --]

Re: [PATCH v11 00/13] Intel SGX1 support

[Josh Triplett]

On Sun, Dec 09, 2018 at 09:06:00PM +0100, Pavel Machek wrote:
...
> > > > The default permissions for the device are 600.
> > > 
> > > Good. This does not belong to non-root.
> > 
> > There are entirely legitimate use cases for using this as an
> > unprivileged user. However, that'll be up to system and distribution
> > policy, which can evolve over time, and it makes sense for the *initial*
> > kernel permission to start out root-only and then adjust permissions via
> > udev.
> 
> Agreed.
> 
> > Building a software certificate store. Hardening key-agent software like
> > ssh-agent or gpg-agent. Building a challenge-response authentication
> > system. Providing more assurance that your server infrastructure is
> > uncompromised. Offloading computation to a system without having to
> > fully trust that system.
> 
> I think you can do the crypto stuff... as crypto already verifies the
> results. But I don't think you can do the computation offload.

You can, as long as you can do attestation.

> > As one of many possibilities, imagine a distcc that didn't have to trust
> > the compile nodes. The compile nodes could fail to return results at
> > all, but they couldn't alter the results.
> 
> distcc on untrusted nodes ... oh yes, that would be great.
> 
> Except that you can't do it, right? :-).
> 
> First, AFAICT it would be quite hard to get gcc to run under SGX. But
> maybe you have spare month or three and can do it.

Assuming you don't need to #include files, gcc seems quite simple to run
in an enclave: data in, computation inside, data out.

Re: [PATCH v11 00/13] Intel SGX1 support

[Josh Triplett]

On Sun, Dec 09, 2018 at 09:06:00PM +0100, Pavel Machek wrote:
...
> > > > The default permissions for the device are 600.
> > > 
> > > Good. This does not belong to non-root.
> > 
> > There are entirely legitimate use cases for using this as an
> > unprivileged user. However, that'll be up to system and distribution
> > policy, which can evolve over time, and it makes sense for the *initial*
> > kernel permission to start out root-only and then adjust permissions via
> > udev.
> 
> Agreed.
> 
> > Building a software certificate store. Hardening key-agent software like
> > ssh-agent or gpg-agent. Building a challenge-response authentication
> > system. Providing more assurance that your server infrastructure is
> > uncompromised. Offloading computation to a system without having to
> > fully trust that system.
> 
> I think you can do the crypto stuff... as crypto already verifies the
> results. But I don't think you can do the computation offload.

You can, as long as you can do attestation.

> > As one of many possibilities, imagine a distcc that didn't have to trust
> > the compile nodes. The compile nodes could fail to return results at
> > all, but they couldn't alter the results.
> 
> distcc on untrusted nodes ... oh yes, that would be great.
> 
> Except that you can't do it, right? :-).
> 
> First, AFAICT it would be quite hard to get gcc to run under SGX. But
> maybe you have spare month or three and can do it.

Assuming you don't need to #include files, gcc seems quite simple to run
in an enclave: data in, computation inside, data out.

Re: [PATCH v11 00/13] Intel SGX1 support

[Josh Triplett]

On Sun, Dec 09, 2018 at 09:06:00PM +0100, Pavel Machek wrote:
...
> > > > The default permissions for the device are 600.
> > > 
> > > Good. This does not belong to non-root.
> > 
> > There are entirely legitimate use cases for using this as an
> > unprivileged user. However, that'll be up to system and distribution
> > policy, which can evolve over time, and it makes sense for the *initial*
> > kernel permission to start out root-only and then adjust permissions via
> > udev.
> 
> Agreed.
> 
> > Building a software certificate store. Hardening key-agent software like
> > ssh-agent or gpg-agent. Building a challenge-response authentication
> > system. Providing more assurance that your server infrastructure is
> > uncompromised. Offloading computation to a system without having to
> > fully trust that system.
> 
> I think you can do the crypto stuff... as crypto already verifies the
> results. But I don't think you can do the computation offload.

You can, as long as you can do attestation.

> > As one of many possibilities, imagine a distcc that didn't have to trust
> > the compile nodes. The compile nodes could fail to return results at
> > all, but they couldn't alter the results.
> 
> distcc on untrusted nodes ... oh yes, that would be great.
> 
> Except that you can't do it, right? :-).
> 
> First, AFAICT it would be quite hard to get gcc to run under SGX. But
> maybe you have spare month or three and can do it.

Assuming you don't need to #include files, gcc seems quite simple to run
in an enclave: data in, computation inside, data out.

Re: [PATCH v11 00/13] Intel SGX1 support

[Josh Triplett]

On Sun, Dec 09, 2018 at 09:06:00PM +0100, Pavel Machek wrote:
...
> > > > The default permissions for the device are 600.
> > > 
> > > Good. This does not belong to non-root.
> > 
> > There are entirely legitimate use cases for using this as an
> > unprivileged user. However, that'll be up to system and distribution
> > policy, which can evolve over time, and it makes sense for the *initial*
> > kernel permission to start out root-only and then adjust permissions via
> > udev.
> 
> Agreed.
> 
> > Building a software certificate store. Hardening key-agent software like
> > ssh-agent or gpg-agent. Building a challenge-response authentication
> > system. Providing more assurance that your server infrastructure is
> > uncompromised. Offloading computation to a system without having to
> > fully trust that system.
> 
> I think you can do the crypto stuff... as crypto already verifies the
> results. But I don't think you can do the computation offload.

You can, as long as you can do attestation.

> > As one of many possibilities, imagine a distcc that didn't have to trust
> > the compile nodes. The compile nodes could fail to return results at
> > all, but they couldn't alter the results.
> 
> distcc on untrusted nodes ... oh yes, that would be great.
> 
> Except that you can't do it, right? :-).
> 
> First, AFAICT it would be quite hard to get gcc to run under SGX. But
> maybe you have spare month or three and can do it.

Assuming you don't need to #include files, gcc seems quite simple to run
in an enclave: data in, computation inside, data out.

Re: [PATCH v11 00/13] Intel SGX1 support

[Pavel Machek]

[-- Attachment #1: Type: text/plain, Size: 2220 bytes --]

On Sun 2018-12-09 23:47:17, Josh Triplett wrote:
> On Sun, Dec 09, 2018 at 09:06:00PM +0100, Pavel Machek wrote:
> ...
> > > > > The default permissions for the device are 600.
> > > > 
> > > > Good. This does not belong to non-root.
> > > 
> > > There are entirely legitimate use cases for using this as an
> > > unprivileged user. However, that'll be up to system and distribution
> > > policy, which can evolve over time, and it makes sense for the *initial*
> > > kernel permission to start out root-only and then adjust permissions via
> > > udev.
> > 
> > Agreed.
> > 
> > > Building a software certificate store. Hardening key-agent software like
> > > ssh-agent or gpg-agent. Building a challenge-response authentication
> > > system. Providing more assurance that your server infrastructure is
> > > uncompromised. Offloading computation to a system without having to
> > > fully trust that system.
> > 
> > I think you can do the crypto stuff... as crypto already verifies the
> > results. But I don't think you can do the computation offload.
> 
> You can, as long as you can do attestation.

You can not, because random errors are very easy to trigger for person
with physical access, as I explained in the part of email you
stripped.

> > > As one of many possibilities, imagine a distcc that didn't have to trust
> > > the compile nodes. The compile nodes could fail to return results at
> > > all, but they couldn't alter the results.
> > 
> > distcc on untrusted nodes ... oh yes, that would be great.
> > 
> > Except that you can't do it, right? :-).
> > 
> > First, AFAICT it would be quite hard to get gcc to run under SGX. But
> > maybe you have spare month or three and can do it.
> 
> Assuming you don't need to #include files, gcc seems quite simple to run
> in an enclave: data in, computation inside, data out.

So is there a plan to run dynamically linked binaries inside enclave?
Or maybe even python/shell scripts? It looked to me like virtual
memory will be "interesting" for enclaves.


								Pavel
-- 
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 181 bytes --]

Re: [PATCH v11 00/13] Intel SGX1 support

[Pavel Machek]

[-- Attachment #1: Type: text/plain, Size: 2220 bytes --]

On Sun 2018-12-09 23:47:17, Josh Triplett wrote:
> On Sun, Dec 09, 2018 at 09:06:00PM +0100, Pavel Machek wrote:
> ...
> > > > > The default permissions for the device are 600.
> > > > 
> > > > Good. This does not belong to non-root.
> > > 
> > > There are entirely legitimate use cases for using this as an
> > > unprivileged user. However, that'll be up to system and distribution
> > > policy, which can evolve over time, and it makes sense for the *initial*
> > > kernel permission to start out root-only and then adjust permissions via
> > > udev.
> > 
> > Agreed.
> > 
> > > Building a software certificate store. Hardening key-agent software like
> > > ssh-agent or gpg-agent. Building a challenge-response authentication
> > > system. Providing more assurance that your server infrastructure is
> > > uncompromised. Offloading computation to a system without having to
> > > fully trust that system.
> > 
> > I think you can do the crypto stuff... as crypto already verifies the
> > results. But I don't think you can do the computation offload.
> 
> You can, as long as you can do attestation.

You can not, because random errors are very easy to trigger for person
with physical access, as I explained in the part of email you
stripped.

> > > As one of many possibilities, imagine a distcc that didn't have to trust
> > > the compile nodes. The compile nodes could fail to return results at
> > > all, but they couldn't alter the results.
> > 
> > distcc on untrusted nodes ... oh yes, that would be great.
> > 
> > Except that you can't do it, right? :-).
> > 
> > First, AFAICT it would be quite hard to get gcc to run under SGX. But
> > maybe you have spare month or three and can do it.
> 
> Assuming you don't need to #include files, gcc seems quite simple to run
> in an enclave: data in, computation inside, data out.

So is there a plan to run dynamically linked binaries inside enclave?
Or maybe even python/shell scripts? It looked to me like virtual
memory will be "interesting" for enclaves.


								Pavel
-- 
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 181 bytes --]

Re: [PATCH v11 00/13] Intel SGX1 support

[Pavel Machek]

[-- Attachment #1: Type: text/plain, Size: 2220 bytes --]

On Sun 2018-12-09 23:47:17, Josh Triplett wrote:
> On Sun, Dec 09, 2018 at 09:06:00PM +0100, Pavel Machek wrote:
> ...
> > > > > The default permissions for the device are 600.
> > > > 
> > > > Good. This does not belong to non-root.
> > > 
> > > There are entirely legitimate use cases for using this as an
> > > unprivileged user. However, that'll be up to system and distribution
> > > policy, which can evolve over time, and it makes sense for the *initial*
> > > kernel permission to start out root-only and then adjust permissions via
> > > udev.
> > 
> > Agreed.
> > 
> > > Building a software certificate store. Hardening key-agent software like
> > > ssh-agent or gpg-agent. Building a challenge-response authentication
> > > system. Providing more assurance that your server infrastructure is
> > > uncompromised. Offloading computation to a system without having to
> > > fully trust that system.
> > 
> > I think you can do the crypto stuff... as crypto already verifies the
> > results. But I don't think you can do the computation offload.
> 
> You can, as long as you can do attestation.

You can not, because random errors are very easy to trigger for person
with physical access, as I explained in the part of email you
stripped.

> > > As one of many possibilities, imagine a distcc that didn't have to trust
> > > the compile nodes. The compile nodes could fail to return results at
> > > all, but they couldn't alter the results.
> > 
> > distcc on untrusted nodes ... oh yes, that would be great.
> > 
> > Except that you can't do it, right? :-).
> > 
> > First, AFAICT it would be quite hard to get gcc to run under SGX. But
> > maybe you have spare month or three and can do it.
> 
> Assuming you don't need to #include files, gcc seems quite simple to run
> in an enclave: data in, computation inside, data out.

So is there a plan to run dynamically linked binaries inside enclave?
Or maybe even python/shell scripts? It looked to me like virtual
memory will be "interesting" for enclaves.


								Pavel
-- 
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 181 bytes --]

Re: [PATCH v11 00/13] Intel SGX1 support

[Pavel Machek]

[-- Attachment #1: Type: text/plain, Size: 2220 bytes --]

On Sun 2018-12-09 23:47:17, Josh Triplett wrote:
> On Sun, Dec 09, 2018 at 09:06:00PM +0100, Pavel Machek wrote:
> ...
> > > > > The default permissions for the device are 600.
> > > > 
> > > > Good. This does not belong to non-root.
> > > 
> > > There are entirely legitimate use cases for using this as an
> > > unprivileged user. However, that'll be up to system and distribution
> > > policy, which can evolve over time, and it makes sense for the *initial*
> > > kernel permission to start out root-only and then adjust permissions via
> > > udev.
> > 
> > Agreed.
> > 
> > > Building a software certificate store. Hardening key-agent software like
> > > ssh-agent or gpg-agent. Building a challenge-response authentication
> > > system. Providing more assurance that your server infrastructure is
> > > uncompromised. Offloading computation to a system without having to
> > > fully trust that system.
> > 
> > I think you can do the crypto stuff... as crypto already verifies the
> > results. But I don't think you can do the computation offload.
> 
> You can, as long as you can do attestation.

You can not, because random errors are very easy to trigger for person
with physical access, as I explained in the part of email you
stripped.

> > > As one of many possibilities, imagine a distcc that didn't have to trust
> > > the compile nodes. The compile nodes could fail to return results at
> > > all, but they couldn't alter the results.
> > 
> > distcc on untrusted nodes ... oh yes, that would be great.
> > 
> > Except that you can't do it, right? :-).
> > 
> > First, AFAICT it would be quite hard to get gcc to run under SGX. But
> > maybe you have spare month or three and can do it.
> 
> Assuming you don't need to #include files, gcc seems quite simple to run
> in an enclave: data in, computation inside, data out.

So is there a plan to run dynamically linked binaries inside enclave?
Or maybe even python/shell scripts? It looked to me like virtual
memory will be "interesting" for enclaves.


								Pavel
-- 
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 181 bytes --]

Re: [PATCH v11 00/13] Intel SGX1 support

[Josh Triplett]

On Mon, Dec 10, 2018 at 09:27:04AM +0100, Pavel Machek wrote:
> On Sun 2018-12-09 23:47:17, Josh Triplett wrote:
> > On Sun, Dec 09, 2018 at 09:06:00PM +0100, Pavel Machek wrote:
> > ...
> > > > > > The default permissions for the device are 600.
> > > > > 
> > > > > Good. This does not belong to non-root.
> > > > 
> > > > There are entirely legitimate use cases for using this as an
> > > > unprivileged user. However, that'll be up to system and distribution
> > > > policy, which can evolve over time, and it makes sense for the *initial*
> > > > kernel permission to start out root-only and then adjust permissions via
> > > > udev.
> > > 
> > > Agreed.
> > > 
> > > > Building a software certificate store. Hardening key-agent software like
> > > > ssh-agent or gpg-agent. Building a challenge-response authentication
> > > > system. Providing more assurance that your server infrastructure is
> > > > uncompromised. Offloading computation to a system without having to
> > > > fully trust that system.
> > > 
> > > I think you can do the crypto stuff... as crypto already verifies the
> > > results. But I don't think you can do the computation offload.
> > 
> > You can, as long as you can do attestation.
> 
> You can not, because random errors are very easy to trigger for person
> with physical access,

Random errors can also just happen, so if you're concerned about that
you might want to build each object on two different machines and
compare. Good luck generating the *same* random errors on two machines.

(And, of course, someone can also DoS you in any number of other ways,
such as accepting data and then never sending back a result. So you'll
need timeouts and failovers.)

> > > > As one of many possibilities, imagine a distcc that didn't have to trust
> > > > the compile nodes. The compile nodes could fail to return results at
> > > > all, but they couldn't alter the results.
> > > 
> > > distcc on untrusted nodes ... oh yes, that would be great.
> > > 
> > > Except that you can't do it, right? :-).
> > > 
> > > First, AFAICT it would be quite hard to get gcc to run under SGX. But
> > > maybe you have spare month or three and can do it.
> > 
> > Assuming you don't need to #include files, gcc seems quite simple to run
> > in an enclave: data in, computation inside, data out.
> 
> So is there a plan to run dynamically linked binaries inside enclave?

I've seen some approaches for that, but you could also just statically
link your compiler. (Since you'd need attestation for all the individual
libraries, you'd need to know the versions of all those libraries, so
you might as well just statically link.)

> Or maybe even python/shell scripts? It looked to me like virtual
> memory will be "interesting" for enclaves.

Memory management doesn't seem that hard to deal with.

Re: [PATCH v11 00/13] Intel SGX1 support

[Josh Triplett]

On Mon, Dec 10, 2018 at 09:27:04AM +0100, Pavel Machek wrote:
> On Sun 2018-12-09 23:47:17, Josh Triplett wrote:
> > On Sun, Dec 09, 2018 at 09:06:00PM +0100, Pavel Machek wrote:
> > ...
> > > > > > The default permissions for the device are 600.
> > > > > 
> > > > > Good. This does not belong to non-root.
> > > > 
> > > > There are entirely legitimate use cases for using this as an
> > > > unprivileged user. However, that'll be up to system and distribution
> > > > policy, which can evolve over time, and it makes sense for the *initial*
> > > > kernel permission to start out root-only and then adjust permissions via
> > > > udev.
> > > 
> > > Agreed.
> > > 
> > > > Building a software certificate store. Hardening key-agent software like
> > > > ssh-agent or gpg-agent. Building a challenge-response authentication
> > > > system. Providing more assurance that your server infrastructure is
> > > > uncompromised. Offloading computation to a system without having to
> > > > fully trust that system.
> > > 
> > > I think you can do the crypto stuff... as crypto already verifies the
> > > results. But I don't think you can do the computation offload.
> > 
> > You can, as long as you can do attestation.
> 
> You can not, because random errors are very easy to trigger for person
> with physical access,

Random errors can also just happen, so if you're concerned about that
you might want to build each object on two different machines and
compare. Good luck generating the *same* random errors on two machines.

(And, of course, someone can also DoS you in any number of other ways,
such as accepting data and then never sending back a result. So you'll
need timeouts and failovers.)

> > > > As one of many possibilities, imagine a distcc that didn't have to trust
> > > > the compile nodes. The compile nodes could fail to return results at
> > > > all, but they couldn't alter the results.
> > > 
> > > distcc on untrusted nodes ... oh yes, that would be great.
> > > 
> > > Except that you can't do it, right? :-).
> > > 
> > > First, AFAICT it would be quite hard to get gcc to run under SGX. But
> > > maybe you have spare month or three and can do it.
> > 
> > Assuming you don't need to #include files, gcc seems quite simple to run
> > in an enclave: data in, computation inside, data out.
> 
> So is there a plan to run dynamically linked binaries inside enclave?

I've seen some approaches for that, but you could also just statically
link your compiler. (Since you'd need attestation for all the individual
libraries, you'd need to know the versions of all those libraries, so
you might as well just statically link.)

> Or maybe even python/shell scripts? It looked to me like virtual
> memory will be "interesting" for enclaves.

Memory management doesn't seem that hard to deal with.

Re: [PATCH v11 00/13] Intel SGX1 support

[Josh Triplett]

On Mon, Dec 10, 2018 at 09:27:04AM +0100, Pavel Machek wrote:
> On Sun 2018-12-09 23:47:17, Josh Triplett wrote:
> > On Sun, Dec 09, 2018 at 09:06:00PM +0100, Pavel Machek wrote:
> > ...
> > > > > > The default permissions for the device are 600.
> > > > > 
> > > > > Good. This does not belong to non-root.
> > > > 
> > > > There are entirely legitimate use cases for using this as an
> > > > unprivileged user. However, that'll be up to system and distribution
> > > > policy, which can evolve over time, and it makes sense for the *initial*
> > > > kernel permission to start out root-only and then adjust permissions via
> > > > udev.
> > > 
> > > Agreed.
> > > 
> > > > Building a software certificate store. Hardening key-agent software like
> > > > ssh-agent or gpg-agent. Building a challenge-response authentication
> > > > system. Providing more assurance that your server infrastructure is
> > > > uncompromised. Offloading computation to a system without having to
> > > > fully trust that system.
> > > 
> > > I think you can do the crypto stuff... as crypto already verifies the
> > > results. But I don't think you can do the computation offload.
> > 
> > You can, as long as you can do attestation.
> 
> You can not, because random errors are very easy to trigger for person
> with physical access,

Random errors can also just happen, so if you're concerned about that
you might want to build each object on two different machines and
compare. Good luck generating the *same* random errors on two machines.

(And, of course, someone can also DoS you in any number of other ways,
such as accepting data and then never sending back a result. So you'll
need timeouts and failovers.)

> > > > As one of many possibilities, imagine a distcc that didn't have to trust
> > > > the compile nodes. The compile nodes could fail to return results at
> > > > all, but they couldn't alter the results.
> > > 
> > > distcc on untrusted nodes ... oh yes, that would be great.
> > > 
> > > Except that you can't do it, right? :-).
> > > 
> > > First, AFAICT it would be quite hard to get gcc to run under SGX. But
> > > maybe you have spare month or three and can do it.
> > 
> > Assuming you don't need to #include files, gcc seems quite simple to run
> > in an enclave: data in, computation inside, data out.
> 
> So is there a plan to run dynamically linked binaries inside enclave?

I've seen some approaches for that, but you could also just statically
link your compiler. (Since you'd need attestation for all the individual
libraries, you'd need to know the versions of all those libraries, so
you might as well just statically link.)

> Or maybe even python/shell scripts? It looked to me like virtual
> memory will be "interesting" for enclaves.

Memory management doesn't seem that hard to deal with.

Re: [PATCH v11 00/13] Intel SGX1 support

[Josh Triplett]

On Mon, Dec 10, 2018 at 09:27:04AM +0100, Pavel Machek wrote:
> On Sun 2018-12-09 23:47:17, Josh Triplett wrote:
> > On Sun, Dec 09, 2018 at 09:06:00PM +0100, Pavel Machek wrote:
> > ...
> > > > > > The default permissions for the device are 600.
> > > > > 
> > > > > Good. This does not belong to non-root.
> > > > 
> > > > There are entirely legitimate use cases for using this as an
> > > > unprivileged user. However, that'll be up to system and distribution
> > > > policy, which can evolve over time, and it makes sense for the *initial*
> > > > kernel permission to start out root-only and then adjust permissions via
> > > > udev.
> > > 
> > > Agreed.
> > > 
> > > > Building a software certificate store. Hardening key-agent software like
> > > > ssh-agent or gpg-agent. Building a challenge-response authentication
> > > > system. Providing more assurance that your server infrastructure is
> > > > uncompromised. Offloading computation to a system without having to
> > > > fully trust that system.
> > > 
> > > I think you can do the crypto stuff... as crypto already verifies the
> > > results. But I don't think you can do the computation offload.
> > 
> > You can, as long as you can do attestation.
> 
> You can not, because random errors are very easy to trigger for person
> with physical access,

Random errors can also just happen, so if you're concerned about that
you might want to build each object on two different machines and
compare. Good luck generating the *same* random errors on two machines.

(And, of course, someone can also DoS you in any number of other ways,
such as accepting data and then never sending back a result. So you'll
need timeouts and failovers.)

> > > > As one of many possibilities, imagine a distcc that didn't have to trust
> > > > the compile nodes. The compile nodes could fail to return results at
> > > > all, but they couldn't alter the results.
> > > 
> > > distcc on untrusted nodes ... oh yes, that would be great.
> > > 
> > > Except that you can't do it, right? :-).
> > > 
> > > First, AFAICT it would be quite hard to get gcc to run under SGX. But
> > > maybe you have spare month or three and can do it.
> > 
> > Assuming you don't need to #include files, gcc seems quite simple to run
> > in an enclave: data in, computation inside, data out.
> 
> So is there a plan to run dynamically linked binaries inside enclave?

I've seen some approaches for that, but you could also just statically
link your compiler. (Since you'd need attestation for all the individual
libraries, you'd need to know the versions of all those libraries, so
you might as well just statically link.)

> Or maybe even python/shell scripts? It looked to me like virtual
> memory will be "interesting" for enclaves.

Memory management doesn't seem that hard to deal with.

Re: [PATCH v11 00/13] Intel SGX1 support

[Dave Hansen]

On 12/10/18 3:12 PM, Josh Triplett wrote:
>> Or maybe even python/shell scripts? It looked to me like virtual
>> memory will be "interesting" for enclaves.
> Memory management doesn't seem that hard to deal with.

The problems are:

1. SGX enclave memory (EPC) is statically allocated at boot and can't
   grow or shrink
2. EPC is much smaller than regular RAM
3. The core VM has no comprehension of EPC use, thus can not help
   with its algorithms, like the LRU
4. The SGX driver implements its own VM which is substantially simpler
   than the core VM, but less feature-rich, fast, or scalable

Re: [PATCH v11 00/13] Intel SGX1 support

[Dave Hansen]

On 12/10/18 3:12 PM, Josh Triplett wrote:
>> Or maybe even python/shell scripts? It looked to me like virtual
>> memory will be "interesting" for enclaves.
> Memory management doesn't seem that hard to deal with.

The problems are:

1. SGX enclave memory (EPC) is statically allocated at boot and can't
   grow or shrink
2. EPC is much smaller than regular RAM
3. The core VM has no comprehension of EPC use, thus can not help
   with its algorithms, like the LRU
4. The SGX driver implements its own VM which is substantially simpler
   than the core VM, but less feature-rich, fast, or scalable

Re: [PATCH v11 00/13] Intel SGX1 support

[Dave Hansen]

On 12/10/18 3:12 PM, Josh Triplett wrote:
>> Or maybe even python/shell scripts? It looked to me like virtual
>> memory will be "interesting" for enclaves.
> Memory management doesn't seem that hard to deal with.

The problems are:

1. SGX enclave memory (EPC) is statically allocated at boot and can't
   grow or shrink
2. EPC is much smaller than regular RAM
3. The core VM has no comprehension of EPC use, thus can not help
   with its algorithms, like the LRU
4. The SGX driver implements its own VM which is substantially simpler
   than the core VM, but less feature-rich, fast, or scalable

Re: [PATCH v11 00/13] Intel SGX1 support

[Dave Hansen]

On 12/10/18 3:12 PM, Josh Triplett wrote:
>> Or maybe even python/shell scripts? It looked to me like virtual
>> memory will be "interesting" for enclaves.
> Memory management doesn't seem that hard to deal with.

The problems are:

1. SGX enclave memory (EPC) is statically allocated at boot and can't
   grow or shrink
2. EPC is much smaller than regular RAM
3. The core VM has no comprehension of EPC use, thus can not help
   with its algorithms, like the LRU
4. The SGX driver implements its own VM which is substantially simpler
   than the core VM, but less feature-rich, fast, or scalable

Re: [PATCH v11 00/13] Intel SGX1 support

[Sean Christopherson]

On Tue, Dec 11, 2018 at 10:10:38AM -0800, Dave Hansen wrote:
> On 12/10/18 3:12 PM, Josh Triplett wrote:
> >> Or maybe even python/shell scripts? It looked to me like virtual
> >> memory will be "interesting" for enclaves.
> > Memory management doesn't seem that hard to deal with.
> 
> The problems are:
> 
> 1. SGX enclave memory (EPC) is statically allocated at boot and can't
>    grow or shrink
> 2. EPC is much smaller than regular RAM
> 3. The core VM has no comprehension of EPC use, thus can not help
>    with its algorithms, like the LRU
> 4. The SGX driver implements its own VM which is substantially simpler
>    than the core VM, but less feature-rich, fast, or scalable

I'd also add:

  5. Swapping EPC pages can only be done through SGX specific ISA that
     has strict concurrency requirements and enforces TLB flushing.
  6. There are specialized types of EPC pages that have different
     swapping requirements than regular EPC pages.
  7. EPC pages that are exposed to a KVM guest have yet another set of
     swapping requirements.

In other words, extending the core VM to SGX EPC is painfully difficult.

Re: [PATCH v11 00/13] Intel SGX1 support

[Sean Christopherson]

On Tue, Dec 11, 2018 at 10:10:38AM -0800, Dave Hansen wrote:
> On 12/10/18 3:12 PM, Josh Triplett wrote:
> >> Or maybe even python/shell scripts? It looked to me like virtual
> >> memory will be "interesting" for enclaves.
> > Memory management doesn't seem that hard to deal with.
> 
> The problems are:
> 
> 1. SGX enclave memory (EPC) is statically allocated at boot and can't
>    grow or shrink
> 2. EPC is much smaller than regular RAM
> 3. The core VM has no comprehension of EPC use, thus can not help
>    with its algorithms, like the LRU
> 4. The SGX driver implements its own VM which is substantially simpler
>    than the core VM, but less feature-rich, fast, or scalable

I'd also add:

  5. Swapping EPC pages can only be done through SGX specific ISA that
     has strict concurrency requirements and enforces TLB flushing.
  6. There are specialized types of EPC pages that have different
     swapping requirements than regular EPC pages.
  7. EPC pages that are exposed to a KVM guest have yet another set of
     swapping requirements.

In other words, extending the core VM to SGX EPC is painfully difficult.

Re: [PATCH v11 00/13] Intel SGX1 support

[Sean Christopherson]

On Tue, Dec 11, 2018 at 10:10:38AM -0800, Dave Hansen wrote:
> On 12/10/18 3:12 PM, Josh Triplett wrote:
> >> Or maybe even python/shell scripts? It looked to me like virtual
> >> memory will be "interesting" for enclaves.
> > Memory management doesn't seem that hard to deal with.
> 
> The problems are:
> 
> 1. SGX enclave memory (EPC) is statically allocated at boot and can't
>    grow or shrink
> 2. EPC is much smaller than regular RAM
> 3. The core VM has no comprehension of EPC use, thus can not help
>    with its algorithms, like the LRU
> 4. The SGX driver implements its own VM which is substantially simpler
>    than the core VM, but less feature-rich, fast, or scalable

I'd also add:

  5. Swapping EPC pages can only be done through SGX specific ISA that
     has strict concurrency requirements and enforces TLB flushing.
  6. There are specialized types of EPC pages that have different
     swapping requirements than regular EPC pages.
  7. EPC pages that are exposed to a KVM guest have yet another set of
     swapping requirements.

In other words, extending the core VM to SGX EPC is painfully difficult.

Re: [PATCH v11 00/13] Intel SGX1 support

[Sean Christopherson]

On Tue, Dec 11, 2018 at 10:10:38AM -0800, Dave Hansen wrote:
> On 12/10/18 3:12 PM, Josh Triplett wrote:
> >> Or maybe even python/shell scripts? It looked to me like virtual
> >> memory will be "interesting" for enclaves.
> > Memory management doesn't seem that hard to deal with.
> 
> The problems are:
> 
> 1. SGX enclave memory (EPC) is statically allocated at boot and can't
>    grow or shrink
> 2. EPC is much smaller than regular RAM
> 3. The core VM has no comprehension of EPC use, thus can not help
>    with its algorithms, like the LRU
> 4. The SGX driver implements its own VM which is substantially simpler
>    than the core VM, but less feature-rich, fast, or scalable

I'd also add:

  5. Swapping EPC pages can only be done through SGX specific ISA that
     has strict concurrency requirements and enforces TLB flushing.
  6. There are specialized types of EPC pages that have different
     swapping requirements than regular EPC pages.
  7. EPC pages that are exposed to a KVM guest have yet another set of
     swapping requirements.

In other words, extending the core VM to SGX EPC is painfully difficult.