[RFC PATCH v1 00/18] x86: Secure Memory Encryption (AMD)

[RFC PATCH v1 00/18] x86: Secure Memory Encryption (AMD)

[Tom Lendacky]

This RFC patch series provides support for AMD's new Secure Memory
Encryption (SME) feature.

SME can be used to mark individual pages of memory as encrypted through the
page tables. A page of memory that is marked encrypted will be automatically
decrypted when read from DRAM and will be automatically encrypted when
written to DRAM. Details on SME can found in the links below.

The SME feature is identified through a CPUID function and enabled through
the SYSCFG MSR. Once enabled, page table entries will determine how the
memory is accessed. If a page table entry has the memory encryption mask set,
then that memory will be accessed as encrypted memory. The memory encryption
mask (as well as other related information) is determined from settings
returned through the same CPUID function that identifies the presence of the
feature.

The approach that this patch series takes is to encrypt everything possible
starting early in the boot where the kernel is encrypted. Using the page
table macros the encryption mask can be incorporated into all page table
entries and page allocations. By updating the protection map, userspace
allocations are also marked encrypted. Certain data must be accounted for
as having been placed in memory before SME was enabled (EFI, initrd, etc.)
and accessed accordingly.

This patch series is a pre-cursor to another AMD processor feature called
Secure Encrypted Virtualization (SEV). The support for SEV will build upon
the SME support and will be submitted later. Details on SEV can be found
in the links below.

The following links provide additional detail:

AMD Memory Encryption whitepaper:
   http://amd-dev.wpengine.netdna-cdn.com/wordpress/media/2013/12/AMD_Memory_Encryption_Whitepaper_v7-Public.pdf

AMD64 Architecture Programmer's Manual:
   http://support.amd.com/TechDocs/24593.pdf
   SME is section 7.10
   SEV is section 15.34

This patch series is based off of the master branch of tip.
  Commit 8d54fcebd9b3 ("Merge branch 'x86/urgent'")

---

Tom Lendacky (18):
      x86: Set the write-protect cache mode for AMD processors
      x86: Secure Memory Encryption (SME) build enablement
      x86: Secure Memory Encryption (SME) support
      x86: Add the Secure Memory Encryption cpu feature
      x86: Handle reduction in physical address size with SME
      x86: Provide general kernel support for memory encryption
      x86: Extend the early_memmap support with additional attrs
      x86: Add support for early encryption/decryption of memory
      x86: Insure that memory areas are encrypted when possible
      x86/efi: Access EFI related tables in the clear
      x86: Decrypt trampoline area if memory encryption is active
      x86: Access device tree in the clear
      x86: DMA support for memory encryption
      iommu/amd: AMD IOMMU support for memory encryption
      x86: Enable memory encryption on the APs
      x86: Do not specify encrypted memory for VGA mapping
      x86/kvm: Enable Secure Memory Encryption of nested page tables
      x86: Add support to turn on Secure Memory Encryption


 Documentation/kernel-parameters.txt  |    3 
 arch/x86/Kconfig                     |    9 +
 arch/x86/include/asm/cacheflush.h    |    3 
 arch/x86/include/asm/cpufeature.h    |    1 
 arch/x86/include/asm/cpufeatures.h   |    5 
 arch/x86/include/asm/dma-mapping.h   |    5 
 arch/x86/include/asm/fixmap.h        |   16 ++
 arch/x86/include/asm/kvm_host.h      |    2 
 arch/x86/include/asm/mem_encrypt.h   |   99 ++++++++++
 arch/x86/include/asm/msr-index.h     |    2 
 arch/x86/include/asm/pgtable_types.h |   49 +++--
 arch/x86/include/asm/processor.h     |    3 
 arch/x86/include/asm/realmode.h      |   12 +
 arch/x86/include/asm/vga.h           |   13 +
 arch/x86/kernel/Makefile             |    2 
 arch/x86/kernel/asm-offsets.c        |    2 
 arch/x86/kernel/cpu/common.c         |    2 
 arch/x86/kernel/cpu/scattered.c      |    1 
 arch/x86/kernel/devicetree.c         |    6 -
 arch/x86/kernel/espfix_64.c          |    2 
 arch/x86/kernel/head64.c             |  100 +++++++++-
 arch/x86/kernel/head_64.S            |   42 +++-
 arch/x86/kernel/machine_kexec_64.c   |    2 
 arch/x86/kernel/mem_encrypt.S        |  343 ++++++++++++++++++++++++++++++++++
 arch/x86/kernel/pci-dma.c            |   11 +
 arch/x86/kernel/pci-nommu.c          |    2 
 arch/x86/kernel/pci-swiotlb.c        |    8 +
 arch/x86/kernel/setup.c              |   14 +
 arch/x86/kernel/x8664_ksyms_64.c     |    6 +
 arch/x86/kvm/mmu.c                   |    7 -
 arch/x86/kvm/vmx.c                   |    2 
 arch/x86/kvm/x86.c                   |    3 
 arch/x86/mm/Makefile                 |    1 
 arch/x86/mm/fault.c                  |    5 
 arch/x86/mm/ioremap.c                |   31 +++
 arch/x86/mm/kasan_init_64.c          |    4 
 arch/x86/mm/mem_encrypt.c            |  201 ++++++++++++++++++++
 arch/x86/mm/pageattr.c               |   78 ++++++++
 arch/x86/mm/pat.c                    |   11 +
 arch/x86/platform/efi/efi.c          |   26 +--
 arch/x86/platform/efi/efi_64.c       |    9 +
 arch/x86/platform/efi/quirks.c       |   12 +
 arch/x86/realmode/init.c             |   13 +
 arch/x86/realmode/rm/trampoline_64.S |   14 +
 drivers/firmware/efi/efi.c           |   18 +-
 drivers/firmware/efi/esrt.c          |   12 +
 drivers/iommu/amd_iommu.c            |   10 +
 include/asm-generic/early_ioremap.h  |    2 
 include/linux/efi.h                  |    3 
 include/linux/swiotlb.h              |    1 
 init/main.c                          |    6 +
 lib/swiotlb.c                        |   64 ++++++
 mm/early_ioremap.c                   |   15 +
 53 files changed, 1217 insertions(+), 96 deletions(-)
 create mode 100644 arch/x86/include/asm/mem_encrypt.h
 create mode 100644 arch/x86/kernel/mem_encrypt.S
 create mode 100644 arch/x86/mm/mem_encrypt.c

-- 
Tom Lendacky

Re: [RFC PATCH v1 00/18] x86: Secure Memory Encryption (AMD)

[Pavel Machek]

Hi!

> This RFC patch series provides support for AMD's new Secure Memory
> Encryption (SME) feature.
> 
> SME can be used to mark individual pages of memory as encrypted through the
> page tables. A page of memory that is marked encrypted will be automatically
> decrypted when read from DRAM and will be automatically encrypted when
> written to DRAM. Details on SME can found in the links below.

Well, actually brief summary should go to changelog and probably to the documentation,
too...

Why would I want SME on my system? My system seems to work without it.

Does it protect against cold boot attacks? Rowhammer (I guess not?)

Does it cost some performance?

Does it break debugging over JTAG?

> The approach that this patch series takes is to encrypt everything possible
> starting early in the boot where the kernel is encrypted. Using the page
> table macros the encryption mask can be incorporated into all page table
> entries and page allocations. By updating the protection map, userspace
> allocations are also marked encrypted. Certain data must be accounted for
> as having been placed in memory before SME was enabled (EFI, initrd, etc.)
> and accessed accordingly.

Do you also need to do something special for device DMA?

Thanks,

									Pavel
-- 
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html

Re: [RFC PATCH v1 00/18] x86: Secure Memory Encryption (AMD)

[Borislav Petkov]

On Tue, Mar 22, 2016 at 02:00:58PM +0100, Pavel Machek wrote:
> Why would I want SME on my system? My system seems to work without it.

Your system doesn't have it and SME is default off.

-- 
Regards/Gruss,
    Boris.

ECO tip #101: Trim your mails when you reply.

Re: [RFC PATCH v1 00/18] x86: Secure Memory Encryption (AMD)

[Pavel Machek]

On Wed 2016-04-27 16:05:20, Borislav Petkov wrote:
> On Tue, Mar 22, 2016 at 02:00:58PM +0100, Pavel Machek wrote:
> > Why would I want SME on my system? My system seems to work without it.
> 
> Your system doesn't have it and SME is default off.

That does not answer the question. "Why would I want SME on my
system?".

And that answer should go to Documentation/.

									Pavel
-- 
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html

Re: [RFC PATCH v1 00/18] x86: Secure Memory Encryption (AMD)

[Borislav Petkov]

On Wed, Apr 27, 2016 at 04:30:45PM +0200, Pavel Machek wrote:
> That does not answer the question. "Why would I want SME on my
> system?".

Because your question wasn't formulated properly. Here's some text from
the 0th mail which you could've found on your own:

"The following links provide additional detail:

AMD Memory Encryption whitepaper:
   http://amd-dev.wpengine.netdna-cdn.com/wordpress/media/2013/12/AMD_Memory_Encryption_Whitepaper_v7-Public.pdf
"

> And that answer should go to Documentation/.

It will.

-- 
Regards/Gruss,
    Boris.

ECO tip #101: Trim your mails when you reply.

Re: [RFC PATCH v1 00/18] x86: Secure Memory Encryption (AMD)

[Pavel Machek]

On Wed 2016-04-27 16:39:51, Borislav Petkov wrote:
> On Wed, Apr 27, 2016 at 04:30:45PM +0200, Pavel Machek wrote:
> > That does not answer the question. "Why would I want SME on my
> > system?".
> 
> Because your question wasn't formulated properly. Here's some text from
> the 0th mail which you could've found on your own:

> "The following links provide additional detail:
> 
> AMD Memory Encryption whitepaper:
>    http://amd-dev.wpengine.netdna-cdn.com/wordpress/media/2013/12/AMD_Memory_Encryption_Whitepaper_v7-Public.pdf
> "
> 
> > And that answer should go to Documentation/.
> 
> It will.

Yeah? So why not include it in reply instead of flaming me?

									Pavel
-- 
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html

Re: [RFC PATCH v1 00/18] x86: Secure Memory Encryption (AMD)

[Pavel Machek]

On Wed 2016-04-27 16:39:51, Borislav Petkov wrote:
> On Wed, Apr 27, 2016 at 04:30:45PM +0200, Pavel Machek wrote:
> > That does not answer the question. "Why would I want SME on my
> > system?".
> 
> Because your question wasn't formulated properly. Here's some text from
> the 0th mail which you could've found on your own:
> 
> "The following links provide additional detail:
> 
> AMD Memory Encryption whitepaper:
>    http://amd-dev.wpengine.netdna-cdn.com/wordpress/media/2013/12/AMD_Memory_Encryption_Whitepaper_v7-Public.pdf
> "


Unfortunately that document is marketing junk.

AFAICT:

SME can protect against cold boot attack and snooping at DRAM
level. That's pretty much it.

Does the AES encryption take the address as a parameter?

SEV may protect against passive attack on the VM. For active attack,
they claim it will "probably" crash the VM, but we already know that
is untrue, see the work on gaining root using rowhammer. In this case,
attacker can choose which address to damage and has precise control of
timing.

Best regards,
                                                                Pavel
								

-- 
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html

Re: [RFC PATCH v1 00/18] x86: Secure Memory Encryption (AMD)

[Tom Lendacky]

On 03/22/2016 08:00 AM, Pavel Machek wrote:
> Hi!
> 
>> This RFC patch series provides support for AMD's new Secure Memory
>> Encryption (SME) feature.
>>
>> SME can be used to mark individual pages of memory as encrypted through the
>> page tables. A page of memory that is marked encrypted will be automatically
>> decrypted when read from DRAM and will be automatically encrypted when
>> written to DRAM. Details on SME can found in the links below.
> 
> Well, actually brief summary should go to changelog and probably to the documentation,
> too...
> 
> Why would I want SME on my system? My system seems to work without it.

The whitepaper explains the technologies and the value they provide.
It's up to you to decide if you'd want to use it.

> 
> Does it protect against cold boot attacks? Rowhammer (I guess not?)

It does protect well against cold boot attacks. It might offer some
protection against Rowhammer since if a bit got flipped the entire
16B chunk would be decrypted differently.

> 
> Does it cost some performance?

The whitepaper talks about this a little, but it has a minimal impact
on system performance when accessing an encrypted page.

> 
> Does it break debugging over JTAG?
> 
>> The approach that this patch series takes is to encrypt everything possible
>> starting early in the boot where the kernel is encrypted. Using the page
>> table macros the encryption mask can be incorporated into all page table
>> entries and page allocations. By updating the protection map, userspace
>> allocations are also marked encrypted. Certain data must be accounted for
>> as having been placed in memory before SME was enabled (EFI, initrd, etc.)
>> and accessed accordingly.
> 
> Do you also need to do something special for device DMA?

DMA should function normally unless the device does not support the
addressing requirements when SME is active. When the encryption mask is
applied (bit 47 of the physical address in this case), if the device
doesn't support 48 bit or higher DMA then swiotlb bounce buffers will
be used.

Thanks,
Tom

> 
> Thanks,
> 
> 									Pavel
> 

[RFC PATCH v1 01/18] x86: Set the write-protect cache mode for AMD processors

[Tom Lendacky]

For AMD processors that support PAT, set the write-protect cache mode
(_PAGE_CACHE_MODE_WP) entry to the actual write-protect value (x05).

Signed-off-by: Tom Lendacky <thomas.lendacky@amd.com>
---
 arch/x86/mm/pat.c |   11 +++++++++--
 1 file changed, 9 insertions(+), 2 deletions(-)

diff --git a/arch/x86/mm/pat.c b/arch/x86/mm/pat.c
index fb0604f..dda78ed 100644
--- a/arch/x86/mm/pat.c
+++ b/arch/x86/mm/pat.c
@@ -345,6 +345,8 @@ void pat_init(void)
 		 * we lose performance without causing a correctness issue.
 		 * Pentium 4 erratum N46 is an example for such an erratum,
 		 * although we try not to use PAT at all on affected CPUs.
+		 * AMD processors support write-protect so initialize the
+		 * PAT slot 5 appropriately.
 		 *
 		 *  PTE encoding:
 		 *      PAT
@@ -356,7 +358,7 @@ void pat_init(void)
 		 *      010    2    UC-: _PAGE_CACHE_MODE_UC_MINUS
 		 *      011    3    UC : _PAGE_CACHE_MODE_UC
 		 *      100    4    WB : Reserved
-		 *      101    5    WC : Reserved
+		 *      101    5    WC : Reserved (AMD: _PAGE_CACHE_MODE_WP)
 		 *      110    6    UC-: Reserved
 		 *      111    7    WT : _PAGE_CACHE_MODE_WT
 		 *
@@ -364,7 +366,12 @@ void pat_init(void)
 		 * corresponding types in the presence of PAT errata.
 		 */
 		pat = PAT(0, WB) | PAT(1, WC) | PAT(2, UC_MINUS) | PAT(3, UC) |
-		      PAT(4, WB) | PAT(5, WC) | PAT(6, UC_MINUS) | PAT(7, WT);
+		      PAT(4, WB) | PAT(6, UC_MINUS) | PAT(7, WT);
+
+		if (c->x86_vendor == X86_VENDOR_AMD)
+			pat |= PAT(5, WP);
+		else
+			pat |= PAT(5, WC);
 	}
 
 	if (!boot_cpu_done) {

Re: [RFC PATCH v1 01/18] x86: Set the write-protect cache mode for AMD processors

[Andy Lutomirski]

On Tue, Apr 26, 2016 at 3:56 PM, Tom Lendacky <thomas.lendacky@amd.com> wrote:
> For AMD processors that support PAT, set the write-protect cache mode
> (_PAGE_CACHE_MODE_WP) entry to the actual write-protect value (x05).

What's the purpose of using the WP memory type?

--Andy

Re: [RFC PATCH v1 01/18] x86: Set the write-protect cache mode for AMD processors

[Tom Lendacky]

On 04/27/2016 09:33 AM, Andy Lutomirski wrote:
> On Tue, Apr 26, 2016 at 3:56 PM, Tom Lendacky <thomas.lendacky@amd.com> wrote:
>> For AMD processors that support PAT, set the write-protect cache mode
>> (_PAGE_CACHE_MODE_WP) entry to the actual write-protect value (x05).
> 
> What's the purpose of using the WP memory type?

The WP memory type is used for encrypting or decrypting data "in place".
The use of the WP on the source data will prevent any of the source
data from being cached.  Refer to section 7.10.8 "Encrypt-in-Place" in
the AMD64 APM link provided in the cover letter.

This memory type will be used in subsequent patches for this purpose.

Thanks,
Tom

> 
> --Andy
> 

Re: [RFC PATCH v1 01/18] x86: Set the write-protect cache mode for AMD processors

[Andy Lutomirski]

On Wed, Apr 27, 2016 at 7:44 AM, Tom Lendacky <thomas.lendacky@amd.com> wrote:
> On 04/27/2016 09:33 AM, Andy Lutomirski wrote:
>> On Tue, Apr 26, 2016 at 3:56 PM, Tom Lendacky <thomas.lendacky@amd.com> wrote:
>>> For AMD processors that support PAT, set the write-protect cache mode
>>> (_PAGE_CACHE_MODE_WP) entry to the actual write-protect value (x05).
>>
>> What's the purpose of using the WP memory type?
>
> The WP memory type is used for encrypting or decrypting data "in place".
> The use of the WP on the source data will prevent any of the source
> data from being cached.  Refer to section 7.10.8 "Encrypt-in-Place" in
> the AMD64 APM link provided in the cover letter.
>
> This memory type will be used in subsequent patches for this purpose.

OK.

Why AMD-only?  I thought Intel supported WP, too.

--Andy

Re: [RFC PATCH v1 01/18] x86: Set the write-protect cache mode for AMD processors

[Tom Lendacky]

On 04/27/2016 09:47 AM, Andy Lutomirski wrote:
> On Wed, Apr 27, 2016 at 7:44 AM, Tom Lendacky <thomas.lendacky@amd.com> wrote:
>> On 04/27/2016 09:33 AM, Andy Lutomirski wrote:
>>> On Tue, Apr 26, 2016 at 3:56 PM, Tom Lendacky <thomas.lendacky@amd.com> wrote:
>>>> For AMD processors that support PAT, set the write-protect cache mode
>>>> (_PAGE_CACHE_MODE_WP) entry to the actual write-protect value (x05).
>>>
>>> What's the purpose of using the WP memory type?
>>
>> The WP memory type is used for encrypting or decrypting data "in place".
>> The use of the WP on the source data will prevent any of the source
>> data from being cached.  Refer to section 7.10.8 "Encrypt-in-Place" in
>> the AMD64 APM link provided in the cover letter.
>>
>> This memory type will be used in subsequent patches for this purpose.
> 
> OK.
> 
> Why AMD-only?  I thought Intel supported WP, too.

Just me being conservative. If there aren't any objections from the
Intel folks about it we can remove the vendor check and just set it.

Thanks,
Tom

> 
> --Andy
> 

Re: [RFC PATCH v1 01/18] x86: Set the write-protect cache mode for AMD processors

[Andy Lutomirski]

On Wed, Apr 27, 2016 at 8:05 AM, Tom Lendacky <thomas.lendacky@amd.com> wrote:
> On 04/27/2016 09:47 AM, Andy Lutomirski wrote:
>> On Wed, Apr 27, 2016 at 7:44 AM, Tom Lendacky <thomas.lendacky@amd.com> wrote:
>>> On 04/27/2016 09:33 AM, Andy Lutomirski wrote:
>>>> On Tue, Apr 26, 2016 at 3:56 PM, Tom Lendacky <thomas.lendacky@amd.com> wrote:
>>>>> For AMD processors that support PAT, set the write-protect cache mode
>>>>> (_PAGE_CACHE_MODE_WP) entry to the actual write-protect value (x05).
>>>>
>>>> What's the purpose of using the WP memory type?
>>>
>>> The WP memory type is used for encrypting or decrypting data "in place".
>>> The use of the WP on the source data will prevent any of the source
>>> data from being cached.  Refer to section 7.10.8 "Encrypt-in-Place" in
>>> the AMD64 APM link provided in the cover letter.
>>>
>>> This memory type will be used in subsequent patches for this purpose.
>>
>> OK.
>>
>> Why AMD-only?  I thought Intel supported WP, too.
>
> Just me being conservative. If there aren't any objections from the
> Intel folks about it we can remove the vendor check and just set it.

I think there are some errata that will cause high PAT references to
incorrectly reference the low parts of the table, but I don't recall
any that go the other way around.  So merely setting WP in a high
entry should be harmless unless something tries to use it.

>
> Thanks,
> Tom
>
>>
>> --Andy
>>



-- 
Andy Lutomirski
AMA Capital Management, LLC

Re: [RFC PATCH v1 01/18] x86: Set the write-protect cache mode for AMD processors

[Borislav Petkov]

On Wed, Apr 27, 2016 at 08:12:56AM -0700, Andy Lutomirski wrote:
> I think there are some errata

Isn't that addressed by the first branch of the if-test in pat_init():

        if ((c->x86_vendor == X86_VENDOR_INTEL) &&
            (((c->x86 == 0x6) && (c->x86_model <= 0xd)) ||
             ((c->x86 == 0xf) && (c->x86_model <= 0x6)))) {


-- 
Regards/Gruss,
    Boris.

ECO tip #101: Trim your mails when you reply.

Re: [RFC PATCH v1 01/18] x86: Set the write-protect cache mode for AMD processors

[Andy Lutomirski]

On Wed, Apr 27, 2016 at 8:31 AM, Borislav Petkov <bp@alien8.de> wrote:
> On Wed, Apr 27, 2016 at 08:12:56AM -0700, Andy Lutomirski wrote:
>> I think there are some errata
>
> Isn't that addressed by the first branch of the if-test in pat_init():
>
>         if ((c->x86_vendor == X86_VENDOR_INTEL) &&
>             (((c->x86 == 0x6) && (c->x86_model <= 0xd)) ||
>              ((c->x86 == 0xf) && (c->x86_model <= 0x6)))) {
>

That's the intent, but I'm unconvinced that it's complete.  The reason
that WT is in slot 7 is that if it accidentally ends up using the slot
3 entry instead of 7 (e.g. if a 2M page gets confused due to an
erratum we didn't handle or similar), then it falls back to UC, which
is safe.

But this is mostly moot in this case.  There is no safe fallback for
WP, but it doesn't really matter, because no one will actually try to
use it except on a system will full PAT support anyway.  So I'm not
really concerned.

>
> --
> Regards/Gruss,
>     Boris.
>
> ECO tip #101: Trim your mails when you reply.



-- 
Andy Lutomirski
AMA Capital Management, LLC

[RFC PATCH v1 02/18] x86: Secure Memory Encryption (SME) build enablement

[Tom Lendacky]

Provide the Kconfig support to build the SME support in the kernel.

Signed-off-by: Tom Lendacky <thomas.lendacky@amd.com>
---
 arch/x86/Kconfig |    9 +++++++++
 1 file changed, 9 insertions(+)

diff --git a/arch/x86/Kconfig b/arch/x86/Kconfig
index 7bb1574..13249b5 100644
--- a/arch/x86/Kconfig
+++ b/arch/x86/Kconfig
@@ -1356,6 +1356,15 @@ config X86_DIRECT_GBPAGES
 	  supports them), so don't confuse the user by printing
 	  that we have them enabled.
 
+config AMD_MEM_ENCRYPT
+	bool "Secure Memory Encryption support for AMD"
+	depends on X86_64 && CPU_SUP_AMD
+	---help---
+	  Say yes to enable the encryption of system memory. This requires
+	  an AMD processor that supports Secure Memory Encryption (SME).
+	  The encryption of system memory is disabled by default but can be
+	  enabled with the mem_encrypt=on command line option.
+
 # Common NUMA Features
 config NUMA
 	bool "Numa Memory Allocation and Scheduler Support"

Re: [RFC PATCH v1 02/18] x86: Secure Memory Encryption (SME) build enablement

[Pavel Machek]

On Tue 2016-04-26 17:56:14, Tom Lendacky wrote:
> Provide the Kconfig support to build the SME support in the kernel.


Probably should go last in the series?

> Signed-off-by: Tom Lendacky <thomas.lendacky@amd.com>
> ---
>  arch/x86/Kconfig |    9 +++++++++
>  1 file changed, 9 insertions(+)
> 
> diff --git a/arch/x86/Kconfig b/arch/x86/Kconfig
> index 7bb1574..13249b5 100644
> --- a/arch/x86/Kconfig
> +++ b/arch/x86/Kconfig
> @@ -1356,6 +1356,15 @@ config X86_DIRECT_GBPAGES
>  	  supports them), so don't confuse the user by printing
>  	  that we have them enabled.
>  
> +config AMD_MEM_ENCRYPT
> +	bool "Secure Memory Encryption support for AMD"
> +	depends on X86_64 && CPU_SUP_AMD
> +	---help---
> +	  Say yes to enable the encryption of system memory. This requires
> +	  an AMD processor that supports Secure Memory Encryption (SME).
> +	  The encryption of system memory is disabled by default but can be
> +	  enabled with the mem_encrypt=on command line option.
> +
>  # Common NUMA Features
>  config NUMA
>  	bool "Numa Memory Allocation and Scheduler Support"

-- 
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html

Re: [RFC PATCH v1 02/18] x86: Secure Memory Encryption (SME) build enablement

[Tom Lendacky]

On 03/22/2016 08:01 AM, Pavel Machek wrote:
> On Tue 2016-04-26 17:56:14, Tom Lendacky wrote:
>> Provide the Kconfig support to build the SME support in the kernel.
> 
> 
> Probably should go last in the series?

Yeah, I've seen arguments both ways for this. Doing it early
allows compiling and testing with it enabled and doing it late
doesn't enable anything until it's all there. I just chose the
former.

Thanks,
Tom

> 
>> Signed-off-by: Tom Lendacky <thomas.lendacky@amd.com>
>> ---
>>  arch/x86/Kconfig |    9 +++++++++
>>  1 file changed, 9 insertions(+)
>>
>> diff --git a/arch/x86/Kconfig b/arch/x86/Kconfig
>> index 7bb1574..13249b5 100644
>> --- a/arch/x86/Kconfig
>> +++ b/arch/x86/Kconfig
>> @@ -1356,6 +1356,15 @@ config X86_DIRECT_GBPAGES
>>  	  supports them), so don't confuse the user by printing
>>  	  that we have them enabled.
>>  
>> +config AMD_MEM_ENCRYPT
>> +	bool "Secure Memory Encryption support for AMD"
>> +	depends on X86_64 && CPU_SUP_AMD
>> +	---help---
>> +	  Say yes to enable the encryption of system memory. This requires
>> +	  an AMD processor that supports Secure Memory Encryption (SME).
>> +	  The encryption of system memory is disabled by default but can be
>> +	  enabled with the mem_encrypt=on command line option.
>> +
>>  # Common NUMA Features
>>  config NUMA
>>  	bool "Numa Memory Allocation and Scheduler Support"
> 

Re: [RFC PATCH v1 02/18] x86: Secure Memory Encryption (SME) build enablement

[Pavel Machek]

On Wed 2016-04-27 10:17:36, Tom Lendacky wrote:
> On 03/22/2016 08:01 AM, Pavel Machek wrote:
> > On Tue 2016-04-26 17:56:14, Tom Lendacky wrote:
> >> Provide the Kconfig support to build the SME support in the kernel.
> > 
> > 
> > Probably should go last in the series?
> 
> Yeah, I've seen arguments both ways for this. Doing it early
> allows compiling and testing with it enabled and doing it late
> doesn't enable anything until it's all there. I just chose the
> former.

Doing it early will break bisect, right?
							Pavel

> >> +config AMD_MEM_ENCRYPT
> >> +	bool "Secure Memory Encryption support for AMD"
> >> +	depends on X86_64 && CPU_SUP_AMD
> >> +	---help---
> >> +	  Say yes to enable the encryption of system memory. This requires
> >> +	  an AMD processor that supports Secure Memory Encryption (SME).
> >> +	  The encryption of system memory is disabled by default but can be
> >> +	  enabled with the mem_encrypt=on command line option.
> >> +
> >>  # Common NUMA Features
> >>  config NUMA
> >>  	bool "Numa Memory Allocation and Scheduler Support"
> > 

-- 
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html

Re: [RFC PATCH v1 02/18] x86: Secure Memory Encryption (SME) build enablement

[Borislav Petkov]

On Wed, Apr 27, 2016 at 05:30:10PM +0200, Pavel Machek wrote:
> Doing it early will break bisect, right?

How exactly? Please do tell.

-- 
Regards/Gruss,
    Boris.

ECO tip #101: Trim your mails when you reply.

Re: [RFC PATCH v1 02/18] x86: Secure Memory Encryption (SME) build enablement

[Pavel Machek]

On Wed 2016-04-27 17:41:40, Borislav Petkov wrote:
> On Wed, Apr 27, 2016 at 05:30:10PM +0200, Pavel Machek wrote:
> > Doing it early will break bisect, right?
> 
> How exactly? Please do tell.

Hey look, SME slowed down 30% since being initially merged into
kernel!
									Pavel
-- 
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html

Re: [RFC PATCH v1 02/18] x86: Secure Memory Encryption (SME) build enablement

[Robin Murphy]

On 27/04/16 17:41, Pavel Machek wrote:
> On Wed 2016-04-27 17:41:40, Borislav Petkov wrote:
>> On Wed, Apr 27, 2016 at 05:30:10PM +0200, Pavel Machek wrote:
>>> Doing it early will break bisect, right?
>>
>> How exactly? Please do tell.
>
> Hey look, SME slowed down 30% since being initially merged into
> kernel!

As opposed to "well, bisection shows these n+1 complicated changes are 
all fine and the crash is down to this Kconfig patch", presumably. I'm 
sure we all love spending a whole afternoon only to find that, right? :P

Robin.

> 									Pavel
>

Re: [RFC PATCH v1 02/18] x86: Secure Memory Encryption (SME) build enablement

[Borislav Petkov]

On Wed, Apr 27, 2016 at 06:41:37PM +0200, Pavel Machek wrote:
> Hey look, SME slowed down 30% since being initially merged into
> kernel!

How is that breaking bisection?

-- 
Regards/Gruss,
    Boris.

ECO tip #101: Trim your mails when you reply.

[RFC PATCH v1 03/18] x86: Secure Memory Encryption (SME) support

[Tom Lendacky]

Provide support for Secure Memory Encryption (SME). This initial support
defines the memory encryption mask as a variable for quick access and an
accessor for retrieving the number of physical addressing bits lost if
SME is enabled.

Signed-off-by: Tom Lendacky <thomas.lendacky@amd.com>
---
 arch/x86/include/asm/mem_encrypt.h |   37 ++++++++++++++++++++++++++++++++++++
 arch/x86/kernel/Makefile           |    2 ++
 arch/x86/kernel/mem_encrypt.S      |   29 ++++++++++++++++++++++++++++
 arch/x86/kernel/x8664_ksyms_64.c   |    6 ++++++
 4 files changed, 74 insertions(+)
 create mode 100644 arch/x86/include/asm/mem_encrypt.h
 create mode 100644 arch/x86/kernel/mem_encrypt.S

diff --git a/arch/x86/include/asm/mem_encrypt.h b/arch/x86/include/asm/mem_encrypt.h
new file mode 100644
index 0000000..747fc52
--- /dev/null
+++ b/arch/x86/include/asm/mem_encrypt.h
@@ -0,0 +1,37 @@
+/*
+ * AMD Memory Encryption Support
+ *
+ * Copyright (C) 2016 Advanced Micro Devices, Inc.
+ *
+ * Author: Tom Lendacky <thomas.lendacky@amd.com>
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License version 2 as
+ * published by the Free Software Foundation.
+ */
+
+#ifndef __X86_MEM_ENCRYPT_H__
+#define __X86_MEM_ENCRYPT_H__
+
+#ifndef __ASSEMBLY__
+
+#ifdef CONFIG_AMD_MEM_ENCRYPT
+
+extern unsigned long sme_me_mask;
+
+u8 sme_get_me_loss(void);
+
+#else	/* !CONFIG_AMD_MEM_ENCRYPT */
+
+#define sme_me_mask		0UL
+
+static inline u8 sme_get_me_loss(void)
+{
+	return 0;
+}
+
+#endif	/* CONFIG_AMD_MEM_ENCRYPT */
+
+#endif	/* __ASSEMBLY__ */
+
+#endif	/* __X86_MEM_ENCRYPT_H__ */
diff --git a/arch/x86/kernel/Makefile b/arch/x86/kernel/Makefile
index 9abf855..11536d9 100644
--- a/arch/x86/kernel/Makefile
+++ b/arch/x86/kernel/Makefile
@@ -126,6 +126,8 @@ obj-$(CONFIG_EFI)			+= sysfb_efi.o
 obj-$(CONFIG_PERF_EVENTS)		+= perf_regs.o
 obj-$(CONFIG_TRACING)			+= tracepoint.o
 
+obj-y					+= mem_encrypt.o
+
 ###
 # 64 bit specific files
 ifeq ($(CONFIG_X86_64),y)
diff --git a/arch/x86/kernel/mem_encrypt.S b/arch/x86/kernel/mem_encrypt.S
new file mode 100644
index 0000000..ef7f325
--- /dev/null
+++ b/arch/x86/kernel/mem_encrypt.S
@@ -0,0 +1,29 @@
+/*
+ * AMD Memory Encryption Support
+ *
+ * Copyright (C) 2016 Advanced Micro Devices, Inc.
+ *
+ * Author: Tom Lendacky <thomas.lendacky@amd.com>
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License version 2 as
+ * published by the Free Software Foundation.
+ */
+
+#include <linux/linkage.h>
+
+	.text
+	.code64
+ENTRY(sme_get_me_loss)
+	xor	%rax, %rax
+	mov	sme_me_loss(%rip), %al
+	ret
+ENDPROC(sme_get_me_loss)
+
+	.data
+	.align 16
+ENTRY(sme_me_mask)
+	.quad	0x0000000000000000
+sme_me_loss:
+	.byte	0x00
+	.align	8
diff --git a/arch/x86/kernel/x8664_ksyms_64.c b/arch/x86/kernel/x8664_ksyms_64.c
index cd05942..72cb689 100644
--- a/arch/x86/kernel/x8664_ksyms_64.c
+++ b/arch/x86/kernel/x8664_ksyms_64.c
@@ -11,6 +11,7 @@
 #include <asm/uaccess.h>
 #include <asm/desc.h>
 #include <asm/ftrace.h>
+#include <asm/mem_encrypt.h>
 
 #ifdef CONFIG_FUNCTION_TRACER
 /* mcount and __fentry__ are defined in assembly */
@@ -79,3 +80,8 @@ EXPORT_SYMBOL(native_load_gs_index);
 EXPORT_SYMBOL(___preempt_schedule);
 EXPORT_SYMBOL(___preempt_schedule_notrace);
 #endif
+
+#ifdef CONFIG_AMD_MEM_ENCRYPT
+EXPORT_SYMBOL_GPL(sme_me_mask);
+EXPORT_SYMBOL_GPL(sme_get_me_loss);
+#endif

Re: [RFC PATCH v1 03/18] x86: Secure Memory Encryption (SME) support

[Pavel Machek]

On Tue 2016-04-26 17:56:26, Tom Lendacky wrote:
> Provide support for Secure Memory Encryption (SME). This initial support
> defines the memory encryption mask as a variable for quick access and an
> accessor for retrieving the number of physical addressing bits lost if
> SME is enabled.
> 
> Signed-off-by: Tom Lendacky <thomas.lendacky@amd.com>
> ---
>  arch/x86/include/asm/mem_encrypt.h |   37 ++++++++++++++++++++++++++++++++++++
>  arch/x86/kernel/Makefile           |    2 ++
>  arch/x86/kernel/mem_encrypt.S      |   29 ++++++++++++++++++++++++++++
>  arch/x86/kernel/x8664_ksyms_64.c   |    6 ++++++
>  4 files changed, 74 insertions(+)
>  create mode 100644 arch/x86/include/asm/mem_encrypt.h
>  create mode 100644 arch/x86/kernel/mem_encrypt.S
> 
> index 0000000..ef7f325
> --- /dev/null
> +++ b/arch/x86/kernel/mem_encrypt.S
> @@ -0,0 +1,29 @@
> +/*
> + * AMD Memory Encryption Support
> + *
> + * Copyright (C) 2016 Advanced Micro Devices, Inc.
> + *
> + * Author: Tom Lendacky <thomas.lendacky@amd.com>
> + *
> + * This program is free software; you can redistribute it and/or modify
> + * it under the terms of the GNU General Public License version 2 as
> + * published by the Free Software Foundation.
> + */
> +
> +#include <linux/linkage.h>
> +
> +	.text
> +	.code64
> +ENTRY(sme_get_me_loss)
> +	xor	%rax, %rax
> +	mov	sme_me_loss(%rip), %al
> +	ret
> +ENDPROC(sme_get_me_loss)

Does this really need to be implemented in assembly?


-- 
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html

Re: [RFC PATCH v1 03/18] x86: Secure Memory Encryption (SME) support

[Tom Lendacky]

On 03/22/2016 08:03 AM, Pavel Machek wrote:
> On Tue 2016-04-26 17:56:26, Tom Lendacky wrote:
>> Provide support for Secure Memory Encryption (SME). This initial support
>> defines the memory encryption mask as a variable for quick access and an
>> accessor for retrieving the number of physical addressing bits lost if
>> SME is enabled.
>>
>> Signed-off-by: Tom Lendacky <thomas.lendacky@amd.com>
>> ---
>>  arch/x86/include/asm/mem_encrypt.h |   37 ++++++++++++++++++++++++++++++++++++
>>  arch/x86/kernel/Makefile           |    2 ++
>>  arch/x86/kernel/mem_encrypt.S      |   29 ++++++++++++++++++++++++++++
>>  arch/x86/kernel/x8664_ksyms_64.c   |    6 ++++++
>>  4 files changed, 74 insertions(+)
>>  create mode 100644 arch/x86/include/asm/mem_encrypt.h
>>  create mode 100644 arch/x86/kernel/mem_encrypt.S
>>
>> index 0000000..ef7f325
>> --- /dev/null
>> +++ b/arch/x86/kernel/mem_encrypt.S
>> @@ -0,0 +1,29 @@
>> +/*
>> + * AMD Memory Encryption Support
>> + *
>> + * Copyright (C) 2016 Advanced Micro Devices, Inc.
>> + *
>> + * Author: Tom Lendacky <thomas.lendacky@amd.com>
>> + *
>> + * This program is free software; you can redistribute it and/or modify
>> + * it under the terms of the GNU General Public License version 2 as
>> + * published by the Free Software Foundation.
>> + */
>> +
>> +#include <linux/linkage.h>
>> +
>> +	.text
>> +	.code64
>> +ENTRY(sme_get_me_loss)
>> +	xor	%rax, %rax
>> +	mov	sme_me_loss(%rip), %al
>> +	ret
>> +ENDPROC(sme_get_me_loss)
> 
> Does this really need to be implemented in assembly?

That particular routine probably doesn't need to be in assembly. But
since it was such a simple routine I put it there because a later
patch derives the value in this file.

Thanks,
Tom

> 
> 

[RFC PATCH v1 04/18] x86: Add the Secure Memory Encryption cpu feature

[Tom Lendacky]

Update the cpu features to include identifying and reporting on the
Secure Memory Encryption feature.

Signed-off-by: Tom Lendacky <thomas.lendacky@amd.com>
---
 arch/x86/include/asm/cpufeature.h  |    1 +
 arch/x86/include/asm/cpufeatures.h |    5 ++++-
 arch/x86/kernel/cpu/scattered.c    |    1 +
 3 files changed, 6 insertions(+), 1 deletion(-)

diff --git a/arch/x86/include/asm/cpufeature.h b/arch/x86/include/asm/cpufeature.h
index 07c942d..e27e352 100644
--- a/arch/x86/include/asm/cpufeature.h
+++ b/arch/x86/include/asm/cpufeature.h
@@ -27,6 +27,7 @@ enum cpuid_leafs
 	CPUID_6_EAX,
 	CPUID_8000_000A_EDX,
 	CPUID_7_ECX,
+	CPUID_8000_001F_EAX,
 };
 
 #ifdef CONFIG_X86_FEATURE_NAMES
diff --git a/arch/x86/include/asm/cpufeatures.h b/arch/x86/include/asm/cpufeatures.h
index 47b5056..4aea205 100644
--- a/arch/x86/include/asm/cpufeatures.h
+++ b/arch/x86/include/asm/cpufeatures.h
@@ -12,7 +12,7 @@
 /*
  * Defines x86 CPU feature bits
  */
-#define NCAPINTS	17	/* N 32-bit words worth of info */
+#define NCAPINTS	18	/* N 32-bit words worth of info */
 #define NBUGINTS	1	/* N 32-bit bug flags */
 
 /*
@@ -282,6 +282,9 @@
 #define X86_FEATURE_PKU		(16*32+ 3) /* Protection Keys for Userspace */
 #define X86_FEATURE_OSPKE	(16*32+ 4) /* OS Protection Keys Enable */
 
+/* AMD SME Feature Identification, CPUID level 0x8000001f (eax), word 17 */
+#define X86_FEATURE_SME		(17*32+ 0) /* Secure Memory Encryption support */
+
 /*
  * BUG word(s)
  */
diff --git a/arch/x86/kernel/cpu/scattered.c b/arch/x86/kernel/cpu/scattered.c
index 8cb57df..d86d9a5 100644
--- a/arch/x86/kernel/cpu/scattered.c
+++ b/arch/x86/kernel/cpu/scattered.c
@@ -37,6 +37,7 @@ void init_scattered_cpuid_features(struct cpuinfo_x86 *c)
 		{ X86_FEATURE_HW_PSTATE,	CR_EDX, 7, 0x80000007, 0 },
 		{ X86_FEATURE_CPB,		CR_EDX, 9, 0x80000007, 0 },
 		{ X86_FEATURE_PROC_FEEDBACK,	CR_EDX,11, 0x80000007, 0 },
+		{ X86_FEATURE_SME,		CR_EAX, 0, 0x8000001f, 0 },
 		{ 0, 0, 0, 0, 0 }
 	};
 

[RFC PATCH v1 05/18] x86: Handle reduction in physical address size with SME

[Tom Lendacky]

When System Memory Encryption (SME) is enabled, the physical address
space is reduced. Adjust the x86_phys_bits value to reflect this
reduction.

Signed-off-by: Tom Lendacky <thomas.lendacky@amd.com>
---
 arch/x86/kernel/cpu/common.c |    2 ++
 1 file changed, 2 insertions(+)

diff --git a/arch/x86/kernel/cpu/common.c b/arch/x86/kernel/cpu/common.c
index 6bfa36d..b49e7fc 100644
--- a/arch/x86/kernel/cpu/common.c
+++ b/arch/x86/kernel/cpu/common.c
@@ -43,6 +43,7 @@
 #include <asm/pat.h>
 #include <asm/microcode.h>
 #include <asm/microcode_intel.h>
+#include <asm/mem_encrypt.h>
 
 #ifdef CONFIG_X86_LOCAL_APIC
 #include <asm/uv/uv.h>
@@ -722,6 +723,7 @@ void get_cpu_cap(struct cpuinfo_x86 *c)
 
 		c->x86_virt_bits = (eax >> 8) & 0xff;
 		c->x86_phys_bits = eax & 0xff;
+		c->x86_phys_bits -= sme_get_me_loss();
 		c->x86_capability[CPUID_8000_0008_EBX] = ebx;
 	}
 #ifdef CONFIG_X86_32

[RFC PATCH v1 06/18] x86: Provide general kernel support for memory encryption

[Tom Lendacky]

Adding general kernel support for memory encryption includes:
- Modify and create some page table macros to include the Secure Memory
  Encryption (SME) memory encryption mask
- Update kernel boot support to call an SME routine that checks for and
  sets the SME capability (the SME routine will grow later and for now
  is just a stub routine)
- Update kernel boot support to call an SME routine that encrypts the
  kernel (the SME routine will grow later and for now is just a stub
  routine)
- Provide an SME initialization routine to update the protection map with
  the memory encryption mask so that it is used by default

Signed-off-by: Tom Lendacky <thomas.lendacky@amd.com>
---
 arch/x86/include/asm/fixmap.h        |    7 ++++++
 arch/x86/include/asm/mem_encrypt.h   |   18 +++++++++++++++
 arch/x86/include/asm/pgtable_types.h |   41 ++++++++++++++++++++++-----------
 arch/x86/include/asm/processor.h     |    3 ++
 arch/x86/kernel/espfix_64.c          |    2 +-
 arch/x86/kernel/head64.c             |   10 ++++++--
 arch/x86/kernel/head_64.S            |   42 ++++++++++++++++++++++++++--------
 arch/x86/kernel/machine_kexec_64.c   |    2 +-
 arch/x86/kernel/mem_encrypt.S        |    8 ++++++
 arch/x86/mm/Makefile                 |    1 +
 arch/x86/mm/fault.c                  |    5 ++--
 arch/x86/mm/ioremap.c                |    3 ++
 arch/x86/mm/kasan_init_64.c          |    4 ++-
 arch/x86/mm/mem_encrypt.c            |   30 ++++++++++++++++++++++++
 arch/x86/mm/pageattr.c               |    3 ++
 15 files changed, 145 insertions(+), 34 deletions(-)
 create mode 100644 arch/x86/mm/mem_encrypt.c

diff --git a/arch/x86/include/asm/fixmap.h b/arch/x86/include/asm/fixmap.h
index 8554f96..83e91f0 100644
--- a/arch/x86/include/asm/fixmap.h
+++ b/arch/x86/include/asm/fixmap.h
@@ -153,6 +153,13 @@ static inline void __set_fixmap(enum fixed_addresses idx,
 }
 #endif
 
+/*
+ * Fixmap settings used with memory encryption
+ *   - FIXMAP_PAGE_NOCACHE is used for MMIO so make sure the memory
+ *     encryption mask is not part of the page attributes
+ */
+#define FIXMAP_PAGE_NOCACHE PAGE_KERNEL_IO_NOCACHE
+
 #include <asm-generic/fixmap.h>
 
 #define __late_set_fixmap(idx, phys, flags) __set_fixmap(idx, phys, flags)
diff --git a/arch/x86/include/asm/mem_encrypt.h b/arch/x86/include/asm/mem_encrypt.h
index 747fc52..9f3e762 100644
--- a/arch/x86/include/asm/mem_encrypt.h
+++ b/arch/x86/include/asm/mem_encrypt.h
@@ -15,12 +15,21 @@
 
 #ifndef __ASSEMBLY__
 
+#include <linux/init.h>
+
 #ifdef CONFIG_AMD_MEM_ENCRYPT
 
 extern unsigned long sme_me_mask;
 
 u8 sme_get_me_loss(void);
 
+void __init sme_early_init(void);
+
+#define __sme_pa(x)		(__pa((x)) | sme_me_mask)
+#define __sme_pa_nodebug(x)	(__pa_nodebug((x)) | sme_me_mask)
+
+#define __sme_va(x)		(__va((x) & ~sme_me_mask))
+
 #else	/* !CONFIG_AMD_MEM_ENCRYPT */
 
 #define sme_me_mask		0UL
@@ -30,6 +39,15 @@ static inline u8 sme_get_me_loss(void)
 	return 0;
 }
 
+static inline void __init sme_early_init(void)
+{
+}
+
+#define __sme_pa		__pa
+#define __sme_pa_nodebug	__pa_nodebug
+
+#define __sme_va		__va
+
 #endif	/* CONFIG_AMD_MEM_ENCRYPT */
 
 #endif	/* __ASSEMBLY__ */
diff --git a/arch/x86/include/asm/pgtable_types.h b/arch/x86/include/asm/pgtable_types.h
index 7b5efe2..fda7877 100644
--- a/arch/x86/include/asm/pgtable_types.h
+++ b/arch/x86/include/asm/pgtable_types.h
@@ -3,6 +3,7 @@
 
 #include <linux/const.h>
 #include <asm/page_types.h>
+#include <asm/mem_encrypt.h>
 
 #define FIRST_USER_ADDRESS	0UL
 
@@ -115,9 +116,9 @@
 
 #define _PAGE_PROTNONE	(_AT(pteval_t, 1) << _PAGE_BIT_PROTNONE)
 
-#define _PAGE_TABLE	(_PAGE_PRESENT | _PAGE_RW | _PAGE_USER |	\
+#define __PAGE_TABLE	(_PAGE_PRESENT | _PAGE_RW | _PAGE_USER |	\
 			 _PAGE_ACCESSED | _PAGE_DIRTY)
-#define _KERNPG_TABLE	(_PAGE_PRESENT | _PAGE_RW | _PAGE_ACCESSED |	\
+#define __KERNPG_TABLE	(_PAGE_PRESENT | _PAGE_RW | _PAGE_ACCESSED |	\
 			 _PAGE_DIRTY)
 
 /*
@@ -185,18 +186,30 @@ enum page_cache_mode {
 #define __PAGE_KERNEL_IO		(__PAGE_KERNEL)
 #define __PAGE_KERNEL_IO_NOCACHE	(__PAGE_KERNEL_NOCACHE)
 
-#define PAGE_KERNEL			__pgprot(__PAGE_KERNEL)
-#define PAGE_KERNEL_RO			__pgprot(__PAGE_KERNEL_RO)
-#define PAGE_KERNEL_EXEC		__pgprot(__PAGE_KERNEL_EXEC)
-#define PAGE_KERNEL_RX			__pgprot(__PAGE_KERNEL_RX)
-#define PAGE_KERNEL_NOCACHE		__pgprot(__PAGE_KERNEL_NOCACHE)
-#define PAGE_KERNEL_LARGE		__pgprot(__PAGE_KERNEL_LARGE)
-#define PAGE_KERNEL_LARGE_EXEC		__pgprot(__PAGE_KERNEL_LARGE_EXEC)
-#define PAGE_KERNEL_VSYSCALL		__pgprot(__PAGE_KERNEL_VSYSCALL)
-#define PAGE_KERNEL_VVAR		__pgprot(__PAGE_KERNEL_VVAR)
-
-#define PAGE_KERNEL_IO			__pgprot(__PAGE_KERNEL_IO)
-#define PAGE_KERNEL_IO_NOCACHE		__pgprot(__PAGE_KERNEL_IO_NOCACHE)
+#ifndef __ASSEMBLY__
+
+#define _PAGE_ENC	sme_me_mask
+
+/* Redefine macros to inclue the memory encryption mask */
+#define _PAGE_TABLE	(_PAGE_PRESENT | _PAGE_RW | _PAGE_USER |	\
+			 _PAGE_ACCESSED | _PAGE_DIRTY | _PAGE_ENC)
+#define _KERNPG_TABLE	(_PAGE_PRESENT | _PAGE_RW | _PAGE_ACCESSED |	\
+			 _PAGE_DIRTY | _PAGE_ENC)
+
+#define PAGE_KERNEL		__pgprot(__PAGE_KERNEL | _PAGE_ENC)
+#define PAGE_KERNEL_RO		__pgprot(__PAGE_KERNEL_RO | _PAGE_ENC)
+#define PAGE_KERNEL_EXEC	__pgprot(__PAGE_KERNEL_EXEC | _PAGE_ENC)
+#define PAGE_KERNEL_RX		__pgprot(__PAGE_KERNEL_RX | _PAGE_ENC)
+#define PAGE_KERNEL_NOCACHE	__pgprot(__PAGE_KERNEL_NOCACHE | _PAGE_ENC)
+#define PAGE_KERNEL_LARGE	__pgprot(__PAGE_KERNEL_LARGE | _PAGE_ENC)
+#define PAGE_KERNEL_LARGE_EXEC	__pgprot(__PAGE_KERNEL_LARGE_EXEC | _PAGE_ENC)
+#define PAGE_KERNEL_VSYSCALL	__pgprot(__PAGE_KERNEL_VSYSCALL | _PAGE_ENC)
+#define PAGE_KERNEL_VVAR	__pgprot(__PAGE_KERNEL_VVAR | _PAGE_ENC)
+
+#define PAGE_KERNEL_IO		__pgprot(__PAGE_KERNEL_IO)
+#define PAGE_KERNEL_IO_NOCACHE	__pgprot(__PAGE_KERNEL_IO_NOCACHE)
+
+#endif	/* __ASSEMBLY__ */
 
 /*         xwr */
 #define __P000	PAGE_NONE
diff --git a/arch/x86/include/asm/processor.h b/arch/x86/include/asm/processor.h
index 8d326e8..1fae737 100644
--- a/arch/x86/include/asm/processor.h
+++ b/arch/x86/include/asm/processor.h
@@ -22,6 +22,7 @@ struct vm86;
 #include <asm/nops.h>
 #include <asm/special_insns.h>
 #include <asm/fpu/types.h>
+#include <asm/mem_encrypt.h>
 
 #include <linux/personality.h>
 #include <linux/cache.h>
@@ -207,7 +208,7 @@ static inline void native_cpuid(unsigned int *eax, unsigned int *ebx,
 
 static inline void load_cr3(pgd_t *pgdir)
 {
-	write_cr3(__pa(pgdir));
+	write_cr3(__sme_pa(pgdir));
 }
 
 #ifdef CONFIG_X86_32
diff --git a/arch/x86/kernel/espfix_64.c b/arch/x86/kernel/espfix_64.c
index 4d38416..3385377 100644
--- a/arch/x86/kernel/espfix_64.c
+++ b/arch/x86/kernel/espfix_64.c
@@ -193,7 +193,7 @@ void init_espfix_ap(int cpu)
 
 	pte_p = pte_offset_kernel(&pmd, addr);
 	stack_page = page_address(alloc_pages_node(node, GFP_KERNEL, 0));
-	pte = __pte(__pa(stack_page) | (__PAGE_KERNEL_RO & ptemask));
+	pte = __pte(__pa(stack_page) | ((__PAGE_KERNEL_RO | _PAGE_ENC) & ptemask));
 	for (n = 0; n < ESPFIX_PTE_CLONES; n++)
 		set_pte(&pte_p[n*PTE_STRIDE], pte);
 
diff --git a/arch/x86/kernel/head64.c b/arch/x86/kernel/head64.c
index b72fb0b..3516f9f 100644
--- a/arch/x86/kernel/head64.c
+++ b/arch/x86/kernel/head64.c
@@ -28,6 +28,7 @@
 #include <asm/bootparam_utils.h>
 #include <asm/microcode.h>
 #include <asm/kasan.h>
+#include <asm/mem_encrypt.h>
 
 /*
  * Manage page tables very early on.
@@ -42,7 +43,7 @@ static void __init reset_early_page_tables(void)
 {
 	memset(early_level4_pgt, 0, sizeof(pgd_t)*(PTRS_PER_PGD-1));
 	next_early_pgt = 0;
-	write_cr3(__pa_nodebug(early_level4_pgt));
+	write_cr3(__sme_pa_nodebug(early_level4_pgt));
 }
 
 /* Create a new PMD entry */
@@ -54,7 +55,7 @@ int __init early_make_pgtable(unsigned long address)
 	pmdval_t pmd, *pmd_p;
 
 	/* Invalid address or early pgt is done ?  */
-	if (physaddr >= MAXMEM || read_cr3() != __pa_nodebug(early_level4_pgt))
+	if (physaddr >= MAXMEM || read_cr3() != __sme_pa_nodebug(early_level4_pgt))
 		return -1;
 
 again:
@@ -157,6 +158,11 @@ asmlinkage __visible void __init x86_64_start_kernel(char * real_mode_data)
 
 	clear_page(init_level4_pgt);
 
+	/* Update the early_pmd_flags with the memory encryption mask */
+	early_pmd_flags |= _PAGE_ENC;
+
+	sme_early_init();
+
 	kasan_early_init();
 
 	for (i = 0; i < NUM_EXCEPTION_VECTORS; i++)
diff --git a/arch/x86/kernel/head_64.S b/arch/x86/kernel/head_64.S
index 5df831e..0f3ad72 100644
--- a/arch/x86/kernel/head_64.S
+++ b/arch/x86/kernel/head_64.S
@@ -95,6 +95,13 @@ startup_64:
 	jnz	bad_address
 
 	/*
+	 * Enable memory encryption (if available). Add the memory encryption
+	 * mask to %rbp to include it in the the page table fixup.
+	 */
+	call	sme_enable
+	addq	sme_me_mask(%rip), %rbp
+
+	/*
 	 * Fixup the physical addresses in the page table
 	 */
 	addq	%rbp, early_level4_pgt + (L4_START_KERNEL*8)(%rip)
@@ -116,7 +123,8 @@ startup_64:
 	movq	%rdi, %rax
 	shrq	$PGDIR_SHIFT, %rax
 
-	leaq	(4096 + _KERNPG_TABLE)(%rbx), %rdx
+	leaq	(4096 + __KERNPG_TABLE)(%rbx), %rdx
+	addq	sme_me_mask(%rip), %rdx		/* Apply mem encryption mask */
 	movq	%rdx, 0(%rbx,%rax,8)
 	movq	%rdx, 8(%rbx,%rax,8)
 
@@ -133,6 +141,7 @@ startup_64:
 	movq	%rdi, %rax
 	shrq	$PMD_SHIFT, %rdi
 	addq	$(__PAGE_KERNEL_LARGE_EXEC & ~_PAGE_GLOBAL), %rax
+	addq	sme_me_mask(%rip), %rax		/* Apply mem encryption mask */
 	leaq	(_end - 1)(%rip), %rcx
 	shrq	$PMD_SHIFT, %rcx
 	subq	%rdi, %rcx
@@ -163,9 +172,19 @@ startup_64:
 	cmp	%r8, %rdi
 	jne	1b
 
-	/* Fixup phys_base */
+	/*
+	 * Fixup phys_base, remove the memory encryption mask from %rbp
+	 * to obtain the true physical address.
+	 */
+	subq	sme_me_mask(%rip), %rbp
 	addq	%rbp, phys_base(%rip)
 
+	/*
+	 * The page tables have been updated with the memory encryption mask,
+	 * so encrypt the kernel if memory encryption is active
+	 */
+	call	sme_encrypt_kernel
+
 	movq	$(early_level4_pgt - __START_KERNEL_map), %rax
 	jmp 1f
 ENTRY(secondary_startup_64)
@@ -189,6 +208,9 @@ ENTRY(secondary_startup_64)
 	movq	$(init_level4_pgt - __START_KERNEL_map), %rax
 1:
 
+	/* Add the memory encryption mask to RAX */
+	addq	sme_me_mask(%rip), %rax
+
 	/* Enable PAE mode and PGE */
 	movl	$(X86_CR4_PAE | X86_CR4_PGE), %ecx
 	movq	%rcx, %cr4
@@ -416,7 +438,7 @@ GLOBAL(name)
 	__INITDATA
 NEXT_PAGE(early_level4_pgt)
 	.fill	511,8,0
-	.quad	level3_kernel_pgt - __START_KERNEL_map + _PAGE_TABLE
+	.quad	level3_kernel_pgt - __START_KERNEL_map + __PAGE_TABLE
 
 NEXT_PAGE(early_dynamic_pgts)
 	.fill	512*EARLY_DYNAMIC_PAGE_TABLES,8,0
@@ -428,15 +450,15 @@ NEXT_PAGE(init_level4_pgt)
 	.fill	512,8,0
 #else
 NEXT_PAGE(init_level4_pgt)
-	.quad   level3_ident_pgt - __START_KERNEL_map + _KERNPG_TABLE
+	.quad   level3_ident_pgt - __START_KERNEL_map + __KERNPG_TABLE
 	.org    init_level4_pgt + L4_PAGE_OFFSET*8, 0
-	.quad   level3_ident_pgt - __START_KERNEL_map + _KERNPG_TABLE
+	.quad   level3_ident_pgt - __START_KERNEL_map + __KERNPG_TABLE
 	.org    init_level4_pgt + L4_START_KERNEL*8, 0
 	/* (2^48-(2*1024*1024*1024))/(2^39) = 511 */
-	.quad   level3_kernel_pgt - __START_KERNEL_map + _PAGE_TABLE
+	.quad   level3_kernel_pgt - __START_KERNEL_map + __PAGE_TABLE
 
 NEXT_PAGE(level3_ident_pgt)
-	.quad	level2_ident_pgt - __START_KERNEL_map + _KERNPG_TABLE
+	.quad	level2_ident_pgt - __START_KERNEL_map + __KERNPG_TABLE
 	.fill	511, 8, 0
 NEXT_PAGE(level2_ident_pgt)
 	/* Since I easily can, map the first 1G.
@@ -448,8 +470,8 @@ NEXT_PAGE(level2_ident_pgt)
 NEXT_PAGE(level3_kernel_pgt)
 	.fill	L3_START_KERNEL,8,0
 	/* (2^48-(2*1024*1024*1024)-((2^39)*511))/(2^30) = 510 */
-	.quad	level2_kernel_pgt - __START_KERNEL_map + _KERNPG_TABLE
-	.quad	level2_fixmap_pgt - __START_KERNEL_map + _PAGE_TABLE
+	.quad	level2_kernel_pgt - __START_KERNEL_map + __KERNPG_TABLE
+	.quad	level2_fixmap_pgt - __START_KERNEL_map + __PAGE_TABLE
 
 NEXT_PAGE(level2_kernel_pgt)
 	/*
@@ -467,7 +489,7 @@ NEXT_PAGE(level2_kernel_pgt)
 
 NEXT_PAGE(level2_fixmap_pgt)
 	.fill	506,8,0
-	.quad	level1_fixmap_pgt - __START_KERNEL_map + _PAGE_TABLE
+	.quad	level1_fixmap_pgt - __START_KERNEL_map + __PAGE_TABLE
 	/* 8MB reserved for vsyscalls + a 2MB hole = 4 + 1 entries */
 	.fill	5,8,0
 
diff --git a/arch/x86/kernel/machine_kexec_64.c b/arch/x86/kernel/machine_kexec_64.c
index ba7fbba..eb2faee 100644
--- a/arch/x86/kernel/machine_kexec_64.c
+++ b/arch/x86/kernel/machine_kexec_64.c
@@ -103,7 +103,7 @@ static int init_pgtable(struct kimage *image, unsigned long start_pgtable)
 	struct x86_mapping_info info = {
 		.alloc_pgt_page	= alloc_pgt_page,
 		.context	= image,
-		.pmd_flag	= __PAGE_KERNEL_LARGE_EXEC,
+		.pmd_flag	= __PAGE_KERNEL_LARGE_EXEC | _PAGE_ENC,
 	};
 	unsigned long mstart, mend;
 	pgd_t *level4p;
diff --git a/arch/x86/kernel/mem_encrypt.S b/arch/x86/kernel/mem_encrypt.S
index ef7f325..f2e0536 100644
--- a/arch/x86/kernel/mem_encrypt.S
+++ b/arch/x86/kernel/mem_encrypt.S
@@ -14,6 +14,14 @@
 
 	.text
 	.code64
+ENTRY(sme_enable)
+	ret
+ENDPROC(sme_enable)
+
+ENTRY(sme_encrypt_kernel)
+	ret
+ENDPROC(sme_encrypt_kernel)
+
 ENTRY(sme_get_me_loss)
 	xor	%rax, %rax
 	mov	sme_me_loss(%rip), %al
diff --git a/arch/x86/mm/Makefile b/arch/x86/mm/Makefile
index f989132..21f8ea3 100644
--- a/arch/x86/mm/Makefile
+++ b/arch/x86/mm/Makefile
@@ -39,3 +39,4 @@ obj-$(CONFIG_NUMA_EMU)		+= numa_emulation.o
 obj-$(CONFIG_X86_INTEL_MPX)	+= mpx.o
 obj-$(CONFIG_X86_INTEL_MEMORY_PROTECTION_KEYS) += pkeys.o
 
+obj-$(CONFIG_AMD_MEM_ENCRYPT)	+= mem_encrypt.o
diff --git a/arch/x86/mm/fault.c b/arch/x86/mm/fault.c
index 5ce1ed0..9e93a0d 100644
--- a/arch/x86/mm/fault.c
+++ b/arch/x86/mm/fault.c
@@ -23,6 +23,7 @@
 #include <asm/vsyscall.h>		/* emulate_vsyscall		*/
 #include <asm/vm86.h>			/* struct vm86			*/
 #include <asm/mmu_context.h>		/* vma_pkey()			*/
+#include <asm/mem_encrypt.h>		/* __sme_va()			*/
 
 #define CREATE_TRACE_POINTS
 #include <asm/trace/exceptions.h>
@@ -523,7 +524,7 @@ static int bad_address(void *p)
 
 static void dump_pagetable(unsigned long address)
 {
-	pgd_t *base = __va(read_cr3() & PHYSICAL_PAGE_MASK);
+	pgd_t *base = __sme_va(read_cr3() & PHYSICAL_PAGE_MASK);
 	pgd_t *pgd = base + pgd_index(address);
 	pud_t *pud;
 	pmd_t *pmd;
@@ -659,7 +660,7 @@ show_fault_oops(struct pt_regs *regs, unsigned long error_code,
 		pgd_t *pgd;
 		pte_t *pte;
 
-		pgd = __va(read_cr3() & PHYSICAL_PAGE_MASK);
+		pgd = __sme_va(read_cr3() & PHYSICAL_PAGE_MASK);
 		pgd += pgd_index(address);
 
 		pte = lookup_address_in_pgd(pgd, address, &level);
diff --git a/arch/x86/mm/ioremap.c b/arch/x86/mm/ioremap.c
index f089491..77dadf5 100644
--- a/arch/x86/mm/ioremap.c
+++ b/arch/x86/mm/ioremap.c
@@ -21,6 +21,7 @@
 #include <asm/tlbflush.h>
 #include <asm/pgalloc.h>
 #include <asm/pat.h>
+#include <asm/mem_encrypt.h>
 
 #include "physaddr.h"
 
@@ -424,7 +425,7 @@ static pte_t bm_pte[PAGE_SIZE/sizeof(pte_t)] __page_aligned_bss;
 static inline pmd_t * __init early_ioremap_pmd(unsigned long addr)
 {
 	/* Don't assume we're using swapper_pg_dir at this point */
-	pgd_t *base = __va(read_cr3());
+	pgd_t *base = __sme_va(read_cr3());
 	pgd_t *pgd = &base[pgd_index(addr)];
 	pud_t *pud = pud_offset(pgd, addr);
 	pmd_t *pmd = pmd_offset(pud, addr);
diff --git a/arch/x86/mm/kasan_init_64.c b/arch/x86/mm/kasan_init_64.c
index 1b1110f..7102408 100644
--- a/arch/x86/mm/kasan_init_64.c
+++ b/arch/x86/mm/kasan_init_64.c
@@ -68,7 +68,7 @@ static struct notifier_block kasan_die_notifier = {
 void __init kasan_early_init(void)
 {
 	int i;
-	pteval_t pte_val = __pa_nodebug(kasan_zero_page) | __PAGE_KERNEL;
+	pteval_t pte_val = __pa_nodebug(kasan_zero_page) | __PAGE_KERNEL | _PAGE_ENC;
 	pmdval_t pmd_val = __pa_nodebug(kasan_zero_pte) | _KERNPG_TABLE;
 	pudval_t pud_val = __pa_nodebug(kasan_zero_pmd) | _KERNPG_TABLE;
 
@@ -130,7 +130,7 @@ void __init kasan_init(void)
 	 */
 	memset(kasan_zero_page, 0, PAGE_SIZE);
 	for (i = 0; i < PTRS_PER_PTE; i++) {
-		pte_t pte = __pte(__pa(kasan_zero_page) | __PAGE_KERNEL_RO);
+		pte_t pte = __pte(__pa(kasan_zero_page) | __PAGE_KERNEL_RO | _PAGE_ENC);
 		set_pte(&kasan_zero_pte[i], pte);
 	}
 	/* Flush TLBs again to be sure that write protection applied. */
diff --git a/arch/x86/mm/mem_encrypt.c b/arch/x86/mm/mem_encrypt.c
new file mode 100644
index 0000000..00eb705
--- /dev/null
+++ b/arch/x86/mm/mem_encrypt.c
@@ -0,0 +1,30 @@
+/*
+ * AMD Memory Encryption Support
+ *
+ * Copyright (C) 2016 Advanced Micro Devices, Inc.
+ *
+ * Author: Tom Lendacky <thomas.lendacky@amd.com>
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License version 2 as
+ * published by the Free Software Foundation.
+ */
+
+#include <linux/init.h>
+#include <linux/mm.h>
+
+#include <asm/mem_encrypt.h>
+
+void __init sme_early_init(void)
+{
+	unsigned int i;
+
+	if (!sme_me_mask)
+		return;
+
+	__supported_pte_mask |= sme_me_mask;
+
+	/* Update the protection map with memory encryption mask */
+	for (i = 0; i < ARRAY_SIZE(protection_map); i++)
+		protection_map[i] = __pgprot(pgprot_val(protection_map[i]) | sme_me_mask);
+}
diff --git a/arch/x86/mm/pageattr.c b/arch/x86/mm/pageattr.c
index bbf462f..c055302 100644
--- a/arch/x86/mm/pageattr.c
+++ b/arch/x86/mm/pageattr.c
@@ -1976,6 +1976,9 @@ int kernel_map_pages_in_pgd(pgd_t *pgd, u64 pfn, unsigned long address,
 	if (!(page_flags & _PAGE_RW))
 		cpa.mask_clr = __pgprot(_PAGE_RW);
 
+	if (!(page_flags & _PAGE_ENC))
+		cpa.mask_clr = __pgprot(pgprot_val(cpa.mask_clr) | _PAGE_ENC);
+
 	cpa.mask_set = __pgprot(_PAGE_PRESENT | page_flags);
 
 	retval = __change_page_attr_set_clr(&cpa, 0);

[RFC PATCH v1 07/18] x86: Extend the early_memmap support with additional attrs

[Tom Lendacky]

Add to the early_memmap support to be able to specify encrypted and
un-encrypted mappings with and without write-protection. The use of
write-protection is necessary when encrypting data "in place". The
write-protect attribute is considered cacheable for loads, but not
stores. This implies that the hardware will never give the core a
dirty line with this memtype.

Signed-off-by: Tom Lendacky <thomas.lendacky@amd.com>
---
 arch/x86/include/asm/fixmap.h        |    9 +++++++++
 arch/x86/include/asm/pgtable_types.h |    8 ++++++++
 arch/x86/mm/ioremap.c                |   28 ++++++++++++++++++++++++++++
 include/asm-generic/early_ioremap.h  |    2 ++
 mm/early_ioremap.c                   |   15 +++++++++++++++
 5 files changed, 62 insertions(+)

diff --git a/arch/x86/include/asm/fixmap.h b/arch/x86/include/asm/fixmap.h
index 83e91f0..4d41878 100644
--- a/arch/x86/include/asm/fixmap.h
+++ b/arch/x86/include/asm/fixmap.h
@@ -160,6 +160,15 @@ static inline void __set_fixmap(enum fixed_addresses idx,
  */
 #define FIXMAP_PAGE_NOCACHE PAGE_KERNEL_IO_NOCACHE
 
+void __init *early_memremap_enc(resource_size_t phys_addr,
+				unsigned long size);
+void __init *early_memremap_enc_wp(resource_size_t phys_addr,
+				   unsigned long size);
+void __init *early_memremap_dec(resource_size_t phys_addr,
+				unsigned long size);
+void __init *early_memremap_dec_wp(resource_size_t phys_addr,
+				   unsigned long size);
+
 #include <asm-generic/fixmap.h>
 
 #define __late_set_fixmap(idx, phys, flags) __set_fixmap(idx, phys, flags)
diff --git a/arch/x86/include/asm/pgtable_types.h b/arch/x86/include/asm/pgtable_types.h
index fda7877..6291248 100644
--- a/arch/x86/include/asm/pgtable_types.h
+++ b/arch/x86/include/asm/pgtable_types.h
@@ -154,6 +154,7 @@ enum page_cache_mode {
 
 #define _PAGE_CACHE_MASK	(_PAGE_PAT | _PAGE_PCD | _PAGE_PWT)
 #define _PAGE_NOCACHE		(cachemode2protval(_PAGE_CACHE_MODE_UC))
+#define _PAGE_CACHE_WP		(cachemode2protval(_PAGE_CACHE_MODE_WP))
 
 #define PAGE_NONE	__pgprot(_PAGE_PROTNONE | _PAGE_ACCESSED)
 #define PAGE_SHARED	__pgprot(_PAGE_PRESENT | _PAGE_RW | _PAGE_USER | \
@@ -182,6 +183,7 @@ enum page_cache_mode {
 #define __PAGE_KERNEL_VVAR		(__PAGE_KERNEL_RO | _PAGE_USER)
 #define __PAGE_KERNEL_LARGE		(__PAGE_KERNEL | _PAGE_PSE)
 #define __PAGE_KERNEL_LARGE_EXEC	(__PAGE_KERNEL_EXEC | _PAGE_PSE)
+#define __PAGE_KERNEL_WP		(__PAGE_KERNEL | _PAGE_CACHE_WP)
 
 #define __PAGE_KERNEL_IO		(__PAGE_KERNEL)
 #define __PAGE_KERNEL_IO_NOCACHE	(__PAGE_KERNEL_NOCACHE)
@@ -196,6 +198,12 @@ enum page_cache_mode {
 #define _KERNPG_TABLE	(_PAGE_PRESENT | _PAGE_RW | _PAGE_ACCESSED |	\
 			 _PAGE_DIRTY | _PAGE_ENC)
 
+#define __PAGE_KERNEL_ENC	(__PAGE_KERNEL | _PAGE_ENC)
+#define __PAGE_KERNEL_ENC_WP	(__PAGE_KERNEL_WP | _PAGE_ENC)
+
+#define __PAGE_KERNEL_DEC	(__PAGE_KERNEL)
+#define __PAGE_KERNEL_DEC_WP	(__PAGE_KERNEL_WP)
+
 #define PAGE_KERNEL		__pgprot(__PAGE_KERNEL | _PAGE_ENC)
 #define PAGE_KERNEL_RO		__pgprot(__PAGE_KERNEL_RO | _PAGE_ENC)
 #define PAGE_KERNEL_EXEC	__pgprot(__PAGE_KERNEL_EXEC | _PAGE_ENC)
diff --git a/arch/x86/mm/ioremap.c b/arch/x86/mm/ioremap.c
index 77dadf5..14c7ed5 100644
--- a/arch/x86/mm/ioremap.c
+++ b/arch/x86/mm/ioremap.c
@@ -420,6 +420,34 @@ void unxlate_dev_mem_ptr(phys_addr_t phys, void *addr)
 	iounmap((void __iomem *)((unsigned long)addr & PAGE_MASK));
 }
 
+/* Remap memory with encryption */
+void __init *early_memremap_enc(resource_size_t phys_addr,
+				unsigned long size)
+{
+	return early_memremap_prot(phys_addr, size, __PAGE_KERNEL_ENC);
+}
+
+/* Remap memory with encryption and write-protected */
+void __init *early_memremap_enc_wp(resource_size_t phys_addr,
+				   unsigned long size)
+{
+	return early_memremap_prot(phys_addr, size, __PAGE_KERNEL_ENC_WP);
+}
+
+/* Remap memory without encryption */
+void __init *early_memremap_dec(resource_size_t phys_addr,
+				unsigned long size)
+{
+	return early_memremap_prot(phys_addr, size, __PAGE_KERNEL_DEC);
+}
+
+/* Remap memory without encryption and write-protected */
+void __init *early_memremap_dec_wp(resource_size_t phys_addr,
+				   unsigned long size)
+{
+	return early_memremap_prot(phys_addr, size, __PAGE_KERNEL_DEC_WP);
+}
+
 static pte_t bm_pte[PAGE_SIZE/sizeof(pte_t)] __page_aligned_bss;
 
 static inline pmd_t * __init early_ioremap_pmd(unsigned long addr)
diff --git a/include/asm-generic/early_ioremap.h b/include/asm-generic/early_ioremap.h
index 734ad4d..2edef8d 100644
--- a/include/asm-generic/early_ioremap.h
+++ b/include/asm-generic/early_ioremap.h
@@ -13,6 +13,8 @@ extern void *early_memremap(resource_size_t phys_addr,
 			    unsigned long size);
 extern void *early_memremap_ro(resource_size_t phys_addr,
 			       unsigned long size);
+extern void *early_memremap_prot(resource_size_t phys_addr,
+				 unsigned long size, unsigned long prot_val);
 extern void early_iounmap(void __iomem *addr, unsigned long size);
 extern void early_memunmap(void *addr, unsigned long size);
 
diff --git a/mm/early_ioremap.c b/mm/early_ioremap.c
index 6d5717b..d71b98b 100644
--- a/mm/early_ioremap.c
+++ b/mm/early_ioremap.c
@@ -226,6 +226,14 @@ early_memremap_ro(resource_size_t phys_addr, unsigned long size)
 }
 #endif
 
+void __init *
+early_memremap_prot(resource_size_t phys_addr, unsigned long size,
+		    unsigned long prot_val)
+{
+	return (__force void *)__early_ioremap(phys_addr, size,
+					       __pgprot(prot_val));
+}
+
 #define MAX_MAP_CHUNK	(NR_FIX_BTMAPS << PAGE_SHIFT)
 
 void __init copy_from_early_mem(void *dest, phys_addr_t src, unsigned long size)
@@ -267,6 +275,13 @@ early_memremap_ro(resource_size_t phys_addr, unsigned long size)
 	return (void *)phys_addr;
 }
 
+void __init *
+early_memremap_prot(resource_size_t phys_addr, unsigned long size,
+		    unsigned long prot_val)
+{
+	return (void *)phys_addr;
+}
+
 void __init early_iounmap(void __iomem *addr, unsigned long size)
 {
 }

[RFC PATCH v1 08/18] x86: Add support for early encryption/decryption of memory

[Tom Lendacky]

This adds support to be able to either encrypt or decrypt data during
the early stages of booting the kernel. This does not change the memory
encryption attribute - it is used for ensuring that data present in
either an encrypted or un-encrypted memory area is in the proper state
(for example the initrd will have been loaded by the boot loader and
will not be encrypted, but the memory that it resides in is marked as
encrypted).

Signed-off-by: Tom Lendacky <thomas.lendacky@amd.com>
---
 arch/x86/include/asm/mem_encrypt.h |   15 ++++++
 arch/x86/mm/mem_encrypt.c          |   89 ++++++++++++++++++++++++++++++++++++
 2 files changed, 104 insertions(+)

diff --git a/arch/x86/include/asm/mem_encrypt.h b/arch/x86/include/asm/mem_encrypt.h
index 9f3e762..2785493 100644
--- a/arch/x86/include/asm/mem_encrypt.h
+++ b/arch/x86/include/asm/mem_encrypt.h
@@ -23,6 +23,11 @@ extern unsigned long sme_me_mask;
 
 u8 sme_get_me_loss(void);
 
+void __init sme_early_mem_enc(resource_size_t paddr,
+			      unsigned long size);
+void __init sme_early_mem_dec(resource_size_t paddr,
+			      unsigned long size);
+
 void __init sme_early_init(void);
 
 #define __sme_pa(x)		(__pa((x)) | sme_me_mask)
@@ -39,6 +44,16 @@ static inline u8 sme_get_me_loss(void)
 	return 0;
 }
 
+static inline void __init sme_early_mem_enc(resource_size_t paddr,
+					    unsigned long size)
+{
+}
+
+static inline void __init sme_early_mem_dec(resource_size_t paddr,
+					    unsigned long size)
+{
+}
+
 static inline void __init sme_early_init(void)
 {
 }
diff --git a/arch/x86/mm/mem_encrypt.c b/arch/x86/mm/mem_encrypt.c
index 00eb705..5f19ede 100644
--- a/arch/x86/mm/mem_encrypt.c
+++ b/arch/x86/mm/mem_encrypt.c
@@ -14,6 +14,95 @@
 #include <linux/mm.h>
 
 #include <asm/mem_encrypt.h>
+#include <asm/tlbflush.h>
+#include <asm/fixmap.h>
+
+/* Buffer used for early in-place encryption by BSP, no locking needed */
+static char me_early_buffer[PAGE_SIZE] __aligned(PAGE_SIZE);
+
+void __init sme_early_mem_enc(resource_size_t paddr, unsigned long size)
+{
+	void *src, *dst;
+	size_t len;
+
+	if (!sme_me_mask)
+		return;
+
+	local_flush_tlb();
+	wbinvd();
+
+	/*
+	 * There are limited number of early mapping slots, so map (at most)
+	 * one page at time.
+	 */
+	while (size) {
+		len = min_t(size_t, sizeof(me_early_buffer), size);
+
+		/* Create a mapping for non-encrypted write-protected memory */
+		src = early_memremap_dec_wp(paddr, len);
+
+		/* Create a mapping for encrypted memory */
+		dst = early_memremap_enc(paddr, len);
+
+		/*
+		 * If a mapping can't be obtained to perform the encryption,
+		 * then encrypted access to that area will end up causing
+		 * a crash.
+		 */
+		BUG_ON(!src || !dst);
+
+		memcpy(me_early_buffer, src, len);
+		memcpy(dst, me_early_buffer, len);
+
+		early_memunmap(dst, len);
+		early_memunmap(src, len);
+
+		paddr += len;
+		size -= len;
+	}
+}
+
+void __init sme_early_mem_dec(resource_size_t paddr, unsigned long size)
+{
+	void *src, *dst;
+	size_t len;
+
+	if (!sme_me_mask)
+		return;
+
+	local_flush_tlb();
+	wbinvd();
+
+	/*
+	 * There are limited number of early mapping slots, so map (at most)
+	 * one page at time.
+	 */
+	while (size) {
+		len = min_t(size_t, sizeof(me_early_buffer), size);
+
+		/* Create a mapping for encrypted write-protected memory */
+		src = early_memremap_enc_wp(paddr, len);
+
+		/* Create a mapping for non-encrypted memory */
+		dst = early_memremap_dec(paddr, len);
+
+		/*
+		 * If a mapping can't be obtained to perform the decryption,
+		 * then un-encrypted access to that area will end up causing
+		 * a crash.
+		 */
+		BUG_ON(!src || !dst);
+
+		memcpy(me_early_buffer, src, len);
+		memcpy(dst, me_early_buffer, len);
+
+		early_memunmap(dst, len);
+		early_memunmap(src, len);
+
+		paddr += len;
+		size -= len;
+	}
+}
 
 void __init sme_early_init(void)
 {

[RFC PATCH v1 09/18] x86: Insure that memory areas are encrypted when possible

[Tom Lendacky]

Encrypt memory areas in place when possible (e.g. zero page, etc.) so
that special handling isn't needed afterwards.

Signed-off-by: Tom Lendacky <thomas.lendacky@amd.com>
---
 arch/x86/kernel/head64.c |   90 +++++++++++++++++++++++++++++++++++++++++++---
 arch/x86/kernel/setup.c  |    8 ++++
 2 files changed, 93 insertions(+), 5 deletions(-)

diff --git a/arch/x86/kernel/head64.c b/arch/x86/kernel/head64.c
index 3516f9f..ac3a2bf 100644
--- a/arch/x86/kernel/head64.c
+++ b/arch/x86/kernel/head64.c
@@ -47,12 +47,12 @@ static void __init reset_early_page_tables(void)
 }
 
 /* Create a new PMD entry */
-int __init early_make_pgtable(unsigned long address)
+static int __init __early_make_pgtable(unsigned long address, pmdval_t pmd)
 {
 	unsigned long physaddr = address - __PAGE_OFFSET;
 	pgdval_t pgd, *pgd_p;
 	pudval_t pud, *pud_p;
-	pmdval_t pmd, *pmd_p;
+	pmdval_t *pmd_p;
 
 	/* Invalid address or early pgt is done ?  */
 	if (physaddr >= MAXMEM || read_cr3() != __sme_pa_nodebug(early_level4_pgt))
@@ -94,12 +94,92 @@ again:
 		memset(pmd_p, 0, sizeof(*pmd_p) * PTRS_PER_PMD);
 		*pud_p = (pudval_t)pmd_p - __START_KERNEL_map + phys_base + _KERNPG_TABLE;
 	}
-	pmd = (physaddr & PMD_MASK) + early_pmd_flags;
 	pmd_p[pmd_index(address)] = pmd;
 
 	return 0;
 }
 
+int __init early_make_pgtable(unsigned long address)
+{
+	unsigned long physaddr = address - __PAGE_OFFSET;
+	pmdval_t pmd;
+
+	pmd = (physaddr & PMD_MASK) + early_pmd_flags;
+
+	return __early_make_pgtable(address, pmd);
+}
+
+static void __init create_unencrypted_mapping(void *address, unsigned long size)
+{
+	unsigned long physaddr = (unsigned long)address - __PAGE_OFFSET;
+	pmdval_t pmd_flags, pmd;
+
+	if (!sme_me_mask)
+		return;
+
+	/* Clear the encryption mask from the early_pmd_flags */
+	pmd_flags = early_pmd_flags & ~sme_me_mask;
+
+	do {
+		pmd = (physaddr & PMD_MASK) + pmd_flags;
+		__early_make_pgtable((unsigned long)address, pmd);
+
+		address += PMD_SIZE;
+		physaddr += PMD_SIZE;
+		size = (size < PMD_SIZE) ? 0 : size - PMD_SIZE;
+	} while (size);
+}
+
+static void __init __clear_mapping(unsigned long address)
+{
+	unsigned long physaddr = address - __PAGE_OFFSET;
+	pgdval_t pgd, *pgd_p;
+	pudval_t pud, *pud_p;
+	pmdval_t *pmd_p;
+
+	/* Invalid address or early pgt is done ?  */
+	if (physaddr >= MAXMEM ||
+	    read_cr3() != __sme_pa_nodebug(early_level4_pgt))
+		return;
+
+	pgd_p = &early_level4_pgt[pgd_index(address)].pgd;
+	pgd = *pgd_p;
+
+	if (!pgd)
+		return;
+
+	/*
+	 * The use of __START_KERNEL_map rather than __PAGE_OFFSET here matches
+	 * __early_make_pgtable where the entry was created.
+	 */
+	pud_p = (pudval_t *)((pgd & PTE_PFN_MASK) + __START_KERNEL_map - phys_base);
+	pud_p += pud_index(address);
+	pud = *pud_p;
+
+	if (!pud)
+		return;
+
+	pmd_p = (pmdval_t *)((pud & PTE_PFN_MASK) + __START_KERNEL_map - phys_base);
+	pmd_p[pmd_index(address)] = 0;
+}
+
+static void __init clear_mapping(void *address, unsigned long size)
+{
+	do {
+		__clear_mapping((unsigned long)address);
+
+		address += PMD_SIZE;
+		size = (size < PMD_SIZE) ? 0 : size - PMD_SIZE;
+	} while (size);
+}
+
+static void __init sme_memcpy(void *dst, void *src, unsigned long size)
+{
+	create_unencrypted_mapping(src, size);
+	memcpy(dst, src, size);
+	clear_mapping(src, size);
+}
+
 /* Don't add a printk in there. printk relies on the PDA which is not initialized 
    yet. */
 static void __init clear_bss(void)
@@ -122,12 +202,12 @@ static void __init copy_bootdata(char *real_mode_data)
 	char * command_line;
 	unsigned long cmd_line_ptr;
 
-	memcpy(&boot_params, real_mode_data, sizeof boot_params);
+	sme_memcpy(&boot_params, real_mode_data, sizeof boot_params);
 	sanitize_boot_params(&boot_params);
 	cmd_line_ptr = get_cmd_line_ptr();
 	if (cmd_line_ptr) {
 		command_line = __va(cmd_line_ptr);
-		memcpy(boot_command_line, command_line, COMMAND_LINE_SIZE);
+		sme_memcpy(boot_command_line, command_line, COMMAND_LINE_SIZE);
 	}
 }
 
diff --git a/arch/x86/kernel/setup.c b/arch/x86/kernel/setup.c
index 2367ae0..1d29cf9 100644
--- a/arch/x86/kernel/setup.c
+++ b/arch/x86/kernel/setup.c
@@ -113,6 +113,7 @@
 #include <asm/prom.h>
 #include <asm/microcode.h>
 #include <asm/mmu_context.h>
+#include <asm/mem_encrypt.h>
 
 /*
  * max_low_pfn_mapped: highest direct mapped pfn under 4GB
@@ -375,6 +376,13 @@ static void __init reserve_initrd(void)
 	    !ramdisk_image || !ramdisk_size)
 		return;		/* No initrd provided by bootloader */
 
+	/*
+	 * This memory is marked encrypted by the kernel but the ramdisk
+	 * was loaded in the clear by the bootloader, so make sure that
+	 * the ramdisk image is encrypted.
+	 */
+	sme_early_mem_enc(ramdisk_image, ramdisk_end - ramdisk_image);
+
 	initrd_start = 0;
 
 	mapped_size = memblock_mem_size(max_pfn_mapped);

[RFC PATCH v1 10/18] x86/efi: Access EFI related tables in the clear

[Tom Lendacky]

The EFI tables are not encrypted and need to be accessed as such. Be sure
to memmap them without the encryption attribute set. For EFI support that
lives outside of the arch/x86 tree, create a routine that uses the __weak
attribute so that it can be overridden by an architecture specific routine.

When freeing boot services related memory, since it has been mapped as
un-encrypted, be sure to change the mapping to encrypted for future use.

Signed-off-by: Tom Lendacky <thomas.lendacky@amd.com>
---
 arch/x86/include/asm/cacheflush.h  |    3 +
 arch/x86/include/asm/mem_encrypt.h |   22 +++++++++++
 arch/x86/kernel/setup.c            |    6 +--
 arch/x86/mm/mem_encrypt.c          |   56 +++++++++++++++++++++++++++
 arch/x86/mm/pageattr.c             |   75 ++++++++++++++++++++++++++++++++++++
 arch/x86/platform/efi/efi.c        |   26 +++++++-----
 arch/x86/platform/efi/efi_64.c     |    9 +++-
 arch/x86/platform/efi/quirks.c     |   12 +++++-
 drivers/firmware/efi/efi.c         |   18 +++++++--
 drivers/firmware/efi/esrt.c        |   12 +++---
 include/linux/efi.h                |    3 +
 11 files changed, 212 insertions(+), 30 deletions(-)

diff --git a/arch/x86/include/asm/cacheflush.h b/arch/x86/include/asm/cacheflush.h
index 61518cf..bfb08e5 100644
--- a/arch/x86/include/asm/cacheflush.h
+++ b/arch/x86/include/asm/cacheflush.h
@@ -13,6 +13,7 @@
  * Executability : eXeutable, NoteXecutable
  * Read/Write    : ReadOnly, ReadWrite
  * Presence      : NotPresent
+ * Encryption    : ENCrypted, DECrypted
  *
  * Within a category, the attributes are mutually exclusive.
  *
@@ -48,6 +49,8 @@ int set_memory_ro(unsigned long addr, int numpages);
 int set_memory_rw(unsigned long addr, int numpages);
 int set_memory_np(unsigned long addr, int numpages);
 int set_memory_4k(unsigned long addr, int numpages);
+int set_memory_enc(unsigned long addr, int numpages);
+int set_memory_dec(unsigned long addr, int numpages);
 
 int set_memory_array_uc(unsigned long *addr, int addrinarray);
 int set_memory_array_wc(unsigned long *addr, int addrinarray);
diff --git a/arch/x86/include/asm/mem_encrypt.h b/arch/x86/include/asm/mem_encrypt.h
index 2785493..42868f5 100644
--- a/arch/x86/include/asm/mem_encrypt.h
+++ b/arch/x86/include/asm/mem_encrypt.h
@@ -23,13 +23,23 @@ extern unsigned long sme_me_mask;
 
 u8 sme_get_me_loss(void);
 
+int sme_set_mem_enc(void *vaddr, unsigned long size);
+int sme_set_mem_dec(void *vaddr, unsigned long size);
+
 void __init sme_early_mem_enc(resource_size_t paddr,
 			      unsigned long size);
 void __init sme_early_mem_dec(resource_size_t paddr,
 			      unsigned long size);
 
+void __init *sme_early_memremap(resource_size_t paddr,
+				unsigned long size);
+
 void __init sme_early_init(void);
 
+/* Architecture __weak replacement functions */
+void __init *efi_me_early_memremap(resource_size_t paddr,
+				   unsigned long size);
+
 #define __sme_pa(x)		(__pa((x)) | sme_me_mask)
 #define __sme_pa_nodebug(x)	(__pa_nodebug((x)) | sme_me_mask)
 
@@ -44,6 +54,16 @@ static inline u8 sme_get_me_loss(void)
 	return 0;
 }
 
+static inline int sme_set_mem_enc(void *vaddr, unsigned long size)
+{
+	return 0;
+}
+
+static inline int sme_set_mem_dec(void *vaddr, unsigned long size)
+{
+	return 0;
+}
+
 static inline void __init sme_early_mem_enc(resource_size_t paddr,
 					    unsigned long size)
 {
@@ -63,6 +83,8 @@ static inline void __init sme_early_init(void)
 
 #define __sme_va		__va
 
+#define sme_early_memremap	early_memremap
+
 #endif	/* CONFIG_AMD_MEM_ENCRYPT */
 
 #endif	/* __ASSEMBLY__ */
diff --git a/arch/x86/kernel/setup.c b/arch/x86/kernel/setup.c
index 1d29cf9..2e460fb 100644
--- a/arch/x86/kernel/setup.c
+++ b/arch/x86/kernel/setup.c
@@ -424,7 +424,7 @@ static void __init parse_setup_data(void)
 	while (pa_data) {
 		u32 data_len, data_type;
 
-		data = early_memremap(pa_data, sizeof(*data));
+		data = sme_early_memremap(pa_data, sizeof(*data));
 		data_len = data->len + sizeof(struct setup_data);
 		data_type = data->type;
 		pa_next = data->next;
@@ -457,7 +457,7 @@ static void __init e820_reserve_setup_data(void)
 		return;
 
 	while (pa_data) {
-		data = early_memremap(pa_data, sizeof(*data));
+		data = sme_early_memremap(pa_data, sizeof(*data));
 		e820_update_range(pa_data, sizeof(*data)+data->len,
 			 E820_RAM, E820_RESERVED_KERN);
 		pa_data = data->next;
@@ -477,7 +477,7 @@ static void __init memblock_x86_reserve_range_setup_data(void)
 
 	pa_data = boot_params.hdr.setup_data;
 	while (pa_data) {
-		data = early_memremap(pa_data, sizeof(*data));
+		data = sme_early_memremap(pa_data, sizeof(*data));
 		memblock_reserve(pa_data, sizeof(*data) + data->len);
 		pa_data = data->next;
 		early_memunmap(data, sizeof(*data));
diff --git a/arch/x86/mm/mem_encrypt.c b/arch/x86/mm/mem_encrypt.c
index 5f19ede..7d56d1b 100644
--- a/arch/x86/mm/mem_encrypt.c
+++ b/arch/x86/mm/mem_encrypt.c
@@ -14,12 +14,55 @@
 #include <linux/mm.h>
 
 #include <asm/mem_encrypt.h>
+#include <asm/cacheflush.h>
 #include <asm/tlbflush.h>
 #include <asm/fixmap.h>
 
 /* Buffer used for early in-place encryption by BSP, no locking needed */
 static char me_early_buffer[PAGE_SIZE] __aligned(PAGE_SIZE);
 
+int sme_set_mem_enc(void *vaddr, unsigned long size)
+{
+	unsigned long addr, numpages;
+
+	if (!sme_me_mask)
+		return 0;
+
+	addr = (unsigned long)vaddr & PAGE_MASK;
+	numpages = PAGE_ALIGN(size) >> PAGE_SHIFT;
+
+	/*
+	 * The set_memory_xxx functions take an integer for numpages, make
+	 * sure it doesn't exceed that.
+	 */
+	if (numpages > INT_MAX)
+		return -EINVAL;
+
+	return set_memory_enc(addr, numpages);
+}
+EXPORT_SYMBOL_GPL(sme_set_mem_enc);
+
+int sme_set_mem_dec(void *vaddr, unsigned long size)
+{
+	unsigned long addr, numpages;
+
+	if (!sme_me_mask)
+		return 0;
+
+	addr = (unsigned long)vaddr & PAGE_MASK;
+	numpages = PAGE_ALIGN(size) >> PAGE_SHIFT;
+
+	/*
+	 * The set_memory_xxx functions take an integer for numpages, make
+	 * sure it doesn't exceed that.
+	 */
+	if (numpages > INT_MAX)
+		return -EINVAL;
+
+	return set_memory_dec(addr, numpages);
+}
+EXPORT_SYMBOL_GPL(sme_set_mem_dec);
+
 void __init sme_early_mem_enc(resource_size_t paddr, unsigned long size)
 {
 	void *src, *dst;
@@ -104,6 +147,12 @@ void __init sme_early_mem_dec(resource_size_t paddr, unsigned long size)
 	}
 }
 
+void __init *sme_early_memremap(resource_size_t paddr,
+				unsigned long size)
+{
+	return early_memremap_dec(paddr, size);
+}
+
 void __init sme_early_init(void)
 {
 	unsigned int i;
@@ -117,3 +166,10 @@ void __init sme_early_init(void)
 	for (i = 0; i < ARRAY_SIZE(protection_map); i++)
 		protection_map[i] = __pgprot(pgprot_val(protection_map[i]) | sme_me_mask);
 }
+
+/* Architecture __weak replacement functions */
+void __init *efi_me_early_memremap(resource_size_t paddr,
+				   unsigned long size)
+{
+	return sme_early_memremap(paddr, size);
+}
diff --git a/arch/x86/mm/pageattr.c b/arch/x86/mm/pageattr.c
index c055302..0384fb3 100644
--- a/arch/x86/mm/pageattr.c
+++ b/arch/x86/mm/pageattr.c
@@ -1731,6 +1731,81 @@ int set_memory_4k(unsigned long addr, int numpages)
 					__pgprot(0), 1, 0, NULL);
 }
 
+static int __set_memory_enc_dec(struct cpa_data *cpa)
+{
+	unsigned long addr;
+	int numpages;
+	int ret;
+
+	if (*cpa->vaddr & ~PAGE_MASK) {
+		*cpa->vaddr &= PAGE_MASK;
+
+		/* People should not be passing in unaligned addresses */
+		WARN_ON_ONCE(1);
+	}
+
+	addr = *cpa->vaddr;
+	numpages = cpa->numpages;
+
+	/* Must avoid aliasing mappings in the highmem code */
+	kmap_flush_unused();
+	vm_unmap_aliases();
+
+	ret = __change_page_attr_set_clr(cpa, 1);
+
+	/* Check whether we really changed something */
+	if (!(cpa->flags & CPA_FLUSHTLB))
+		goto out;
+
+	/*
+	 * On success we use CLFLUSH, when the CPU supports it to
+	 * avoid the WBINVD.
+	 */
+	if (!ret && static_cpu_has(X86_FEATURE_CLFLUSH))
+		cpa_flush_range(addr, numpages, 1);
+	else
+		cpa_flush_all(1);
+
+out:
+	return ret;
+}
+
+int set_memory_enc(unsigned long addr, int numpages)
+{
+	struct cpa_data cpa;
+
+	if (!sme_me_mask)
+		return 0;
+
+	memset(&cpa, 0, sizeof(cpa));
+	cpa.vaddr = &addr;
+	cpa.numpages = numpages;
+	cpa.mask_set = __pgprot(_PAGE_ENC);
+	cpa.mask_clr = __pgprot(0);
+	cpa.pgd = init_mm.pgd;
+
+	return __set_memory_enc_dec(&cpa);
+}
+EXPORT_SYMBOL(set_memory_enc);
+
+int set_memory_dec(unsigned long addr, int numpages)
+{
+	struct cpa_data cpa;
+
+	if (!sme_me_mask)
+		return 0;
+
+	memset(&cpa, 0, sizeof(cpa));
+	cpa.vaddr = &addr;
+	cpa.numpages = numpages;
+	cpa.mask_set = __pgprot(0);
+	cpa.mask_clr = __pgprot(_PAGE_ENC);
+	cpa.pgd = init_mm.pgd;
+
+	return __set_memory_enc_dec(&cpa);
+}
+EXPORT_SYMBOL(set_memory_dec);
+
 int set_pages_uc(struct page *page, int numpages)
 {
 	unsigned long addr = (unsigned long)page_address(page);
diff --git a/arch/x86/platform/efi/efi.c b/arch/x86/platform/efi/efi.c
index 994a7df8..871b213 100644
--- a/arch/x86/platform/efi/efi.c
+++ b/arch/x86/platform/efi/efi.c
@@ -53,6 +53,7 @@
 #include <asm/x86_init.h>
 #include <asm/rtc.h>
 #include <asm/uv/uv.h>
+#include <asm/mem_encrypt.h>
 
 #define EFI_DEBUG
 
@@ -261,12 +262,12 @@ static int __init efi_systab_init(void *phys)
 		u64 tmp = 0;
 
 		if (efi_setup) {
-			data = early_memremap(efi_setup, sizeof(*data));
+			data = sme_early_memremap(efi_setup, sizeof(*data));
 			if (!data)
 				return -ENOMEM;
 		}
-		systab64 = early_memremap((unsigned long)phys,
-					 sizeof(*systab64));
+		systab64 = sme_early_memremap((unsigned long)phys,
+					      sizeof(*systab64));
 		if (systab64 == NULL) {
 			pr_err("Couldn't map the system table!\n");
 			if (data)
@@ -314,8 +315,8 @@ static int __init efi_systab_init(void *phys)
 	} else {
 		efi_system_table_32_t *systab32;
 
-		systab32 = early_memremap((unsigned long)phys,
-					 sizeof(*systab32));
+		systab32 = sme_early_memremap((unsigned long)phys,
+					      sizeof(*systab32));
 		if (systab32 == NULL) {
 			pr_err("Couldn't map the system table!\n");
 			return -ENOMEM;
@@ -361,8 +362,8 @@ static int __init efi_runtime_init32(void)
 {
 	efi_runtime_services_32_t *runtime;
 
-	runtime = early_memremap((unsigned long)efi.systab->runtime,
-			sizeof(efi_runtime_services_32_t));
+	runtime = sme_early_memremap((unsigned long)efi.systab->runtime,
+				     sizeof(efi_runtime_services_32_t));
 	if (!runtime) {
 		pr_err("Could not map the runtime service table!\n");
 		return -ENOMEM;
@@ -385,8 +386,8 @@ static int __init efi_runtime_init64(void)
 {
 	efi_runtime_services_64_t *runtime;
 
-	runtime = early_memremap((unsigned long)efi.systab->runtime,
-			sizeof(efi_runtime_services_64_t));
+	runtime = sme_early_memremap((unsigned long)efi.systab->runtime,
+				     sizeof(efi_runtime_services_64_t));
 	if (!runtime) {
 		pr_err("Could not map the runtime service table!\n");
 		return -ENOMEM;
@@ -444,8 +445,8 @@ static int __init efi_memmap_init(void)
 		return 0;
 
 	/* Map the EFI memory map */
-	memmap.map = early_memremap((unsigned long)memmap.phys_map,
-				   memmap.nr_map * memmap.desc_size);
+	memmap.map = sme_early_memremap((unsigned long)memmap.phys_map,
+					memmap.nr_map * memmap.desc_size);
 	if (memmap.map == NULL) {
 		pr_err("Could not map the memory map!\n");
 		return -ENOMEM;
@@ -490,7 +491,7 @@ void __init efi_init(void)
 	/*
 	 * Show what we know for posterity
 	 */
-	c16 = tmp = early_memremap(efi.systab->fw_vendor, 2);
+	c16 = tmp = sme_early_memremap(efi.systab->fw_vendor, 2);
 	if (c16) {
 		for (i = 0; i < sizeof(vendor) - 1 && *c16; ++i)
 			vendor[i] = *c16++;
@@ -690,6 +691,7 @@ static void *realloc_pages(void *old_memmap, int old_shift)
 	ret = (void *)__get_free_pages(GFP_KERNEL, old_shift + 1);
 	if (!ret)
 		goto out;
+	sme_set_mem_dec(ret, PAGE_SIZE << (old_shift + 1));
 
 	/*
 	 * A first-time allocation doesn't have anything to copy.
diff --git a/arch/x86/platform/efi/efi_64.c b/arch/x86/platform/efi/efi_64.c
index 49e4dd4..834a992 100644
--- a/arch/x86/platform/efi/efi_64.c
+++ b/arch/x86/platform/efi/efi_64.c
@@ -223,7 +223,7 @@ int __init efi_setup_page_tables(unsigned long pa_memmap, unsigned num_pages)
 	if (efi_enabled(EFI_OLD_MEMMAP))
 		return 0;
 
-	efi_scratch.efi_pgt = (pgd_t *)__pa(efi_pgd);
+	efi_scratch.efi_pgt = (pgd_t *)__sme_pa(efi_pgd);
 	pgd = efi_pgd;
 
 	/*
@@ -262,7 +262,8 @@ int __init efi_setup_page_tables(unsigned long pa_memmap, unsigned num_pages)
 		pfn = md->phys_addr >> PAGE_SHIFT;
 		npages = md->num_pages;
 
-		if (kernel_map_pages_in_pgd(pgd, pfn, md->phys_addr, npages, _PAGE_RW)) {
+		if (kernel_map_pages_in_pgd(pgd, pfn, md->phys_addr, npages,
+					    _PAGE_RW | _PAGE_ENC)) {
 			pr_err("Failed to map 1:1 memory\n");
 			return 1;
 		}
@@ -272,6 +273,7 @@ int __init efi_setup_page_tables(unsigned long pa_memmap, unsigned num_pages)
 	if (!page)
 		panic("Unable to allocate EFI runtime stack < 4GB\n");
 
+	sme_set_mem_dec(page_address(page), PAGE_SIZE);
 	efi_scratch.phys_stack = virt_to_phys(page_address(page));
 	efi_scratch.phys_stack += PAGE_SIZE; /* stack grows down */
 
@@ -279,7 +281,8 @@ int __init efi_setup_page_tables(unsigned long pa_memmap, unsigned num_pages)
 	text = __pa(_text);
 	pfn = text >> PAGE_SHIFT;
 
-	if (kernel_map_pages_in_pgd(pgd, pfn, text, npages, _PAGE_RW)) {
+	if (kernel_map_pages_in_pgd(pgd, pfn, text, npages,
+				    _PAGE_RW | _PAGE_ENC)) {
 		pr_err("Failed to map kernel text 1:1\n");
 		return 1;
 	}
diff --git a/arch/x86/platform/efi/quirks.c b/arch/x86/platform/efi/quirks.c
index ab50ada..dde4fb6b 100644
--- a/arch/x86/platform/efi/quirks.c
+++ b/arch/x86/platform/efi/quirks.c
@@ -13,6 +13,7 @@
 #include <linux/dmi.h>
 #include <asm/efi.h>
 #include <asm/uv/uv.h>
+#include <asm/mem_encrypt.h>
 
 #define EFI_MIN_RESERVE 5120
 
@@ -265,6 +266,13 @@ void __init efi_free_boot_services(void)
 		if (md->attribute & EFI_MEMORY_RUNTIME)
 			continue;
 
+		/*
+		 * Change the mapping to encrypted memory before freeing.
+		 * This insures any future allocations of this mapped area
+		 * are used encrypted.
+		 */
+		sme_set_mem_enc(__va(start), size);
+
 		free_bootmem_late(start, size);
 	}
 
@@ -292,7 +300,7 @@ int __init efi_reuse_config(u64 tables, int nr_tables)
 	if (!efi_enabled(EFI_64BIT))
 		return 0;
 
-	data = early_memremap(efi_setup, sizeof(*data));
+	data = sme_early_memremap(efi_setup, sizeof(*data));
 	if (!data) {
 		ret = -ENOMEM;
 		goto out;
@@ -303,7 +311,7 @@ int __init efi_reuse_config(u64 tables, int nr_tables)
 
 	sz = sizeof(efi_config_table_64_t);
 
-	p = tablep = early_memremap(tables, nr_tables * sz);
+	p = tablep = sme_early_memremap(tables, nr_tables * sz);
 	if (!p) {
 		pr_err("Could not map Configuration table!\n");
 		ret = -ENOMEM;
diff --git a/drivers/firmware/efi/efi.c b/drivers/firmware/efi/efi.c
index 3a69ed5..25010c7 100644
--- a/drivers/firmware/efi/efi.c
+++ b/drivers/firmware/efi/efi.c
@@ -76,6 +76,16 @@ static int __init parse_efi_cmdline(char *str)
 }
 early_param("efi", parse_efi_cmdline);
 
+/*
+ * If memory encryption is supported, then an override to this function
+ * will be provided.
+ */
+void __weak __init *efi_me_early_memremap(resource_size_t phys_addr,
+					  unsigned long size)
+{
+	return early_memremap(phys_addr, size);
+}
+
 struct kobject *efi_kobj;
 
 /*
@@ -289,9 +299,9 @@ int __init efi_mem_desc_lookup(u64 phys_addr, efi_memory_desc_t *out_md)
 		 * So just always get our own virtual map on the CPU.
 		 *
 		 */
-		md = early_memremap(p, sizeof (*md));
+		md = efi_me_early_memremap(p, sizeof (*md));
 		if (!md) {
-			pr_err_once("early_memremap(%pa, %zu) failed.\n",
+			pr_err_once("efi_me_early_memremap(%pa, %zu) failed.\n",
 				    &p, sizeof (*md));
 			return -ENOMEM;
 		}
@@ -431,8 +441,8 @@ int __init efi_config_init(efi_config_table_type_t *arch_tables)
 	/*
 	 * Let's see what config tables the firmware passed to us.
 	 */
-	config_tables = early_memremap(efi.systab->tables,
-				       efi.systab->nr_tables * sz);
+	config_tables = efi_me_early_memremap(efi.systab->tables,
+					      efi.systab->nr_tables * sz);
 	if (config_tables == NULL) {
 		pr_err("Could not map Configuration table!\n");
 		return -ENOMEM;
diff --git a/drivers/firmware/efi/esrt.c b/drivers/firmware/efi/esrt.c
index 75feb3f..7a96bc6 100644
--- a/drivers/firmware/efi/esrt.c
+++ b/drivers/firmware/efi/esrt.c
@@ -273,10 +273,10 @@ void __init efi_esrt_init(void)
 		return;
 	}
 
-	va = early_memremap(efi.esrt, size);
+	va = efi_me_early_memremap(efi.esrt, size);
 	if (!va) {
-		pr_err("early_memremap(%p, %zu) failed.\n", (void *)efi.esrt,
-		       size);
+		pr_err("efi_me_early_memremap(%p, %zu) failed.\n",
+		       (void *)efi.esrt, size);
 		return;
 	}
 
@@ -323,10 +323,10 @@ void __init efi_esrt_init(void)
 	/* remap it with our (plausible) new pages */
 	early_memunmap(va, size);
 	size += entries_size;
-	va = early_memremap(efi.esrt, size);
+	va = efi_me_early_memremap(efi.esrt, size);
 	if (!va) {
-		pr_err("early_memremap(%p, %zu) failed.\n", (void *)efi.esrt,
-		       size);
+		pr_err("efi_me_early_memremap(%p, %zu) failed.\n",
+		       (void *)efi.esrt, size);
 		return;
 	}
 
diff --git a/include/linux/efi.h b/include/linux/efi.h
index 1626474..557c774 100644
--- a/include/linux/efi.h
+++ b/include/linux/efi.h
@@ -957,6 +957,9 @@ extern void __init efi_fake_memmap(void);
 static inline void efi_fake_memmap(void) { }
 #endif
 
+extern void __weak __init *efi_me_early_memremap(resource_size_t phys_addr,
+						 unsigned long size);
+
 /* Iterate through an efi_memory_map */
 #define for_each_efi_memory_desc(m, md)					   \
 	for ((md) = (m)->map;						   \

Re: [RFC PATCH v1 10/18] x86/efi: Access EFI related tables in the clear

[Matt Fleming]

On Tue, 26 Apr, at 05:57:40PM, Tom Lendacky wrote:
> The EFI tables are not encrypted and need to be accessed as such. Be sure
> to memmap them without the encryption attribute set. For EFI support that
> lives outside of the arch/x86 tree, create a routine that uses the __weak
> attribute so that it can be overridden by an architecture specific routine.
> 
> When freeing boot services related memory, since it has been mapped as
> un-encrypted, be sure to change the mapping to encrypted for future use.
> 
> Signed-off-by: Tom Lendacky <thomas.lendacky@amd.com>
> ---
>  arch/x86/include/asm/cacheflush.h  |    3 +
>  arch/x86/include/asm/mem_encrypt.h |   22 +++++++++++
>  arch/x86/kernel/setup.c            |    6 +--
>  arch/x86/mm/mem_encrypt.c          |   56 +++++++++++++++++++++++++++
>  arch/x86/mm/pageattr.c             |   75 ++++++++++++++++++++++++++++++++++++
>  arch/x86/platform/efi/efi.c        |   26 +++++++-----
>  arch/x86/platform/efi/efi_64.c     |    9 +++-
>  arch/x86/platform/efi/quirks.c     |   12 +++++-
>  drivers/firmware/efi/efi.c         |   18 +++++++--
>  drivers/firmware/efi/esrt.c        |   12 +++---
>  include/linux/efi.h                |    3 +
>  11 files changed, 212 insertions(+), 30 deletions(-)

The size of this change is completely unexpected. Why is there so much
churn to workaround this new feature?

Is it not possible to maintain some kind of kernel virtual address
mapping so memremap*() and friends can figure out when to twiddle the
mapping attributes and map with/without encryption?

These API changes place an undue burden on developers that don't even
care about SME.

Re: [RFC PATCH v1 10/18] x86/efi: Access EFI related tables in the clear

[Borislav Petkov]

On Tue, May 10, 2016 at 02:43:58PM +0100, Matt Fleming wrote:
> Is it not possible to maintain some kind of kernel virtual address
> mapping so memremap*() and friends can figure out when to twiddle the
> mapping attributes and map with/without encryption?

I guess we can move the sme_* specific stuff one indirection layer
below, i.e., in the *memremap() routines so that callers don't have to
care... That should keep the churn down...

-- 
Regards/Gruss,
    Boris.

ECO tip #101: Trim your mails when you reply.

Re: [RFC PATCH v1 10/18] x86/efi: Access EFI related tables in the clear

[Tom Lendacky]

On 05/10/2016 08:57 AM, Borislav Petkov wrote:
> On Tue, May 10, 2016 at 02:43:58PM +0100, Matt Fleming wrote:
>> Is it not possible to maintain some kind of kernel virtual address
>> mapping so memremap*() and friends can figure out when to twiddle the
>> mapping attributes and map with/without encryption?
> 
> I guess we can move the sme_* specific stuff one indirection layer
> below, i.e., in the *memremap() routines so that callers don't have to
> care... That should keep the churn down...
> 

We could do that, but we'll have to generate that list of addresses so
that it can be checked against the range being mapped.  Since this is
part of early memmap support searching that list every time might not be
too bad. I'll have to look into that and see what that looks like.

Thanks,
Tom

Re: [RFC PATCH v1 10/18] x86/efi: Access EFI related tables in the clear

[Tom Lendacky]

On 05/12/2016 01:20 PM, Tom Lendacky wrote:
> On 05/10/2016 08:57 AM, Borislav Petkov wrote:
>> On Tue, May 10, 2016 at 02:43:58PM +0100, Matt Fleming wrote:
>>> Is it not possible to maintain some kind of kernel virtual address
>>> mapping so memremap*() and friends can figure out when to twiddle the
>>> mapping attributes and map with/without encryption?
>>
>> I guess we can move the sme_* specific stuff one indirection layer
>> below, i.e., in the *memremap() routines so that callers don't have to
>> care... That should keep the churn down...
>>
> 
> We could do that, but we'll have to generate that list of addresses so
> that it can be checked against the range being mapped.  Since this is
> part of early memmap support searching that list every time might not be
> too bad. I'll have to look into that and see what that looks like.

I looked into this and this would be a large change also to parse tables
and build lists.  It occurred to me that this could all be taken care of
if the early_memremap calls were changed to early_ioremap calls. Looking
in the git log I see that they were originally early_ioremap calls but
were changed to early_memremap calls with this commit:

commit abc93f8eb6e4 ("efi: Use early_mem*() instead of early_io*()")

Looking at the early_memremap code and the early_ioremap code they both
call __early_ioremap so I don't see how this change makes any
difference (especially since FIXMAP_PAGE_NORMAL and FIXMAP_PAGE_IO are
identical in this case).

Is it safe to change these back to early_ioremap calls (at least on
x86)?

Thanks,
Tom

> 
> Thanks,
> Tom
> 

Re: [RFC PATCH v1 10/18] x86/efi: Access EFI related tables in the clear

[Daniel Kiper]

On Tue, May 24, 2016 at 09:54:31AM -0500, Tom Lendacky wrote:
> On 05/12/2016 01:20 PM, Tom Lendacky wrote:
> > On 05/10/2016 08:57 AM, Borislav Petkov wrote:
> >> On Tue, May 10, 2016 at 02:43:58PM +0100, Matt Fleming wrote:
> >>> Is it not possible to maintain some kind of kernel virtual address
> >>> mapping so memremap*() and friends can figure out when to twiddle the
> >>> mapping attributes and map with/without encryption?
> >>
> >> I guess we can move the sme_* specific stuff one indirection layer
> >> below, i.e., in the *memremap() routines so that callers don't have to
> >> care... That should keep the churn down...
> >>
> >
> > We could do that, but we'll have to generate that list of addresses so
> > that it can be checked against the range being mapped.  Since this is
> > part of early memmap support searching that list every time might not be
> > too bad. I'll have to look into that and see what that looks like.
>
> I looked into this and this would be a large change also to parse tables
> and build lists.  It occurred to me that this could all be taken care of
> if the early_memremap calls were changed to early_ioremap calls. Looking
> in the git log I see that they were originally early_ioremap calls but
> were changed to early_memremap calls with this commit:
>
> commit abc93f8eb6e4 ("efi: Use early_mem*() instead of early_io*()")
>
> Looking at the early_memremap code and the early_ioremap code they both
> call __early_ioremap so I don't see how this change makes any
> difference (especially since FIXMAP_PAGE_NORMAL and FIXMAP_PAGE_IO are
> identical in this case).
>
> Is it safe to change these back to early_ioremap calls (at least on
> x86)?

Commit f955371ca9d3986bca100666041fcfa9b6d21962 (x86: remove the Xen-specific
_PAGE_IOMAP PTE flag) made commit abc93f8eb6e4 unnecessary. Though, IMO, it
is still valid code cleanup. So, if it is not very strongly needed I would
not revert this change.

Daniel

Re: [RFC PATCH v1 10/18] x86/efi: Access EFI related tables in the clear

[Matt Fleming]

On Tue, 24 May, at 09:54:31AM, Tom Lendacky wrote:
> 
> I looked into this and this would be a large change also to parse tables
> and build lists.  It occurred to me that this could all be taken care of
> if the early_memremap calls were changed to early_ioremap calls. Looking
> in the git log I see that they were originally early_ioremap calls but
> were changed to early_memremap calls with this commit:
> 
> commit abc93f8eb6e4 ("efi: Use early_mem*() instead of early_io*()")
> 
> Looking at the early_memremap code and the early_ioremap code they both
> call __early_ioremap so I don't see how this change makes any
> difference (especially since FIXMAP_PAGE_NORMAL and FIXMAP_PAGE_IO are
> identical in this case).
> 
> Is it safe to change these back to early_ioremap calls (at least on
> x86)?

I really don't want to begin mixing early_ioremap() calls and
early_memremap() calls for any of the EFI code if it can be avoided.

There is slow but steady progress to move more and more of the
architecture specific EFI code out into generic code. Swapping
early_memremap() for early_ioremap() would be a step backwards,
because FIXMAP_PAGE_NORMAL and FIXMAP_PAGE_IO are not identical on
ARM/arm64.

Could you point me at the patch that in this series that fixes up
early_ioremap() to work with mem encrypt/decrypt? I took another
(quick) look through but couldn't find it.

Re: [RFC PATCH v1 10/18] x86/efi: Access EFI related tables in the clear

[Tom Lendacky]

On 05/25/2016 02:30 PM, Matt Fleming wrote:
> On Tue, 24 May, at 09:54:31AM, Tom Lendacky wrote:
>>
>> I looked into this and this would be a large change also to parse tables
>> and build lists.  It occurred to me that this could all be taken care of
>> if the early_memremap calls were changed to early_ioremap calls. Looking
>> in the git log I see that they were originally early_ioremap calls but
>> were changed to early_memremap calls with this commit:
>>
>> commit abc93f8eb6e4 ("efi: Use early_mem*() instead of early_io*()")
>>
>> Looking at the early_memremap code and the early_ioremap code they both
>> call __early_ioremap so I don't see how this change makes any
>> difference (especially since FIXMAP_PAGE_NORMAL and FIXMAP_PAGE_IO are
>> identical in this case).
>>
>> Is it safe to change these back to early_ioremap calls (at least on
>> x86)?
> 
> I really don't want to begin mixing early_ioremap() calls and
> early_memremap() calls for any of the EFI code if it can be avoided.

I definitely wouldn't mix them, it would be all or nothing.

> 
> There is slow but steady progress to move more and more of the
> architecture specific EFI code out into generic code. Swapping
> early_memremap() for early_ioremap() would be a step backwards,
> because FIXMAP_PAGE_NORMAL and FIXMAP_PAGE_IO are not identical on
> ARM/arm64.

Maybe adding something similar to __acpi_map_table would be more
appropriate?

> 
> Could you point me at the patch that in this series that fixes up
> early_ioremap() to work with mem encrypt/decrypt? I took another
> (quick) look through but couldn't find it.

The patch in question is patch 6/18 where PAGE_KERNEL is changed to
include the _PAGE_ENC attribute (the encryption mask). This now
makes FIXMAP_PAGE_NORMAL contain the encryption mask while
FIXMAP_PAGE_IO does not. In this way, anything mapped using the
early_ioremap call won't be mapped encrypted.

Thanks,
Tom

> 

Re: [RFC PATCH v1 10/18] x86/efi: Access EFI related tables in the clear

[Matt Fleming]

(Sorry for the delay)

On Thu, 26 May, at 08:45:58AM, Tom Lendacky wrote:
> 
> The patch in question is patch 6/18 where PAGE_KERNEL is changed to
> include the _PAGE_ENC attribute (the encryption mask). This now
> makes FIXMAP_PAGE_NORMAL contain the encryption mask while
> FIXMAP_PAGE_IO does not. In this way, anything mapped using the
> early_ioremap call won't be mapped encrypted.

There are semantics attached to early_ioremap() that do not apply in
this case; that you're mapping an MMIO region but for EFI we just care
about noting where the firmware (not the kernel) populated the region
with data. Similar problems exist for other early boot code such as
the devicetree stuff.

early_ioremap() is not the answer.

What you really want is just some way to distinguish kernel-owned
regions from those owned by "somebody else".

I have no problem updating early_memremap() to take a @flags argument
to make that distinction, provided that the naming is generic and not
tied to AMD's SME technology via an "sme" prefix/suffix.

And making it generic should allow it to be easily sprinkled into the
shared architecture code in drivers/firmware/efi/ without issue.

I'm going to follow up with some additional comments/questions on
PATCH 10.

Re: [RFC PATCH v1 10/18] x86/efi: Access EFI related tables in the clear

[Tom Lendacky]

On 06/08/2016 05:07 AM, Matt Fleming wrote:
> (Sorry for the delay)

No worries, thanks for all the feedback.

> 
> On Thu, 26 May, at 08:45:58AM, Tom Lendacky wrote:
>>
>> The patch in question is patch 6/18 where PAGE_KERNEL is changed to
>> include the _PAGE_ENC attribute (the encryption mask). This now
>> makes FIXMAP_PAGE_NORMAL contain the encryption mask while
>> FIXMAP_PAGE_IO does not. In this way, anything mapped using the
>> early_ioremap call won't be mapped encrypted.
> 
> There are semantics attached to early_ioremap() that do not apply in
> this case; that you're mapping an MMIO region but for EFI we just care
> about noting where the firmware (not the kernel) populated the region
> with data. Similar problems exist for other early boot code such as
> the devicetree stuff.
> 
> early_ioremap() is not the answer.
> 
> What you really want is just some way to distinguish kernel-owned
> regions from those owned by "somebody else".
> 
> I have no problem updating early_memremap() to take a @flags argument
> to make that distinction, provided that the naming is generic and not
> tied to AMD's SME technology via an "sme" prefix/suffix.

So maybe something along the lines of an enum that would have entries
(initially) like KERNEL_DATA (equal to zero) and EFI_DATA. Others could
be added later as needed.

Would you then want to allow the protection attributes to be updated
by architecture specific code through something like a __weak function?
In the x86 case I can add this function as a non-SME specific function
that would initially just have the SME-related mask modification in it.

Thanks,
Tom

> 
> And making it generic should allow it to be easily sprinkled into the
> shared architecture code in drivers/firmware/efi/ without issue.
> 
> I'm going to follow up with some additional comments/questions on
> PATCH 10.
> 

Re: [RFC PATCH v1 10/18] x86/efi: Access EFI related tables in the clear

[Matt Fleming]

On Thu, 09 Jun, at 11:16:40AM, Tom Lendacky wrote:
> 
> So maybe something along the lines of an enum that would have entries
> (initially) like KERNEL_DATA (equal to zero) and EFI_DATA. Others could
> be added later as needed.
 
Sure, that works for me, though maybe BOOT_DATA would be more
applicable considering the devicetree case too.

> Would you then want to allow the protection attributes to be updated
> by architecture specific code through something like a __weak function?
> In the x86 case I can add this function as a non-SME specific function
> that would initially just have the SME-related mask modification in it.

Would we need a new function? Couldn't we just have a new
FIXMAP_PAGE_* constant? e.g. would something like this work?

---

enum memremap_owner {
	KERNEL_DATA = 0,
	BOOT_DATA,
};

void __init *
early_memremap(resource_size_t phys_addr, unsigned long size,
	       enum memremap_owner owner)
{
	pgprot_t prot;

	switch (owner) {
	case BOOT_DATA:
		prot = FIXMAP_PAGE_BOOT;
		break;
	case KERNEL_DATA:	/* FALLTHROUGH */
	default:
		prot = FIXMAP_PAGE_NORMAL;
		
	}

	return (__force void *)__early_ioremap(phys_addr, size, prot);
}

Re: [RFC PATCH v1 10/18] x86/efi: Access EFI related tables in the clear

[Matt Fleming]

On Mon, 13 Jun, at 01:03:22PM, Matt Fleming wrote:
> 
> Would we need a new function? Couldn't we just have a new
> FIXMAP_PAGE_* constant? e.g. would something like this work?
> 
> ---
> 
> enum memremap_owner {
> 	KERNEL_DATA = 0,
> 	BOOT_DATA,
> };
> 
> void __init *
> early_memremap(resource_size_t phys_addr, unsigned long size,
> 	       enum memremap_owner owner)
> {
> 	pgprot_t prot;
> 
> 	switch (owner) {
> 	case BOOT_DATA:
> 		prot = FIXMAP_PAGE_BOOT;
> 		break;
> 	case KERNEL_DATA:	/* FALLTHROUGH */
> 	default:
> 		prot = FIXMAP_PAGE_NORMAL;
> 		
> 	}
> 
> 	return (__force void *)__early_ioremap(phys_addr, size, prot);
> }

Although it occurs to me that if there's a trivial 1:1 mapping between
memremap_owner and FIXMAP_PAGE_* we might as well just add a new
early_memremap_boot() that uses the correct FIXMAP_PAGE_* constant,
akin to early_memremap_ro().

Re: [RFC PATCH v1 10/18] x86/efi: Access EFI related tables in the clear

[Tom Lendacky]

On 06/13/2016 07:03 AM, Matt Fleming wrote:
> On Thu, 09 Jun, at 11:16:40AM, Tom Lendacky wrote:
>>
>> So maybe something along the lines of an enum that would have entries
>> (initially) like KERNEL_DATA (equal to zero) and EFI_DATA. Others could
>> be added later as needed.
>  
> Sure, that works for me, though maybe BOOT_DATA would be more
> applicable considering the devicetree case too.
> 
>> Would you then want to allow the protection attributes to be updated
>> by architecture specific code through something like a __weak function?
>> In the x86 case I can add this function as a non-SME specific function
>> that would initially just have the SME-related mask modification in it.
> 
> Would we need a new function? Couldn't we just have a new
> FIXMAP_PAGE_* constant? e.g. would something like this work?

Looking forward to the virtualization support (SEV), the VM will be
completely encrypted from the time it is started. In this case all of
the UEFI data will be encrypted and I would need to insure that the
mapping reflects that. When I do the SEV patches, I can change the
FIXMAP #define to add some logic to return a value, so I think the
FIXMAP_PAGE_ idea can work.

Thanks,
Tom

> 
> ---
> 
> enum memremap_owner {
> 	KERNEL_DATA = 0,
> 	BOOT_DATA,
> };
> 
> void __init *
> early_memremap(resource_size_t phys_addr, unsigned long size,
> 	       enum memremap_owner owner)
> {
> 	pgprot_t prot;
> 
> 	switch (owner) {
> 	case BOOT_DATA:
> 		prot = FIXMAP_PAGE_BOOT;
> 		break;
> 	case KERNEL_DATA:	/* FALLTHROUGH */
> 	default:
> 		prot = FIXMAP_PAGE_NORMAL;
> 		
> 	}
> 
> 	return (__force void *)__early_ioremap(phys_addr, size, prot);
> }
> 

Re: [RFC PATCH v1 10/18] x86/efi: Access EFI related tables in the clear

[Matt Fleming]

On Tue, 26 Apr, at 05:57:40PM, Tom Lendacky wrote:
> The EFI tables are not encrypted and need to be accessed as such. Be sure
> to memmap them without the encryption attribute set. For EFI support that
> lives outside of the arch/x86 tree, create a routine that uses the __weak
> attribute so that it can be overridden by an architecture specific routine.
> 
> When freeing boot services related memory, since it has been mapped as
> un-encrypted, be sure to change the mapping to encrypted for future use.
> 
> Signed-off-by: Tom Lendacky <thomas.lendacky@amd.com>
> ---
>  arch/x86/include/asm/cacheflush.h  |    3 +
>  arch/x86/include/asm/mem_encrypt.h |   22 +++++++++++
>  arch/x86/kernel/setup.c            |    6 +--
>  arch/x86/mm/mem_encrypt.c          |   56 +++++++++++++++++++++++++++
>  arch/x86/mm/pageattr.c             |   75 ++++++++++++++++++++++++++++++++++++
>  arch/x86/platform/efi/efi.c        |   26 +++++++-----
>  arch/x86/platform/efi/efi_64.c     |    9 +++-
>  arch/x86/platform/efi/quirks.c     |   12 +++++-
>  drivers/firmware/efi/efi.c         |   18 +++++++--
>  drivers/firmware/efi/esrt.c        |   12 +++---
>  include/linux/efi.h                |    3 +
>  11 files changed, 212 insertions(+), 30 deletions(-)

[...]

> diff --git a/arch/x86/platform/efi/efi.c b/arch/x86/platform/efi/efi.c
> index 994a7df8..871b213 100644
> --- a/arch/x86/platform/efi/efi.c
> +++ b/arch/x86/platform/efi/efi.c
> @@ -53,6 +53,7 @@
>  #include <asm/x86_init.h>
>  #include <asm/rtc.h>
>  #include <asm/uv/uv.h>
> +#include <asm/mem_encrypt.h>
>  
>  #define EFI_DEBUG
>  
> @@ -261,12 +262,12 @@ static int __init efi_systab_init(void *phys)
>  		u64 tmp = 0;
>  
>  		if (efi_setup) {
> -			data = early_memremap(efi_setup, sizeof(*data));
> +			data = sme_early_memremap(efi_setup, sizeof(*data));
>  			if (!data)
>  				return -ENOMEM;
>  		}

Beware, this data comes from a previous kernel that kexec'd this
kernel. Unless you've updated bzImage64_load() to allocate an
unencrypted region 'efi_setup' will in fact be encrypted.

> @@ -690,6 +691,7 @@ static void *realloc_pages(void *old_memmap, int old_shift)
>  	ret = (void *)__get_free_pages(GFP_KERNEL, old_shift + 1);
>  	if (!ret)
>  		goto out;
> +	sme_set_mem_dec(ret, PAGE_SIZE << (old_shift + 1));
>  
>  	/*
>  	 * A first-time allocation doesn't have anything to copy.

I'm not sure why it's necessary to mark this region as unencrypted,
because at this point the kernel controls the platform and when we
call into the firmware it should be using our page tables. I wouldn't
expect the firmware to mess with the SYSCFG MSR either.

Have you come across a situation where the above was required?

> diff --git a/arch/x86/platform/efi/efi_64.c b/arch/x86/platform/efi/efi_64.c
> index 49e4dd4..834a992 100644
> --- a/arch/x86/platform/efi/efi_64.c
> +++ b/arch/x86/platform/efi/efi_64.c
> @@ -223,7 +223,7 @@ int __init efi_setup_page_tables(unsigned long pa_memmap, unsigned num_pages)
>  	if (efi_enabled(EFI_OLD_MEMMAP))
>  		return 0;
>  
> -	efi_scratch.efi_pgt = (pgd_t *)__pa(efi_pgd);
> +	efi_scratch.efi_pgt = (pgd_t *)__sme_pa(efi_pgd);
>  	pgd = efi_pgd;
>  
>  	/*

Huh? Why does __pa() now OR in sme_mas_mask? I thought SME only
required the page table structures to be modified, not the end
address?

> @@ -262,7 +262,8 @@ int __init efi_setup_page_tables(unsigned long pa_memmap, unsigned num_pages)
>  		pfn = md->phys_addr >> PAGE_SHIFT;
>  		npages = md->num_pages;
>  
> -		if (kernel_map_pages_in_pgd(pgd, pfn, md->phys_addr, npages, _PAGE_RW)) {
> +		if (kernel_map_pages_in_pgd(pgd, pfn, md->phys_addr, npages,
> +					    _PAGE_RW | _PAGE_ENC)) {
>  			pr_err("Failed to map 1:1 memory\n");
>  			return 1;
>  		}

Could you push the _PAGE_ENC addition down into
kernel_map_pages_in_pgd()? Other flags are also handled that way, see
_PAGE_PRESENT.

> @@ -272,6 +273,7 @@ int __init efi_setup_page_tables(unsigned long pa_memmap, unsigned num_pages)
>  	if (!page)
>  		panic("Unable to allocate EFI runtime stack < 4GB\n");
>  
> +	sme_set_mem_dec(page_address(page), PAGE_SIZE);
>  	efi_scratch.phys_stack = virt_to_phys(page_address(page));
>  	efi_scratch.phys_stack += PAGE_SIZE; /* stack grows down */
>  

We should not need to mark the stack as unencrypted, the firmware
should respect our SME settings, right?

> diff --git a/arch/x86/platform/efi/quirks.c b/arch/x86/platform/efi/quirks.c
> index ab50ada..dde4fb6b 100644
> --- a/arch/x86/platform/efi/quirks.c
> +++ b/arch/x86/platform/efi/quirks.c
> @@ -13,6 +13,7 @@
>  #include <linux/dmi.h>
>  #include <asm/efi.h>
>  #include <asm/uv/uv.h>
> +#include <asm/mem_encrypt.h>
>  
>  #define EFI_MIN_RESERVE 5120
>  
> @@ -265,6 +266,13 @@ void __init efi_free_boot_services(void)
>  		if (md->attribute & EFI_MEMORY_RUNTIME)
>  			continue;
>  
> +		/*
> +		 * Change the mapping to encrypted memory before freeing.
> +		 * This insures any future allocations of this mapped area
> +		 * are used encrypted.
> +		 */
> +		sme_set_mem_enc(__va(start), size);
> +
>  		free_bootmem_late(start, size);
>  	}
>  

I don't think it's necessary to have to mark the __va() mapping of
these regions as encrypted at this point. They should be setup that
way initially.

The reason is that it'd be a bug if these regions were accessed via
the __va() mappings before this point. Unless there's something I'm
missing.

> diff --git a/drivers/firmware/efi/efi.c b/drivers/firmware/efi/efi.c
> index 3a69ed5..25010c7 100644
> --- a/drivers/firmware/efi/efi.c
> +++ b/drivers/firmware/efi/efi.c
> @@ -76,6 +76,16 @@ static int __init parse_efi_cmdline(char *str)
>  }
>  early_param("efi", parse_efi_cmdline);
>  
> +/*
> + * If memory encryption is supported, then an override to this function
> + * will be provided.
> + */
> +void __weak __init *efi_me_early_memremap(resource_size_t phys_addr,
> +					  unsigned long size)
> +{
> +	return early_memremap(phys_addr, size);
> +}
> +
>  struct kobject *efi_kobj;
>  
>  /*

Like I said in my other mail, I'd much prefer to see this buried in
arch/x86 by passing a flag to early_memremap() which can be parsed in
arch directories.

Re: [RFC PATCH v1 10/18] x86/efi: Access EFI related tables in the clear

[Tom Lendacky]

On 06/08/2016 06:18 AM, Matt Fleming wrote:
> On Tue, 26 Apr, at 05:57:40PM, Tom Lendacky wrote:
>> The EFI tables are not encrypted and need to be accessed as such. Be sure
>> to memmap them without the encryption attribute set. For EFI support that
>> lives outside of the arch/x86 tree, create a routine that uses the __weak
>> attribute so that it can be overridden by an architecture specific routine.
>>
>> When freeing boot services related memory, since it has been mapped as
>> un-encrypted, be sure to change the mapping to encrypted for future use.
>>
>> Signed-off-by: Tom Lendacky <thomas.lendacky@amd.com>
>> ---
>>  arch/x86/include/asm/cacheflush.h  |    3 +
>>  arch/x86/include/asm/mem_encrypt.h |   22 +++++++++++
>>  arch/x86/kernel/setup.c            |    6 +--
>>  arch/x86/mm/mem_encrypt.c          |   56 +++++++++++++++++++++++++++
>>  arch/x86/mm/pageattr.c             |   75 ++++++++++++++++++++++++++++++++++++
>>  arch/x86/platform/efi/efi.c        |   26 +++++++-----
>>  arch/x86/platform/efi/efi_64.c     |    9 +++-
>>  arch/x86/platform/efi/quirks.c     |   12 +++++-
>>  drivers/firmware/efi/efi.c         |   18 +++++++--
>>  drivers/firmware/efi/esrt.c        |   12 +++---
>>  include/linux/efi.h                |    3 +
>>  11 files changed, 212 insertions(+), 30 deletions(-)
> 
> [...]
> 
>> diff --git a/arch/x86/platform/efi/efi.c b/arch/x86/platform/efi/efi.c
>> index 994a7df8..871b213 100644
>> --- a/arch/x86/platform/efi/efi.c
>> +++ b/arch/x86/platform/efi/efi.c
>> @@ -53,6 +53,7 @@
>>  #include <asm/x86_init.h>
>>  #include <asm/rtc.h>
>>  #include <asm/uv/uv.h>
>> +#include <asm/mem_encrypt.h>
>>  
>>  #define EFI_DEBUG
>>  
>> @@ -261,12 +262,12 @@ static int __init efi_systab_init(void *phys)
>>  		u64 tmp = 0;
>>  
>>  		if (efi_setup) {
>> -			data = early_memremap(efi_setup, sizeof(*data));
>> +			data = sme_early_memremap(efi_setup, sizeof(*data));
>>  			if (!data)
>>  				return -ENOMEM;
>>  		}
> 
> Beware, this data comes from a previous kernel that kexec'd this
> kernel. Unless you've updated bzImage64_load() to allocate an
> unencrypted region 'efi_setup' will in fact be encrypted.

Yes, I missed the kexec path originally and need to take that into
account in general.

> 
>> @@ -690,6 +691,7 @@ static void *realloc_pages(void *old_memmap, int old_shift)
>>  	ret = (void *)__get_free_pages(GFP_KERNEL, old_shift + 1);
>>  	if (!ret)
>>  		goto out;
>> +	sme_set_mem_dec(ret, PAGE_SIZE << (old_shift + 1));
>>  
>>  	/*
>>  	 * A first-time allocation doesn't have anything to copy.
> 
> I'm not sure why it's necessary to mark this region as unencrypted,
> because at this point the kernel controls the platform and when we
> call into the firmware it should be using our page tables. I wouldn't
> expect the firmware to mess with the SYSCFG MSR either.
> 
> Have you come across a situation where the above was required?

I was trying to play it safe here, but as you say, the firmware should
be using our page tables so we can get rid of this call. The problem
will actually be if we transition to a 32-bit efi. The encryption bit
will be lost in cr3 and so the pgd table will have to be un-encrypted.
The entries in the pgd can have the encryption bit set so I would only
need to worry about the pgd itself. I'll have to update the
efi_alloc_page_tables routine.

> 
>> diff --git a/arch/x86/platform/efi/efi_64.c b/arch/x86/platform/efi/efi_64.c
>> index 49e4dd4..834a992 100644
>> --- a/arch/x86/platform/efi/efi_64.c
>> +++ b/arch/x86/platform/efi/efi_64.c
>> @@ -223,7 +223,7 @@ int __init efi_setup_page_tables(unsigned long pa_memmap, unsigned num_pages)
>>  	if (efi_enabled(EFI_OLD_MEMMAP))
>>  		return 0;
>>  
>> -	efi_scratch.efi_pgt = (pgd_t *)__pa(efi_pgd);
>> +	efi_scratch.efi_pgt = (pgd_t *)__sme_pa(efi_pgd);
>>  	pgd = efi_pgd;
>>  
>>  	/*
> 
> Huh? Why does __pa() now OR in sme_mas_mask? I thought SME only
> required the page table structures to be modified, not the end
> address?

The encryption bit in the cr3 register will indicate if the pgd table
is encrypted or not. Based on my comment above about the pgd having
to be un-encrypted in case we have to transition to 32-bit efi, this
can be removed.

> 
>> @@ -262,7 +262,8 @@ int __init efi_setup_page_tables(unsigned long pa_memmap, unsigned num_pages)
>>  		pfn = md->phys_addr >> PAGE_SHIFT;
>>  		npages = md->num_pages;
>>  
>> -		if (kernel_map_pages_in_pgd(pgd, pfn, md->phys_addr, npages, _PAGE_RW)) {
>> +		if (kernel_map_pages_in_pgd(pgd, pfn, md->phys_addr, npages,
>> +					    _PAGE_RW | _PAGE_ENC)) {
>>  			pr_err("Failed to map 1:1 memory\n");
>>  			return 1;
>>  		}
> 
> Could you push the _PAGE_ENC addition down into
> kernel_map_pages_in_pgd()? Other flags are also handled that way, see
> _PAGE_PRESENT.

I'll look into this a bit more. From looking at it I don't want the
_PAGE_ENC bit set for the memmap unless it gets re-allocated (which
I missed in these patches). Let me see what I can do with this.

> 
>> @@ -272,6 +273,7 @@ int __init efi_setup_page_tables(unsigned long pa_memmap, unsigned num_pages)
>>  	if (!page)
>>  		panic("Unable to allocate EFI runtime stack < 4GB\n");
>>  
>> +	sme_set_mem_dec(page_address(page), PAGE_SIZE);
>>  	efi_scratch.phys_stack = virt_to_phys(page_address(page));
>>  	efi_scratch.phys_stack += PAGE_SIZE; /* stack grows down */
>>  
> 
> We should not need to mark the stack as unencrypted, the firmware
> should respect our SME settings, right?

Yup, you're correct. I think we can get rid of this call, too.

> 
>> diff --git a/arch/x86/platform/efi/quirks.c b/arch/x86/platform/efi/quirks.c
>> index ab50ada..dde4fb6b 100644
>> --- a/arch/x86/platform/efi/quirks.c
>> +++ b/arch/x86/platform/efi/quirks.c
>> @@ -13,6 +13,7 @@
>>  #include <linux/dmi.h>
>>  #include <asm/efi.h>
>>  #include <asm/uv/uv.h>
>> +#include <asm/mem_encrypt.h>
>>  
>>  #define EFI_MIN_RESERVE 5120
>>  
>> @@ -265,6 +266,13 @@ void __init efi_free_boot_services(void)
>>  		if (md->attribute & EFI_MEMORY_RUNTIME)
>>  			continue;
>>  
>> +		/*
>> +		 * Change the mapping to encrypted memory before freeing.
>> +		 * This insures any future allocations of this mapped area
>> +		 * are used encrypted.
>> +		 */
>> +		sme_set_mem_enc(__va(start), size);
>> +
>>  		free_bootmem_late(start, size);
>>  	}
>>  
> 
> I don't think it's necessary to have to mark the __va() mapping of
> these regions as encrypted at this point. They should be setup that
> way initially.
> 
> The reason is that it'd be a bug if these regions were accessed via
> the __va() mappings before this point. Unless there's something I'm
> missing.

I'll look further into this, but I saw that this area of virtual memory
was mapped un-encrypted and after freeing the boot services the
mappings were somehow reused as un-encrypted for DMA which assumes
(unless using swiotlb) encrypted. This resulted in DMA data being
transferred in as encrypted and then accessed un-encrypted.

Thanks,
Tom

> 
>> diff --git a/drivers/firmware/efi/efi.c b/drivers/firmware/efi/efi.c
>> index 3a69ed5..25010c7 100644
>> --- a/drivers/firmware/efi/efi.c
>> +++ b/drivers/firmware/efi/efi.c
>> @@ -76,6 +76,16 @@ static int __init parse_efi_cmdline(char *str)
>>  }
>>  early_param("efi", parse_efi_cmdline);
>>  
>> +/*
>> + * If memory encryption is supported, then an override to this function
>> + * will be provided.
>> + */
>> +void __weak __init *efi_me_early_memremap(resource_size_t phys_addr,
>> +					  unsigned long size)
>> +{
>> +	return early_memremap(phys_addr, size);
>> +}
>> +
>>  struct kobject *efi_kobj;
>>  
>>  /*
> 
> Like I said in my other mail, I'd much prefer to see this buried in
> arch/x86 by passing a flag to early_memremap() which can be parsed in
> arch directories.
> 

Re: [RFC PATCH v1 10/18] x86/efi: Access EFI related tables in the clear

[Matt Fleming]

On Thu, 09 Jun, at 01:33:30PM, Tom Lendacky wrote:
> 
> I was trying to play it safe here, but as you say, the firmware should
> be using our page tables so we can get rid of this call. The problem
> will actually be if we transition to a 32-bit efi. The encryption bit
> will be lost in cr3 and so the pgd table will have to be un-encrypted.
> The entries in the pgd can have the encryption bit set so I would only
> need to worry about the pgd itself. I'll have to update the
> efi_alloc_page_tables routine.
 
Interesting, I hadn't expected 32-bit EFI to be an option for
platforms with the SME technology. I'd assumed we could just ignore
that.

Are you saying that the encryption bit isn't supported in 32-bit
compatibility mode? We don't do a "full" switch to 32-bit protected
mode when in mixed mode, just load a 32-bit code segment descriptor.
The page tables are not modified at all.

> The encryption bit in the cr3 register will indicate if the pgd table
> is encrypted or not. Based on my comment above about the pgd having
> to be un-encrypted in case we have to transition to 32-bit efi, this
> can be removed.
 
I'm not (yet) sure that the pgd needs to be unencrypted for 32-bit EFI
when running a 64-bit kernel. In the AMD Programmer's Manual, Section
7.10.3 Operating Modes seems to indicate that running encrypted should
work fine.

> I'll look into this a bit more. From looking at it I don't want the
> _PAGE_ENC bit set for the memmap unless it gets re-allocated (which
> I missed in these patches). Let me see what I can do with this.
 
I don't understand your comment about re-allocating the memmap.

The kernel builds its own EFI memory map at runtime, initially based
on the memory map provided by the firmware. We always allocate a new
memory map.

In efi_setup_page_tables() we're building our own page tables, which
should be encrypted, and mapping EFI regions described by the memmap
into those page tables.

So unless we're mapping an MMIO region (in which case _PAGE_PCD is set
in @flags for kernel_map_pages_in_pgd()) I would expect _PAGE_ENC to
be set.

> I'll look further into this, but I saw that this area of virtual memory
> was mapped un-encrypted and after freeing the boot services the
> mappings were somehow reused as un-encrypted for DMA which assumes
> (unless using swiotlb) encrypted. This resulted in DMA data being
> transferred in as encrypted and then accessed un-encrypted.

That the mappings were re-used isn't a surprise.

efi_free_boot_services() lifts the reservation that was put in place
during efi_reserve_boot_services() and releases the pages to the
kernel's memory allocators.

What is surprising is that they were marked unencrypted at all.
There's nothing special about these pages as far as the __va() region
is concerned.

Re: [RFC PATCH v1 10/18] x86/efi: Access EFI related tables in the clear

[Tom Lendacky]

On 06/13/2016 08:51 AM, Matt Fleming wrote:
> On Thu, 09 Jun, at 01:33:30PM, Tom Lendacky wrote:
>>
>> I was trying to play it safe here, but as you say, the firmware should
>> be using our page tables so we can get rid of this call. The problem
>> will actually be if we transition to a 32-bit efi. The encryption bit
>> will be lost in cr3 and so the pgd table will have to be un-encrypted.
>> The entries in the pgd can have the encryption bit set so I would only
>> need to worry about the pgd itself. I'll have to update the
>> efi_alloc_page_tables routine.
>  
> Interesting, I hadn't expected 32-bit EFI to be an option for
> platforms with the SME technology. I'd assumed we could just ignore
> that.

We may be able to do that.

> 
> Are you saying that the encryption bit isn't supported in 32-bit
> compatibility mode? We don't do a "full" switch to 32-bit protected
> mode when in mixed mode, just load a 32-bit code segment descriptor.
> The page tables are not modified at all.

The encryption bit is supported in 32-bit compatibility mode and since
we're not doing the "full" switch the cr3 register will remain as a
64-bit register so we can leave the pgd table encrypted.

> 
>> The encryption bit in the cr3 register will indicate if the pgd table
>> is encrypted or not. Based on my comment above about the pgd having
>> to be un-encrypted in case we have to transition to 32-bit efi, this
>> can be removed.
>  
> I'm not (yet) sure that the pgd needs to be unencrypted for 32-bit EFI
> when running a 64-bit kernel. In the AMD Programmer's Manual, Section
> 7.10.3 Operating Modes seems to indicate that running encrypted should
> work fine.
> 
>> I'll look into this a bit more. From looking at it I don't want the
>> _PAGE_ENC bit set for the memmap unless it gets re-allocated (which
>> I missed in these patches). Let me see what I can do with this.
>  
> I don't understand your comment about re-allocating the memmap.
> 
> The kernel builds its own EFI memory map at runtime, initially based
> on the memory map provided by the firmware. We always allocate a new
> memory map.

Sorry, I mis-interpreted the efi_map_regions function/loop and see
that the memmap is always allocated by the kernel.

> 
> In efi_setup_page_tables() we're building our own page tables, which
> should be encrypted, and mapping EFI regions described by the memmap
> into those page tables.
> 
> So unless we're mapping an MMIO region (in which case _PAGE_PCD is set
> in @flags for kernel_map_pages_in_pgd()) I would expect _PAGE_ENC to
> be set.
> 
>> I'll look further into this, but I saw that this area of virtual memory
>> was mapped un-encrypted and after freeing the boot services the
>> mappings were somehow reused as un-encrypted for DMA which assumes
>> (unless using swiotlb) encrypted. This resulted in DMA data being
>> transferred in as encrypted and then accessed un-encrypted.
> 
> That the mappings were re-used isn't a surprise.
> 
> efi_free_boot_services() lifts the reservation that was put in place
> during efi_reserve_boot_services() and releases the pages to the
> kernel's memory allocators.
> 
> What is surprising is that they were marked unencrypted at all.
> There's nothing special about these pages as far as the __va() region
> is concerned.

Right, let me keep looking into this to see if I can pin down what
was (or is) happening.

Thanks,
Tom

> 

Re: [RFC PATCH v1 10/18] x86/efi: Access EFI related tables in the clear

[Tom Lendacky]

On 06/15/2016 08:17 AM, Tom Lendacky wrote:
> On 06/13/2016 08:51 AM, Matt Fleming wrote:
>> On Thu, 09 Jun, at 01:33:30PM, Tom Lendacky wrote:
>>>

[...]

>>
>>> I'll look further into this, but I saw that this area of virtual memory
>>> was mapped un-encrypted and after freeing the boot services the
>>> mappings were somehow reused as un-encrypted for DMA which assumes
>>> (unless using swiotlb) encrypted. This resulted in DMA data being
>>> transferred in as encrypted and then accessed un-encrypted.
>>
>> That the mappings were re-used isn't a surprise.
>>
>> efi_free_boot_services() lifts the reservation that was put in place
>> during efi_reserve_boot_services() and releases the pages to the
>> kernel's memory allocators.
>>
>> What is surprising is that they were marked unencrypted at all.
>> There's nothing special about these pages as far as the __va() region
>> is concerned.
> 
> Right, let me keep looking into this to see if I can pin down what
> was (or is) happening.

Ok, I think this was happening before the commit to build our own
EFI page table structures:

commit 67a9108ed ("x86/efi: Build our own page table structures")

Before this commit the boot services ended up mapped into the kernel
page table entries as un-encrypted during efi_map_regions() and I needed
to change those entries back to encrypted. With your change above,
this appears to no longer be needed.

Thanks,
Tom

> 
> Thanks,
> Tom
> 
>>

Re: [RFC PATCH v1 10/18] x86/efi: Access EFI related tables in the clear

[Matt Fleming]

On Thu, 16 Jun, at 09:38:31AM, Tom Lendacky wrote:
> 
> Ok, I think this was happening before the commit to build our own
> EFI page table structures:
> 
> commit 67a9108ed ("x86/efi: Build our own page table structures")
> 
> Before this commit the boot services ended up mapped into the kernel
> page table entries as un-encrypted during efi_map_regions() and I needed
> to change those entries back to encrypted. With your change above,
> this appears to no longer be needed.

Great news! Things are as they should be ;)

[RFC PATCH v1 11/18] x86: Decrypt trampoline area if memory encryption is active

[Tom Lendacky]

When Secure Memory Encryption is enabled, the trampoline area must not
be encrypted. A cpu running in real mode will not be able to decrypt
memory that has been encrypted because it will not be able to use addresses
with the memory encryption mask.

Signed-off-by: Tom Lendacky <thomas.lendacky@amd.com>
---
 arch/x86/realmode/init.c |    9 +++++++++
 1 file changed, 9 insertions(+)

diff --git a/arch/x86/realmode/init.c b/arch/x86/realmode/init.c
index 0b7a63d..85b145c 100644
--- a/arch/x86/realmode/init.c
+++ b/arch/x86/realmode/init.c
@@ -4,6 +4,7 @@
 #include <asm/cacheflush.h>
 #include <asm/pgtable.h>
 #include <asm/realmode.h>
+#include <asm/mem_encrypt.h>
 
 struct real_mode_header *real_mode_header;
 u32 *trampoline_cr4_features;
@@ -113,6 +114,14 @@ static int __init set_real_mode_permissions(void)
 	unsigned long text_start =
 		(unsigned long) __va(real_mode_header->text_start);
 
+	/*
+	 * If memory encryption is active, the trampoline area will need to
+	 * be in non-encrypted memory in order to bring up other processors
+	 * successfully.
+	 */
+	sme_early_mem_dec(__pa(base), size);
+	sme_set_mem_dec(base, size);
+
 	set_memory_nx((unsigned long) base, size >> PAGE_SHIFT);
 	set_memory_ro((unsigned long) base, ro_size >> PAGE_SHIFT);
 	set_memory_x((unsigned long) text_start, text_size >> PAGE_SHIFT);

[RFC PATCH v1 12/18] x86: Access device tree in the clear

[Tom Lendacky]

The device tree is not encrypted and needs to be accessed as such. Be sure
to memmap it without the encryption mask set.

Signed-off-by: Tom Lendacky <thomas.lendacky@amd.com>
---
 arch/x86/kernel/devicetree.c |    6 ++++--
 1 file changed, 4 insertions(+), 2 deletions(-)

diff --git a/arch/x86/kernel/devicetree.c b/arch/x86/kernel/devicetree.c
index 3fe45f8..ff11f7a 100644
--- a/arch/x86/kernel/devicetree.c
+++ b/arch/x86/kernel/devicetree.c
@@ -22,6 +22,7 @@
 #include <asm/pci_x86.h>
 #include <asm/setup.h>
 #include <asm/i8259.h>
+#include <asm/mem_encrypt.h>
 
 __initdata u64 initial_dtb;
 char __initdata cmd_line[COMMAND_LINE_SIZE];
@@ -276,11 +277,12 @@ static void __init x86_flattree_get_config(void)
 
 	map_len = max(PAGE_SIZE - (initial_dtb & ~PAGE_MASK), (u64)128);
 
-	initial_boot_params = dt = early_memremap(initial_dtb, map_len);
+	initial_boot_params = dt = sme_early_memremap(initial_dtb, map_len);
 	size = of_get_flat_dt_size();
 	if (map_len < size) {
 		early_memunmap(dt, map_len);
-		initial_boot_params = dt = early_memremap(initial_dtb, size);
+		initial_boot_params = dt = sme_early_memremap(initial_dtb,
+							      size);
 		map_len = size;
 	}
 

[RFC PATCH v1 13/18] x86: DMA support for memory encryption

[Tom Lendacky]

Since DMA addresses will effectively look like 48-bit addresses when the
memory encryption mask is set, SWIOTLB is needed if the DMA mask of the
device performing the DMA does not support 48-bits. SWIOTLB will be
initialized to create un-encrypted bounce buffers for use by these devices.

Signed-off-by: Tom Lendacky <thomas.lendacky@amd.com>
---
 arch/x86/include/asm/dma-mapping.h |    5 ++-
 arch/x86/include/asm/mem_encrypt.h |    5 +++
 arch/x86/kernel/pci-dma.c          |   11 ++++--
 arch/x86/kernel/pci-nommu.c        |    2 +
 arch/x86/kernel/pci-swiotlb.c      |    8 +++--
 arch/x86/mm/mem_encrypt.c          |   21 ++++++++++++
 include/linux/swiotlb.h            |    1 +
 init/main.c                        |    6 +++
 lib/swiotlb.c                      |   64 ++++++++++++++++++++++++++++++++----
 9 files changed, 106 insertions(+), 17 deletions(-)

diff --git a/arch/x86/include/asm/dma-mapping.h b/arch/x86/include/asm/dma-mapping.h
index 3a27b93..33a4f6d 100644
--- a/arch/x86/include/asm/dma-mapping.h
+++ b/arch/x86/include/asm/dma-mapping.h
@@ -13,6 +13,7 @@
 #include <asm/io.h>
 #include <asm/swiotlb.h>
 #include <linux/dma-contiguous.h>
+#include <asm/mem_encrypt.h>
 
 #ifdef CONFIG_ISA
 # define ISA_DMA_BIT_MASK DMA_BIT_MASK(24)
@@ -70,12 +71,12 @@ static inline bool dma_capable(struct device *dev, dma_addr_t addr, size_t size)
 
 static inline dma_addr_t phys_to_dma(struct device *dev, phys_addr_t paddr)
 {
-	return paddr;
+	return paddr | sme_me_mask;
 }
 
 static inline phys_addr_t dma_to_phys(struct device *dev, dma_addr_t daddr)
 {
-	return daddr;
+	return daddr & ~sme_me_mask;
 }
 #endif /* CONFIG_X86_DMA_REMAP */
 
diff --git a/arch/x86/include/asm/mem_encrypt.h b/arch/x86/include/asm/mem_encrypt.h
index 42868f5..d17d8cf 100644
--- a/arch/x86/include/asm/mem_encrypt.h
+++ b/arch/x86/include/asm/mem_encrypt.h
@@ -37,6 +37,11 @@ void __init *sme_early_memremap(resource_size_t paddr,
 void __init sme_early_init(void);
 
 /* Architecture __weak replacement functions */
+void __init mem_encrypt_init(void);
+
+unsigned long swiotlb_get_me_mask(void);
+void swiotlb_set_mem_dec(void *vaddr, unsigned long size);
+
 void __init *efi_me_early_memremap(resource_size_t paddr,
 				   unsigned long size);
 
diff --git a/arch/x86/kernel/pci-dma.c b/arch/x86/kernel/pci-dma.c
index 6ba014c..bd1daae 100644
--- a/arch/x86/kernel/pci-dma.c
+++ b/arch/x86/kernel/pci-dma.c
@@ -92,9 +92,12 @@ again:
 	/* CMA can be used only in the context which permits sleeping */
 	if (gfpflags_allow_blocking(flag)) {
 		page = dma_alloc_from_contiguous(dev, count, get_order(size));
-		if (page && page_to_phys(page) + size > dma_mask) {
-			dma_release_from_contiguous(dev, page, count);
-			page = NULL;
+		if (page) {
+			addr = phys_to_dma(dev, page_to_phys(page));
+			if (addr + size > dma_mask) {
+				dma_release_from_contiguous(dev, page, count);
+				page = NULL;
+			}
 		}
 	}
 	/* fallback */
@@ -103,7 +106,7 @@ again:
 	if (!page)
 		return NULL;
 
-	addr = page_to_phys(page);
+	addr = phys_to_dma(dev, page_to_phys(page));
 	if (addr + size > dma_mask) {
 		__free_pages(page, get_order(size));
 
diff --git a/arch/x86/kernel/pci-nommu.c b/arch/x86/kernel/pci-nommu.c
index da15918..ca2b820 100644
--- a/arch/x86/kernel/pci-nommu.c
+++ b/arch/x86/kernel/pci-nommu.c
@@ -30,7 +30,7 @@ static dma_addr_t nommu_map_page(struct device *dev, struct page *page,
 				 enum dma_data_direction dir,
 				 struct dma_attrs *attrs)
 {
-	dma_addr_t bus = page_to_phys(page) + offset;
+	dma_addr_t bus = phys_to_dma(dev, page_to_phys(page)) + offset;
 	WARN_ON(size == 0);
 	if (!check_addr("map_single", dev, bus, size))
 		return DMA_ERROR_CODE;
diff --git a/arch/x86/kernel/pci-swiotlb.c b/arch/x86/kernel/pci-swiotlb.c
index 7c577a1..0ae083d 100644
--- a/arch/x86/kernel/pci-swiotlb.c
+++ b/arch/x86/kernel/pci-swiotlb.c
@@ -12,6 +12,8 @@
 #include <asm/dma.h>
 #include <asm/xen/swiotlb-xen.h>
 #include <asm/iommu_table.h>
+#include <asm/mem_encrypt.h>
+
 int swiotlb __read_mostly;
 
 void *x86_swiotlb_alloc_coherent(struct device *hwdev, size_t size,
@@ -64,13 +66,15 @@ static struct dma_map_ops swiotlb_dma_ops = {
  * pci_swiotlb_detect_override - set swiotlb to 1 if necessary
  *
  * This returns non-zero if we are forced to use swiotlb (by the boot
- * option).
+ * option). If memory encryption is enabled then swiotlb will be set
+ * to 1 so that bounce buffers are allocated and used for devices that
+ * do not support the addressing range required for the encryption mask.
  */
 int __init pci_swiotlb_detect_override(void)
 {
 	int use_swiotlb = swiotlb | swiotlb_force;
 
-	if (swiotlb_force)
+	if (swiotlb_force || sme_me_mask)
 		swiotlb = 1;
 
 	return use_swiotlb;
diff --git a/arch/x86/mm/mem_encrypt.c b/arch/x86/mm/mem_encrypt.c
index 7d56d1b..594dc65 100644
--- a/arch/x86/mm/mem_encrypt.c
+++ b/arch/x86/mm/mem_encrypt.c
@@ -12,6 +12,8 @@
 
 #include <linux/init.h>
 #include <linux/mm.h>
+#include <linux/dma-mapping.h>
+#include <linux/swiotlb.h>
 
 #include <asm/mem_encrypt.h>
 #include <asm/cacheflush.h>
@@ -168,6 +170,25 @@ void __init sme_early_init(void)
 }
 
 /* Architecture __weak replacement functions */
+void __init mem_encrypt_init(void)
+{
+	if (!sme_me_mask)
+		return;
+
+	/* Make SWIOTLB use an unencrypted DMA area */
+	swiotlb_clear_encryption();
+}
+
+unsigned long swiotlb_get_me_mask(void)
+{
+	return sme_me_mask;
+}
+
+void swiotlb_set_mem_dec(void *vaddr, unsigned long size)
+{
+	sme_set_mem_dec(vaddr, size);
+}
+
 void __init *efi_me_early_memremap(resource_size_t paddr,
 				   unsigned long size)
 {
diff --git a/include/linux/swiotlb.h b/include/linux/swiotlb.h
index 017fced..121b9de 100644
--- a/include/linux/swiotlb.h
+++ b/include/linux/swiotlb.h
@@ -30,6 +30,7 @@ int swiotlb_init_with_tbl(char *tlb, unsigned long nslabs, int verbose);
 extern unsigned long swiotlb_nr_tbl(void);
 unsigned long swiotlb_size_or_default(void);
 extern int swiotlb_late_init_with_tbl(char *tlb, unsigned long nslabs);
+extern void __init swiotlb_clear_encryption(void);
 
 /*
  * Enumeration for sync targets
diff --git a/init/main.c b/init/main.c
index b3c6e36..1013d1c 100644
--- a/init/main.c
+++ b/init/main.c
@@ -458,6 +458,10 @@ void __init __weak thread_info_cache_init(void)
 }
 #endif
 
+void __init __weak mem_encrypt_init(void)
+{
+}
+
 /*
  * Set up kernel memory allocators
  */
@@ -597,6 +601,8 @@ asmlinkage __visible void __init start_kernel(void)
 	 */
 	locking_selftest();
 
+	mem_encrypt_init();
+
 #ifdef CONFIG_BLK_DEV_INITRD
 	if (initrd_start && !initrd_below_start_ok &&
 	    page_to_pfn(virt_to_page((void *)initrd_start)) < min_low_pfn) {
diff --git a/lib/swiotlb.c b/lib/swiotlb.c
index 76f29ec..339ffdc 100644
--- a/lib/swiotlb.c
+++ b/lib/swiotlb.c
@@ -131,6 +131,26 @@ unsigned long swiotlb_size_or_default(void)
 	return size ? size : (IO_TLB_DEFAULT_SIZE);
 }
 
+/*
+ * Support for memory encryption. If memory encryption is supported, then an
+ * override to these functions will be provided.
+ */
+unsigned long __weak swiotlb_get_me_mask(void)
+{
+	return 0;
+}
+
+void __weak swiotlb_set_mem_dec(void *vaddr, unsigned long size)
+{
+}
+
+/* For swiotlb, clear memory encryption mask from dma addresses */
+static dma_addr_t swiotlb_phys_to_dma(struct device *hwdev,
+				      phys_addr_t address)
+{
+	return phys_to_dma(hwdev, address) & ~swiotlb_get_me_mask();
+}
+
 /* Note that this doesn't work with highmem page */
 static dma_addr_t swiotlb_virt_to_bus(struct device *hwdev,
 				      volatile void *address)
@@ -159,6 +179,30 @@ void swiotlb_print_info(void)
 	       bytes >> 20, vstart, vend - 1);
 }
 
+/*
+ * If memory encryption is active, the DMA address for an encrypted page may
+ * be beyond the range of the device. If bounce buffers are required be sure
+ * that they are not on an encrypted page. This should be called before the
+ * iotlb area is used.
+ */
+void __init swiotlb_clear_encryption(void)
+{
+	void *vaddr;
+	unsigned long bytes;
+
+	if (no_iotlb_memory || !io_tlb_start || late_alloc)
+		return;
+
+	vaddr = phys_to_virt(io_tlb_start);
+	bytes = PAGE_ALIGN(io_tlb_nslabs << IO_TLB_SHIFT);
+	swiotlb_set_mem_dec(vaddr, bytes);
+	memset(vaddr, 0, bytes);
+
+	vaddr = phys_to_virt(io_tlb_overflow_buffer);
+	bytes = PAGE_ALIGN(io_tlb_overflow);
+	swiotlb_set_mem_dec(vaddr, bytes);
+}
+
 int __init swiotlb_init_with_tbl(char *tlb, unsigned long nslabs, int verbose)
 {
 	void *v_overflow_buffer;
@@ -294,6 +338,8 @@ swiotlb_late_init_with_tbl(char *tlb, unsigned long nslabs)
 	io_tlb_start = virt_to_phys(tlb);
 	io_tlb_end = io_tlb_start + bytes;
 
+	/* Keep TLB in unencrypted memory if memory encryption is active */
+	swiotlb_set_mem_dec(tlb, bytes);
 	memset(tlb, 0, bytes);
 
 	/*
@@ -304,6 +350,8 @@ swiotlb_late_init_with_tbl(char *tlb, unsigned long nslabs)
 	if (!v_overflow_buffer)
 		goto cleanup2;
 
+	/* Keep overflow in unencrypted memory if memory encryption is active */
+	swiotlb_set_mem_dec(v_overflow_buffer, io_tlb_overflow);
 	io_tlb_overflow_buffer = virt_to_phys(v_overflow_buffer);
 
 	/*
@@ -541,7 +589,7 @@ static phys_addr_t
 map_single(struct device *hwdev, phys_addr_t phys, size_t size,
 	   enum dma_data_direction dir)
 {
-	dma_addr_t start_dma_addr = phys_to_dma(hwdev, io_tlb_start);
+	dma_addr_t start_dma_addr = swiotlb_phys_to_dma(hwdev, io_tlb_start);
 
 	return swiotlb_tbl_map_single(hwdev, start_dma_addr, phys, size, dir);
 }
@@ -659,7 +707,7 @@ swiotlb_alloc_coherent(struct device *hwdev, size_t size,
 			goto err_warn;
 
 		ret = phys_to_virt(paddr);
-		dev_addr = phys_to_dma(hwdev, paddr);
+		dev_addr = swiotlb_phys_to_dma(hwdev, paddr);
 
 		/* Confirm address can be DMA'd by device */
 		if (dev_addr + size - 1 > dma_mask) {
@@ -758,15 +806,15 @@ dma_addr_t swiotlb_map_page(struct device *dev, struct page *page,
 	map = map_single(dev, phys, size, dir);
 	if (map == SWIOTLB_MAP_ERROR) {
 		swiotlb_full(dev, size, dir, 1);
-		return phys_to_dma(dev, io_tlb_overflow_buffer);
+		return swiotlb_phys_to_dma(dev, io_tlb_overflow_buffer);
 	}
 
-	dev_addr = phys_to_dma(dev, map);
+	dev_addr = swiotlb_phys_to_dma(dev, map);
 
 	/* Ensure that the address returned is DMA'ble */
 	if (!dma_capable(dev, dev_addr, size)) {
 		swiotlb_tbl_unmap_single(dev, map, size, dir);
-		return phys_to_dma(dev, io_tlb_overflow_buffer);
+		return swiotlb_phys_to_dma(dev, io_tlb_overflow_buffer);
 	}
 
 	return dev_addr;
@@ -901,7 +949,7 @@ swiotlb_map_sg_attrs(struct device *hwdev, struct scatterlist *sgl, int nelems,
 				sg_dma_len(sgl) = 0;
 				return 0;
 			}
-			sg->dma_address = phys_to_dma(hwdev, map);
+			sg->dma_address = swiotlb_phys_to_dma(hwdev, map);
 		} else
 			sg->dma_address = dev_addr;
 		sg_dma_len(sg) = sg->length;
@@ -984,7 +1032,7 @@ EXPORT_SYMBOL(swiotlb_sync_sg_for_device);
 int
 swiotlb_dma_mapping_error(struct device *hwdev, dma_addr_t dma_addr)
 {
-	return (dma_addr == phys_to_dma(hwdev, io_tlb_overflow_buffer));
+	return (dma_addr == swiotlb_phys_to_dma(hwdev, io_tlb_overflow_buffer));
 }
 EXPORT_SYMBOL(swiotlb_dma_mapping_error);
 
@@ -997,6 +1045,6 @@ EXPORT_SYMBOL(swiotlb_dma_mapping_error);
 int
 swiotlb_dma_supported(struct device *hwdev, u64 mask)
 {
-	return phys_to_dma(hwdev, io_tlb_end - 1) <= mask;
+	return swiotlb_phys_to_dma(hwdev, io_tlb_end - 1) <= mask;
 }
 EXPORT_SYMBOL(swiotlb_dma_supported);

Re: [RFC PATCH v1 13/18] x86: DMA support for memory encryption

[Tom Lendacky]

On 04/29/2016 02:17 AM, Konrad Rzeszutek Wilk wrote:
> On Tue, Apr 26, 2016 at 05:58:12PM -0500, Tom Lendacky wrote:
>> Since DMA addresses will effectively look like 48-bit addresses when the
>> memory encryption mask is set, SWIOTLB is needed if the DMA mask of the
>> device performing the DMA does not support 48-bits. SWIOTLB will be
>> initialized to create un-encrypted bounce buffers for use by these devices.
>>
> 
> 
> I presume the sme_me_mask does not use the lower 48 bits?

The sme_me_mask will actually be bit 47. So, when applied, the address
will become a 48-bit address.

> 
> 
> ..snip..
>> diff --git a/arch/x86/mm/mem_encrypt.c b/arch/x86/mm/mem_encrypt.c
>> index 7d56d1b..594dc65 100644
>> --- a/arch/x86/mm/mem_encrypt.c
>> +++ b/arch/x86/mm/mem_encrypt.c
>> @@ -12,6 +12,8 @@
>>  
>>  #include <linux/init.h>
>>  #include <linux/mm.h>
>> +#include <linux/dma-mapping.h>
>> +#include <linux/swiotlb.h>
>>  
>>  #include <asm/mem_encrypt.h>
>>  #include <asm/cacheflush.h>
>> @@ -168,6 +170,25 @@ void __init sme_early_init(void)
>>  }
>>  
>>  /* Architecture __weak replacement functions */
>> +void __init mem_encrypt_init(void)
>> +{
>> +	if (!sme_me_mask)
>> +		return;
>> +
>> +	/* Make SWIOTLB use an unencrypted DMA area */
>> +	swiotlb_clear_encryption();
>> +}
>> +
>> +unsigned long swiotlb_get_me_mask(void)
>> +{
>> +	return sme_me_mask;
>> +}
>> +
>> +void swiotlb_set_mem_dec(void *vaddr, unsigned long size)
>> +{
>> +	sme_set_mem_dec(vaddr, size);
>> +}
>> +
>>  void __init *efi_me_early_memremap(resource_size_t paddr,
>>  				   unsigned long size)
>>  {
>> diff --git a/include/linux/swiotlb.h b/include/linux/swiotlb.h
>> index 017fced..121b9de 100644
>> --- a/include/linux/swiotlb.h
>> +++ b/include/linux/swiotlb.h
>> @@ -30,6 +30,7 @@ int swiotlb_init_with_tbl(char *tlb, unsigned long nslabs, int verbose);
>>  extern unsigned long swiotlb_nr_tbl(void);
>>  unsigned long swiotlb_size_or_default(void);
>>  extern int swiotlb_late_init_with_tbl(char *tlb, unsigned long nslabs);
>> +extern void __init swiotlb_clear_encryption(void);
>>  
>>  /*
>>   * Enumeration for sync targets
>> diff --git a/init/main.c b/init/main.c
>> index b3c6e36..1013d1c 100644
>> --- a/init/main.c
>> +++ b/init/main.c
>> @@ -458,6 +458,10 @@ void __init __weak thread_info_cache_init(void)
>>  }
>>  #endif
>>  
>> +void __init __weak mem_encrypt_init(void)
>> +{
>> +}
>> +
>>  /*
>>   * Set up kernel memory allocators
>>   */
>> @@ -597,6 +601,8 @@ asmlinkage __visible void __init start_kernel(void)
>>  	 */
>>  	locking_selftest();
>>  
>> +	mem_encrypt_init();
>> +
>>  #ifdef CONFIG_BLK_DEV_INITRD
>>  	if (initrd_start && !initrd_below_start_ok &&
>>  	    page_to_pfn(virt_to_page((void *)initrd_start)) < min_low_pfn) {
> 
> What happens if devices use the bounce buffer before mem_encrypt_init()?

The call to mem_encrypt_init is early in the boot process, I may have
overlooked something, but what devices would be performing DMA before
this?

Thanks,
Tom

> 
> ..snip..
> 

Re: [RFC PATCH v1 13/18] x86: DMA support for memory encryption

[Tom Lendacky]

On 04/29/2016 11:27 AM, Konrad Rzeszutek Wilk wrote:
> On Fri, Apr 29, 2016 at 10:12:45AM -0500, Tom Lendacky wrote:
>> On 04/29/2016 02:17 AM, Konrad Rzeszutek Wilk wrote:
>>> On Tue, Apr 26, 2016 at 05:58:12PM -0500, Tom Lendacky wrote:
>>>> Since DMA addresses will effectively look like 48-bit addresses when the
>>>> memory encryption mask is set, SWIOTLB is needed if the DMA mask of the
>>>> device performing the DMA does not support 48-bits. SWIOTLB will be
>>>> initialized to create un-encrypted bounce buffers for use by these devices.
>>>>
>>>
>>>
>>> I presume the sme_me_mask does not use the lower 48 bits?
>>
>> The sme_me_mask will actually be bit 47. So, when applied, the address
>> will become a 48-bit address.
>>
>>>
>>>
>>> ..snip..
>>>> diff --git a/arch/x86/mm/mem_encrypt.c b/arch/x86/mm/mem_encrypt.c
>>>> index 7d56d1b..594dc65 100644
>>>> --- a/arch/x86/mm/mem_encrypt.c
>>>> +++ b/arch/x86/mm/mem_encrypt.c
>>>> @@ -12,6 +12,8 @@
>>>>  
>>>>  #include <linux/init.h>
>>>>  #include <linux/mm.h>
>>>> +#include <linux/dma-mapping.h>
>>>> +#include <linux/swiotlb.h>
>>>>  
>>>>  #include <asm/mem_encrypt.h>
>>>>  #include <asm/cacheflush.h>
>>>> @@ -168,6 +170,25 @@ void __init sme_early_init(void)
>>>>  }
>>>>  
>>>>  /* Architecture __weak replacement functions */
>>>> +void __init mem_encrypt_init(void)
>>>> +{
>>>> +	if (!sme_me_mask)
>>>> +		return;
>>>> +
>>>> +	/* Make SWIOTLB use an unencrypted DMA area */
>>>> +	swiotlb_clear_encryption();
>>>> +}
>>>> +
>>>> +unsigned long swiotlb_get_me_mask(void)
>>>> +{
>>>> +	return sme_me_mask;
>>>> +}
>>>> +
>>>> +void swiotlb_set_mem_dec(void *vaddr, unsigned long size)
>>>> +{
>>>> +	sme_set_mem_dec(vaddr, size);
>>>> +}
>>>> +
>>>>  void __init *efi_me_early_memremap(resource_size_t paddr,
>>>>  				   unsigned long size)
>>>>  {
>>>> diff --git a/include/linux/swiotlb.h b/include/linux/swiotlb.h
>>>> index 017fced..121b9de 100644
>>>> --- a/include/linux/swiotlb.h
>>>> +++ b/include/linux/swiotlb.h
>>>> @@ -30,6 +30,7 @@ int swiotlb_init_with_tbl(char *tlb, unsigned long nslabs, int verbose);
>>>>  extern unsigned long swiotlb_nr_tbl(void);
>>>>  unsigned long swiotlb_size_or_default(void);
>>>>  extern int swiotlb_late_init_with_tbl(char *tlb, unsigned long nslabs);
>>>> +extern void __init swiotlb_clear_encryption(void);
>>>>  
>>>>  /*
>>>>   * Enumeration for sync targets
>>>> diff --git a/init/main.c b/init/main.c
>>>> index b3c6e36..1013d1c 100644
>>>> --- a/init/main.c
>>>> +++ b/init/main.c
>>>> @@ -458,6 +458,10 @@ void __init __weak thread_info_cache_init(void)
>>>>  }
>>>>  #endif
>>>>  
>>>> +void __init __weak mem_encrypt_init(void)
>>>> +{
>>>> +}
>>>> +
>>>>  /*
>>>>   * Set up kernel memory allocators
>>>>   */
>>>> @@ -597,6 +601,8 @@ asmlinkage __visible void __init start_kernel(void)
>>>>  	 */
>>>>  	locking_selftest();
>>>>  
>>>> +	mem_encrypt_init();
>>>> +
>>>>  #ifdef CONFIG_BLK_DEV_INITRD
>>>>  	if (initrd_start && !initrd_below_start_ok &&
>>>>  	    page_to_pfn(virt_to_page((void *)initrd_start)) < min_low_pfn) {
>>>
>>> What happens if devices use the bounce buffer before mem_encrypt_init()?
>>
>> The call to mem_encrypt_init is early in the boot process, I may have
>> overlooked something, but what devices would be performing DMA before
>> this?
> 
> I am not saying that you overlooked. Merely wondering if somebody re-orders these
> calls what would happen. It maybe also good to have a comment right before
> mem_encrpyt_init stating what will happen if the device does DMA before the function
> is called.
> 

Ah, ok. Before mem_encrypt_init is called the bounce buffers will be
marked as encrypted in the page tables. The use of the bounce buffers
will not have the memory encryption bit as part of the DMA address so a
device will DMA into memory in the clear. When the bounce buffer is
copied to the original buffer it will be accessed by a virtual address
that has the memory encryption bit set in the page tables. So the
plaintext data that was DMA'd in will be decrypted resulting in invalid
data in the destination buffer.

I'll be sure to add a comment before the call.

Thanks,
Tom

[RFC PATCH v1 14/18] iommu/amd: AMD IOMMU support for memory encryption

[Tom Lendacky]

Add support to the AMD IOMMU driver to set the memory encryption mask if
memory encryption is enabled.

Signed-off-by: Tom Lendacky <thomas.lendacky@amd.com>
---
 arch/x86/include/asm/mem_encrypt.h |    2 ++
 arch/x86/mm/mem_encrypt.c          |    5 +++++
 drivers/iommu/amd_iommu.c          |   10 ++++++++++
 3 files changed, 17 insertions(+)

diff --git a/arch/x86/include/asm/mem_encrypt.h b/arch/x86/include/asm/mem_encrypt.h
index d17d8cf..55163e4 100644
--- a/arch/x86/include/asm/mem_encrypt.h
+++ b/arch/x86/include/asm/mem_encrypt.h
@@ -39,6 +39,8 @@ void __init sme_early_init(void);
 /* Architecture __weak replacement functions */
 void __init mem_encrypt_init(void);
 
+unsigned long amd_iommu_get_me_mask(void);
+
 unsigned long swiotlb_get_me_mask(void);
 void swiotlb_set_mem_dec(void *vaddr, unsigned long size);
 
diff --git a/arch/x86/mm/mem_encrypt.c b/arch/x86/mm/mem_encrypt.c
index 594dc65..6efceb8 100644
--- a/arch/x86/mm/mem_encrypt.c
+++ b/arch/x86/mm/mem_encrypt.c
@@ -179,6 +179,11 @@ void __init mem_encrypt_init(void)
 	swiotlb_clear_encryption();
 }
 
+unsigned long amd_iommu_get_me_mask(void)
+{
+	return sme_me_mask;
+}
+
 unsigned long swiotlb_get_me_mask(void)
 {
 	return sme_me_mask;
diff --git a/drivers/iommu/amd_iommu.c b/drivers/iommu/amd_iommu.c
index 5efadad..5dc8f52 100644
--- a/drivers/iommu/amd_iommu.c
+++ b/drivers/iommu/amd_iommu.c
@@ -156,6 +156,15 @@ struct dma_ops_domain {
 	struct aperture_range *aperture[APERTURE_MAX_RANGES];
 };
 
+/*
+ * Support for memory encryption. If memory encryption is supported, then an
+ * override to this function will be provided.
+ */
+unsigned long __weak amd_iommu_get_me_mask(void)
+{
+	return 0;
+}
+
 /****************************************************************************
  *
  * Helper functions
@@ -2612,6 +2621,7 @@ static dma_addr_t __map_single(struct device *dev,
 	if (address == DMA_ERROR_CODE)
 		goto out;
 
+	paddr |= amd_iommu_get_me_mask();
 	start = address;
 	for (i = 0; i < pages; ++i) {
 		ret = dma_ops_domain_map(dma_dom, start, paddr, dir);

[RFC PATCH v1 15/18] x86: Enable memory encryption on the APs

[Tom Lendacky]

Add support to set the memory encryption enable flag on the APs during
realmode initialization. When an AP is started it checks this flag, and
if set, enables memory encryption on its core.

Signed-off-by: Tom Lendacky <thomas.lendacky@amd.com>
---
 arch/x86/include/asm/msr-index.h     |    2 ++
 arch/x86/include/asm/realmode.h      |   12 ++++++++++++
 arch/x86/realmode/init.c             |    4 ++++
 arch/x86/realmode/rm/trampoline_64.S |   14 ++++++++++++++
 4 files changed, 32 insertions(+)

diff --git a/arch/x86/include/asm/msr-index.h b/arch/x86/include/asm/msr-index.h
index 94555b4..b73182b 100644
--- a/arch/x86/include/asm/msr-index.h
+++ b/arch/x86/include/asm/msr-index.h
@@ -349,6 +349,8 @@
 #define MSR_K8_TOP_MEM1			0xc001001a
 #define MSR_K8_TOP_MEM2			0xc001001d
 #define MSR_K8_SYSCFG			0xc0010010
+#define MSR_K8_SYSCFG_MEM_ENCRYPT_BIT	23
+#define MSR_K8_SYSCFG_MEM_ENCRYPT	(1ULL << MSR_K8_SYSCFG_MEM_ENCRYPT_BIT)
 #define MSR_K8_INT_PENDING_MSG		0xc0010055
 /* C1E active bits in int pending message */
 #define K8_INTP_C1E_ACTIVE_MASK		0x18000000
diff --git a/arch/x86/include/asm/realmode.h b/arch/x86/include/asm/realmode.h
index 9c6b890..e24d2ec 100644
--- a/arch/x86/include/asm/realmode.h
+++ b/arch/x86/include/asm/realmode.h
@@ -1,6 +1,15 @@
 #ifndef _ARCH_X86_REALMODE_H
 #define _ARCH_X86_REALMODE_H
 
+/*
+ * Flag bit definitions for use with the flags field of the trampoline header
+ * when configured for X86_64
+ */
+#define TH_FLAGS_MEM_ENCRYPT_BIT	0
+#define TH_FLAGS_MEM_ENCRYPT		(1ULL << TH_FLAGS_MEM_ENCRYPT_BIT)
+
+#ifndef __ASSEMBLY__
+
 #include <linux/types.h>
 #include <asm/io.h>
 
@@ -38,6 +47,7 @@ struct trampoline_header {
 	u64 start;
 	u64 efer;
 	u32 cr4;
+	u32 flags;
 #endif
 };
 
@@ -61,4 +71,6 @@ extern unsigned char secondary_startup_64[];
 void reserve_real_mode(void);
 void setup_real_mode(void);
 
+#endif /* __ASSEMBLY__ */
+
 #endif /* _ARCH_X86_REALMODE_H */
diff --git a/arch/x86/realmode/init.c b/arch/x86/realmode/init.c
index 85b145c..657532b 100644
--- a/arch/x86/realmode/init.c
+++ b/arch/x86/realmode/init.c
@@ -84,6 +84,10 @@ void __init setup_real_mode(void)
 	trampoline_cr4_features = &trampoline_header->cr4;
 	*trampoline_cr4_features = __read_cr4();
 
+	trampoline_header->flags = 0;
+	if (sme_me_mask)
+		trampoline_header->flags |= TH_FLAGS_MEM_ENCRYPT;
+
 	trampoline_pgd = (u64 *) __va(real_mode_header->trampoline_pgd);
 	trampoline_pgd[0] = init_level4_pgt[pgd_index(__PAGE_OFFSET)].pgd;
 	trampoline_pgd[511] = init_level4_pgt[511].pgd;
diff --git a/arch/x86/realmode/rm/trampoline_64.S b/arch/x86/realmode/rm/trampoline_64.S
index dac7b20..8d84167 100644
--- a/arch/x86/realmode/rm/trampoline_64.S
+++ b/arch/x86/realmode/rm/trampoline_64.S
@@ -30,6 +30,7 @@
 #include <asm/msr.h>
 #include <asm/segment.h>
 #include <asm/processor-flags.h>
+#include <asm/realmode.h>
 #include "realmode.h"
 
 	.text
@@ -109,6 +110,18 @@ ENTRY(startup_32)
 	movl	$(X86_CR0_PG | X86_CR0_WP | X86_CR0_PE), %eax
 	movl	%eax, %cr0
 
+	# Check for and enable memory encryption support
+	movl	pa_tr_flags, %eax
+	bt	$TH_FLAGS_MEM_ENCRYPT_BIT, pa_tr_flags
+	jnc	.Ldone
+	movl	$MSR_K8_SYSCFG, %ecx
+	rdmsr
+	bt	$MSR_K8_SYSCFG_MEM_ENCRYPT_BIT, %eax
+	jc	.Ldone
+	bts	$MSR_K8_SYSCFG_MEM_ENCRYPT_BIT, %eax
+	wrmsr
+.Ldone:
+
 	/*
 	 * At this point we're in long mode but in 32bit compatibility mode
 	 * with EFER.LME = 1, CS.L = 0, CS.D = 1 (and in turn
@@ -147,6 +160,7 @@ GLOBAL(trampoline_header)
 	tr_start:		.space	8
 	GLOBAL(tr_efer)		.space	8
 	GLOBAL(tr_cr4)		.space	4
+	GLOBAL(tr_flags)	.space	4
 END(trampoline_header)
 
 #include "trampoline_common.S"

Re: [RFC PATCH v1 15/18] x86: Enable memory encryption on the APs

[Huang, Kai]

On 4/27/2016 10:58 AM, Tom Lendacky wrote:
> Add support to set the memory encryption enable flag on the APs during
> realmode initialization. When an AP is started it checks this flag, and
> if set, enables memory encryption on its core.
>
> Signed-off-by: Tom Lendacky <thomas.lendacky@amd.com>
> ---
>  arch/x86/include/asm/msr-index.h     |    2 ++
>  arch/x86/include/asm/realmode.h      |   12 ++++++++++++
>  arch/x86/realmode/init.c             |    4 ++++
>  arch/x86/realmode/rm/trampoline_64.S |   14 ++++++++++++++
>  4 files changed, 32 insertions(+)
>
> diff --git a/arch/x86/include/asm/msr-index.h b/arch/x86/include/asm/msr-index.h
> index 94555b4..b73182b 100644
> --- a/arch/x86/include/asm/msr-index.h
> +++ b/arch/x86/include/asm/msr-index.h
> @@ -349,6 +349,8 @@
>  #define MSR_K8_TOP_MEM1			0xc001001a
>  #define MSR_K8_TOP_MEM2			0xc001001d
>  #define MSR_K8_SYSCFG			0xc0010010
> +#define MSR_K8_SYSCFG_MEM_ENCRYPT_BIT	23
> +#define MSR_K8_SYSCFG_MEM_ENCRYPT	(1ULL << MSR_K8_SYSCFG_MEM_ENCRYPT_BIT)
>  #define MSR_K8_INT_PENDING_MSG		0xc0010055
>  /* C1E active bits in int pending message */
>  #define K8_INTP_C1E_ACTIVE_MASK		0x18000000
> diff --git a/arch/x86/include/asm/realmode.h b/arch/x86/include/asm/realmode.h
> index 9c6b890..e24d2ec 100644
> --- a/arch/x86/include/asm/realmode.h
> +++ b/arch/x86/include/asm/realmode.h
> @@ -1,6 +1,15 @@
>  #ifndef _ARCH_X86_REALMODE_H
>  #define _ARCH_X86_REALMODE_H
>
> +/*
> + * Flag bit definitions for use with the flags field of the trampoline header
> + * when configured for X86_64
> + */
> +#define TH_FLAGS_MEM_ENCRYPT_BIT	0
> +#define TH_FLAGS_MEM_ENCRYPT		(1ULL << TH_FLAGS_MEM_ENCRYPT_BIT)

Would mind change it to a more vendor specific name, such as 
AMD_MEM_ENCRYPT, or SME_MEM_ENCRYPT?

> +
> +#ifndef __ASSEMBLY__
> +
>  #include <linux/types.h>
>  #include <asm/io.h>
>
> @@ -38,6 +47,7 @@ struct trampoline_header {
>  	u64 start;
>  	u64 efer;
>  	u32 cr4;
> +	u32 flags;
>  #endif
>  };
>
> @@ -61,4 +71,6 @@ extern unsigned char secondary_startup_64[];
>  void reserve_real_mode(void);
>  void setup_real_mode(void);
>
> +#endif /* __ASSEMBLY__ */
> +
>  #endif /* _ARCH_X86_REALMODE_H */
> diff --git a/arch/x86/realmode/init.c b/arch/x86/realmode/init.c
> index 85b145c..657532b 100644
> --- a/arch/x86/realmode/init.c
> +++ b/arch/x86/realmode/init.c
> @@ -84,6 +84,10 @@ void __init setup_real_mode(void)
>  	trampoline_cr4_features = &trampoline_header->cr4;
>  	*trampoline_cr4_features = __read_cr4();
>
> +	trampoline_header->flags = 0;
> +	if (sme_me_mask)
> +		trampoline_header->flags |= TH_FLAGS_MEM_ENCRYPT;
> +
>  	trampoline_pgd = (u64 *) __va(real_mode_header->trampoline_pgd);
>  	trampoline_pgd[0] = init_level4_pgt[pgd_index(__PAGE_OFFSET)].pgd;
>  	trampoline_pgd[511] = init_level4_pgt[511].pgd;
> diff --git a/arch/x86/realmode/rm/trampoline_64.S b/arch/x86/realmode/rm/trampoline_64.S
> index dac7b20..8d84167 100644
> --- a/arch/x86/realmode/rm/trampoline_64.S
> +++ b/arch/x86/realmode/rm/trampoline_64.S
> @@ -30,6 +30,7 @@
>  #include <asm/msr.h>
>  #include <asm/segment.h>
>  #include <asm/processor-flags.h>
> +#include <asm/realmode.h>
>  #include "realmode.h"
>
>  	.text
> @@ -109,6 +110,18 @@ ENTRY(startup_32)
>  	movl	$(X86_CR0_PG | X86_CR0_WP | X86_CR0_PE), %eax
>  	movl	%eax, %cr0
>
> +	# Check for and enable memory encryption support
> +	movl	pa_tr_flags, %eax
> +	bt	$TH_FLAGS_MEM_ENCRYPT_BIT, pa_tr_flags

pa_tr_flags -> %eax ? Otherwise looks the previous line is useless.

Thanks,
-Kai

> +	jnc	.Ldone
> +	movl	$MSR_K8_SYSCFG, %ecx
> +	rdmsr
> +	bt	$MSR_K8_SYSCFG_MEM_ENCRYPT_BIT, %eax
> +	jc	.Ldone
> +	bts	$MSR_K8_SYSCFG_MEM_ENCRYPT_BIT, %eax
> +	wrmsr
> +.Ldone:
> +
>  	/*
>  	 * At this point we're in long mode but in 32bit compatibility mode
>  	 * with EFER.LME = 1, CS.L = 0, CS.D = 1 (and in turn
> @@ -147,6 +160,7 @@ GLOBAL(trampoline_header)
>  	tr_start:		.space	8
>  	GLOBAL(tr_efer)		.space	8
>  	GLOBAL(tr_cr4)		.space	4
> +	GLOBAL(tr_flags)	.space	4
>  END(trampoline_header)
>
>  #include "trampoline_common.S"
>
>

Re: [RFC PATCH v1 15/18] x86: Enable memory encryption on the APs

[Tom Lendacky]

On 05/01/2016 05:10 PM, Huang, Kai wrote:
> 
> 
> On 4/27/2016 10:58 AM, Tom Lendacky wrote:
>> Add support to set the memory encryption enable flag on the APs during
>> realmode initialization. When an AP is started it checks this flag, and
>> if set, enables memory encryption on its core.
>>
>> Signed-off-by: Tom Lendacky <thomas.lendacky@amd.com>
>> ---
>>  arch/x86/include/asm/msr-index.h     |    2 ++
>>  arch/x86/include/asm/realmode.h      |   12 ++++++++++++
>>  arch/x86/realmode/init.c             |    4 ++++
>>  arch/x86/realmode/rm/trampoline_64.S |   14 ++++++++++++++
>>  4 files changed, 32 insertions(+)
>>
>> diff --git a/arch/x86/include/asm/msr-index.h
>> b/arch/x86/include/asm/msr-index.h
>> index 94555b4..b73182b 100644
>> --- a/arch/x86/include/asm/msr-index.h
>> +++ b/arch/x86/include/asm/msr-index.h
>> @@ -349,6 +349,8 @@
>>  #define MSR_K8_TOP_MEM1            0xc001001a
>>  #define MSR_K8_TOP_MEM2            0xc001001d
>>  #define MSR_K8_SYSCFG            0xc0010010
>> +#define MSR_K8_SYSCFG_MEM_ENCRYPT_BIT    23
>> +#define MSR_K8_SYSCFG_MEM_ENCRYPT    (1ULL <<
>> MSR_K8_SYSCFG_MEM_ENCRYPT_BIT)
>>  #define MSR_K8_INT_PENDING_MSG        0xc0010055
>>  /* C1E active bits in int pending message */
>>  #define K8_INTP_C1E_ACTIVE_MASK        0x18000000
>> diff --git a/arch/x86/include/asm/realmode.h
>> b/arch/x86/include/asm/realmode.h
>> index 9c6b890..e24d2ec 100644
>> --- a/arch/x86/include/asm/realmode.h
>> +++ b/arch/x86/include/asm/realmode.h
>> @@ -1,6 +1,15 @@
>>  #ifndef _ARCH_X86_REALMODE_H
>>  #define _ARCH_X86_REALMODE_H
>>
>> +/*
>> + * Flag bit definitions for use with the flags field of the
>> trampoline header
>> + * when configured for X86_64
>> + */
>> +#define TH_FLAGS_MEM_ENCRYPT_BIT    0
>> +#define TH_FLAGS_MEM_ENCRYPT        (1ULL << TH_FLAGS_MEM_ENCRYPT_BIT)
> 
> Would mind change it to a more vendor specific name, such as
> AMD_MEM_ENCRYPT, or SME_MEM_ENCRYPT?

Yup, that can be done.

> 
>> +
>> +#ifndef __ASSEMBLY__
>> +
>>  #include <linux/types.h>
>>  #include <asm/io.h>
>>
>> @@ -38,6 +47,7 @@ struct trampoline_header {
>>      u64 start;
>>      u64 efer;
>>      u32 cr4;
>> +    u32 flags;
>>  #endif
>>  };
>>
>> @@ -61,4 +71,6 @@ extern unsigned char secondary_startup_64[];
>>  void reserve_real_mode(void);
>>  void setup_real_mode(void);
>>
>> +#endif /* __ASSEMBLY__ */
>> +
>>  #endif /* _ARCH_X86_REALMODE_H */
>> diff --git a/arch/x86/realmode/init.c b/arch/x86/realmode/init.c
>> index 85b145c..657532b 100644
>> --- a/arch/x86/realmode/init.c
>> +++ b/arch/x86/realmode/init.c
>> @@ -84,6 +84,10 @@ void __init setup_real_mode(void)
>>      trampoline_cr4_features = &trampoline_header->cr4;
>>      *trampoline_cr4_features = __read_cr4();
>>
>> +    trampoline_header->flags = 0;
>> +    if (sme_me_mask)
>> +        trampoline_header->flags |= TH_FLAGS_MEM_ENCRYPT;
>> +
>>      trampoline_pgd = (u64 *) __va(real_mode_header->trampoline_pgd);
>>      trampoline_pgd[0] = init_level4_pgt[pgd_index(__PAGE_OFFSET)].pgd;
>>      trampoline_pgd[511] = init_level4_pgt[511].pgd;
>> diff --git a/arch/x86/realmode/rm/trampoline_64.S
>> b/arch/x86/realmode/rm/trampoline_64.S
>> index dac7b20..8d84167 100644
>> --- a/arch/x86/realmode/rm/trampoline_64.S
>> +++ b/arch/x86/realmode/rm/trampoline_64.S
>> @@ -30,6 +30,7 @@
>>  #include <asm/msr.h>
>>  #include <asm/segment.h>
>>  #include <asm/processor-flags.h>
>> +#include <asm/realmode.h>
>>  #include "realmode.h"
>>
>>      .text
>> @@ -109,6 +110,18 @@ ENTRY(startup_32)
>>      movl    $(X86_CR0_PG | X86_CR0_WP | X86_CR0_PE), %eax
>>      movl    %eax, %cr0
>>
>> +    # Check for and enable memory encryption support
>> +    movl    pa_tr_flags, %eax
>> +    bt    $TH_FLAGS_MEM_ENCRYPT_BIT, pa_tr_flags
> 
> pa_tr_flags -> %eax ? Otherwise looks the previous line is useless.

Yes, I overlooked that. I'll take care of it.

Thanks,
Tom

> 
> Thanks,
> -Kai
> 
>> +    jnc    .Ldone
>> +    movl    $MSR_K8_SYSCFG, %ecx
>> +    rdmsr
>> +    bt    $MSR_K8_SYSCFG_MEM_ENCRYPT_BIT, %eax
>> +    jc    .Ldone
>> +    bts    $MSR_K8_SYSCFG_MEM_ENCRYPT_BIT, %eax
>> +    wrmsr
>> +.Ldone:
>> +
>>      /*
>>       * At this point we're in long mode but in 32bit compatibility mode
>>       * with EFER.LME = 1, CS.L = 0, CS.D = 1 (and in turn
>> @@ -147,6 +160,7 @@ GLOBAL(trampoline_header)
>>      tr_start:        .space    8
>>      GLOBAL(tr_efer)        .space    8
>>      GLOBAL(tr_cr4)        .space    4
>> +    GLOBAL(tr_flags)    .space    4
>>  END(trampoline_header)
>>
>>  #include "trampoline_common.S"
>>
>>

[RFC PATCH v1 16/18] x86: Do not specify encrypted memory for VGA mapping

[Tom Lendacky]

Since the VGA memory needs to be accessed unencrypted be sure that the
memory encryption mask is not set for the VGA range being mapped.

Signed-off-by: Tom Lendacky <thomas.lendacky@amd.com>
---
 arch/x86/include/asm/vga.h |   13 +++++++++++++
 1 file changed, 13 insertions(+)

diff --git a/arch/x86/include/asm/vga.h b/arch/x86/include/asm/vga.h
index c4b9dc2..55fe164 100644
--- a/arch/x86/include/asm/vga.h
+++ b/arch/x86/include/asm/vga.h
@@ -7,12 +7,25 @@
 #ifndef _ASM_X86_VGA_H
 #define _ASM_X86_VGA_H
 
+#include <asm/mem_encrypt.h>
+
 /*
  *	On the PC, we can just recalculate addresses and then
  *	access the videoram directly without any black magic.
+ *	To support memory encryption however, we need to access
+ *	the videoram as un-encrypted memory.
  */
 
+#ifdef CONFIG_AMD_MEM_ENCRYPT
+#define VGA_MAP_MEM(x, s)					\
+({								\
+	unsigned long start = (unsigned long)phys_to_virt(x);	\
+	sme_set_mem_dec((void *)start, s);			\
+	start;							\
+})
+#else
 #define VGA_MAP_MEM(x, s) (unsigned long)phys_to_virt(x)
+#endif
 
 #define vga_readb(x) (*(x))
 #define vga_writeb(x, y) (*(y) = (x))

[RFC PATCH v1 17/18] x86/kvm: Enable Secure Memory Encryption of nested page tables

[Tom Lendacky]

Update the KVM support to include the memory encryption mask when creating
and using nested page tables.

Signed-off-by: Tom Lendacky <thomas.lendacky@amd.com>
---
 arch/x86/include/asm/kvm_host.h |    2 +-
 arch/x86/kvm/mmu.c              |    7 +++++--
 arch/x86/kvm/vmx.c              |    2 +-
 arch/x86/kvm/x86.c              |    3 ++-
 4 files changed, 9 insertions(+), 5 deletions(-)

diff --git a/arch/x86/include/asm/kvm_host.h b/arch/x86/include/asm/kvm_host.h
index b7e3944..75f1e30 100644
--- a/arch/x86/include/asm/kvm_host.h
+++ b/arch/x86/include/asm/kvm_host.h
@@ -1012,7 +1012,7 @@ void kvm_mmu_setup(struct kvm_vcpu *vcpu);
 void kvm_mmu_init_vm(struct kvm *kvm);
 void kvm_mmu_uninit_vm(struct kvm *kvm);
 void kvm_mmu_set_mask_ptes(u64 user_mask, u64 accessed_mask,
-		u64 dirty_mask, u64 nx_mask, u64 x_mask);
+		u64 dirty_mask, u64 nx_mask, u64 x_mask, u64 me_mask);
 
 void kvm_mmu_reset_context(struct kvm_vcpu *vcpu);
 void kvm_mmu_slot_remove_write_access(struct kvm *kvm,
diff --git a/arch/x86/kvm/mmu.c b/arch/x86/kvm/mmu.c
index 4c6972f..5c7d939 100644
--- a/arch/x86/kvm/mmu.c
+++ b/arch/x86/kvm/mmu.c
@@ -121,7 +121,7 @@ module_param(dbg, bool, 0644);
 					    * PT32_LEVEL_BITS))) - 1))
 
 #define PT64_PERM_MASK (PT_PRESENT_MASK | PT_WRITABLE_MASK | shadow_user_mask \
-			| shadow_x_mask | shadow_nx_mask)
+			| shadow_x_mask | shadow_nx_mask | shadow_me_mask)
 
 #define ACC_EXEC_MASK    1
 #define ACC_WRITE_MASK   PT_WRITABLE_MASK
@@ -175,6 +175,7 @@ static u64 __read_mostly shadow_user_mask;
 static u64 __read_mostly shadow_accessed_mask;
 static u64 __read_mostly shadow_dirty_mask;
 static u64 __read_mostly shadow_mmio_mask;
+static u64 __read_mostly shadow_me_mask;
 
 static void mmu_spte_set(u64 *sptep, u64 spte);
 static void mmu_free_roots(struct kvm_vcpu *vcpu);
@@ -282,13 +283,14 @@ static bool check_mmio_spte(struct kvm_vcpu *vcpu, u64 spte)
 }
 
 void kvm_mmu_set_mask_ptes(u64 user_mask, u64 accessed_mask,
-		u64 dirty_mask, u64 nx_mask, u64 x_mask)
+		u64 dirty_mask, u64 nx_mask, u64 x_mask, u64 me_mask)
 {
 	shadow_user_mask = user_mask;
 	shadow_accessed_mask = accessed_mask;
 	shadow_dirty_mask = dirty_mask;
 	shadow_nx_mask = nx_mask;
 	shadow_x_mask = x_mask;
+	shadow_me_mask = me_mask;
 }
 EXPORT_SYMBOL_GPL(kvm_mmu_set_mask_ptes);
 
@@ -2549,6 +2551,7 @@ static int set_spte(struct kvm_vcpu *vcpu, u64 *sptep,
 		pte_access &= ~ACC_WRITE_MASK;
 
 	spte |= (u64)pfn << PAGE_SHIFT;
+	spte |= shadow_me_mask;
 
 	if (pte_access & ACC_WRITE_MASK) {
 
diff --git a/arch/x86/kvm/vmx.c b/arch/x86/kvm/vmx.c
index d5908bd..5d8eb4b 100644
--- a/arch/x86/kvm/vmx.c
+++ b/arch/x86/kvm/vmx.c
@@ -6351,7 +6351,7 @@ static __init int hardware_setup(void)
 		kvm_mmu_set_mask_ptes(0ull,
 			(enable_ept_ad_bits) ? VMX_EPT_ACCESS_BIT : 0ull,
 			(enable_ept_ad_bits) ? VMX_EPT_DIRTY_BIT : 0ull,
-			0ull, VMX_EPT_EXECUTABLE_MASK);
+			0ull, VMX_EPT_EXECUTABLE_MASK, 0ull);
 		ept_set_mmio_spte_mask();
 		kvm_enable_tdp();
 	} else
diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c
index 12f33e6..9432e27 100644
--- a/arch/x86/kvm/x86.c
+++ b/arch/x86/kvm/x86.c
@@ -67,6 +67,7 @@
 #include <asm/pvclock.h>
 #include <asm/div64.h>
 #include <asm/irq_remapping.h>
+#include <asm/mem_encrypt.h>
 
 #define MAX_IO_MSRS 256
 #define KVM_MAX_MCE_BANKS 32
@@ -5859,7 +5860,7 @@ int kvm_arch_init(void *opaque)
 	kvm_x86_ops = ops;
 
 	kvm_mmu_set_mask_ptes(PT_USER_MASK, PT_ACCESSED_MASK,
-			PT_DIRTY_MASK, PT64_NX_MASK, 0);
+			PT_DIRTY_MASK, PT64_NX_MASK, 0, sme_me_mask);
 
 	kvm_timer_init();
 

[RFC PATCH v1 18/18] x86: Add support to turn on Secure Memory Encryption

[Tom Lendacky]

This patch adds the support to check for and enable SME when available
on the processor and when the mem_encrypt=on command line option is set.
This consists of setting the encryption mask, calculating the number of
physical bits of addressing lost and encrypting the kernel "in place."

Signed-off-by: Tom Lendacky <thomas.lendacky@amd.com>
---
 Documentation/kernel-parameters.txt |    3 
 arch/x86/kernel/asm-offsets.c       |    2 
 arch/x86/kernel/mem_encrypt.S       |  306 +++++++++++++++++++++++++++++++++++
 3 files changed, 311 insertions(+)

diff --git a/Documentation/kernel-parameters.txt b/Documentation/kernel-parameters.txt
index 8ba7f82..0a2678a 100644
--- a/Documentation/kernel-parameters.txt
+++ b/Documentation/kernel-parameters.txt
@@ -2210,6 +2210,9 @@ bytes respectively. Such letter suffixes can also be entirely omitted.
 			memory contents and reserves bad memory
 			regions that are detected.
 
+	mem_encrypt=on	[X86_64] Enable memory encryption on processors
+			that support this feature.
+
 	meye.*=		[HW] Set MotionEye Camera parameters
 			See Documentation/video4linux/meye.txt.
 
diff --git a/arch/x86/kernel/asm-offsets.c b/arch/x86/kernel/asm-offsets.c
index 5c04246..a0f76de 100644
--- a/arch/x86/kernel/asm-offsets.c
+++ b/arch/x86/kernel/asm-offsets.c
@@ -82,6 +82,8 @@ void common(void) {
 	OFFSET(BP_kernel_alignment, boot_params, hdr.kernel_alignment);
 	OFFSET(BP_pref_address, boot_params, hdr.pref_address);
 	OFFSET(BP_code32_start, boot_params, hdr.code32_start);
+	OFFSET(BP_cmd_line_ptr, boot_params, hdr.cmd_line_ptr);
+	OFFSET(BP_ext_cmd_line_ptr, boot_params, ext_cmd_line_ptr);
 
 	BLANK();
 	DEFINE(PTREGS_SIZE, sizeof(struct pt_regs));
diff --git a/arch/x86/kernel/mem_encrypt.S b/arch/x86/kernel/mem_encrypt.S
index f2e0536..4d3326d 100644
--- a/arch/x86/kernel/mem_encrypt.S
+++ b/arch/x86/kernel/mem_encrypt.S
@@ -12,13 +12,236 @@
 
 #include <linux/linkage.h>
 
+#include <asm/processor-flags.h>
+#include <asm/pgtable.h>
+#include <asm/page.h>
+#include <asm/msr.h>
+#include <asm/asm-offsets.h>
+
 	.text
 	.code64
 ENTRY(sme_enable)
+#ifdef CONFIG_AMD_MEM_ENCRYPT
+	/* Check for AMD processor */
+	xorl	%eax, %eax
+	cpuid
+	cmpl    $0x68747541, %ebx	# AuthenticAMD
+	jne     .Lno_mem_encrypt
+	cmpl    $0x69746e65, %edx
+	jne     .Lno_mem_encrypt
+	cmpl    $0x444d4163, %ecx
+	jne     .Lno_mem_encrypt
+
+	/* Check for memory encryption leaf */
+	movl	$0x80000000, %eax
+	cpuid
+	cmpl	$0x8000001f, %eax
+	jb	.Lno_mem_encrypt
+
+	/*
+	 * Check for memory encryption feature:
+	 *   CPUID Fn8000_001F[EAX] - Bit 0
+	 */
+	movl	$0x8000001f, %eax
+	cpuid
+	bt	$0, %eax
+	jnc	.Lno_mem_encrypt
+
+	/* Check for the mem_encrypt=on command line option */
+	push	%rsi			/* Save RSI (real_mode_data) */
+	movl	BP_ext_cmd_line_ptr(%rsi), %ecx
+	shlq	$32, %rcx
+	movl	BP_cmd_line_ptr(%rsi), %edi
+	addq	%rcx, %rdi
+	leaq	mem_encrypt_enable_option(%rip), %rsi
+	call	cmdline_find_option_bool
+	pop	%rsi			/* Restore RSI (real_mode_data) */
+	testl	%eax, %eax
+	jz	.Lno_mem_encrypt
+
+	/*
+	 * Get memory encryption information:
+	 *   CPUID Fn8000_001F[EBX] - Bits 5:0
+	 *     Pagetable bit position used to indicate encryption
+	 */
+	movl	%ebx, %ecx
+	andl	$0x3f, %ecx
+	jz	.Lno_mem_encrypt
+	bts	%ecx, sme_me_mask(%rip)
+	shrl	$6, %ebx
+
+	/*
+	 * Get memory encryption information:
+	 *   CPUID Fn8000_001F[EBX] - Bits 11:6
+	 *     Reduction in physical address space (in bits) when enabled
+	 */
+	movl	%ebx, %ecx
+	andl	$0x3f, %ecx
+	movb	%cl, sme_me_loss(%rip)
+
+	/*
+	 * Enable memory encryption through the SYSCFG MSR
+	 */
+	movl	$MSR_K8_SYSCFG, %ecx
+	rdmsr
+	bt	$MSR_K8_SYSCFG_MEM_ENCRYPT_BIT, %eax
+	jc	.Lmem_encrypt_exit
+	bts	$MSR_K8_SYSCFG_MEM_ENCRYPT_BIT, %eax
+	wrmsr
+	jmp	.Lmem_encrypt_exit
+
+.Lno_mem_encrypt:
+	/* Did not get enabled, clear settings */
+	movq	$0, sme_me_mask(%rip)
+	movb	$0, sme_me_loss(%rip)
+
+.Lmem_encrypt_exit:
+#endif	/* CONFIG_AMD_MEM_ENCRYPT */
+
 	ret
 ENDPROC(sme_enable)
 
 ENTRY(sme_encrypt_kernel)
+#ifdef CONFIG_AMD_MEM_ENCRYPT
+	cmpq	$0, sme_me_mask(%rip)
+	jz	.Lencrypt_exit
+
+	/*
+	 * Encrypt the kernel.
+	 * Pagetables for performing kernel encryption:
+	 *   0x0000000000 - 0x00FFFFFFFF will map just the memory occupied by
+	 *				 the kernel as encrypted memory
+	 *   0x8000000000 - 0x80FFFFFFFF will map all memory as write-protected,
+	 *				 non-encrypted
+	 *
+	 * The use of write-protected memory will prevent any of the
+	 * non-encrypted memory from being cached.
+	 *
+	 * 0x00... and 0x80... represent the first and second PGD entries.
+	 *
+	 * This collection of entries will be created in an area outside
+	 * of the area that is being encrypted (outside the kernel) and
+	 * requires 11 4K pages:
+	 *   1 - PGD
+	 *   2 - PUDs (1 for each mapping)
+	 *   8 - PMDs (4 for each mapping)
+	 */
+	leaq	_end(%rip), %rdi
+	addq	$~PMD_PAGE_MASK, %rdi
+	andq	$PMD_PAGE_MASK, %rdi	/* RDI points to the new PGD */
+
+	/* Clear the pagetable memory */
+	movq	%rdi, %rbx		/* Save pointer to PGD */
+	movl	$(4096 * 11), %ecx
+	xorl	%eax, %eax
+	rep	stosb
+	movq	%rbx, %rdi		/* Restore pointer to PGD */
+
+	/* Set up PGD entries for the two mappings */
+	leaq	(0x1000 + 0x03)(%rdi), %rbx	/* PUD for encrypted kernel */
+	movq	%rbx, (%rdi)
+	leaq	(0x2000 + 0x03)(%rdi), %rbx	/* PUD for unencrypted kernel */
+	movq	%rbx, 8(%rdi)
+
+	/* Set up PUD entries (4 per mapping) for the two mappings */
+	leaq	(0x3000 + 0x03)(%rdi), %rbx	/* PMD for encrypted kernel */
+	leaq	(0x7000 + 0x03)(%rdi), %rdx	/* PMD for unencrypted kernel */
+	xorq	%rcx, %rcx
+1:
+	/* Populate the PUD entries in each mapping */
+	movq	%rbx, 0x1000(%rdi, %rcx, 8)
+	movq	%rdx, 0x2000(%rdi, %rcx, 8)
+	addq	$0x1000, %rbx
+	addq	$0x1000, %rdx
+	incq	%rcx
+	cmpq	$4, %rcx
+	jb	1b
+
+	/*
+	 * Set up PMD entries (4GB worth) for the two mappings.
+	 *   For the encrypted kernel mapping, when R11 is above RDX
+	 *   and below RDI then we know we are in the kernel and we
+	 *   set the encryption mask for that PMD entry.
+	 *
+	 *   The use of _PAGE_PAT and _PAGE_PWT will provide for the
+	 *   write-protected mapping.
+	 */
+	movq	sme_me_mask(%rip), %r10
+	movq	$__PAGE_KERNEL_LARGE_EXEC, %r11
+	andq	$~_PAGE_GLOBAL, %r11
+	movq	%r11, %r12
+	andq	$~_PAGE_CACHE_MASK, %r12
+	orq	$(_PAGE_PAT | _PAGE_PWT), %r12	/* PA5 index */
+	xorq	%rcx, %rcx
+	leaq	_text(%rip), %rdx	/* RDX points to start of kernel */
+1:
+	/* Populate the PMD entries in each mapping */
+	movq	%r11, 0x3000(%rdi, %rcx, 8)
+	movq	%r12, 0x7000(%rdi, %rcx, 8)
+
+	/*
+	 * Check if we are in the kernel range, and if so, set the
+	 * memory encryption mask.
+	 */
+	cmpq	%r11, %rdx
+	jae	2f
+	cmpq	%r11, %rdi
+	jbe	2f
+	orq	%r10, 0x3000(%rdi, %rcx, 8)
+2:
+	addq	$PMD_SIZE, %r11
+	addq	$PMD_SIZE, %r12
+	incq	%rcx
+	cmpq	$2048, %rcx
+	jb	1b
+
+	/*
+	 * Set up a one page stack in the non-encrypted memory area.
+	 *   Set RAX to point to the next page in memory after all the
+	 *   page tables. The stack grows from the bottom so point to
+	 *   the end of the page.
+	 */
+	leaq	(4096 * 11)(%rdi), %rax
+	addq	$PAGE_SIZE, %rax
+	movq	%rsp, %rbp
+	movq	%rax, %rsp
+	push	%rbp			/* Save original stack pointer */
+
+	push	%rsi			/* Save RSI (real mode data) */
+
+	/*
+	 * Copy encryption routine into safe memory
+	 *   - RAX points to the page after all the page tables and stack
+	 *     where the routine will copied
+	 *   - RDI points to the PGD table
+	 *   - Setup registers for call
+	 * and then call it
+	 */
+	movq	%rdi, %rbx
+
+	leaq	.Lencrypt_start(%rip), %rsi
+	movq	%rax, %rdi
+	movq	$(.Lencrypt_stop - .Lencrypt_start), %rcx
+	rep	movsb
+
+	leaq	_text(%rip), %rsi	/* Kernel start */
+	movq	%rbx, %rcx		/* New PGD start */
+	subq	%rsi, %rcx		/* Size of area to encrypt */
+
+	movq	%rsi, %rdi		/* Encrypted kernel space start */
+	movq	$1, %rsi
+	shlq	$PGDIR_SHIFT, %rsi
+	addq	%rdi, %rsi		/* Non-encrypted kernel start */
+
+	/* Call the encryption routine */
+	call	*%rax
+
+	pop	%rsi			/* Restore RSI (real mode data ) */
+
+	pop	%rsp			/* Restore original stack pointer */
+.Lencrypt_exit:
+#endif	/* CONFIG_AMD_MEM_ENCRYPT */
+
 	ret
 ENDPROC(sme_encrypt_kernel)
 
@@ -28,6 +251,87 @@ ENTRY(sme_get_me_loss)
 	ret
 ENDPROC(sme_get_me_loss)
 
+#ifdef CONFIG_AMD_MEM_ENCRYPT
+/*
+ * Routine used to encrypt kernel.
+ *   This routine must be run outside of the kernel proper since
+ *   the kernel will be encrypted during the process. So this
+ *   routine is defined here and then copied to an area outside
+ *   of the kernel where it will remain and run un-encrypted
+ *   during execution.
+ *
+ *   On entry the registers must be:
+ *   - RAX points to this routine
+ *   - RBX points to new PGD to use
+ *   - RCX contains the kernel length
+ *   - RSI points to the non-encrypted kernel space
+ *   - RDI points to the encrypted kernel space
+ *
+ * The kernel will be encrypted by copying from the non-encrypted
+ * kernel space to a temporary buffer and then copying from the
+ * temporary buffer back to the encrypted kernel space. The physical
+ * addresses of the two kernel space mappings are the same which
+ * results in the kernel being encrypted "in place".
+ */
+.Lencrypt_start:
+	/* Enable the new page tables */
+	mov	%rbx, %cr3
+
+	/* Flush any global TLBs */
+	mov	%cr4, %rbx
+	andq	$~X86_CR4_PGE, %rbx
+	mov	%rbx, %cr4
+	orq	$X86_CR4_PGE, %rbx
+	mov	%rbx, %cr4
+
+	/* Set the PAT register PA5 entry to write-protect */
+	push	%rax
+	push	%rcx
+	movl	$MSR_IA32_CR_PAT, %ecx
+	rdmsr
+	push	%rdx			/* Save original PAT value */
+	andl	$0xffff00ff, %edx	/* Clear PA5 */
+	orl	$0x00000500, %edx	/* Set PA5 to WP */
+	wrmsr
+	pop	%rdx			/* RDX contains original PAT value */
+	pop	%rcx
+	pop	%rax
+
+	movq	%rsi, %r10		/* Save source address */
+	movq	%rdi, %r11		/* Save destination address */
+	movq	%rcx, %r12		/* Save length */
+	addq	$PAGE_SIZE, %rax	/* RAX now points to temp copy page */
+
+	wbinvd				/* Invalidate any cache entries */
+
+	/* Copy/encrypt 2MB at a time */
+1:
+	movq	%r10, %rsi
+	movq	%rax, %rdi
+	movq	$PMD_PAGE_SIZE, %rcx
+	rep	movsb
+
+	movq	%rax, %rsi
+	movq	%r11, %rdi
+	movq	$PMD_PAGE_SIZE, %rcx
+	rep	movsb
+
+	addq	$PMD_PAGE_SIZE, %r10
+	addq	$PMD_PAGE_SIZE, %r11
+	subq	$PMD_PAGE_SIZE, %r12
+	jnz	1b
+
+	/* Restore PAT register */
+	push	%rdx
+	movl	$MSR_IA32_CR_PAT, %ecx
+	rdmsr
+	pop	%rdx
+	wrmsr
+
+	ret
+.Lencrypt_stop:
+#endif	/* CONFIG_AMD_MEM_ENCRYPT */
+
 	.data
 	.align 16
 ENTRY(sme_me_mask)
@@ -35,3 +339,5 @@ ENTRY(sme_me_mask)
 sme_me_loss:
 	.byte	0x00
 	.align	8
+mem_encrypt_enable_option:
+	.asciz "mem_encrypt=on"

Re: [RFC PATCH v1 18/18] x86: Add support to turn on Secure Memory Encryption

[Pavel Machek]

Hi!

> This patch adds the support to check for and enable SME when available
> on the processor and when the mem_encrypt=on command line option is set.
> This consists of setting the encryption mask, calculating the number of
> physical bits of addressing lost and encrypting the kernel "in place."
> 
> Signed-off-by: Tom Lendacky <thomas.lendacky@amd.com>
> ---
>  Documentation/kernel-parameters.txt |    3 
>  arch/x86/kernel/asm-offsets.c       |    2 
>  arch/x86/kernel/mem_encrypt.S       |  306 +++++++++++++++++++++++++++++++++++
>  3 files changed, 311 insertions(+)
> 
> diff --git a/arch/x86/kernel/mem_encrypt.S b/arch/x86/kernel/mem_encrypt.S
> index f2e0536..4d3326d 100644
> --- a/arch/x86/kernel/mem_encrypt.S
> +++ b/arch/x86/kernel/mem_encrypt.S
> @@ -12,13 +12,236 @@
>  
>  #include <linux/linkage.h>
>  
> +#include <asm/processor-flags.h>
> +#include <asm/pgtable.h>
> +#include <asm/page.h>
> +#include <asm/msr.h>
> +#include <asm/asm-offsets.h>
> +
>  	.text
>  	.code64
>  ENTRY(sme_enable)
> +#ifdef CONFIG_AMD_MEM_ENCRYPT
> +	/* Check for AMD processor */
> +	xorl	%eax, %eax
> +	cpuid
> +	cmpl    $0x68747541, %ebx	# AuthenticAMD
> +	jne     .Lno_mem_encrypt
> +	cmpl    $0x69746e65, %edx
> +	jne     .Lno_mem_encrypt
> +	cmpl    $0x444d4163, %ecx
> +	jne     .Lno_mem_encrypt
> +
> +	/* Check for memory encryption leaf */
> +	movl	$0x80000000, %eax
> +	cpuid
> +	cmpl	$0x8000001f, %eax
> +	jb	.Lno_mem_encrypt
> +
> +	/*
> +	 * Check for memory encryption feature:
> +	 *   CPUID Fn8000_001F[EAX] - Bit 0
> +	 */
> +	movl	$0x8000001f, %eax
> +	cpuid
> +	bt	$0, %eax
> +	jnc	.Lno_mem_encrypt
> +
> +	/* Check for the mem_encrypt=on command line option */
> +	push	%rsi			/* Save RSI (real_mode_data) */
> +	movl	BP_ext_cmd_line_ptr(%rsi), %ecx
> +	shlq	$32, %rcx
> +	movl	BP_cmd_line_ptr(%rsi), %edi
> +	addq	%rcx, %rdi
> +	leaq	mem_encrypt_enable_option(%rip), %rsi
> +	call	cmdline_find_option_bool
> +	pop	%rsi			/* Restore RSI (real_mode_data) */
> +	testl	%eax, %eax
> +	jz	.Lno_mem_encrypt

Can you move parts to C here, so that it is more readable?

								Pavel
-- 
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html

Re: [RFC PATCH v1 00/18] x86: Secure Memory Encryption (AMD)

[Andy Lutomirski]

On Tue, Apr 26, 2016 at 3:55 PM, Tom Lendacky <thomas.lendacky@amd.com> wrote:
> This RFC patch series provides support for AMD's new Secure Memory
> Encryption (SME) feature.
>
> SME can be used to mark individual pages of memory as encrypted through the
> page tables. A page of memory that is marked encrypted will be automatically
> decrypted when read from DRAM and will be automatically encrypted when
> written to DRAM. Details on SME can found in the links below.

Having read through the docs briefly, some questions:

1. How does the crypto work?  Is it straight AES-ECB?  Is it a
tweakable mode?  If so, what does into the tweak?  For example, if I
swap the ciphertext of two pages, does the plaintext of the pages get
swapped?  If not, why not?

2. In SEV mode, how does the hypervisor relocate a physical backing
page?  Does it simple move it and update the 2nd-level page tables?
If so, is the result of decryption guaranteed to be garbage if it
relocates a page and re-inserts it at the wrong guest physical
address?

3. In SEV mode, does anything prevent the hypervisor from resuming a
guest with the wrong ASID, or is this all relying on the resulting
corruption of the guest code and data to cause a crash?

4. As I understand it, the caches are all unencrypted, and they're
tagged with the physical address, *including* the SME bit (bit 47).
In SEV mode, are they also tagged with the ASID?  I.e. if I have a
page in cache for ASID 1 and I try to read it with ASID 2, will I get
a fresh copy decrypted with ASID 2's key?  If so, will the old ASID 1
copy be evicted, or will it stay in cache and be non-coherent?

--Andy

Re: [RFC PATCH v1 00/18] x86: Secure Memory Encryption (AMD)

[Tom Lendacky]

On 04/27/2016 09:39 AM, Andy Lutomirski wrote:
> On Tue, Apr 26, 2016 at 3:55 PM, Tom Lendacky <thomas.lendacky@amd.com> wrote:
>> This RFC patch series provides support for AMD's new Secure Memory
>> Encryption (SME) feature.
>>
>> SME can be used to mark individual pages of memory as encrypted through the
>> page tables. A page of memory that is marked encrypted will be automatically
>> decrypted when read from DRAM and will be automatically encrypted when
>> written to DRAM. Details on SME can found in the links below.
> 
> Having read through the docs briefly, some questions:
> 
> 1. How does the crypto work?  Is it straight AES-ECB?  Is it a
> tweakable mode?  If so, what does into the tweak?  For example, if I
> swap the ciphertext of two pages, does the plaintext of the pages get
> swapped?  If not, why not?

The AES crypto uses a tweak such that two identical plaintexts at
different locations will have different ciphertext. So swapping the
ciphertext of two pages will not result in the plaintext being swapped.

> 
> 2. In SEV mode, how does the hypervisor relocate a physical backing
> page?  Does it simple move it and update the 2nd-level page tables?
> If so, is the result of decryption guaranteed to be garbage if it
> relocates a page and re-inserts it at the wrong guest physical
> address?

For SEV mode, relocating a physical backing page takes extra steps.
There are APIs that are used to have the AMD Secure Processor create a
transportable encrypted page that can then be moved to a new location
in memory. After moving it to the new location the APIs are used to
haves the AMD Secure Processor re-encrypt the page for use with the
guests SEV key. Based on #1 above, just moving a page without invoking
the necessary APIs will result in the decryption returning garbage.

> 
> 3. In SEV mode, does anything prevent the hypervisor from resuming a
> guest with the wrong ASID, or is this all relying on the resulting
> corruption of the guest code and data to cause a crash?

There is nothing that prevents resuming a guest with the wrong ASID.
This relies on the resulting corruption of the guest code/data to
cause a crash.

> 
> 4. As I understand it, the caches are all unencrypted, and they're
> tagged with the physical address, *including* the SME bit (bit 47).
> In SEV mode, are they also tagged with the ASID?  I.e. if I have a
> page in cache for ASID 1 and I try to read it with ASID 2, will I get
> a fresh copy decrypted with ASID 2's key?  If so, will the old ASID 1
> copy be evicted, or will it stay in cache and be non-coherent?

In SEV mode, the caches are tagged with the ASID. So if you try to read
a cached page with a different ASID, it would result in a cache miss
for that ASID and will instead fetch from memory and decrypt using
the that ASID's key.

Thanks,
Tom

> 
> --Andy
> 

Re: [RFC PATCH v1 00/18] x86: Secure Memory Encryption (AMD)

[Andy Lutomirski]

On Wed, Apr 27, 2016 at 1:10 PM, Tom Lendacky <thomas.lendacky@amd.com> wrote:
> On 04/27/2016 09:39 AM, Andy Lutomirski wrote:
>> On Tue, Apr 26, 2016 at 3:55 PM, Tom Lendacky <thomas.lendacky@amd.com> wrote:
>>> This RFC patch series provides support for AMD's new Secure Memory
>>> Encryption (SME) feature.
>>>
>>> SME can be used to mark individual pages of memory as encrypted through the
>>> page tables. A page of memory that is marked encrypted will be automatically
>>> decrypted when read from DRAM and will be automatically encrypted when
>>> written to DRAM. Details on SME can found in the links below.
>>
>> Having read through the docs briefly, some questions:
>>
>> 1. How does the crypto work?  Is it straight AES-ECB?  Is it a
>> tweakable mode?  If so, what does into the tweak?  For example, if I
>> swap the ciphertext of two pages, does the plaintext of the pages get
>> swapped?  If not, why not?
>
> The AES crypto uses a tweak such that two identical plaintexts at
> different locations will have different ciphertext. So swapping the
> ciphertext of two pages will not result in the plaintext being swapped.

OK, makes sense.

>
>>
>> 2. In SEV mode, how does the hypervisor relocate a physical backing
>> page?  Does it simple move it and update the 2nd-level page tables?
>> If so, is the result of decryption guaranteed to be garbage if it
>> relocates a page and re-inserts it at the wrong guest physical
>> address?
>
> For SEV mode, relocating a physical backing page takes extra steps.
> There are APIs that are used to have the AMD Secure Processor create a
> transportable encrypted page that can then be moved to a new location
> in memory. After moving it to the new location the APIs are used to
> haves the AMD Secure Processor re-encrypt the page for use with the
> guests SEV key. Based on #1 above, just moving a page without invoking
> the necessary APIs will result in the decryption returning garbage.
>
>>
>> 3. In SEV mode, does anything prevent the hypervisor from resuming a
>> guest with the wrong ASID, or is this all relying on the resulting
>> corruption of the guest code and data to cause a crash?
>
> There is nothing that prevents resuming a guest with the wrong ASID.
> This relies on the resulting corruption of the guest code/data to
> cause a crash.

This all seems somewhat useful, but I almost guarantee that if there
is ever anything economically important (or important for national
security reasons, or simply something that sounds fun for an
enterprising kid to break) that it *will* be broken in many creative
ways.

Someone will break it by replaying old data through the VM, either to
confuse control flow or to use some part of the VM code as an oracle
with which to attack another part.

Someone else will break it by installing a #UD / #PF handler and using
the resulting exceptions as an oracle.

A third clever person will break it by carefully constructing a
scenario in which randomizing 16 bytes of data has a high probability
of letting then pwn your system.  (For example, what if the secured VM
creates an RSA key and you can carefully interrupt it right after
generating p and q.  Replace 16 bytes from the middle of both p and q
(32 bytes total) with random garbage.  With reasonably high
probability, the resulting p and q will no longer be prime.)

Depending on how strong your ASID protection is, a fourth clever
person will break it by replacing a bunch of the VM out from under the
target while leaving the sensitive data in place and then will use
some existing exploit or design issue to gain code execution in the
modified VM.

Also, I really hope that your tweakable cipher mode is at least CCA2
secure, because attackers can absolutely hit it with adaptive chosen
ciphertext attacks.  (Actually, attackers can alternate between
adaptive chosen ciphertext and adaptive chosen plaintext.)

And did the SEV implementation remember to encrypt the guest register
state?  Because, if not, everything of importance will leak out
through the VMCB and/or GPRs.

But I guess it's better than nothing.

--Andy

Re: [RFC PATCH v1 00/18] x86: Secure Memory Encryption (AMD)

[Paolo Bonzini]

On 02/05/2016 20:31, Andy Lutomirski wrote:
> And did the SEV implementation remember to encrypt the guest register
> state?  Because, if not, everything of importance will leak out
> through the VMCB and/or GPRs.

No, it doesn't.  And SEV is very limited unless you paravirtualize
everything.

For example, the hypervisor needs to read some instruction bytes from
memory, and instruction bytes are always encrypted (15.34.5 in the APM).
 So you're pretty much restricted to IN/OUT operations (not even
INS/OUTS) on emulated (non-assigned) devices, paravirtualized MSRs, and
hypercalls.  These are the only operations that connect the guest and
the hypervisor, where the vmexit doesn't have the need to e.g. walk
guest page tables (also always encrypted).  It possibly can be made to
work once the guest boots, and a modern UEFI firmware probably can cope
with it too just like a kernel can, but you need to ensure that your
hardware has no memory BARs for example.  And I/O port space is not very
abundant.

Even in order to emulate I/O ports or RDMSR/WRMSR or process hypercalls,
the hypervisor needs to read the GPRs.  The VMCB doesn't store guest
GPRs, not even on SEV-enabled processors.  Accordingly, the hypervisor
has access to the guest GPRs on every exit.

In general, SEV provides mitigation only.  Even if the hypervisor cannot
write known plaintext directly to memory, an accomplice virtual machine
can e.g. use the network to spray the attacked VM's memory.  At least
it's not as easy as "disable NX under the guest's feet and redirect RIP"
(pte.nx is reserved if efer.nxe=0, all you get is a #PF).  But the
hypervisor can still disable SMEP and SMAP, it can use hardware
breakpoints to leak information through the registers, and it can do all
the other attacks you mentioned.  If AMD had rdrand/rdseed, it could
replace the output with not so random values, and so on.

It's surely better than nothing, but "encryption that really is nothing
more than mitigation" is pretty weird.  I'm waiting for cloud vendors to
sell this as the best thing since sliced bread, when in reality it's
just mitigation.  I wonder how wise it is to merge SEV in its current
state---and since security is not my specialty I am definitely looking
for advice on this.

Paolo

ps: I'm now reminded of this patch:

    commit dab429a798a8ab3377136e09dda55ea75a41648d
    Author: David Kaplan <David.Kaplan@amd.com>
    Date:   Mon Mar 2 13:43:37 2015 -0600

    kvm: svm: make wbinvd faster

    No need to re-decode WBINVD since we know what it is from the
    intercept.

    Signed-off-by: David Kaplan <David.Kaplan@amd.com>
    [extracted from larger unlrelated patch, forward ported,
     tested,style cleanup]
    Signed-off-by: Joel Schopp <joel.schopp@amd.com>
    Reviewed-by: Radim Krčmář <rkrcmar@redhat.com>
    Signed-off-by: Marcelo Tosatti <mtosatti@redhat.com>

and I wonder if the larger unlrelated patch had anything to do with SEV!

Re: [RFC PATCH v1 00/18] x86: Secure Memory Encryption (AMD)

[Tom Lendacky]

On 05/09/2016 10:13 AM, Paolo Bonzini wrote:
> 
> 
> On 02/05/2016 20:31, Andy Lutomirski wrote:
>> And did the SEV implementation remember to encrypt the guest register
>> state?  Because, if not, everything of importance will leak out
>> through the VMCB and/or GPRs.
> 
> No, it doesn't.  And SEV is very limited unless you paravirtualize
> everything.
> 
> For example, the hypervisor needs to read some instruction bytes from
> memory, and instruction bytes are always encrypted (15.34.5 in the APM).
>  So you're pretty much restricted to IN/OUT operations (not even
> INS/OUTS) on emulated (non-assigned) devices, paravirtualized MSRs, and
> hypercalls.  These are the only operations that connect the guest and
> the hypervisor, where the vmexit doesn't have the need to e.g. walk
> guest page tables (also always encrypted).  It possibly can be made to
> work once the guest boots, and a modern UEFI firmware probably can cope
> with it too just like a kernel can, but you need to ensure that your
> hardware has no memory BARs for example.  And I/O port space is not very
> abundant.

The instruction bytes stored in the VMCB at offset 0xd0 for a data
side #NPF are stored un-encrypted (which is not clearly documented in
the APM). This allows for the hypervisor to perform MMIO on emulated
devices. Because the hardware provides enough information on VMEXIT
events, such as exit codes, decode assist, etc., the hypervisor has
the information it needs to perform the operation without having to
read the guest pagetables and/or the guest instruction stream from
guest memory. There are a few minor corner cases (e.g. rep ins) and
there will be more info on those when the SEV patches are submitted.

> 
> Even in order to emulate I/O ports or RDMSR/WRMSR or process hypercalls,
> the hypervisor needs to read the GPRs.  The VMCB doesn't store guest
> GPRs, not even on SEV-enabled processors.  Accordingly, the hypervisor
> has access to the guest GPRs on every exit.

In this initial version of SEV support the hardware does not encrypt
the guest save state and the hypervisor does have access to the GPRs.

> 
> In general, SEV provides mitigation only.  Even if the hypervisor cannot
> write known plaintext directly to memory, an accomplice virtual machine
> can e.g. use the network to spray the attacked VM's memory.  At least

Can you elaborate further on this? The accomplice VM will not have
access to the encryption key of the target VM and cannot accomplish
any spraying that the hypervisor itself cannot do.

> it's not as easy as "disable NX under the guest's feet and redirect RIP"
> (pte.nx is reserved if efer.nxe=0, all you get is a #PF).  But the
> hypervisor can still disable SMEP and SMAP, it can use hardware
> breakpoints to leak information through the registers, and it can do all
> the other attacks you mentioned.  If AMD had rdrand/rdseed, it could
> replace the output with not so random values, and so on.

AMD added support for the rdrand in some of the later fam16h models.

> 
> It's surely better than nothing, but "encryption that really is nothing
> more than mitigation" is pretty weird.  I'm waiting for cloud vendors to
> sell this as the best thing since sliced bread, when in reality it's
> just mitigation.  I wonder how wise it is to merge SEV in its current
> state---and since security is not my specialty I am definitely looking
> for advice on this.
> 

In this first generation of SEV, we are targeting a threat model very
similar to the one used by SMEP and SMAP. Specifically, SEV protects a
guest from a benign but vulnerable hypervisor, where a malicious guest
or unprivileged process exploits a system/hypervisor interface in an
attempt to read or modify the guest's memory.  But, like SMEP and SMAP,
if an attacker has the ability to arbitrarily execute code in the
kernel, he would be able to circumvent the control. AMD has a vision
for this generation of SEV to be foundational to future generations
that defend against stronger attacks.

Thanks,
Tom

> Paolo
> 
> ps: I'm now reminded of this patch:
> 
>     commit dab429a798a8ab3377136e09dda55ea75a41648d
>     Author: David Kaplan <David.Kaplan@amd.com>
>     Date:   Mon Mar 2 13:43:37 2015 -0600
> 
>     kvm: svm: make wbinvd faster
> 
>     No need to re-decode WBINVD since we know what it is from the
>     intercept.
> 
>     Signed-off-by: David Kaplan <David.Kaplan@amd.com>
>     [extracted from larger unlrelated patch, forward ported,
>      tested,style cleanup]
>     Signed-off-by: Joel Schopp <joel.schopp@amd.com>
>     Reviewed-by: Radim Krčmář <rkrcmar@redhat.com>
>     Signed-off-by: Marcelo Tosatti <mtosatti@redhat.com>
> 
> and I wonder if the larger unlrelated patch had anything to do with SEV!
> 

Re: [RFC PATCH v1 00/18] x86: Secure Memory Encryption (AMD)

[Paolo Bonzini]

On 09/05/2016 23:08, Tom Lendacky wrote:
> On 05/09/2016 10:13 AM, Paolo Bonzini wrote:
>>
>>
>> On 02/05/2016 20:31, Andy Lutomirski wrote:
>>> And did the SEV implementation remember to encrypt the guest register
>>> state?  Because, if not, everything of importance will leak out
>>> through the VMCB and/or GPRs.
>>
>> No, it doesn't.  And SEV is very limited unless you paravirtualize
>> everything.
>>
>> For example, the hypervisor needs to read some instruction bytes from
>> memory, and instruction bytes are always encrypted (15.34.5 in the APM).
>>  So you're pretty much restricted to IN/OUT operations (not even
>> INS/OUTS) on emulated (non-assigned) devices, paravirtualized MSRs, and
>> hypercalls.  These are the only operations that connect the guest and
>> the hypervisor, where the vmexit doesn't have the need to e.g. walk
>> guest page tables (also always encrypted).  It possibly can be made to
>> work once the guest boots, and a modern UEFI firmware probably can cope
>> with it too just like a kernel can, but you need to ensure that your
>> hardware has no memory BARs for example.  And I/O port space is not very
>> abundant.
> 
> The instruction bytes stored in the VMCB at offset 0xd0 for a data
> side #NPF are stored un-encrypted (which is not clearly documented in
> the APM). This allows for the hypervisor to perform MMIO on emulated
> devices. Because the hardware provides enough information on VMEXIT
> events, such as exit codes, decode assist, etc., the hypervisor has
> the information it needs to perform the operation

Ok, that helps.

>> In general, SEV provides mitigation only.  Even if the hypervisor cannot
>> write known plaintext directly to memory, an accomplice virtual machine
>> can e.g. use the network to spray the attacked VM's memory.  At least
> 
> Can you elaborate further on this? The accomplice VM will not have
> access to the encryption key of the target VM and cannot accomplish
> any spraying that the hypervisor itself cannot do.

It can send plaintext packets that will be stored encrypted in memory.
(Of course the hypervisor can do that too if it has access to the guest
network).  This was my first thought on attacking SEV, but luckily NX is
designed well.

> In this first generation of SEV, we are targeting a threat model very
> similar to the one used by SMEP and SMAP.

And that's great!  However, it is very different from "virtual machines
need not fully trust the hypervisor and administrator of their host
system" as said in the whitepaper.  SEV protects pretty well from
sibling VMs, but by design this generation of SEV leaks a lot of
information to an evil host---probably more than enough to mount a ROP
attack or to do evil stuff that Andy outlined.

My problem is that people will read AMD's whitepaper, not your message
on LKML, and may put more trust in SEV than (for now) they should.

Thanks,

Paolo

Re: [RFC PATCH v1 00/18] x86: Secure Memory Encryption (AMD)

[Borislav Petkov]

On Tue, May 10, 2016 at 01:23:35PM +0200, Paolo Bonzini wrote:
> It can send plaintext packets that will be stored encrypted in memory.
> (Of course the hypervisor can do that too if it has access to the guest
> network).

And then what?

You need to find out where exactly (which pages) got the packets. If at
all. I don't think you can do that from another VM, you probably are
more lucky if you're the hypervisor. But I'm no security guy so I'm
genuinely asking...

In any case, it sounds hard to do.

> And that's great!  However, it is very different from "virtual machines
> need not fully trust the hypervisor and administrator of their host
> system" as said in the whitepaper.

You know, those documents can be corrected ... :)

> SEV protects pretty well from sibling VMs, but by design
> this generation of SEV leaks a lot of information to an evil
> host---probably more than enough to mount a ROP attack or to do evil
> stuff that Andy outlined.
>
> My problem is that people will read AMD's whitepaper, not your message
> on LKML, and may put more trust in SEV than (for now) they should.

So if people rely on only one security feature, then they get what they
deserve. And even non-security people like me know that proper security
is layering of multiple features/mechanisms which should take care of
aspects only, not of everything. And not a single magic wand which makes
sh*t secure. :)

So let's please get real: the feature is pretty elegant IMO and gives
you a lot more security than before.

Can it be made better/cover more aspects?

Absolutely and it is a safe bet that it will be. You don't just
implement stuff like that in hw to not improve on it in future
iterations. It is like with all hardware features, they get improved
with time and CPU revisions.

Now, can people please look at the actual code and complain about stuff
that bothers them codewise? We've tried to make it as unobtrusive as
possible to the rest of the kernel but improvement suggestions are
always welcome!

:-)

Thanks.

-- 
Regards/Gruss,
    Boris.

ECO tip #101: Trim your mails when you reply.

RE: [RFC PATCH v1 00/18] x86: Secure Memory Encryption (AMD)

[Elliott, Robert (Persistent Memory)]

> -----Original Message-----
> From: linux-kernel-owner@vger.kernel.org [mailto:linux-kernel-
> owner@vger.kernel.org] On Behalf Of Tom Lendacky
> Sent: Tuesday, April 26, 2016 5:56 PM
> Subject: [RFC PATCH v1 00/18] x86: Secure Memory Encryption (AMD)
> 
> This RFC patch series provides support for AMD's new Secure Memory
> Encryption (SME) feature.
> 
> SME can be used to mark individual pages of memory as encrypted through the
> page tables. A page of memory that is marked encrypted will be automatically
> decrypted when read from DRAM and will be automatically encrypted when
> written to DRAM. Details on SME can found in the links below.
> 
...
> ...  Certain data must be accounted for
> as having been placed in memory before SME was enabled (EFI, initrd, etc.)
> and accessed accordingly.
> 
...
>       x86/efi: Access EFI related tables in the clear
>       x86: Access device tree in the clear
>       x86: Do not specify encrypted memory for VGA mapping

If the SME encryption key "is created randomly each time a system is booted,"
data on NVDIMMs won't decrypt properly on the next boot.  You need to exclude
persistent memory regions (reported in the UEFI memory map as 
EfiReservedMemoryType with the NV attribute, or as EfiPersistentMemory).

Perhaps the SEV feature will allow key export/import that could work for
NVDIMMs.

---
Robert Elliott, HPE Persistent Memory

Re: [RFC PATCH v1 00/18] x86: Secure Memory Encryption (AMD)

[Tom Lendacky]

On 04/30/2016 01:13 AM, Elliott, Robert (Persistent Memory) wrote:
>> -----Original Message-----
>> From: linux-kernel-owner@vger.kernel.org [mailto:linux-kernel-
>> owner@vger.kernel.org] On Behalf Of Tom Lendacky
>> Sent: Tuesday, April 26, 2016 5:56 PM
>> Subject: [RFC PATCH v1 00/18] x86: Secure Memory Encryption (AMD)
>>
>> This RFC patch series provides support for AMD's new Secure Memory
>> Encryption (SME) feature.
>>
>> SME can be used to mark individual pages of memory as encrypted through the
>> page tables. A page of memory that is marked encrypted will be automatically
>> decrypted when read from DRAM and will be automatically encrypted when
>> written to DRAM. Details on SME can found in the links below.
>>
> ...
>> ...  Certain data must be accounted for
>> as having been placed in memory before SME was enabled (EFI, initrd, etc.)
>> and accessed accordingly.
>>
> ...
>>       x86/efi: Access EFI related tables in the clear
>>       x86: Access device tree in the clear
>>       x86: Do not specify encrypted memory for VGA mapping
> 
> If the SME encryption key "is created randomly each time a system is booted,"
> data on NVDIMMs won't decrypt properly on the next boot.  You need to exclude
> persistent memory regions (reported in the UEFI memory map as 
> EfiReservedMemoryType with the NV attribute, or as EfiPersistentMemory).

The current plan is for the AMD Secure Processor to securely save the
SME encryption key when NVDIMMs are installed on a system. The saved SME
key will be restored if an NVDIMM restore event needs to be performed.
If there isn't an NVDIMM restore event, then the randomly generated key
will be used.

Thanks,
Tom

> 
> Perhaps the SEV feature will allow key export/import that could work for
> NVDIMMs.
> 
> ---
> Robert Elliott, HPE Persistent Memory
> 
> 

[RFC PATCH v1 00/18] x86: Secure Memory Encryption (AMD)

[Tom Lendacky]

This RFC patch series provides support for AMD's new Secure Memory
Encryption (SME) feature.

SME can be used to mark individual pages of memory as encrypted through the
page tables. A page of memory that is marked encrypted will be automatically
decrypted when read from DRAM and will be automatically encrypted when
written to DRAM. Details on SME can found in the links below.

The SME feature is identified through a CPUID function and enabled through
the SYSCFG MSR. Once enabled, page table entries will determine how the
memory is accessed. If a page table entry has the memory encryption mask set,
then that memory will be accessed as encrypted memory. The memory encryption
mask (as well as other related information) is determined from settings
returned through the same CPUID function that identifies the presence of the
feature.

The approach that this patch series takes is to encrypt everything possible
starting early in the boot where the kernel is encrypted. Using the page
table macros the encryption mask can be incorporated into all page table
entries and page allocations. By updating the protection map, userspace
allocations are also marked encrypted. Certain data must be accounted for
as having been placed in memory before SME was enabled (EFI, initrd, etc.)
and accessed accordingly.

This patch series is a pre-cursor to another AMD processor feature called
Secure Encrypted Virtualization (SEV). The support for SEV will build upon
the SME support and will be submitted later. Details on SEV can be found
in the links below.

The following links provide additional detail:

AMD Memory Encryption whitepaper:
   http://amd-dev.wpengine.netdna-cdn.com/wordpress/media/2013/12/AMD_Memory_Encryption_Whitepaper_v7-Public.pdf

AMD64 Architecture Programmer's Manual:
   http://support.amd.com/TechDocs/24593.pdf
   SME is section 7.10
   SEV is section 15.34

This patch series is based off of the master branch of tip.
  Commit 8d54fcebd9b3 ("Merge branch 'x86/urgent'")

---

Tom Lendacky (18):
      x86: Set the write-protect cache mode for AMD processors
      x86: Secure Memory Encryption (SME) build enablement
      x86: Secure Memory Encryption (SME) support
      x86: Add the Secure Memory Encryption cpu feature
      x86: Handle reduction in physical address size with SME
      x86: Provide general kernel support for memory encryption
      x86: Extend the early_memmap support with additional attrs
      x86: Add support for early encryption/decryption of memory
      x86: Insure that memory areas are encrypted when possible
      x86/efi: Access EFI related tables in the clear
      x86: Decrypt trampoline area if memory encryption is active
      x86: Access device tree in the clear
      x86: DMA support for memory encryption
      iommu/amd: AMD IOMMU support for memory encryption
      x86: Enable memory encryption on the APs
      x86: Do not specify encrypted memory for VGA mapping
      x86/kvm: Enable Secure Memory Encryption of nested page tables
      x86: Add support to turn on Secure Memory Encryption


 Documentation/kernel-parameters.txt  |    3 
 arch/x86/Kconfig                     |    9 +
 arch/x86/include/asm/cacheflush.h    |    3 
 arch/x86/include/asm/cpufeature.h    |    1 
 arch/x86/include/asm/cpufeatures.h   |    5 
 arch/x86/include/asm/dma-mapping.h   |    5 
 arch/x86/include/asm/fixmap.h        |   16 ++
 arch/x86/include/asm/kvm_host.h      |    2 
 arch/x86/include/asm/mem_encrypt.h   |   99 ++++++++++
 arch/x86/include/asm/msr-index.h     |    2 
 arch/x86/include/asm/pgtable_types.h |   49 +++--
 arch/x86/include/asm/processor.h     |    3 
 arch/x86/include/asm/realmode.h      |   12 +
 arch/x86/include/asm/vga.h           |   13 +
 arch/x86/kernel/Makefile             |    2 
 arch/x86/kernel/asm-offsets.c        |    2 
 arch/x86/kernel/cpu/common.c         |    2 
 arch/x86/kernel/cpu/scattered.c      |    1 
 arch/x86/kernel/devicetree.c         |    6 -
 arch/x86/kernel/espfix_64.c          |    2 
 arch/x86/kernel/head64.c             |  100 +++++++++-
 arch/x86/kernel/head_64.S            |   42 +++-
 arch/x86/kernel/machine_kexec_64.c   |    2 
 arch/x86/kernel/mem_encrypt.S        |  343 ++++++++++++++++++++++++++++++++++
 arch/x86/kernel/pci-dma.c            |   11 +
 arch/x86/kernel/pci-nommu.c          |    2 
 arch/x86/kernel/pci-swiotlb.c        |    8 +
 arch/x86/kernel/setup.c              |   14 +
 arch/x86/kernel/x8664_ksyms_64.c     |    6 +
 arch/x86/kvm/mmu.c                   |    7 -
 arch/x86/kvm/vmx.c                   |    2 
 arch/x86/kvm/x86.c                   |    3 
 arch/x86/mm/Makefile                 |    1 
 arch/x86/mm/fault.c                  |    5 
 arch/x86/mm/ioremap.c                |   31 +++
 arch/x86/mm/kasan_init_64.c          |    4 
 arch/x86/mm/mem_encrypt.c            |  201 ++++++++++++++++++++
 arch/x86/mm/pageattr.c               |   78 ++++++++
 arch/x86/mm/pat.c                    |   11 +
 arch/x86/platform/efi/efi.c          |   26 +--
 arch/x86/platform/efi/efi_64.c       |    9 +
 arch/x86/platform/efi/quirks.c       |   12 +
 arch/x86/realmode/init.c             |   13 +
 arch/x86/realmode/rm/trampoline_64.S |   14 +
 drivers/firmware/efi/efi.c           |   18 +-
 drivers/firmware/efi/esrt.c          |   12 +
 drivers/iommu/amd_iommu.c            |   10 +
 include/asm-generic/early_ioremap.h  |    2 
 include/linux/efi.h                  |    3 
 include/linux/swiotlb.h              |    1 
 init/main.c                          |    6 +
 lib/swiotlb.c                        |   64 ++++++
 mm/early_ioremap.c                   |   15 +
 53 files changed, 1217 insertions(+), 96 deletions(-)
 create mode 100644 arch/x86/include/asm/mem_encrypt.h
 create mode 100644 arch/x86/kernel/mem_encrypt.S
 create mode 100644 arch/x86/mm/mem_encrypt.c

-- 
Tom Lendacky