From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S932502AbcHVWxs (ORCPT ); Mon, 22 Aug 2016 18:53:48 -0400 Received: from mail-sn1nam02on0068.outbound.protection.outlook.com ([104.47.36.68]:55004 "EHLO NAM02-SN1-obe.outbound.protection.outlook.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1756288AbcHVWxi (ORCPT ); Mon, 22 Aug 2016 18:53:38 -0400 Authentication-Results: spf=none (sender IP is ) smtp.mailfrom=Thomas.Lendacky@amd.com; From: Tom Lendacky Subject: [RFC PATCH v2 20/20] x86: Add support to make use of Secure Memory Encryption To: , , , , , , , , CC: Radim =?utf-8?b?S3LEjW3DocWZ?= , Arnd Bergmann , Jonathan Corbet , Matt Fleming , Joerg Roedel , "Konrad Rzeszutek Wilk" , Andrey Ryabinin , Ingo Molnar , Borislav Petkov , "Andy Lutomirski" , "H. Peter Anvin" , Paolo Bonzini , Alexander Potapenko , "Thomas Gleixner" , Dmitry Vyukov Date: Mon, 22 Aug 2016 17:39:08 -0500 Message-ID: <20160822223908.29880.50365.stgit@tlendack-t1.amdoffice.net> In-Reply-To: <20160822223529.29880.50884.stgit@tlendack-t1.amdoffice.net> References: <20160822223529.29880.50884.stgit@tlendack-t1.amdoffice.net> User-Agent: StGit/0.17.1-dirty MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: 7bit X-Originating-IP: [165.204.77.1] X-ClientProxiedBy: SN1PR0701CA0004.namprd07.prod.outlook.com (10.162.96.14) To BN6PR12MB1139.namprd12.prod.outlook.com (10.168.226.141) X-MS-Office365-Filtering-Correlation-Id: bc5ceace-4355-46d4-baad-08d3cadd2354 X-Microsoft-Exchange-Diagnostics: 1;BN6PR12MB1139;2:c5x5M2UNf22zqyNm1fJXKL7wgyDQRp8J3pSmlOpHmTPuJJflDocboho1haSop9Hq9tgHseEdUOXARPTX+Kt/UIND6gO1JpfU2KCUReyM3Jxnzw/AoaKPIkZLdRmgVhbEKlVMv3GoZuIDXPV5WL4iMBZgnmEwIKmDsq+bTn8JjeCpohV0BkoZVbwLPiYdyIKJ;3:/JZfoXngofieGxS/yF3oARsdJXJ0ruZL36kETB54ZYVRvK6H9me0kIBQ+zU3/Kyuc48W5tcaNVzyicWBsWdkXz4OkDOfwNllnndX1EKl0NQnrZUs9uAgLQrM3JhJ08JB X-Microsoft-Antispam: UriScan:;BCL:0;PCL:0;RULEID:;SRVR:BN6PR12MB1139; X-Microsoft-Exchange-Diagnostics: 1;BN6PR12MB1139;25:hDbXOzA+v0nbUUq2+nav6mZU3+g8AeXbTNGkr9tfUmNWPB3dH3tp/rhtXUPINBjr7iljNFCVCWfL97j2LuMe9XaD5//DyFlIKzP2oZ77o/HwY5bzFcl1o7SbDrEUfbeppvYCWCAN9yMDJQHbTeKvqaZEGl7GG2cO0ANrf0htCfefXvqpG/4i35WnPMuLZ9aQiJOPJ5JzMk3bytTh23/lxNyvUHKRiAe6L42XwsdmxDeMqXkJShczvbV0isbXLOa9GwQWHe2PprNvQlcYAnnCqSLuI90bLgLSaa+JpgBI+z8QN1wb+y92+lSpcHmsdqlPsyqIzS2OML1EuKORHPYIhs4qnI6QefIg79O6x8lRmoie0uOPgqalR6Ye2OxCvkir9z+LFvzFUA26Cm/VxvRhMZcZB46ra6psiHr+J600OUFXcPHdPge3e3zu3aMd00Df2QmkBXMht/qKNfY5vEszCbUc50GlSM/rmxT97LRWNzfHocXc+WpNIIskwjZiM7ebq4TTwZ6r5IacBNo/W3lL/6gh9E7t9vVhNfpzFqBBMwUDPk51YCDPv/UriWQWw20SRpmOFxwGaqSQ/tH5SDsy9iIwniVtYKzbRzTneU440zWUI9DdRT3zqK6OLlnmMExPm/8D94n8NDzKly7nnW7BVArEgBKc6MHTsykPdwK8QoC4L6JNCSBKxqhfC4uao/2QaGJlrYgU25tr3mhmsLgxNvmvbNyxP5jOduNQIg5EVRs3AvvE39tDy0niF7gIRFkNqMQ7kPmLcEsQdwL0X+WKXg== X-Microsoft-Exchange-Diagnostics: 1;BN6PR12MB1139;31:o0GhDjfAHl3p37UEECJdcuTSaS/TmoxLlETCLaxbA0t8bycYnpXDyEfy4buOxkaw0R8mkbn1E0IpPeUO3eeBCCYp/dWW/SXAP1nJi3ilSh7E1eCDE028ubSafac/9eN+HRw5utcjl9LkTw4X3P9s/iFN/SUEWDNMGWqdVXBJzZS6RMHCES4pAiAkeSvDjkALwl6DQjbwcywgR9ss07vF0Jf7TaUwvFE9U7hFotq32vQ=;20:n22wGM2Dj+WnUbhQbydvqWdmer/zIw4ySKfATI8FI68csGwwWn6ZqdlBoYwqavMmeQpXI1SyGJh294kgPOF8zG14vNS7zFNtI9sG38Eb1Ci2dznl35RMxSZS5TlkOB5MNR1iHknqo8IeljAcVsGrPX8nSTVYfajvy4tp/PiVXf2c+E9yzvNrMwVfNzi5h0eRJ7kDGK7kU7aszSiUtN94dbimId+e+OWGiTHjQYreYLiUeznxt4SSD3bUeJz5YcSIPICA9zgUaFOtdb5XwMEThRwh5TaSDMPSpOm2YgGFwXcYNCklwYLBd3sT3C84SDK1cZ1DApvWBlGB7Y7rcfr0mm8ZQupISuCXs7PJ3q+I6QKfql7Q1z3TQ/3NZ+4pfiZrKlvsHrvidz7b+ChcqlrGOZ1rzcNyb/XfbcgcB203cKAdN2xgtXUvbnM0QWar5KFRSKo2daivl+qL5MLjcTCCu0VQAvphWV43lESnkb1PSl3gitBdy9NmINwWUjbwvF3x X-Microsoft-Antispam-PRVS: X-Exchange-Antispam-Report-Test: UriScan:(767451399110); X-Exchange-Antispam-Report-CFA-Test: BCL:0;PCL:0;RULEID:(6040176)(601004)(2401047)(8121501046)(5005006)(3002001)(10201501046)(6055026);SRVR:BN6PR12MB1139;BCL:0;PCL:0;RULEID:;SRVR:BN6PR12MB1139; X-Microsoft-Exchange-Diagnostics: 1;BN6PR12MB1139;4:5lgyV6URyB+94WTg6KOsvGseqlfZ1q1jwXAAPh1tpU4ABIawLYrq3cNJqEhbhjvuDFNUVfB7Rg2+erSnzLUmlmHlpPTvu2GAaFkuwLDDpzklgOzoksEFi/NpM7Q1zH5oqEjtAFBsUgoQ/AHmn803K1hcN16KY8CfnYrfSCpUx54tDrh8V504xdP45+phxNcSPwaxLuhf1BH4BLTv7d4x4bW+EYaAUc9kp8eX/zTuPztsk2K47ah4UvVMQfGCdrMXi4Kwoowy7M2/K9PC/gveOdFhxTlFytpwm+L9WfRbJTtknQK9aWCR4RHD6H/IFm75OzcJjMseigV0xeQk7zG31CfJwyMA4tIjYmDmuZTVEVaL54O3pnOOjZskNZ1rOwuRc0V+y2Bf4vyATYIC+EROx1Mhz+o+RX5nJ6HE1wBrFu2DHr45rPXY1ot6Fr5fjaQV X-Forefront-PRVS: 00429279BA X-Forefront-Antispam-Report: SFV:NSPM;SFS:(10009020)(4630300001)(6009001)(7916002)(189002)(199003)(3846002)(5660300001)(97746001)(50986999)(86362001)(6116002)(77096005)(76176999)(54356999)(97736004)(23676002)(586003)(81166006)(8676002)(2950100001)(92566002)(81156014)(5001770100001)(2201001)(53416004)(189998001)(1076002)(83506001)(33646002)(68736007)(2906002)(229853001)(4001350100001)(4326007)(69596002)(47776003)(7416002)(42186005)(7846002)(105586002)(230700001)(66066001)(7736002)(103116003)(50466002)(305945005)(19580405001)(19580395003)(106356001)(101416001)(9686002)(71626007)(217873001);DIR:OUT;SFP:1101;SCL:1;SRVR:BN6PR12MB1139;H:tlendack-t1.amdoffice.net;FPR:;SPF:None;PTR:InfoNoRecords;MX:1;A:1;LANG:en; X-Microsoft-Exchange-Diagnostics: =?utf-8?B?MTtCTjZQUjEyTUIxMTM5OzIzOnFJOVorZkVma0wzbVI1OTBMSHFaVE1FMjI2?= =?utf-8?B?RVdidklPeWtjbXhuN3lIV1N4cUpGSXQ5UUNRNXFocEUzU1p1eWwyekVpVUh2?= =?utf-8?B?RUVGaEZrekU2bVdWQ0xjNzRWbEhFbTF6WU00M1duMnhRNzZ4N0ROQ2VITS91?= =?utf-8?B?R0VtUFpzdzgvVW5wOFR3QkQ5ODI5MmJSbFdQTWY0MUZMVVVSZlBlTStaK2RZ?= =?utf-8?B?ZkhwekM4TFN1RlFuNnFUV0Vpd0hTeGdDZmRVVXZ1bysxU3FqMzJSa2orRVd1?= =?utf-8?B?ajNPemhDNDlQb2JNMWtqTy9TUCtvbG4yWmZwMnBpeWxDNWYrVDZwMDZQTkNm?= =?utf-8?B?alZYV3FQZkM0QnBaNGw3NWl4dEFpOVVzTFhsc0c5bnN0ZFNDdXJ1WmFDcFM5?= =?utf-8?B?M2xJT0xmS1h2V0NsMW1LdWRGRTFIZ1lKOXVBOFFEZXdPTzZEc1JUNnQ3UXk2?= =?utf-8?B?eERkcnIzQzZhMXA2dnd4S3ZpRVRaaEQ3b3I4alhCZXkyWkliakNTUENoQ002?= =?utf-8?B?OWhCQkhwV3RuT0VUK3RSSjFrRVRwMnBwcEE1VGo2YkFHTE1PR3NKZ0wrMnFF?= =?utf-8?B?bk42S3F0L2ZMZHJOeTFUMXdmUllqa1JrRlNLUGdzalRVWE1KWHJrTkJMbEhw?= =?utf-8?B?d1hqQVQ3VUwwbTIxb2xGWm1KYnFFUnRUcTZmTnlURkZNMlZFRzZxMWJNbzBJ?= =?utf-8?B?aElLRk9ON1NYZTlGZENSYnEzdmN1L2VGUDNhS1ViTWd2eW9Kdm4zYUlKYmc2?= =?utf-8?B?Q3ZQVkZNRjZmV1p6Ymc0VUIyUGt4NU44YnhYUWwvWEszRVJ0SVNQM1hqRjZv?= =?utf-8?B?WWxFZTNjWWdkMnFWZFFLMFcza1JqZG9FeElsVjhRMFI4THplc2pidUJhd3hh?= =?utf-8?B?RkZEcjFDMHFjUHlYNHAvdHdXVHZQMnoxelBpQlRoQ0JGUlpHOHdEZmUzdGN6?= =?utf-8?B?UFFESGlWMytKNDlQV0RzUGFCQzNIenBjZDNvaGpEZ3lvR1ZiTkZoeEpJd1lI?= =?utf-8?B?NXBaV2NrMll2VkQ0cXZsZk9nSk9VRmo4dUdXSlBwOVowSmFCSU5WVEZGV3ds?= =?utf-8?B?bjk4bSt2YnZJWTI5WmVCd2JFeklUeTNPRVNWVDRxOU9nMVZsTGI3WVlXMVJv?= =?utf-8?B?U2tzSS9vaExUdkI0M2JTc0J5N1NOK2graVhJRDd3eGQyOFAxNFJyWFdOTkdE?= =?utf-8?B?V0RISkNSQUpkQ2RoTk01ekc5eEJ0Q2t0bXUvOG93YjhoN3BQRjJua29oRzN3?= =?utf-8?B?ZVUxMVpyYStLS0EzVFczTlQ1UVFBeWZxMEZocG1XWkdiTDJ0b3FJc0pSdElR?= =?utf-8?B?b0xKUldjYjIrLzNZODVkSjh2ZEsvN2ErRDVSdjJJQmg0cUpKcmFLTm44TG9q?= =?utf-8?B?dE5GbXc2d2VESStzSDFCSEtYeFZoOTJOZlRIZVJxc0NoaFR2NFhQNGU1K3oz?= =?utf-8?B?bjNaU3FmU2pEc3lZaThMbWtSN0J3LzhWcy8wOUl4T0NQZXdvTzFqbEMxNmx6?= =?utf-8?B?WklWMWIwb0VPOGJ1U2kwaGNHY3BVMGdMUjFxYXBxUnJOVDlOVVA3eVhUUXgw?= =?utf-8?B?TkFLOXZDeG1FWGkrY2oyUnJ1WmdveVlqYWVvNnVjQzllY2treThWdW5DUGdr?= =?utf-8?B?QzRJNDYrL0RVaklYd0F4VFNteU5WUFJiT1dhSHNobHlnb0lRblFCYlJMb3Vn?= =?utf-8?B?aklXSlFGS1M5KzNIQU5icTZxRFBwRVlON3ZoeEtrY3ZISzB2N3Jibkc4WHBU?= =?utf-8?Q?OBoJY5t9a9gI6Zsr9tfD4TJJ2gXviMhi8KEqk=3D?= X-Microsoft-Exchange-Diagnostics: 1;BN6PR12MB1139;6:t2EmnNxEobI7u26MyGWqtm7ZmsXMieO+QObmEG86xkYwtJXuAp1oZirUU13l3koo8T6zJRd2S9yiAIYBxMvB450oC69POdmFun55ASrWIw6syHgAJrlMloqkclv/mFFyt2AkrhnRv5jeeJ3M2jltK2R/5D2w8Ig3wEpjlCq9PolXoSoFEPT4sn/2LMPpzhTdWDauPWsSgUTLJLBxSzlNMhe3MAhlyze+LQolvZeQMWBOv+PLiJnoO146KNZ2K1h3/Ydnt85GH/iMUXR5ilJznqNMGNsOt/iChkI8jyezb0/JCal10lNdzuJvXdaSOrvVykYJDcPVtbB8L9jAR3lixg==;5:BxnDzUwWj+SGCBOukiyYzqnXCyJJXsq8XV4DukDoHN1R0opQJZYScYZBIATzd1CmmaDpCr+PP+kGKLNXtBv3uNOtxNUsqj2rUZewnFmTvkgOByYJY1AzUt3TFtGxMSDk0fNG22b+rDu2LKoOC8/x7Q==;24:tp3KF3bo4TAbnY2ycRxhnsh+ejHLpnlkEupblMGJ5h9he6iV/mQKCB2hM7nzYE2XCIi9nJlrT5a2DwxpJyagPoHrtATfAYQxilavvWCCk0A=;7:7OplTI4GNvGzrA3l1vNYzgO3DReKvjgQn4ebd2R+PhllXBKzomq8aX6CN+QyRbaHRB64CEDFiluUEEUkQ7wNzw10DoDVUpsWk2zSH6T06Aer561Ec/ov6b8s9py1+am8ArGIWQLJxwxaol3FP2naj3w429x5jSDTMKB20vnS5kRQmYagROw81J5/ez8U4o8e6o3YA45EcJTduJJk8gkTF+c2W1IMopYj2kcyxjN8/e+xwrwrN4KNPnNLU6K34tw3 SpamDiagnosticOutput: 1:99 SpamDiagnosticMetadata: NSPM X-Microsoft-Exchange-Diagnostics: 1;BN6PR12MB1139;20:L1zeJVukPENlRt+oo4Pe5QPip9b5RJI84T9FbpZy9lNtz8rqkuFKuyfTNlSHPQMb4nJBrBJ/O+GRPVa+asIClveB9nXaqCl6l1x4o6oW3wg0JOEAQQ4t/ZLx22enOYFp8QjT+H2lijcGXzgty5KEmIGUAOJt18OkiMiURmYRFqSNDUNH5G1DkhHHEgS1/IwisM2NLe2RNUle3UaUi1t4iiUMDlJ1Nj7B4KBnTRF3uiHlTuMAwPGAie+Dvc8h3rPr X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 22 Aug 2016 22:39:10.6610 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-Transport-CrossTenantHeadersStamped: BN6PR12MB1139 Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org This patch adds the support to check if SME has been enabled and if the mem_encrypt=on command line option is set. If both of these conditions are true, then the encryption mask is set and the kernel is encrypted "in place." Signed-off-by: Tom Lendacky --- Documentation/kernel-parameters.txt | 3 arch/x86/kernel/asm-offsets.c | 2 arch/x86/kernel/mem_encrypt.S | 302 +++++++++++++++++++++++++++++++++++ arch/x86/mm/mem_encrypt.c | 2 4 files changed, 309 insertions(+) diff --git a/Documentation/kernel-parameters.txt b/Documentation/kernel-parameters.txt index 46c030a..a1986c8 100644 --- a/Documentation/kernel-parameters.txt +++ b/Documentation/kernel-parameters.txt @@ -2268,6 +2268,9 @@ bytes respectively. Such letter suffixes can also be entirely omitted. memory contents and reserves bad memory regions that are detected. + mem_encrypt=on [X86_64] Enable memory encryption on processors + that support this feature. + meye.*= [HW] Set MotionEye Camera parameters See Documentation/video4linux/meye.txt. diff --git a/arch/x86/kernel/asm-offsets.c b/arch/x86/kernel/asm-offsets.c index 2bd5c6f..e485ada 100644 --- a/arch/x86/kernel/asm-offsets.c +++ b/arch/x86/kernel/asm-offsets.c @@ -85,6 +85,8 @@ void common(void) { OFFSET(BP_init_size, boot_params, hdr.init_size); OFFSET(BP_pref_address, boot_params, hdr.pref_address); OFFSET(BP_code32_start, boot_params, hdr.code32_start); + OFFSET(BP_cmd_line_ptr, boot_params, hdr.cmd_line_ptr); + OFFSET(BP_ext_cmd_line_ptr, boot_params, ext_cmd_line_ptr); BLANK(); DEFINE(PTREGS_SIZE, sizeof(struct pt_regs)); diff --git a/arch/x86/kernel/mem_encrypt.S b/arch/x86/kernel/mem_encrypt.S index f2e0536..bf9f6a9 100644 --- a/arch/x86/kernel/mem_encrypt.S +++ b/arch/x86/kernel/mem_encrypt.S @@ -12,13 +12,230 @@ #include +#include +#include +#include +#include +#include + .text .code64 ENTRY(sme_enable) +#ifdef CONFIG_AMD_MEM_ENCRYPT + /* Check for AMD processor */ + xorl %eax, %eax + cpuid + cmpl $0x68747541, %ebx # AuthenticAMD + jne .Lmem_encrypt_exit + cmpl $0x69746e65, %edx + jne .Lmem_encrypt_exit + cmpl $0x444d4163, %ecx + jne .Lmem_encrypt_exit + + /* Check for memory encryption leaf */ + movl $0x80000000, %eax + cpuid + cmpl $0x8000001f, %eax + jb .Lmem_encrypt_exit + + /* + * Check for memory encryption feature: + * CPUID Fn8000_001F[EAX] - Bit 0 + * Secure Memory Encryption support + * CPUID Fn8000_001F[EBX] - Bits 5:0 + * Pagetable bit position used to indicate encryption + * CPUID Fn8000_001F[EBX] - Bits 11:6 + * Reduction in physical address space (in bits) when enabled + */ + movl $0x8000001f, %eax + cpuid + bt $0, %eax + jnc .Lmem_encrypt_exit + + /* Check if BIOS/UEFI has allowed memory encryption */ + movl $MSR_K8_SYSCFG, %ecx + rdmsr + bt $MSR_K8_SYSCFG_MEM_ENCRYPT_BIT, %eax + jnc .Lmem_encrypt_exit + + /* Check for the mem_encrypt=on command line option */ + push %rsi /* Save RSI (real_mode_data) */ + push %rbx /* Save CPUID information */ + movl BP_ext_cmd_line_ptr(%rsi), %ecx + shlq $32, %rcx + movl BP_cmd_line_ptr(%rsi), %edi + addq %rcx, %rdi + leaq mem_encrypt_enable_option(%rip), %rsi + call cmdline_find_option_bool + pop %rbx /* Restore CPUID information */ + pop %rsi /* Restore RSI (real_mode_data) */ + testl %eax, %eax + jz .Lno_mem_encrypt + + /* Set memory encryption mask */ + movl %ebx, %ecx + andl $0x3f, %ecx + bts %ecx, sme_me_mask(%rip) + +.Lno_mem_encrypt: + /* + * BIOS/UEFI has allowed memory encryption so we need to set + * the amount of physical address space reduction even if + * the user decides not to use memory encryption. + */ + movl %ebx, %ecx + shrl $6, %ecx + andl $0x3f, %ecx + movb %cl, sme_me_loss(%rip) + +.Lmem_encrypt_exit: +#endif /* CONFIG_AMD_MEM_ENCRYPT */ + ret ENDPROC(sme_enable) ENTRY(sme_encrypt_kernel) +#ifdef CONFIG_AMD_MEM_ENCRYPT + /* If SME is not active then no need to encrypt the kernel */ + cmpq $0, sme_me_mask(%rip) + jz .Lencrypt_exit + + /* + * Encrypt the kernel. + * Pagetables for performing kernel encryption: + * 0x0000000000 - 0x00FFFFFFFF will map just the memory occupied by + * the kernel as encrypted memory + * 0x8000000000 - 0x80FFFFFFFF will map all memory as write-protected, + * non-encrypted + * + * The use of write-protected memory will prevent any of the + * non-encrypted memory from being cached. + * + * 0x00... and 0x80... represent the first and second PGD entries. + * + * This collection of entries will be created in an area outside + * of the area that is being encrypted (outside the kernel) and + * requires 11 4K pages: + * 1 - PGD + * 2 - PUDs (1 for each mapping) + * 8 - PMDs (4 for each mapping) + */ + leaq _end(%rip), %rdi + addq $~PMD_PAGE_MASK, %rdi + andq $PMD_PAGE_MASK, %rdi /* RDI points to the new PGD */ + + /* Clear the pagetable memory */ + movq %rdi, %rbx /* Save pointer to PGD */ + movl $(4096 * 11), %ecx + xorl %eax, %eax + rep stosb + movq %rbx, %rdi /* Restore pointer to PGD */ + + /* Set up PGD entries for the two mappings */ + leaq (0x1000 + 0x03)(%rdi), %rbx /* PUD for encrypted kernel */ + movq %rbx, (%rdi) + leaq (0x2000 + 0x03)(%rdi), %rbx /* PUD for unencrypted kernel */ + movq %rbx, 8(%rdi) + + /* Set up PUD entries (4 per mapping) for the two mappings */ + leaq (0x3000 + 0x03)(%rdi), %rbx /* PMD for encrypted kernel */ + leaq (0x7000 + 0x03)(%rdi), %rdx /* PMD for unencrypted kernel */ + xorq %rcx, %rcx +1: + /* Populate the PUD entries in each mapping */ + movq %rbx, 0x1000(%rdi, %rcx, 8) + movq %rdx, 0x2000(%rdi, %rcx, 8) + addq $0x1000, %rbx + addq $0x1000, %rdx + incq %rcx + cmpq $4, %rcx + jb 1b + + /* + * Set up PMD entries (4GB worth) for the two mappings. + * For the encrypted kernel mapping, when R11 is above RDX + * and below RDI then we know we are in the kernel and we + * set the encryption mask for that PMD entry. + * + * The use of _PAGE_PAT and _PAGE_PWT will provide for the + * write-protected mapping. + */ + movq sme_me_mask(%rip), %r10 + movq $__PAGE_KERNEL_LARGE_EXEC, %r11 + andq $~_PAGE_GLOBAL, %r11 + movq %r11, %r12 + andq $~_PAGE_CACHE_MASK, %r12 + orq $(_PAGE_PAT | _PAGE_PWT), %r12 /* PA5 index */ + xorq %rcx, %rcx + leaq _text(%rip), %rdx /* RDX points to start of kernel */ +1: + /* Populate the PMD entries in each mapping */ + movq %r11, 0x3000(%rdi, %rcx, 8) + movq %r12, 0x7000(%rdi, %rcx, 8) + + /* + * Check if we are in the kernel range, and if so, set the + * memory encryption mask. + */ + cmpq %r11, %rdx + jae 2f + cmpq %r11, %rdi + jbe 2f + orq %r10, 0x3000(%rdi, %rcx, 8) +2: + addq $PMD_SIZE, %r11 + addq $PMD_SIZE, %r12 + incq %rcx + cmpq $2048, %rcx + jb 1b + + /* + * Set up a one page stack in the non-encrypted memory area. + * Set RAX to point to the next page in memory after all the + * page tables. The stack grows from the bottom so point to + * the end of the page. + */ + leaq (4096 * 11)(%rdi), %rax + addq $PAGE_SIZE, %rax + movq %rsp, %rbp + movq %rax, %rsp + push %rbp /* Save original stack pointer */ + + push %rsi /* Save RSI (real mode data) */ + + /* + * Copy encryption routine into safe memory + * - RAX points to the page after all the page tables and stack + * where the routine will copied + * - RDI points to the PGD table + * - Setup registers for call + * and then call it + */ + movq %rdi, %rbx + + leaq .Lencrypt_start(%rip), %rsi + movq %rax, %rdi + movq $(.Lencrypt_stop - .Lencrypt_start), %rcx + rep movsb + + leaq _text(%rip), %rsi /* Kernel start */ + movq %rbx, %rcx /* New PGD start */ + subq %rsi, %rcx /* Size of area to encrypt */ + + movq %rsi, %rdi /* Encrypted kernel space start */ + movq $1, %rsi + shlq $PGDIR_SHIFT, %rsi + addq %rdi, %rsi /* Non-encrypted kernel start */ + + /* Call the encryption routine */ + call *%rax + + pop %rsi /* Restore RSI (real mode data ) */ + + pop %rsp /* Restore original stack pointer */ +.Lencrypt_exit: +#endif /* CONFIG_AMD_MEM_ENCRYPT */ + ret ENDPROC(sme_encrypt_kernel) @@ -28,6 +245,87 @@ ENTRY(sme_get_me_loss) ret ENDPROC(sme_get_me_loss) +#ifdef CONFIG_AMD_MEM_ENCRYPT +/* + * Routine used to encrypt kernel. + * This routine must be run outside of the kernel proper since + * the kernel will be encrypted during the process. So this + * routine is defined here and then copied to an area outside + * of the kernel where it will remain and run un-encrypted + * during execution. + * + * On entry the registers must be: + * - RAX points to this routine + * - RBX points to new PGD to use + * - RCX contains the kernel length + * - RSI points to the non-encrypted kernel space + * - RDI points to the encrypted kernel space + * + * The kernel will be encrypted by copying from the non-encrypted + * kernel space to a temporary buffer and then copying from the + * temporary buffer back to the encrypted kernel space. The physical + * addresses of the two kernel space mappings are the same which + * results in the kernel being encrypted "in place". + */ +.Lencrypt_start: + /* Enable the new page tables */ + mov %rbx, %cr3 + + /* Flush any global TLBs */ + mov %cr4, %rbx + andq $~X86_CR4_PGE, %rbx + mov %rbx, %cr4 + orq $X86_CR4_PGE, %rbx + mov %rbx, %cr4 + + /* Set the PAT register PA5 entry to write-protect */ + push %rax + push %rcx + movl $MSR_IA32_CR_PAT, %ecx + rdmsr + push %rdx /* Save original PAT value */ + andl $0xffff00ff, %edx /* Clear PA5 */ + orl $0x00000500, %edx /* Set PA5 to WP */ + wrmsr + pop %rdx /* RDX contains original PAT value */ + pop %rcx + pop %rax + + movq %rsi, %r10 /* Save source address */ + movq %rdi, %r11 /* Save destination address */ + movq %rcx, %r12 /* Save length */ + addq $PAGE_SIZE, %rax /* RAX now points to temp copy page */ + + wbinvd /* Invalidate any cache entries */ + + /* Copy/encrypt 2MB at a time */ +1: + movq %r10, %rsi + movq %rax, %rdi + movq $PMD_PAGE_SIZE, %rcx + rep movsb + + movq %rax, %rsi + movq %r11, %rdi + movq $PMD_PAGE_SIZE, %rcx + rep movsb + + addq $PMD_PAGE_SIZE, %r10 + addq $PMD_PAGE_SIZE, %r11 + subq $PMD_PAGE_SIZE, %r12 + jnz 1b + + /* Restore PAT register */ + push %rdx + movl $MSR_IA32_CR_PAT, %ecx + rdmsr + pop %rdx + wrmsr + + ret +.Lencrypt_stop: +#endif /* CONFIG_AMD_MEM_ENCRYPT */ + .data .align 16 ENTRY(sme_me_mask) @@ -35,3 +333,7 @@ ENTRY(sme_me_mask) sme_me_loss: .byte 0x00 .align 8 + +mem_encrypt_enable_option: + .asciz "mem_encrypt=on" + .align 8 diff --git a/arch/x86/mm/mem_encrypt.c b/arch/x86/mm/mem_encrypt.c index 2f28d87..1154353 100644 --- a/arch/x86/mm/mem_encrypt.c +++ b/arch/x86/mm/mem_encrypt.c @@ -183,6 +183,8 @@ void __init mem_encrypt_init(void) /* Make SWIOTLB use an unencrypted DMA area */ swiotlb_clear_encryption(); + + pr_info("memory encryption active\n"); } unsigned long amd_iommu_get_me_mask(void)