From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S932821AbcHVWyQ (ORCPT ); Mon, 22 Aug 2016 18:54:16 -0400 Received: from mail-by2nam03on0077.outbound.protection.outlook.com ([104.47.42.77]:49152 "EHLO NAM03-BY2-obe.outbound.protection.outlook.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1752564AbcHVWxJ (ORCPT ); Mon, 22 Aug 2016 18:53:09 -0400 Authentication-Results: spf=none (sender IP is ) smtp.mailfrom=Thomas.Lendacky@amd.com; From: Tom Lendacky Subject: [RFC PATCH v2 10/20] x86: Insure that memory areas are encrypted when possible To: , , , , , , , , CC: Radim =?utf-8?b?S3LEjW3DocWZ?= , Arnd Bergmann , Jonathan Corbet , Matt Fleming , Joerg Roedel , "Konrad Rzeszutek Wilk" , Andrey Ryabinin , Ingo Molnar , Borislav Petkov , "Andy Lutomirski" , "H. Peter Anvin" , Paolo Bonzini , Alexander Potapenko , "Thomas Gleixner" , Dmitry Vyukov Date: Mon, 22 Aug 2016 17:37:23 -0500 Message-ID: <20160822223722.29880.94331.stgit@tlendack-t1.amdoffice.net> In-Reply-To: <20160822223529.29880.50884.stgit@tlendack-t1.amdoffice.net> References: <20160822223529.29880.50884.stgit@tlendack-t1.amdoffice.net> User-Agent: StGit/0.17.1-dirty MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: 7bit X-Originating-IP: [165.204.77.1] X-ClientProxiedBy: MWHPR07CA0004.namprd07.prod.outlook.com (10.172.94.14) To BN6PR12MB1137.namprd12.prod.outlook.com (10.168.226.139) X-MS-Office365-Filtering-Correlation-Id: af372026-e9c3-452e-0226-08d3cadce7c1 X-Microsoft-Exchange-Diagnostics: 1;BN6PR12MB1137;2:A7Bs9rasnsSrrkt+8xmKJilhBs3ZEBkHRIL7QDTEaS1adUvaliLhx4KURdWnmgFZJxz6HcuxCc0xlHRF5zyg5HSTbRulJp2SYc+t4Fr5TZAYamXlepybHDFiQrFrY2ve+EX71IEhzZGkK+ZHz5PMmwaNsjjaKMxMhqdtTHWBwbmq6x67zA9ECslLXDvAhZvV;3:zMDM8LSB/jsqxJKvIOA/U0rEh7HmZQoTPV/PVkyOJTrEI4EaIdvaB0pgWUCEOo49mYtdaNYxIKUcQSynmdPao471CqN/wk60kZNJuMP4SBp8s4INCVb93Dh4vl9PsyTy;25:x3wVNhs1/ksv4AmTOokT5Xg/A9NobLMMNcP54XF++AHG/C4LXU/Go2PurSH/1KZ1WfohQiJ8dg5uSewrJi5HUOTU5OIpJVTlR6DDaRMUZ7xIap25m6hZJFsyzAPKb1IDCZg1E62faMxDdMeFIX4LblaEmw/D83c4JKG+WXvABYzGH0abj30yKl4wwrU/E6IedTsFcUd2bhHD7WAu9wU0d+cQ+OFFBDs0rjiNCOt4SL09O3VftWZfd3E5TBeCiMMw5Zh2psF0ZISikhyCi3QcXQfAXUMMptiujqF4jfr+aGbXHsIHvkkH5CI7AMpUqVCdZHy2j4//5kVtdQ2K+kHRUCdl+0dgSzyJb7ITqavYV89Vm3+SRSIb4dYz8HZhWOH4bnBaXDmOVrR7jwY3ARwIJWJq3hhiL+9dS1TbN0TCFRI= X-Microsoft-Antispam: UriScan:;BCL:0;PCL:0;RULEID:;SRVR:BN6PR12MB1137; X-Microsoft-Exchange-Diagnostics: 1;BN6PR12MB1137;31:8H6AXZcxU/RIybNg+bUkMOWxI6r8pABtwdqipoEl6DP8Og3Wj9rkYURVegMiSd3onjzptnwTOYziHXhpdxG9oXMTYhWHN0H1sw7UwJjqDyhD2EsWokOnvWrBeBaD+hSmBlLw5y5JYxQVx7vBLynb3BLdjn4DlCfLcisjG/5zoSPRZQi07ndklRZlHfrsh21EciGEne9K6bFfEB/OJ996TfgTrKGthuqI6/48L4gZvWc=;20:Ws4NSDPVlEY94W3C7Yg4FqHrQElf9LrxvoaoEixgo/LkIL9kS1sCQ3gG98vOgtwgsD1Q5IYn4hB/VzPZOkv1ZM0ylmFNEepeT9O66tdkSat3R64iwOW2eqrPGF0K1C2RgCBekpQdKjowIB3jBGOh80LPvdM/GsE94oiF6+irWE5+BSfL8B4VezLhdDMGq+DO6FCqiPNE9gu3N34+GpTef9YFOZWUYKwE+NdxxrHc2KMM59w4V2NlhDceiBYZXoAMyfuOs72L9LY5O3smbUH0jIiYmOHuHf8xC7ZU1dHymbM/D1O5QwEiG2ooq5IcgbDh45JxcHBqwYNYPibYefG0Mn6yhSYzKqJ7+kvSHvz9NQmrh/+SNqZzRto7kKc99g7bKaEhEsNSCLs6lndK5s9Y3iGg8DX19rj5gdFaSltSDoraK1NZEL3ztWZrTAut2Capd/SNq+9hOnEhttUOGs5+k0HAJghFp0YhK16a6fwxH7+46mO5G+CIEvEHFMHjm6jM X-Microsoft-Antispam-PRVS: X-Exchange-Antispam-Report-Test: UriScan:(767451399110); X-Exchange-Antispam-Report-CFA-Test: BCL:0;PCL:0;RULEID:(6040176)(601004)(2401047)(5005006)(8121501046)(3002001)(10201501046)(6055026);SRVR:BN6PR12MB1137;BCL:0;PCL:0;RULEID:;SRVR:BN6PR12MB1137; X-Microsoft-Exchange-Diagnostics: 1;BN6PR12MB1137;4:HcOAPy5Xn7ffg1pYnQPf5hQG7H5DRBu/h2I8D+rKTExDN30W++Dg/c8oaIIKbKZtsMNZIzawZCITtyrb0gA9X9w/CR97qUOMo0O/5m2q7cmmslYjQOuESjG+qwr3F/wyjDNz72BasvrpKZdw/DVFSVhSWuunA2Jnsk4qUa5CFIJIVASSf3081bSi7UAo0QffJRapUOUwOd4Oqx9l9pV5QpRkQUKfjZBmtN511qb9JALpmyskXYJi81gLJ786YcWTF1P/JCUhziGDQM9eoXP+Ol4p4z88hdWrMy90e0H416pjF4L7vL516k72N4+JT67XTAmTFNwy7QZBrCcwDBOTGJ82CcdqmmhdhITOkbUB3GVDJis5H8GO8Y4W9wRmOrRTdLXzTtf0loh1HGOzRxOe7H033d04nEPVDwDaukcmbjSvfC7Hx8joSMM3wsfgUMTk X-Forefront-PRVS: 00429279BA X-Forefront-Antispam-Report: SFV:NSPM;SFS:(10009020)(4630300001)(6009001)(7916002)(189002)(199003)(4001350100001)(97736004)(189998001)(5001770100001)(69596002)(19580405001)(19580395003)(83506001)(86362001)(42186005)(97746001)(23676002)(305945005)(586003)(103116003)(2201001)(7846002)(6116002)(47776003)(50466002)(66066001)(229853001)(7736002)(3846002)(106356001)(2906002)(7416002)(81156014)(81166006)(33646002)(8676002)(2950100001)(77096005)(4326007)(50986999)(76176999)(230700001)(54356999)(68736007)(1076002)(105586002)(9686002)(5660300001)(53416004)(101416001)(92566002)(71626007)(217873001);DIR:OUT;SFP:1101;SCL:1;SRVR:BN6PR12MB1137;H:tlendack-t1.amdoffice.net;FPR:;SPF:None;PTR:InfoNoRecords;MX:1;A:1;LANG:en; X-Microsoft-Exchange-Diagnostics: =?utf-8?B?MTtCTjZQUjEyTUIxMTM3OzIzOktWVU9OUHk0dnpEWG01dko1SEprTEpQMTNB?= =?utf-8?B?TjdhTHZyaWVCT1R0am00MG8yZUF0NWYyOFpERkxGQk1zMktoNjRjZjVEeW93?= =?utf-8?B?N0NiUW41RG1lQVhTQS82MEF5TW5VSStqbHNBVlYvbGxVT0lqUW1KcldYZXdQ?= =?utf-8?B?VGFRaXVST2IzTzJLNVAvQUMzcHFZNjdua3lINEtJcUN3ajNna0h6NW82RHNx?= =?utf-8?B?UUN0K3BsajNsOGhxQ2FCc0ppcFEvWlFOd1kvc282Qk9NbWZxM05MTmc5REti?= =?utf-8?B?aXVob1dIN2Y5TzFSV0ZlaEt1aFRnUnQyS3pBdmFRODlPK3Z0SVRqWVhlL2oz?= =?utf-8?B?a1Zyb3MwU0VnQ1E5anpvZVpBNHRFWEpEZDROcC9mM2c3Y1laZlZicFllK1U4?= =?utf-8?B?TTQxOGVwekhabVIyR2ZjQTkwSVQzTnJ1VHoyMHdDbjdjM09MWlREM0Z1Tzk3?= =?utf-8?B?ZkxGeEhEdkVoTXNQcHQrQ3MzZmF4U25EYW1WaGFmQWlLWnpXL1B6WThLdUJB?= =?utf-8?B?OXhrMXh0Vll1MUkzQ3hKQ0hFaFVaR3BpKzdXK3dXdVpNNFo0bmN2Nk9ZRkh3?= =?utf-8?B?c3VwWDNqdllXUk51QjRnU09tckVPL2tDQTQxQnR6R0VzWEM5RmhxdVZocnZR?= =?utf-8?B?Y2czRGZ1WXNmOXBqQXFaMUNCVmtIVUEvQzhCL1NpbXVQUHFjazVYb1h5WGFZ?= =?utf-8?B?TE1DMm1yWjhCc1lkQzQvTzhmRXN1dUNLL1ljUXd2blhWQkFFcm5EU0EwVFpl?= =?utf-8?B?aTYzdnBQWTYrajRiTGhhZDJSU3Y1TVExbllyRUdSbVlJM2pnak5tNndZSita?= =?utf-8?B?ZXQ3RWNlNnRNdkVVbUpwK1I0dFdsYzNEOFloa0ZwbTZGa0FJV2NyeGlNcWVp?= =?utf-8?B?MnFOUk8yU3p4SlVmVTJaVWoxRkowaDdjcmpUdWtJSi9LYnhORGlTMStuVTlE?= =?utf-8?B?NmR2QzVLeVMyUUVuNE5EL3g2cW5odko4R251Vjd2TXI1STJHV3dHNHJ0TTcx?= =?utf-8?B?bW9PQXpJcWlJbzUvYm4xMnNLb3hvZTZGL0lYcjBuQUtVbDR3bTltL0pGUWsz?= =?utf-8?B?eW9PRU93NWlxcXpEeTlqMkY4OEFnRFVSZzYxdGdId2lmR0wvbFdmSlFlWktW?= =?utf-8?B?NFNQYXFJd1h1N2Q3WGhSV2xCckxvREdjMkJNNDVVdlRNSlZVT05JQWJ3SkRh?= =?utf-8?B?UUdCTnZkem5vVGUzTDBYeS9kTzljQ2V2cDJYQ2dhYmlLQjZjcmdqMVB4SEhw?= =?utf-8?B?cFhpZkVnbFVjWXM0R3BjVnJ0ZWZSeWFmZGk1Tkh3cEJzeGxwbDkxMDZOL2VT?= =?utf-8?B?WjdIdjlEOEJhd1p4Y2c5SFNTdmEvMmZEbUxwajBzMjQ1SGQ1Sm82bG1TQnd1?= =?utf-8?B?Y25qcVE1ZHlpMkJKdkd2SUM2TG5yb0VhZHV0ZS9IdmdVL3Rwbm11TW1GdWla?= =?utf-8?B?d3N3b0ZHdzhCMWVUbjNNVXhUSXNaSE0vOEltUUZ0TmxqajJNWW95SHpFSVZE?= =?utf-8?B?VzNPWU1Pb3lrTFRhelFyaG5QK1ZSRndRSzdTdTR1V1VkUVlMMVczdiswVzJz?= =?utf-8?B?Um0reHFMSnA2UDg4Z2JveERJR2t3dWFOWHRkT2VNWDk3b2VuTWF0L1dYUnZw?= =?utf-8?B?dk9wZjNabm9GSE1Jd2hDelQxN0wxaUNGMFJrdE10S0lPRTFyOXd2alFlcVFk?= =?utf-8?B?aFRRV2ZLVXBKbHpjNkRQb2xnNUZibGJHRVhIN3ozakZsUm1GWEI1cXF1S0JH?= =?utf-8?Q?dASFKaYA1Hd1QT+sOQqR/j8TPDXoqZFgl3oAw=3D?= X-Microsoft-Exchange-Diagnostics: 1;BN6PR12MB1137;6:HmJvbrEG5/e1osiRSKlQs+MhNcZEEkWczIbmaVLYR2N0PEB2Wvw1iAcPiAKX3v2Oi/j5YvXr0tSAjcxSqsm8ghCTfrrjRlLaB9mH0ycYc1OA+VALsCJ0uXtetUpR06ZQQoobdGja3uZQmeaEeW1bVVaoH6x1JWrdHg0QwE7MHMg0OxP+kIFbe23z4+nCZ/q5LbpeqiOVXjbqKtyLgSEF4daC5OWPYcbX+ymnTyLtattxdu0ckHx/hjgZcpWzprSWwq/YGQ4Ab3tKuYGjgATRv2PROtnD2JXMnnx6d2la5fHXjNGLe+dus+7ZFNHiYaO8m0xnm38WNsMtmeDNxGi4QQ==;5:VQSSmhaY+ajPn9dQ7NBMt8K5ajWW6gR8gC+EeVl/HCwkmwjiCOL3ux5MpAgPsp/RwzaYCIpPdpKNf2T6NYl4I3pejbRpK1qbba7G2FymarZE+qNwjEQ/8jJwNBXbT8j7XeTARRlv/bJVfdbqtmpJ2Q==;24:aw/TrWU7yZuU8A3yBrPhhv6lwHSlAbEiLZVzEWwOG+GvtUf7NoRwyK5NHsdzcIdMPDd+5TtXsT3mR2WuOZSdt1ZF0NX0mYst3TNEynpjssA=;7:WTYkR2IEGvsla+jgIyLt1vRSNyoZMv6jz3nCZHZriyxmaaVS0ajL/lxGmSKfRrIgFkef3T3VHCE40fd1ABnQE3gKc3ETxa9iEVX0WPPgP3rTPlxKl/1A01IV8iGV5u/n4G7/cxm6YeZY1G6GBsMY4pyRaZE13CbiJLQqWRLO7yJKxWgN5NQoQi1hQXcGUVBoz/qGSHdXD22eeb6eTqKatAQb6KbZcm1O6mQIWyE6GDJR0F6jx2E0kBmAYUOQlXwn SpamDiagnosticOutput: 1:99 SpamDiagnosticMetadata: NSPM X-Microsoft-Exchange-Diagnostics: 1;BN6PR12MB1137;20:r/ZFwCM5PmxXV8+oxcZlgipMttjxodIwvyaOwmQdVe79vPwlRnO5h6u7uef2bxkESjldsvBGijNo8tVbKp+M75DTqKJf+uGhrRheueaid+/7/af2pNsIHTAzrvpzljAB/WdOaSJBLIVd/zmJFGkBJx8bY0eh2J9s2d3T0dHadiFx2WlCtKwS4KRAOIMxEhz4gE/fxvD9OTOM29Rk8DA8NizO/V5g1PJohibQXPZ+f5PYfyp+mQMeGecimNGsF6eZ X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 22 Aug 2016 22:37:28.4454 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-Transport-CrossTenantHeadersStamped: BN6PR12MB1137 Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Encrypt memory areas in place when possible (e.g. zero page, etc.) so that special handling isn't needed afterwards. Signed-off-by: Tom Lendacky --- arch/x86/kernel/head64.c | 93 ++++++++++++++++++++++++++++++++++++++++++++-- arch/x86/kernel/setup.c | 8 ++++ 2 files changed, 96 insertions(+), 5 deletions(-) diff --git a/arch/x86/kernel/head64.c b/arch/x86/kernel/head64.c index 88c7bae..358d7bc 100644 --- a/arch/x86/kernel/head64.c +++ b/arch/x86/kernel/head64.c @@ -47,12 +47,12 @@ static void __init reset_early_page_tables(void) } /* Create a new PMD entry */ -int __init early_make_pgtable(unsigned long address) +static int __init __early_make_pgtable(unsigned long address, pmdval_t pmd) { unsigned long physaddr = address - __PAGE_OFFSET; pgdval_t pgd, *pgd_p; pudval_t pud, *pud_p; - pmdval_t pmd, *pmd_p; + pmdval_t *pmd_p; /* Invalid address or early pgt is done ? */ if (physaddr >= MAXMEM || read_cr3() != __sme_pa_nodebug(early_level4_pgt)) @@ -94,12 +94,95 @@ again: memset(pmd_p, 0, sizeof(*pmd_p) * PTRS_PER_PMD); *pud_p = (pudval_t)pmd_p - __START_KERNEL_map + phys_base + _KERNPG_TABLE; } - pmd = (physaddr & PMD_MASK) + early_pmd_flags; pmd_p[pmd_index(address)] = pmd; return 0; } +int __init early_make_pgtable(unsigned long address) +{ + unsigned long physaddr = address - __PAGE_OFFSET; + pmdval_t pmd; + + pmd = (physaddr & PMD_MASK) + early_pmd_flags; + + return __early_make_pgtable(address, pmd); +} + +static void __init create_unencrypted_mapping(void *address, unsigned long size) +{ + unsigned long physaddr = (unsigned long)address - __PAGE_OFFSET; + pmdval_t pmd_flags, pmd; + + if (!sme_me_mask) + return; + + /* Clear the encryption mask from the early_pmd_flags */ + pmd_flags = early_pmd_flags & ~sme_me_mask; + + do { + pmd = (physaddr & PMD_MASK) + pmd_flags; + __early_make_pgtable((unsigned long)address, pmd); + + address += PMD_SIZE; + physaddr += PMD_SIZE; + size = (size < PMD_SIZE) ? 0 : size - PMD_SIZE; + } while (size); +} + +static void __init __clear_mapping(unsigned long address) +{ + unsigned long physaddr = address - __PAGE_OFFSET; + pgdval_t pgd, *pgd_p; + pudval_t pud, *pud_p; + pmdval_t *pmd_p; + + /* Invalid address or early pgt is done ? */ + if (physaddr >= MAXMEM || + read_cr3() != __sme_pa_nodebug(early_level4_pgt)) + return; + + pgd_p = &early_level4_pgt[pgd_index(address)].pgd; + pgd = *pgd_p; + + if (!pgd) + return; + + /* + * The use of __START_KERNEL_map rather than __PAGE_OFFSET here matches + * __early_make_pgtable where the entry was created. + */ + pud_p = (pudval_t *)((pgd & PTE_PFN_MASK) + __START_KERNEL_map - phys_base); + pud_p += pud_index(address); + pud = *pud_p; + + if (!pud) + return; + + pmd_p = (pmdval_t *)((pud & PTE_PFN_MASK) + __START_KERNEL_map - phys_base); + pmd_p[pmd_index(address)] = 0; +} + +static void __init clear_mapping(void *address, unsigned long size) +{ + if (!sme_me_mask) + return; + + do { + __clear_mapping((unsigned long)address); + + address += PMD_SIZE; + size = (size < PMD_SIZE) ? 0 : size - PMD_SIZE; + } while (size); +} + +static void __init sme_memcpy(void *dst, void *src, unsigned long size) +{ + create_unencrypted_mapping(src, size); + memcpy(dst, src, size); + clear_mapping(src, size); +} + /* Don't add a printk in there. printk relies on the PDA which is not initialized yet. */ static void __init clear_bss(void) @@ -122,12 +205,12 @@ static void __init copy_bootdata(char *real_mode_data) char * command_line; unsigned long cmd_line_ptr; - memcpy(&boot_params, real_mode_data, sizeof boot_params); + sme_memcpy(&boot_params, real_mode_data, sizeof boot_params); sanitize_boot_params(&boot_params); cmd_line_ptr = get_cmd_line_ptr(); if (cmd_line_ptr) { command_line = __va(cmd_line_ptr); - memcpy(boot_command_line, command_line, COMMAND_LINE_SIZE); + sme_memcpy(boot_command_line, command_line, COMMAND_LINE_SIZE); } } diff --git a/arch/x86/kernel/setup.c b/arch/x86/kernel/setup.c index 1489da8..1fdaa11 100644 --- a/arch/x86/kernel/setup.c +++ b/arch/x86/kernel/setup.c @@ -114,6 +114,7 @@ #include #include #include +#include /* * max_low_pfn_mapped: highest direct mapped pfn under 4GB @@ -376,6 +377,13 @@ static void __init reserve_initrd(void) !ramdisk_image || !ramdisk_size) return; /* No initrd provided by bootloader */ + /* + * This memory is marked encrypted by the kernel but the ramdisk + * was loaded in the clear by the bootloader, so make sure that + * the ramdisk image is encrypted. + */ + sme_early_mem_enc(ramdisk_image, ramdisk_end - ramdisk_image); + initrd_start = 0; mapped_size = memblock_mem_size(max_pfn_mapped); From mboxrd@z Thu Jan 1 00:00:00 1970 From: Tom Lendacky Subject: [RFC PATCH v2 10/20] x86: Insure that memory areas are encrypted when possible Date: Mon, 22 Aug 2016 17:37:23 -0500 Message-ID: <20160822223722.29880.94331.stgit@tlendack-t1.amdoffice.net> References: <20160822223529.29880.50884.stgit@tlendack-t1.amdoffice.net> Mime-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: 7bit Return-path: In-Reply-To: <20160822223529.29880.50884.stgit@tlendack-t1.amdoffice.net> Sender: owner-linux-mm@kvack.org To: linux-arch@vger.kernel.org, linux-efi@vger.kernel.org, kvm@vger.kernel.org, linux-doc@vger.kernel.org, x86@kernel.org, linux-kernel@vger.kernel.org, kasan-dev@googlegroups.com, linux-mm@kvack.org, iommu@lists.linux-foundation.org Cc: Radim =?utf-8?b?S3LEjW3DocWZ?= , Arnd Bergmann , Jonathan Corbet , Matt Fleming , Joerg Roedel , Konrad Rzeszutek Wilk , Andrey Ryabinin , Ingo Molnar , Borislav Petkov , Andy Lutomirski , "H. Peter Anvin" , Paolo Bonzini , Alexander Potapenko , Thomas Gleixner , Dmitry Vyukov List-Id: linux-efi@vger.kernel.org Encrypt memory areas in place when possible (e.g. zero page, etc.) so that special handling isn't needed afterwards. Signed-off-by: Tom Lendacky --- arch/x86/kernel/head64.c | 93 ++++++++++++++++++++++++++++++++++++++++++++-- arch/x86/kernel/setup.c | 8 ++++ 2 files changed, 96 insertions(+), 5 deletions(-) diff --git a/arch/x86/kernel/head64.c b/arch/x86/kernel/head64.c index 88c7bae..358d7bc 100644 --- a/arch/x86/kernel/head64.c +++ b/arch/x86/kernel/head64.c @@ -47,12 +47,12 @@ static void __init reset_early_page_tables(void) } /* Create a new PMD entry */ -int __init early_make_pgtable(unsigned long address) +static int __init __early_make_pgtable(unsigned long address, pmdval_t pmd) { unsigned long physaddr = address - __PAGE_OFFSET; pgdval_t pgd, *pgd_p; pudval_t pud, *pud_p; - pmdval_t pmd, *pmd_p; + pmdval_t *pmd_p; /* Invalid address or early pgt is done ? */ if (physaddr >= MAXMEM || read_cr3() != __sme_pa_nodebug(early_level4_pgt)) @@ -94,12 +94,95 @@ again: memset(pmd_p, 0, sizeof(*pmd_p) * PTRS_PER_PMD); *pud_p = (pudval_t)pmd_p - __START_KERNEL_map + phys_base + _KERNPG_TABLE; } - pmd = (physaddr & PMD_MASK) + early_pmd_flags; pmd_p[pmd_index(address)] = pmd; return 0; } +int __init early_make_pgtable(unsigned long address) +{ + unsigned long physaddr = address - __PAGE_OFFSET; + pmdval_t pmd; + + pmd = (physaddr & PMD_MASK) + early_pmd_flags; + + return __early_make_pgtable(address, pmd); +} + +static void __init create_unencrypted_mapping(void *address, unsigned long size) +{ + unsigned long physaddr = (unsigned long)address - __PAGE_OFFSET; + pmdval_t pmd_flags, pmd; + + if (!sme_me_mask) + return; + + /* Clear the encryption mask from the early_pmd_flags */ + pmd_flags = early_pmd_flags & ~sme_me_mask; + + do { + pmd = (physaddr & PMD_MASK) + pmd_flags; + __early_make_pgtable((unsigned long)address, pmd); + + address += PMD_SIZE; + physaddr += PMD_SIZE; + size = (size < PMD_SIZE) ? 0 : size - PMD_SIZE; + } while (size); +} + +static void __init __clear_mapping(unsigned long address) +{ + unsigned long physaddr = address - __PAGE_OFFSET; + pgdval_t pgd, *pgd_p; + pudval_t pud, *pud_p; + pmdval_t *pmd_p; + + /* Invalid address or early pgt is done ? */ + if (physaddr >= MAXMEM || + read_cr3() != __sme_pa_nodebug(early_level4_pgt)) + return; + + pgd_p = &early_level4_pgt[pgd_index(address)].pgd; + pgd = *pgd_p; + + if (!pgd) + return; + + /* + * The use of __START_KERNEL_map rather than __PAGE_OFFSET here matches + * __early_make_pgtable where the entry was created. + */ + pud_p = (pudval_t *)((pgd & PTE_PFN_MASK) + __START_KERNEL_map - phys_base); + pud_p += pud_index(address); + pud = *pud_p; + + if (!pud) + return; + + pmd_p = (pmdval_t *)((pud & PTE_PFN_MASK) + __START_KERNEL_map - phys_base); + pmd_p[pmd_index(address)] = 0; +} + +static void __init clear_mapping(void *address, unsigned long size) +{ + if (!sme_me_mask) + return; + + do { + __clear_mapping((unsigned long)address); + + address += PMD_SIZE; + size = (size < PMD_SIZE) ? 0 : size - PMD_SIZE; + } while (size); +} + +static void __init sme_memcpy(void *dst, void *src, unsigned long size) +{ + create_unencrypted_mapping(src, size); + memcpy(dst, src, size); + clear_mapping(src, size); +} + /* Don't add a printk in there. printk relies on the PDA which is not initialized yet. */ static void __init clear_bss(void) @@ -122,12 +205,12 @@ static void __init copy_bootdata(char *real_mode_data) char * command_line; unsigned long cmd_line_ptr; - memcpy(&boot_params, real_mode_data, sizeof boot_params); + sme_memcpy(&boot_params, real_mode_data, sizeof boot_params); sanitize_boot_params(&boot_params); cmd_line_ptr = get_cmd_line_ptr(); if (cmd_line_ptr) { command_line = __va(cmd_line_ptr); - memcpy(boot_command_line, command_line, COMMAND_LINE_SIZE); + sme_memcpy(boot_command_line, command_line, COMMAND_LINE_SIZE); } } diff --git a/arch/x86/kernel/setup.c b/arch/x86/kernel/setup.c index 1489da8..1fdaa11 100644 --- a/arch/x86/kernel/setup.c +++ b/arch/x86/kernel/setup.c @@ -114,6 +114,7 @@ #include #include #include +#include /* * max_low_pfn_mapped: highest direct mapped pfn under 4GB @@ -376,6 +377,13 @@ static void __init reserve_initrd(void) !ramdisk_image || !ramdisk_size) return; /* No initrd provided by bootloader */ + /* + * This memory is marked encrypted by the kernel but the ramdisk + * was loaded in the clear by the bootloader, so make sure that + * the ramdisk image is encrypted. + */ + sme_early_mem_enc(ramdisk_image, ramdisk_end - ramdisk_image); + initrd_start = 0; mapped_size = memblock_mem_size(max_pfn_mapped); -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: email@kvack.org From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-by2nam03on0077.outbound.protection.outlook.com ([104.47.42.77]:49152 "EHLO NAM03-BY2-obe.outbound.protection.outlook.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1752564AbcHVWxJ (ORCPT ); Mon, 22 Aug 2016 18:53:09 -0400 From: Tom Lendacky Subject: [RFC PATCH v2 10/20] x86: Insure that memory areas are encrypted when possible Date: Mon, 22 Aug 2016 17:37:23 -0500 Message-ID: <20160822223722.29880.94331.stgit@tlendack-t1.amdoffice.net> In-Reply-To: <20160822223529.29880.50884.stgit@tlendack-t1.amdoffice.net> References: <20160822223529.29880.50884.stgit@tlendack-t1.amdoffice.net> MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: 7bit Sender: linux-arch-owner@vger.kernel.org List-ID: To: linux-arch@vger.kernel.org, linux-efi@vger.kernel.org, kvm@vger.kernel.org, linux-doc@vger.kernel.org, x86@kernel.org, linux-kernel@vger.kernel.org, kasan-dev@googlegroups.com, linux-mm@kvack.org, iommu@lists.linux-foundation.org Cc: Radim =?utf-8?b?S3LEjW3DocWZ?= , Arnd Bergmann , Jonathan Corbet , Matt Fleming , Joerg Roedel , Konrad Rzeszutek Wilk , Andrey Ryabinin , Ingo Molnar , Borislav Petkov , Andy Lutomirski , "H. Peter Anvin" , Paolo Bonzini , Alexander Potapenko , Thomas Gleixner , Dmitry Vyukov Message-ID: <20160822223723.8j6z2S4ZHCLvpMy4g4p3SrPVsxlDFiLkrdHuFdadG8Y@z> Encrypt memory areas in place when possible (e.g. zero page, etc.) so that special handling isn't needed afterwards. Signed-off-by: Tom Lendacky --- arch/x86/kernel/head64.c | 93 ++++++++++++++++++++++++++++++++++++++++++++-- arch/x86/kernel/setup.c | 8 ++++ 2 files changed, 96 insertions(+), 5 deletions(-) diff --git a/arch/x86/kernel/head64.c b/arch/x86/kernel/head64.c index 88c7bae..358d7bc 100644 --- a/arch/x86/kernel/head64.c +++ b/arch/x86/kernel/head64.c @@ -47,12 +47,12 @@ static void __init reset_early_page_tables(void) } /* Create a new PMD entry */ -int __init early_make_pgtable(unsigned long address) +static int __init __early_make_pgtable(unsigned long address, pmdval_t pmd) { unsigned long physaddr = address - __PAGE_OFFSET; pgdval_t pgd, *pgd_p; pudval_t pud, *pud_p; - pmdval_t pmd, *pmd_p; + pmdval_t *pmd_p; /* Invalid address or early pgt is done ? */ if (physaddr >= MAXMEM || read_cr3() != __sme_pa_nodebug(early_level4_pgt)) @@ -94,12 +94,95 @@ again: memset(pmd_p, 0, sizeof(*pmd_p) * PTRS_PER_PMD); *pud_p = (pudval_t)pmd_p - __START_KERNEL_map + phys_base + _KERNPG_TABLE; } - pmd = (physaddr & PMD_MASK) + early_pmd_flags; pmd_p[pmd_index(address)] = pmd; return 0; } +int __init early_make_pgtable(unsigned long address) +{ + unsigned long physaddr = address - __PAGE_OFFSET; + pmdval_t pmd; + + pmd = (physaddr & PMD_MASK) + early_pmd_flags; + + return __early_make_pgtable(address, pmd); +} + +static void __init create_unencrypted_mapping(void *address, unsigned long size) +{ + unsigned long physaddr = (unsigned long)address - __PAGE_OFFSET; + pmdval_t pmd_flags, pmd; + + if (!sme_me_mask) + return; + + /* Clear the encryption mask from the early_pmd_flags */ + pmd_flags = early_pmd_flags & ~sme_me_mask; + + do { + pmd = (physaddr & PMD_MASK) + pmd_flags; + __early_make_pgtable((unsigned long)address, pmd); + + address += PMD_SIZE; + physaddr += PMD_SIZE; + size = (size < PMD_SIZE) ? 0 : size - PMD_SIZE; + } while (size); +} + +static void __init __clear_mapping(unsigned long address) +{ + unsigned long physaddr = address - __PAGE_OFFSET; + pgdval_t pgd, *pgd_p; + pudval_t pud, *pud_p; + pmdval_t *pmd_p; + + /* Invalid address or early pgt is done ? */ + if (physaddr >= MAXMEM || + read_cr3() != __sme_pa_nodebug(early_level4_pgt)) + return; + + pgd_p = &early_level4_pgt[pgd_index(address)].pgd; + pgd = *pgd_p; + + if (!pgd) + return; + + /* + * The use of __START_KERNEL_map rather than __PAGE_OFFSET here matches + * __early_make_pgtable where the entry was created. + */ + pud_p = (pudval_t *)((pgd & PTE_PFN_MASK) + __START_KERNEL_map - phys_base); + pud_p += pud_index(address); + pud = *pud_p; + + if (!pud) + return; + + pmd_p = (pmdval_t *)((pud & PTE_PFN_MASK) + __START_KERNEL_map - phys_base); + pmd_p[pmd_index(address)] = 0; +} + +static void __init clear_mapping(void *address, unsigned long size) +{ + if (!sme_me_mask) + return; + + do { + __clear_mapping((unsigned long)address); + + address += PMD_SIZE; + size = (size < PMD_SIZE) ? 0 : size - PMD_SIZE; + } while (size); +} + +static void __init sme_memcpy(void *dst, void *src, unsigned long size) +{ + create_unencrypted_mapping(src, size); + memcpy(dst, src, size); + clear_mapping(src, size); +} + /* Don't add a printk in there. printk relies on the PDA which is not initialized yet. */ static void __init clear_bss(void) @@ -122,12 +205,12 @@ static void __init copy_bootdata(char *real_mode_data) char * command_line; unsigned long cmd_line_ptr; - memcpy(&boot_params, real_mode_data, sizeof boot_params); + sme_memcpy(&boot_params, real_mode_data, sizeof boot_params); sanitize_boot_params(&boot_params); cmd_line_ptr = get_cmd_line_ptr(); if (cmd_line_ptr) { command_line = __va(cmd_line_ptr); - memcpy(boot_command_line, command_line, COMMAND_LINE_SIZE); + sme_memcpy(boot_command_line, command_line, COMMAND_LINE_SIZE); } } diff --git a/arch/x86/kernel/setup.c b/arch/x86/kernel/setup.c index 1489da8..1fdaa11 100644 --- a/arch/x86/kernel/setup.c +++ b/arch/x86/kernel/setup.c @@ -114,6 +114,7 @@ #include #include #include +#include /* * max_low_pfn_mapped: highest direct mapped pfn under 4GB @@ -376,6 +377,13 @@ static void __init reserve_initrd(void) !ramdisk_image || !ramdisk_size) return; /* No initrd provided by bootloader */ + /* + * This memory is marked encrypted by the kernel but the ramdisk + * was loaded in the clear by the bootloader, so make sure that + * the ramdisk image is encrypted. + */ + sme_early_mem_enc(ramdisk_image, ramdisk_end - ramdisk_image); + initrd_start = 0; mapped_size = memblock_mem_size(max_pfn_mapped); From mboxrd@z Thu Jan 1 00:00:00 1970 From: Tom Lendacky Subject: [RFC PATCH v2 10/20] x86: Insure that memory areas are encrypted when possible Date: Mon, 22 Aug 2016 17:37:23 -0500 Message-ID: <20160822223722.29880.94331.stgit@tlendack-t1.amdoffice.net> References: <20160822223529.29880.50884.stgit@tlendack-t1.amdoffice.net> Mime-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: 7bit Cc: Radim =?utf-8?b?S3LEjW3DocWZ?= , Arnd Bergmann , Jonathan Corbet , Matt Fleming , Joerg Roedel , "Konrad Rzeszutek Wilk" , Andrey Ryabinin , Ingo Molnar , Borislav Petkov , "Andy Lutomirski" , "H. Peter Anvin" , Paolo Bonzini , Alexander Potapenko , "Thomas Gleixner" , Dmitry Vyukov To: , , , , , , , , Return-path: In-Reply-To: <20160822223529.29880.50884.stgit@tlendack-t1.amdoffice.net> Sender: owner-linux-mm@kvack.org List-Id: kvm.vger.kernel.org Encrypt memory areas in place when possible (e.g. zero page, etc.) so that special handling isn't needed afterwards. Signed-off-by: Tom Lendacky --- arch/x86/kernel/head64.c | 93 ++++++++++++++++++++++++++++++++++++++++++++-- arch/x86/kernel/setup.c | 8 ++++ 2 files changed, 96 insertions(+), 5 deletions(-) diff --git a/arch/x86/kernel/head64.c b/arch/x86/kernel/head64.c index 88c7bae..358d7bc 100644 --- a/arch/x86/kernel/head64.c +++ b/arch/x86/kernel/head64.c @@ -47,12 +47,12 @@ static void __init reset_early_page_tables(void) } /* Create a new PMD entry */ -int __init early_make_pgtable(unsigned long address) +static int __init __early_make_pgtable(unsigned long address, pmdval_t pmd) { unsigned long physaddr = address - __PAGE_OFFSET; pgdval_t pgd, *pgd_p; pudval_t pud, *pud_p; - pmdval_t pmd, *pmd_p; + pmdval_t *pmd_p; /* Invalid address or early pgt is done ? */ if (physaddr >= MAXMEM || read_cr3() != __sme_pa_nodebug(early_level4_pgt)) @@ -94,12 +94,95 @@ again: memset(pmd_p, 0, sizeof(*pmd_p) * PTRS_PER_PMD); *pud_p = (pudval_t)pmd_p - __START_KERNEL_map + phys_base + _KERNPG_TABLE; } - pmd = (physaddr & PMD_MASK) + early_pmd_flags; pmd_p[pmd_index(address)] = pmd; return 0; } +int __init early_make_pgtable(unsigned long address) +{ + unsigned long physaddr = address - __PAGE_OFFSET; + pmdval_t pmd; + + pmd = (physaddr & PMD_MASK) + early_pmd_flags; + + return __early_make_pgtable(address, pmd); +} + +static void __init create_unencrypted_mapping(void *address, unsigned long size) +{ + unsigned long physaddr = (unsigned long)address - __PAGE_OFFSET; + pmdval_t pmd_flags, pmd; + + if (!sme_me_mask) + return; + + /* Clear the encryption mask from the early_pmd_flags */ + pmd_flags = early_pmd_flags & ~sme_me_mask; + + do { + pmd = (physaddr & PMD_MASK) + pmd_flags; + __early_make_pgtable((unsigned long)address, pmd); + + address += PMD_SIZE; + physaddr += PMD_SIZE; + size = (size < PMD_SIZE) ? 0 : size - PMD_SIZE; + } while (size); +} + +static void __init __clear_mapping(unsigned long address) +{ + unsigned long physaddr = address - __PAGE_OFFSET; + pgdval_t pgd, *pgd_p; + pudval_t pud, *pud_p; + pmdval_t *pmd_p; + + /* Invalid address or early pgt is done ? */ + if (physaddr >= MAXMEM || + read_cr3() != __sme_pa_nodebug(early_level4_pgt)) + return; + + pgd_p = &early_level4_pgt[pgd_index(address)].pgd; + pgd = *pgd_p; + + if (!pgd) + return; + + /* + * The use of __START_KERNEL_map rather than __PAGE_OFFSET here matches + * __early_make_pgtable where the entry was created. + */ + pud_p = (pudval_t *)((pgd & PTE_PFN_MASK) + __START_KERNEL_map - phys_base); + pud_p += pud_index(address); + pud = *pud_p; + + if (!pud) + return; + + pmd_p = (pmdval_t *)((pud & PTE_PFN_MASK) + __START_KERNEL_map - phys_base); + pmd_p[pmd_index(address)] = 0; +} + +static void __init clear_mapping(void *address, unsigned long size) +{ + if (!sme_me_mask) + return; + + do { + __clear_mapping((unsigned long)address); + + address += PMD_SIZE; + size = (size < PMD_SIZE) ? 0 : size - PMD_SIZE; + } while (size); +} + +static void __init sme_memcpy(void *dst, void *src, unsigned long size) +{ + create_unencrypted_mapping(src, size); + memcpy(dst, src, size); + clear_mapping(src, size); +} + /* Don't add a printk in there. printk relies on the PDA which is not initialized yet. */ static void __init clear_bss(void) @@ -122,12 +205,12 @@ static void __init copy_bootdata(char *real_mode_data) char * command_line; unsigned long cmd_line_ptr; - memcpy(&boot_params, real_mode_data, sizeof boot_params); + sme_memcpy(&boot_params, real_mode_data, sizeof boot_params); sanitize_boot_params(&boot_params); cmd_line_ptr = get_cmd_line_ptr(); if (cmd_line_ptr) { command_line = __va(cmd_line_ptr); - memcpy(boot_command_line, command_line, COMMAND_LINE_SIZE); + sme_memcpy(boot_command_line, command_line, COMMAND_LINE_SIZE); } } diff --git a/arch/x86/kernel/setup.c b/arch/x86/kernel/setup.c index 1489da8..1fdaa11 100644 --- a/arch/x86/kernel/setup.c +++ b/arch/x86/kernel/setup.c @@ -114,6 +114,7 @@ #include #include #include +#include /* * max_low_pfn_mapped: highest direct mapped pfn under 4GB @@ -376,6 +377,13 @@ static void __init reserve_initrd(void) !ramdisk_image || !ramdisk_size) return; /* No initrd provided by bootloader */ + /* + * This memory is marked encrypted by the kernel but the ramdisk + * was loaded in the clear by the bootloader, so make sure that + * the ramdisk image is encrypted. + */ + sme_early_mem_enc(ramdisk_image, ramdisk_end - ramdisk_image); + initrd_start = 0; mapped_size = memblock_mem_size(max_pfn_mapped); -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: email@kvack.org