From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1752592AbdFGTVf (ORCPT ); Wed, 7 Jun 2017 15:21:35 -0400 Received: from mail-sn1nam01on0082.outbound.protection.outlook.com ([104.47.32.82]:24216 "EHLO NAM01-SN1-obe.outbound.protection.outlook.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1751897AbdFGTTJ (ORCPT ); Wed, 7 Jun 2017 15:19:09 -0400 Authentication-Results: vger.kernel.org; dkim=none (message not signed) header.d=none;vger.kernel.org; dmarc=none action=none header.from=amd.com; From: Tom Lendacky Subject: [PATCH v6 32/34] x86/mm: Add support to encrypt the kernel in-place To: linux-arch@vger.kernel.org, linux-efi@vger.kernel.org, kvm@vger.kernel.org, linux-doc@vger.kernel.org, x86@kernel.org, kexec@lists.infradead.org, linux-kernel@vger.kernel.org, kasan-dev@googlegroups.com, linux-mm@kvack.org, iommu@lists.linux-foundation.org Cc: Rik van Riel , Radim =?utf-8?b?S3LEjW3DocWZ?= , Toshimitsu Kani , Arnd Bergmann , Jonathan Corbet , Matt Fleming , "Michael S. Tsirkin" , Joerg Roedel , Konrad Rzeszutek Wilk , Paolo Bonzini , Larry Woodman , Brijesh Singh , Ingo Molnar , Borislav Petkov , Andy Lutomirski , "H. Peter Anvin" , Andrey Ryabinin , Alexander Potapenko , Dave Young , Thomas Gleixner , Dmitry Vyukov Date: Wed, 07 Jun 2017 14:18:54 -0500 Message-ID: <20170607191853.28645.85024.stgit@tlendack-t1.amdoffice.net> In-Reply-To: <20170607191309.28645.15241.stgit@tlendack-t1.amdoffice.net> References: <20170607191309.28645.15241.stgit@tlendack-t1.amdoffice.net> User-Agent: StGit/0.17.1-dirty MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: 7bit X-Originating-IP: [165.204.77.1] X-ClientProxiedBy: MWHPR2201CA0046.namprd22.prod.outlook.com (10.172.59.20) To CY4PR12MB1141.namprd12.prod.outlook.com (10.168.163.149) X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CY4PR12MB1141: X-MS-Office365-Filtering-Correlation-Id: 317e4d41-8642-4716-c279-08d4adda0d04 X-MS-Office365-Filtering-HT: Tenant X-Microsoft-Antispam: UriScan:;BCL:0;PCL:0;RULEID:(22001)(48565401081)(201703131423075)(201703031133081);SRVR:CY4PR12MB1141; X-Microsoft-Exchange-Diagnostics: 1;CY4PR12MB1141;3:FWCUbQ5sWZzKazkDc5kNgyk7RAOR/tUDkrkR88677mSiDEOYOR7ACZFHiUO0pTxEOW22/E62s7j3lrR05oimaVhT9BCZaASywY6aArfdSLnPi07fWv7Np6XWv9U2dTadfnDI5O0EBsuPA3G+nsWNyDSR+fqrCxzaolthLkZKKipe/TTNdWdxDxlRQmOUyWgcvtf6XG3f+lW+RDQCpdYbDRd5cuKCHp8pVwU7JWK0spFo7mku5RClSg6W/IQxS+0c6llF0W+xVtiErVA3hdB+n4U2Z53i21UYFz+WNpdoFeRvn5MyfWcAe0i0gv0pTzq+rmkXG2vihpkx1m4vOfUPFJuNEtpnVquMp96cjhpe7YA=;25:B3nr/CiUlBnI1Bg6V9mTeSvCdBoQqqgNCByDwd41FwBR/qW6St+Wjfw33XSQhMD1v58BUtbcNhG/g6pMZ5HOt+GxKWWZo0U/FjjAgVca1DFpO7KYJckfnWOlLtaQZ9pXhXirKufWVVVPc6BBvXYcUdMrq9bKyj1vWCXdBKgTADaN3hQKLbRjHWe5N/xeeljtVeCXqVFOrg3GcCReWyDE1OP67kmuiyNYYL4S8kZVqBCOY3enfIvCz/oRFLXKjoF0fmy1rAABl4K594q4ProZYSAbGEa3LNLT/GowuBCLgzAMHCeB6474SDvGRpa9TlytvgO7WUlSIZaa22bbhV2ExSdIGj91OgusWAdS0FUcLGJYdDp6vBt2Z1d7PMmv1ljA3QTRmiJmgbx/yLkJQPdAczBMABfcREUjf0bE8+MsCJsEPnTEAgW7KFz36uGB6TYiwgTHhTjqg39Uh7L21iUvJt7W9E6dg4tls5EatkaHStY= X-Microsoft-Exchange-Diagnostics: 1;CY4PR12MB1141;31:y5LJI4otGybsm65dhzuKWPW5m4RJCVvm+dZb8X+thI3PNJR3hpHa/7iK1uv+lxsvCLq7IwWRJV/hx9JKcxzVVE6hFM2K6iTBoH4CmfkOpT9QxiwWgdTz71isgrbhAvij5IhnVtXYG0hhyKEPCtUjn50SrZzSGgoT9QtLyH0A2BGc7SSdL6nP1zxKzqDQbdIer6iaZ9/dmonSRaZ9gxzTu0eGiU7n4kIs22uEYaR+fsk=;20:gz07YYBc1XMD9T/qhKXb7xN6sl0UQk13bKZItA4saUXWeapFkScr4xsZzeLIWj9rhkpVd5ITfxmeapDkH7RNYrNgpGmdUsfmX91v7U0EZ/YOkGBeJQUnk2k6TvDda1IcBwyefEpJwqkQ1O/wFyVpOUoGfB1RTy54pLroxOOjOBWWdXeE7DwCfmvVq7fMwlGXJYX//NNPquiquY4xGr/ztep97xr6r2FQwWj1Fxj1Y91c8/jDshVvFDE62iR3a9AhdngvyAkMqzIMhrnXoD4C1piau/7k2XAIVQQr4yTsD4/O+p7YHyhCsDLe7MCVVGJqK2BK+kdxBd1fiY0PELrjsfwd4iX/0GOmdzqOTEKRHCPWCawM5JhQYbUIQV2wfAa1WDAn6RmykivsuOTlrpPTv2pIOGWqvnvKfKt4GzmTvJlZb1Nv+J7sWDmoGX4fLKcJqqbPlb4NUY8EAV/udrQyyCMuWqXnLtDq7QJ8zUOCjxye8iahYeoaNbPIqptc/qkc X-Microsoft-Antispam-PRVS: X-Exchange-Antispam-Report-Test: UriScan:(767451399110); X-Exchange-Antispam-Report-CFA-Test: BCL:0;PCL:0;RULEID:(100000700101)(100105000095)(100000701101)(100105300095)(100000702101)(100105100095)(6040450)(601004)(2401047)(8121501046)(5005006)(93006095)(93001095)(100000703101)(100105400095)(10201501046)(3002001)(6055026)(6041248)(20161123555025)(20161123558100)(20161123564025)(20161123560025)(201703131423075)(201702281528075)(201703061421075)(201703061406153)(20161123562025)(6072148)(100000704101)(100105200095)(100000705101)(100105500095);SRVR:CY4PR12MB1141;BCL:0;PCL:0;RULEID:(100000800101)(100110000095)(100000801101)(100110300095)(100000802101)(100110100095)(100000803101)(100110400095)(100000804101)(100110200095)(100000805101)(100110500095);SRVR:CY4PR12MB1141; X-Microsoft-Exchange-Diagnostics: =?utf-8?B?MTtDWTRQUjEyTUIxMTQxOzQ6aXJxYTJGdDJSTEhnMXNFeFhyZ1JCR3AralNU?= =?utf-8?B?UW1QenNKMzg1MmVQV3VNS1AyeGJ2emdzakxaWHh4MGgzSE9RbVJaakVMK2NV?= =?utf-8?B?ZDFhUVgvcDRCOUllUDBwMGM1MkQ4dkx1VGNhaDN1ZFRIS3lIRmJ5c1RuK0Rh?= =?utf-8?B?NUZoUEVvN1VGR25sNllZekh5dWw0eTJLS1hGQkNXQ3d4OCtLRUFJNjJyUVh5?= =?utf-8?B?c1pUZGlNMXpPN3M4aTB6SHFBNnFXMlpGMGlRSW1mc3VUUVIxNFdqT3Avc2lI?= =?utf-8?B?Vi9rcVM3eUhOSGppVlJSR3NBOGpaMk9Xb2FVUFpqNTRPVlZ3UUFhR2dmbEo0?= =?utf-8?B?dHNWcFNqVVJrOS9FeW1Ra0ZYUWRvd1lHcWFCamNLSXFqNDgvVnFkN3hjb1c4?= =?utf-8?B?dnpJOU1Oa0dRK0ljVURZbnVaTDRjaDVpVloxOU95NEZieC9CWFNsSFFRbmdj?= =?utf-8?B?bE5WQzlxUk9oajlBOVNJRWNkc3lxR2VBY2dSRU1BTzRDaStuS3dkVm0xZ3dZ?= =?utf-8?B?dVU2T216WDBVOXptdlBIdlBPM29kTnIyQ2k4c2hvclN0WGhqUE1CK1A5b2h0?= =?utf-8?B?Wk5TQjdaREY1WnZhY2VXdndRVk5wd3ZFSGJRVGZMb0l1SGU4ZkRVUjM1NlY5?= =?utf-8?B?bDk2NnBXUlBPVjFMNmI0K3lDZmp6L0hBOWdkSEk4bHdqblg5UnVXeEd3N3NE?= =?utf-8?B?UURSaFI4ZE43ZkdOSG5pTThFdXhuV0dlcDljd1NCWThqeUNkTmVaeW5PM29j?= =?utf-8?B?TWw3RGtmQ3UyTzU0VTN5a0p2aTlEZlM0OVFYdUlKdVdQWlF3YngwK013eUxN?= =?utf-8?B?bEFvL2RnMGNSc1NTSmZhL0tvbHEwQjE0akllTUhOV0I2ZFB6ZWp5QmdNekNE?= =?utf-8?B?NGZwZmpWRFN1YTRtY0JzYndUNzZ6L0d6Skg5SklsVXlUZVhuS2ZiOHFOM1FW?= =?utf-8?B?aEU0eGY0RnYrMThtREE0c2wvdzA2OHNZdWdVM1Q5Y1ArcHB3NjA4ek9kbWs4?= =?utf-8?B?dnc5YWYyVE5FSXZUbnJuT3JJbGdodGkxM1NXUDFTNk95TUFrUFVIZ3dFdGdQ?= =?utf-8?B?RUVSaHBFNnBiNUlyNFJsSEdhN0ZZRW1kR05wKzJTRGsrejREaUFCUS9ZYUV4?= =?utf-8?B?SkNYZU95clJPNlRzMFVyQUFsekxtMmpuVGxhMGRUS0lLd3RhLzNUWER6TGF3?= =?utf-8?B?T0JZZDZyL2pGQWdqaWdadUx6eWVIeTNZZElkM0I3MGt0SEtOWVlSVG9OdDAx?= =?utf-8?B?aUkwTzB2YzVXMVBZLzg3Tm1nT28wSXNmM1dEQWJvWEZJeUdJRFY1MkVEcGdp?= =?utf-8?B?cGZqY01DNnU1OEdKTEdERnBaY3ZvVHYvUFMxaFFYR1BSdmlQWU5TaElTdmZm?= =?utf-8?B?Y21jRk1HRGR4eUxiWUkyZ2F5eEJ6bCtIWmczRVhOYm91YXN4b0R3MGhxYWtR?= =?utf-8?B?NzU2aGpoa2NRdHpBbmQzZlZ2UkN0Q2hLZVkzYkR1SDBLVWU0R3g1SU0wK3R2?= =?utf-8?B?b2ZpVks4M1pQMkNUcndoQ25kQlpTdGVaeUFkczJGeU1WZVpPQzNNczA0NlZk?= =?utf-8?B?TFhhNEo5UmVNNW9maUFMck9rUnJZZFNXblZyTUtmZ0tKMGlkK2dMYWoyVzNV?= =?utf-8?Q?T7HgNytJ+mlTwr0jeyp?= X-Forefront-PRVS: 03319F6FEF X-Forefront-Antispam-Report: SFV:NSPM;SFS:(10009020)(4630300001)(6009001)(39450400003)(39840400002)(39860400002)(39410400002)(39850400002)(39400400002)(9686003)(54906002)(55016002)(6506006)(66066001)(4326008)(47776003)(110136004)(103116003)(25786009)(38730400002)(72206003)(478600001)(575784001)(83506001)(305945005)(7736002)(86362001)(76176999)(50466002)(53936002)(54356999)(97746001)(7416002)(7406005)(23676002)(1076002)(6116002)(33646002)(3846002)(50986999)(230700001)(53416004)(42186005)(5660300001)(4001350100001)(8676002)(81166006)(2906002)(2950100002)(6666003)(189998001)(921003)(2004002)(1121003);DIR:OUT;SFP:1101;SCL:1;SRVR:CY4PR12MB1141;H:tlendack-t1.amdoffice.net;FPR:;SPF:None;MLV:sfv;LANG:en; X-Microsoft-Exchange-Diagnostics: =?utf-8?B?MTtDWTRQUjEyTUIxMTQxOzIzOjZFOXZhc0NISGtvK3ZFY0xJOUpVYmlsZ21S?= =?utf-8?B?M0JjQzZ5YWdHbzJsRjd1QWZzdjBHQ2VZNGo2SGRtR1AvM3o5TmExaEtLeTFm?= =?utf-8?B?em9zYmwyUHBpbFlPQkZXZWtnU2twZEM3eFkxbFNkNEdCb0JsRndaWjNFa2xk?= =?utf-8?B?V05jNU5kcTBlcWNFMXIrYWhoR2V0eDFWS2p4SDhJdW9iRVJGOHI3VlBJYmVz?= =?utf-8?B?UkFaMFFKRUl4Vyt6MU9aVHp2ZHlXM0ZBNHQxUTcxUnlicTBlWTcrc1ZmZkJS?= =?utf-8?B?M0ViYmxoVUhmL1R0SURSUmROaTZCbXVjNVd4R0ZtY01hc3h6cDFITWJQTGEy?= =?utf-8?B?TkNkdUNaZDZPaCs4emJoMFZpb3pWd09TQ25zb21nemYyVE5HYjVaM05TRDNw?= =?utf-8?B?alJlaCt3QWExYTNsVUgwYjExSU04QTcvVHJPTkJINyttY0dlSXVub2JiZkV3?= =?utf-8?B?QmdYeUJsUmo3SXg3a05IWXpGR0NSOGlpYWY3RlFjK2xNQU9hcU1BSDB6eXlC?= =?utf-8?B?M2I1eUpETlNCZVJQdmMxd2pPL0FVbkUyYTJ3S1hKbXJBUllNNU9wd0U3cVVj?= =?utf-8?B?TmlUUjJYd2UyOGF2bkNJbW5mQ1hJUUVZRU1Oc1NrUGlGRWsxMloxOWUvRS9l?= =?utf-8?B?bUFIYnhLTlJkaHBXLzFYcjNsdFZlai90TVZSOW42ZjR4bVhFQWlWZ3lNcmNa?= =?utf-8?B?Q1AvWnExWWovNUFPTnNoTlRRN2lMc25IY2lPMVRweWNlckhiakdJL0lEUXl2?= =?utf-8?B?d3hucEJTTGdMMFJDK0xtWDZzZ3NveExkME9ZTlV6QU9YY3lHNG1ya0NDVlRh?= =?utf-8?B?dGg5YWJNSjZPWVRRdEltWFp6emRKRWl5SG4vQm9HM1VLdXcyM0ttTVVyTy9U?= =?utf-8?B?emd5THFDTDlzQVk3NnhVQUJaOHF6SklhY2M2UHhCR0lHdUFLV1JqdXQ0T005?= =?utf-8?B?ZTkwQzR1Yk9MWlRPOGRqT3hCTS9kU1Y0VGdKUEpuTDlTZHlDUFc2WjFWQ3VK?= =?utf-8?B?RFdQV0VlOXlHejVjT0gwYWJ1THVhWGpTUWk1Q2hkTlIrWlBYdTFqck1vdUtU?= =?utf-8?B?dHBDbzZ3VkRpRGF1K0MvNDdoc1RkYnhwZmNNbm43OHdoTjFFbzdYMXN3cGFJ?= =?utf-8?B?QWJhU29tT3A3UUVPSXpiNHA2L2lIZlJKQzJsNVNKQWpWdUZvc0xud1Vma21Y?= =?utf-8?B?dFpaVmNCL0dMbDBlL0xwRkhHM3FGOUxXeHcydmZ1Q3JqRk1iVmlBYU0raC9P?= =?utf-8?B?Vm5OckZEUnRpSXNTTWVVeSt0ME5kR1dqVUx6QS8zajRjL1lxSGVydmQ1UFVi?= =?utf-8?B?dW9VWW85M0dWZk9mVGZ2Q3B0TW51SzJNRC9SbUtqcFhrRzdwbmdQSkl0UUVW?= =?utf-8?B?eHUrdzJqT1EwNHM5MysveEZQQXUyK045VnZTQkhWckVoSGdxbkFkalJieGs4?= =?utf-8?B?Y0N4UTlTcmxDU00xOXVjRStDdkkyUTRuVUlXK2hGdGl1RiszTlpCeWt0dTVx?= =?utf-8?B?T1g5aEs2c1NReDQ4VGFmWHVacFdnaHhpQU81T24xYjhJc0FLTkx1MVFKaUk4?= =?utf-8?B?eGc5Ly9yMzQzZjh3VTRUUVhUTHcrUWduRHlKL3dscmRIRTUwbUJ6cUgrbFVZ?= =?utf-8?B?T2ZCYnVrSGptZHdIT1R6b291NDhqMTR4NzllcG1qV3hkYUp5TkJ1MVppdEIv?= =?utf-8?B?b2MxbGRXKytSS25EcDVOZTJjMkNNcGpLMm9zanBwVk1iZUc5SS9zV3ZSSDZD?= =?utf-8?B?Ykdad3AxTFQ3b2pHc3dXUT09?= X-Microsoft-Exchange-Diagnostics: 1;CY4PR12MB1141;6:y7LS4sPrQv96ltbfHqnq8WRhOAjyJnhrFp9IbymxOk4jxY3A+oBE30sHyYj5zivney7p3VF0oIZi4GOmfLOf+wxmqhhg4jfosRMbd9f4N+cR0QwVu2drQtsBavWS8fKkMV9xUC/eglSMvQZeYvR786/fSVzsgOfFmRZfMN4pmxVr7X499fkrGLamntcjfe/+VKujUCszVM+B07wjeDn4Rcs0aRY8RhZztBWthKrgduiT3Iep3MGBR1Mb6DUgaGvubEYOfGiwTkUFgosemtwY8N46o/S+0BrDflaJnzciumlOXMDQ7+9wCRk6xGcLbNyzm7+cliAlR8jaZzcuQ1gUGpAphRi5wqCmm3NGb3edViMWJ3zy4nZquMVSAohD0PSOHtS1tEhoPHe47iwtVTJGP7rlHiK5KlZbhsVzZIMj8HjXt152SABZduoMBEaaB8MIAn5pY+jVNthKJzaivs5KadMcADyClNbCkW6K07xme63sim9PsS6n0HYwOmGSu6W1oJEQKKn4WFww40PhPVl6ndjSM4yTU53bdSBb8Wl1ZEY= X-Microsoft-Exchange-Diagnostics: 1;CY4PR12MB1141;5:ttkaI6UdQFNhiI0ny4VH6c70Kiqdns/DzvvvXme/j08BMvMXIxAXsPfrWvKHyygDXoW4pl32/E+nGu9C9fKkmkmjvr4gwTK0N0igxSBQDSBrW9YggcJ1Lak7qsrwEcqOf5NsUTBN8G2iS6EHC/CE06qKz8f8djLJY/A/SMMd+FW2Fx8wMm0lGn4UsEBBxihVHrnRtFh4GXdASXTPq2p/gbPIYJ9m7RoEgezAR/ZDTNenRXZRBrF9b9eQ1oD0ae/t+hSSgVNBtwqQg5NrDAPpgNPQS3XuXidxld8z9Co63Z+CUTSa72vHMi7lgpyfnt7rAzR9L6oOiGCAarTiGXN7lDVtYvY2EmEoQA1jikOqRpdipY2oSG+uAGuDtIhurCBMGlq4V+emJVbarexxKrhn9znnQh/35EoALGAMjwKnm+1WLcYMDINtsvC779lsEQgsWY/w63GsTEmtVBt12H7Z4/yM0vu2vuZlvHTcDPVFmOc4gwKJKLexYn33tDBRw7hj;24:1U3Nd+2vhxlqbxpmS8Ob3IuT07T6vefk1RWefaqxX+WMU/sbHrOVBrEVvIMuj9cFI+SZaRW+7p2Ib7LjeTHQpGAKEgPdjYSyTsOlzB8WN2g= SpamDiagnosticOutput: 1:99 SpamDiagnosticMetadata: NSPM X-Microsoft-Exchange-Diagnostics: 1;CY4PR12MB1141;7:YNKDztFWXcEYFOa3000p9FsnoAMXKSKk0ON+p382okUo/J2g45ZHIaliob9FicPd5rkq21Zh+W5jE5nYe0BHCxhsgWVjqAsbL583KcfZv65kAvEb9hkr2FMRWzfb+sCX+dKt9NijeEp/v6flyGo5y8bMAzxfWWybGkzGo977lXIOcKE8c4llN/yx6T+5BY+9y5dm4XySQkKtV2BtWDNxUEkQcyRduCoSf7s76fpnZLGXbSKH0nUI9EpqYiD2x//olj2A7YcZolYzNY0P82A16BHatk8CplEX9rAJF5kfCM+FoQJdBpo0V+ojT06aotjSNLLMvePKPIGBuKp9fXiJKg==;20:/JJTAYH3z6O4xMxqlPFkIOP+eNFEdRWgp9iL59Yk0YV5HowbjQLLJWJM7AsF/tAiP1eOsiMtcZReNLuMwsksQ5zKPVPZ2njRAokcfapwkQCEqK5W3UTA0h2YXixDmjw1NsTfExsWt8ZdZ73gE7ZgVb9kaM/YrjSAPrdPikVZdplmJdQ74+lHZK1qnpBXbE3YR/NZ8dsAQckIVgyLh+setNkXh7v3k+fTrIwN/FsKyNzL4ZBYWWXfS0dQjUWUuweU X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 07 Jun 2017 19:18:57.2880 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-Transport-CrossTenantHeadersStamped: CY4PR12MB1141 Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Add the support to encrypt the kernel in-place. This is done by creating new page mappings for the kernel - a decrypted write-protected mapping and an encrypted mapping. The kernel is encrypted by copying it through a temporary buffer. Signed-off-by: Tom Lendacky --- arch/x86/include/asm/mem_encrypt.h | 6 + arch/x86/mm/Makefile | 2 arch/x86/mm/mem_encrypt.c | 314 ++++++++++++++++++++++++++++++++++++ arch/x86/mm/mem_encrypt_boot.S | 150 +++++++++++++++++ 4 files changed, 472 insertions(+) create mode 100644 arch/x86/mm/mem_encrypt_boot.S diff --git a/arch/x86/include/asm/mem_encrypt.h b/arch/x86/include/asm/mem_encrypt.h index d86e544..e0a8edc 100644 --- a/arch/x86/include/asm/mem_encrypt.h +++ b/arch/x86/include/asm/mem_encrypt.h @@ -21,6 +21,12 @@ extern unsigned long sme_me_mask; +void sme_encrypt_execute(unsigned long encrypted_kernel_vaddr, + unsigned long decrypted_kernel_vaddr, + unsigned long kernel_len, + unsigned long encryption_wa, + unsigned long encryption_pgd); + void __init sme_early_encrypt(resource_size_t paddr, unsigned long size); void __init sme_early_decrypt(resource_size_t paddr, diff --git a/arch/x86/mm/Makefile b/arch/x86/mm/Makefile index 88ee454..47b26ea 100644 --- a/arch/x86/mm/Makefile +++ b/arch/x86/mm/Makefile @@ -38,3 +38,5 @@ obj-$(CONFIG_NUMA_EMU) += numa_emulation.o obj-$(CONFIG_X86_INTEL_MPX) += mpx.o obj-$(CONFIG_X86_INTEL_MEMORY_PROTECTION_KEYS) += pkeys.o obj-$(CONFIG_RANDOMIZE_MEMORY) += kaslr.o + +obj-$(CONFIG_AMD_MEM_ENCRYPT) += mem_encrypt_boot.o diff --git a/arch/x86/mm/mem_encrypt.c b/arch/x86/mm/mem_encrypt.c index 018b58a..6129477 100644 --- a/arch/x86/mm/mem_encrypt.c +++ b/arch/x86/mm/mem_encrypt.c @@ -24,6 +24,8 @@ #include #include #include +#include +#include /* * Since SME related variables are set early in the boot process they must @@ -246,8 +248,320 @@ void swiotlb_set_mem_attributes(void *vaddr, unsigned long size) set_memory_decrypted((unsigned long)vaddr, size >> PAGE_SHIFT); } +static void __init sme_clear_pgd(pgd_t *pgd_base, unsigned long start, + unsigned long end) +{ + unsigned long pgd_start, pgd_end, pgd_size; + pgd_t *pgd_p; + + pgd_start = start & PGDIR_MASK; + pgd_end = end & PGDIR_MASK; + + pgd_size = (((pgd_end - pgd_start) / PGDIR_SIZE) + 1); + pgd_size *= sizeof(pgd_t); + + pgd_p = pgd_base + pgd_index(start); + + memset(pgd_p, 0, pgd_size); +} + +#ifndef CONFIG_X86_5LEVEL +#define native_make_p4d(_x) (p4d_t) { .pgd = native_make_pgd(_x) } +#endif + +#define PGD_FLAGS _KERNPG_TABLE_NOENC +#define P4D_FLAGS _KERNPG_TABLE_NOENC +#define PUD_FLAGS _KERNPG_TABLE_NOENC +#define PMD_FLAGS (__PAGE_KERNEL_LARGE_EXEC & ~_PAGE_GLOBAL) + +static void __init *sme_populate_pgd(pgd_t *pgd_base, void *pgtable_area, + unsigned long vaddr, pmdval_t pmd_val) +{ + pgd_t *pgd_p; + p4d_t *p4d_p; + pud_t *pud_p; + pmd_t *pmd_p; + + pgd_p = pgd_base + pgd_index(vaddr); + if (native_pgd_val(*pgd_p)) { + if (IS_ENABLED(CONFIG_X86_5LEVEL)) + p4d_p = (p4d_t *)(native_pgd_val(*pgd_p) & ~PTE_FLAGS_MASK); + else + pud_p = (pud_t *)(native_pgd_val(*pgd_p) & ~PTE_FLAGS_MASK); + } else { + pgd_t pgd; + + if (IS_ENABLED(CONFIG_X86_5LEVEL)) { + p4d_p = pgtable_area; + memset(p4d_p, 0, sizeof(*p4d_p) * PTRS_PER_P4D); + pgtable_area += sizeof(*p4d_p) * PTRS_PER_P4D; + + pgd = native_make_pgd((pgdval_t)p4d_p + PGD_FLAGS); + } else { + pud_p = pgtable_area; + memset(pud_p, 0, sizeof(*pud_p) * PTRS_PER_PUD); + pgtable_area += sizeof(*pud_p) * PTRS_PER_PUD; + + pgd = native_make_pgd((pgdval_t)pud_p + PGD_FLAGS); + } + native_set_pgd(pgd_p, pgd); + } + + if (IS_ENABLED(CONFIG_X86_5LEVEL)) { + p4d_p += p4d_index(vaddr); + if (native_p4d_val(*p4d_p)) { + pud_p = (pud_t *)(native_p4d_val(*p4d_p) & ~PTE_FLAGS_MASK); + } else { + p4d_t p4d; + + pud_p = pgtable_area; + memset(pud_p, 0, sizeof(*pud_p) * PTRS_PER_PUD); + pgtable_area += sizeof(*pud_p) * PTRS_PER_PUD; + + p4d = native_make_p4d((p4dval_t)pud_p + P4D_FLAGS); + native_set_p4d(p4d_p, p4d); + } + } + + pud_p += pud_index(vaddr); + if (native_pud_val(*pud_p)) { + if (native_pud_val(*pud_p) & _PAGE_PSE) + goto out; + + pmd_p = (pmd_t *)(native_pud_val(*pud_p) & ~PTE_FLAGS_MASK); + } else { + pud_t pud; + + pmd_p = pgtable_area; + memset(pmd_p, 0, sizeof(*pmd_p) * PTRS_PER_PMD); + pgtable_area += sizeof(*pmd_p) * PTRS_PER_PMD; + + pud = native_make_pud((pudval_t)pmd_p + PUD_FLAGS); + native_set_pud(pud_p, pud); + } + + pmd_p += pmd_index(vaddr); + if (!native_pmd_val(*pmd_p) || !(native_pmd_val(*pmd_p) & _PAGE_PSE)) + native_set_pmd(pmd_p, native_make_pmd(pmd_val)); + +out: + return pgtable_area; +} + +static unsigned long __init sme_pgtable_calc(unsigned long len) +{ + unsigned long p4d_size, pud_size, pmd_size; + unsigned long total; + + /* + * Perform a relatively simplistic calculation of the pagetable + * entries that are needed. That mappings will be covered by 2MB + * PMD entries so we can conservatively calculate the required + * number of P4D, PUD and PMD structures needed to perform the + * mappings. Incrementing the count for each covers the case where + * the addresses cross entries. + */ + if (IS_ENABLED(CONFIG_X86_5LEVEL)) { + p4d_size = (ALIGN(len, PGDIR_SIZE) / PGDIR_SIZE) + 1; + p4d_size *= sizeof(p4d_t) * PTRS_PER_P4D; + pud_size = (ALIGN(len, P4D_SIZE) / P4D_SIZE) + 1; + pud_size *= sizeof(pud_t) * PTRS_PER_PUD; + } else { + p4d_size = 0; + pud_size = (ALIGN(len, PGDIR_SIZE) / PGDIR_SIZE) + 1; + pud_size *= sizeof(pud_t) * PTRS_PER_PUD; + } + pmd_size = (ALIGN(len, PUD_SIZE) / PUD_SIZE) + 1; + pmd_size *= sizeof(pmd_t) * PTRS_PER_PMD; + + total = p4d_size + pud_size + pmd_size; + + /* + * Now calculate the added pagetable structures needed to populate + * the new pagetables. + */ + if (IS_ENABLED(CONFIG_X86_5LEVEL)) { + p4d_size = ALIGN(total, PGDIR_SIZE) / PGDIR_SIZE; + p4d_size *= sizeof(p4d_t) * PTRS_PER_P4D; + pud_size = ALIGN(total, P4D_SIZE) / P4D_SIZE; + pud_size *= sizeof(pud_t) * PTRS_PER_PUD; + } else { + p4d_size = 0; + pud_size = ALIGN(total, PGDIR_SIZE) / PGDIR_SIZE; + pud_size *= sizeof(pud_t) * PTRS_PER_PUD; + } + pmd_size = ALIGN(total, PUD_SIZE) / PUD_SIZE; + pmd_size *= sizeof(pmd_t) * PTRS_PER_PMD; + + total += p4d_size + pud_size + pmd_size; + + return total; +} + void __init sme_encrypt_kernel(void) { + unsigned long workarea_start, workarea_end, workarea_len; + unsigned long execute_start, execute_end, execute_len; + unsigned long kernel_start, kernel_end, kernel_len; + unsigned long pgtable_area_len; + unsigned long paddr, pmd_flags; + unsigned long decrypted_base; + void *pgtable_area; + pgd_t *pgd; + + if (!sme_active()) + return; + + /* + * Prepare for encrypting the kernel by building new pagetables with + * the necessary attributes needed to encrypt the kernel in place. + * + * One range of virtual addresses will map the memory occupied + * by the kernel as encrypted. + * + * Another range of virtual addresses will map the memory occupied + * by the kernel as decrypted and write-protected. + * + * The use of write-protect attribute will prevent any of the + * memory from being cached. + */ + + /* Physical addresses gives us the identity mapped virtual addresses */ + kernel_start = __pa_symbol(_text); + kernel_end = ALIGN(__pa_symbol(_end), PMD_PAGE_SIZE); + kernel_len = kernel_end - kernel_start; + + /* Set the encryption workarea to be immediately after the kernel */ + workarea_start = kernel_end; + + /* + * Calculate required number of workarea bytes needed: + * executable encryption area size: + * stack page (PAGE_SIZE) + * encryption routine page (PAGE_SIZE) + * intermediate copy buffer (PMD_PAGE_SIZE) + * pagetable structures for the encryption of the kernel + * pagetable structures for workarea (in case not currently mapped) + */ + execute_start = workarea_start; + execute_end = execute_start + (PAGE_SIZE * 2) + PMD_PAGE_SIZE; + execute_len = execute_end - execute_start; + + /* + * One PGD for both encrypted and decrypted mappings and a set of + * PUDs and PMDs for each of the encrypted and decrypted mappings. + */ + pgtable_area_len = sizeof(pgd_t) * PTRS_PER_PGD; + pgtable_area_len += sme_pgtable_calc(execute_end - kernel_start) * 2; + + /* PUDs and PMDs needed in the current pagetables for the workarea */ + pgtable_area_len += sme_pgtable_calc(execute_len + pgtable_area_len); + + /* + * The total workarea includes the executable encryption area and + * the pagetable area. + */ + workarea_len = execute_len + pgtable_area_len; + workarea_end = workarea_start + workarea_len; + + /* + * Set the address to the start of where newly created pagetable + * structures (PGDs, PUDs and PMDs) will be allocated. New pagetable + * structures are created when the workarea is added to the current + * pagetables and when the new encrypted and decrypted kernel + * mappings are populated. + */ + pgtable_area = (void *)execute_end; + + /* + * Make sure the current pagetable structure has entries for + * addressing the workarea. + */ + pgd = (pgd_t *)native_read_cr3_pa(); + paddr = workarea_start; + while (paddr < workarea_end) { + pgtable_area = sme_populate_pgd(pgd, pgtable_area, + paddr, + paddr + PMD_FLAGS); + + paddr += PMD_PAGE_SIZE; + } + + /* Flush the TLB - no globals so cr3 is enough */ + native_write_cr3(native_read_cr3()); + + /* + * A new pagetable structure is being built to allow for the kernel + * to be encrypted. It starts with an empty PGD that will then be + * populated with new PUDs and PMDs as the encrypted and decrypted + * kernel mappings are created. + */ + pgd = pgtable_area; + memset(pgd, 0, sizeof(*pgd) * PTRS_PER_PGD); + pgtable_area += sizeof(*pgd) * PTRS_PER_PGD; + + /* Add encrypted kernel (identity) mappings */ + pmd_flags = PMD_FLAGS | _PAGE_ENC; + paddr = kernel_start; + while (paddr < kernel_end) { + pgtable_area = sme_populate_pgd(pgd, pgtable_area, + paddr, + paddr + pmd_flags); + + paddr += PMD_PAGE_SIZE; + } + + /* + * A different PGD index/entry must be used to get different + * pagetable entries for the decrypted mapping. Choose the next + * PGD index and convert it to a virtual address to be used as + * the base of the mapping. + */ + decrypted_base = (pgd_index(workarea_end) + 1) & (PTRS_PER_PGD - 1); + decrypted_base <<= PGDIR_SHIFT; + + /* Add decrypted, write-protected kernel (non-identity) mappings */ + pmd_flags = (PMD_FLAGS & ~_PAGE_CACHE_MASK) | (_PAGE_PAT | _PAGE_PWT); + paddr = kernel_start; + while (paddr < kernel_end) { + pgtable_area = sme_populate_pgd(pgd, pgtable_area, + paddr + decrypted_base, + paddr + pmd_flags); + + paddr += PMD_PAGE_SIZE; + } + + /* Add decrypted workarea mappings to both kernel mappings */ + paddr = workarea_start; + while (paddr < workarea_end) { + pgtable_area = sme_populate_pgd(pgd, pgtable_area, + paddr, + paddr + PMD_FLAGS); + + pgtable_area = sme_populate_pgd(pgd, pgtable_area, + paddr + decrypted_base, + paddr + PMD_FLAGS); + + paddr += PMD_PAGE_SIZE; + } + + /* Perform the encryption */ + sme_encrypt_execute(kernel_start, kernel_start + decrypted_base, + kernel_len, workarea_start, (unsigned long)pgd); + + /* + * At this point we are running encrypted. Remove the mappings for + * the decrypted areas - all that is needed for this is to remove + * the PGD entry/entries. + */ + sme_clear_pgd(pgd, kernel_start + decrypted_base, + kernel_end + decrypted_base); + + sme_clear_pgd(pgd, workarea_start + decrypted_base, + workarea_end + decrypted_base); + + /* Flush the TLB - no globals so cr3 is enough */ + native_write_cr3(native_read_cr3()); } unsigned long __init sme_enable(void) diff --git a/arch/x86/mm/mem_encrypt_boot.S b/arch/x86/mm/mem_encrypt_boot.S new file mode 100644 index 0000000..7720b00 --- /dev/null +++ b/arch/x86/mm/mem_encrypt_boot.S @@ -0,0 +1,150 @@ +/* + * AMD Memory Encryption Support + * + * Copyright (C) 2016 Advanced Micro Devices, Inc. + * + * Author: Tom Lendacky + * + * This program is free software; you can redistribute it and/or modify + * it under the terms of the GNU General Public License version 2 as + * published by the Free Software Foundation. + */ + +#include +#include +#include +#include +#include + + .text + .code64 +ENTRY(sme_encrypt_execute) + + /* + * Entry parameters: + * RDI - virtual address for the encrypted kernel mapping + * RSI - virtual address for the decrypted kernel mapping + * RDX - length of kernel + * RCX - virtual address of the encryption workarea, including: + * - stack page (PAGE_SIZE) + * - encryption routine page (PAGE_SIZE) + * - intermediate copy buffer (PMD_PAGE_SIZE) + * R8 - physcial address of the pagetables to use for encryption + */ + + push %rbp + push %r12 + + /* Set up a one page stack in the non-encrypted memory area */ + movq %rsp, %rbp /* Save current stack pointer */ + movq %rcx, %rax /* Workarea stack page */ + movq %rax, %rsp /* Set new stack pointer */ + addq $PAGE_SIZE, %rsp /* Stack grows from the bottom */ + addq $PAGE_SIZE, %rax /* Workarea encryption routine */ + + movq %rdi, %r10 /* Encrypted kernel */ + movq %rsi, %r11 /* Decrypted kernel */ + movq %rdx, %r12 /* Kernel length */ + + /* Copy encryption routine into the workarea */ + movq %rax, %rdi /* Workarea encryption routine */ + leaq __enc_copy(%rip), %rsi /* Encryption routine */ + movq $(.L__enc_copy_end - __enc_copy), %rcx /* Encryption routine length */ + rep movsb + + /* Setup registers for call */ + movq %r10, %rdi /* Encrypted kernel */ + movq %r11, %rsi /* Decrypted kernel */ + movq %r8, %rdx /* Pagetables used for encryption */ + movq %r12, %rcx /* Kernel length */ + movq %rax, %r8 /* Workarea encryption routine */ + addq $PAGE_SIZE, %r8 /* Workarea intermediate copy buffer */ + + call *%rax /* Call the encryption routine */ + + movq %rbp, %rsp /* Restore original stack pointer */ + + pop %r12 + pop %rbp + + ret +ENDPROC(sme_encrypt_execute) + +ENTRY(__enc_copy) +/* + * Routine used to encrypt kernel. + * This routine must be run outside of the kernel proper since + * the kernel will be encrypted during the process. So this + * routine is defined here and then copied to an area outside + * of the kernel where it will remain and run decrypted + * during execution. + * + * On entry the registers must be: + * RDI - virtual address for the encrypted kernel mapping + * RSI - virtual address for the decrypted kernel mapping + * RDX - address of the pagetables to use for encryption + * RCX - length of kernel + * R8 - intermediate copy buffer + * + * RAX - points to this routine + * + * The kernel will be encrypted by copying from the non-encrypted + * kernel space to an intermediate buffer and then copying from the + * intermediate buffer back to the encrypted kernel space. The physical + * addresses of the two kernel space mappings are the same which + * results in the kernel being encrypted "in place". + */ + /* Enable the new page tables */ + mov %rdx, %cr3 + + /* Flush any global TLBs */ + mov %cr4, %rdx + andq $~X86_CR4_PGE, %rdx + mov %rdx, %cr4 + orq $X86_CR4_PGE, %rdx + mov %rdx, %cr4 + + /* Set the PAT register PA5 entry to write-protect */ + push %rcx + movl $MSR_IA32_CR_PAT, %ecx + rdmsr + push %rdx /* Save original PAT value */ + andl $0xffff00ff, %edx /* Clear PA5 */ + orl $0x00000500, %edx /* Set PA5 to WP */ + wrmsr + pop %rdx /* RDX contains original PAT value */ + pop %rcx + + movq %rcx, %r9 /* Save kernel length */ + movq %rdi, %r10 /* Save encrypted kernel address */ + movq %rsi, %r11 /* Save decrypted kernel address */ + + wbinvd /* Invalidate any cache entries */ + + /* Copy/encrypt 2MB at a time */ +1: + movq %r11, %rsi /* Source - decrypted kernel */ + movq %r8, %rdi /* Dest - intermediate copy buffer */ + movq $PMD_PAGE_SIZE, %rcx /* 2MB length */ + rep movsb + + movq %r8, %rsi /* Source - intermediate copy buffer */ + movq %r10, %rdi /* Dest - encrypted kernel */ + movq $PMD_PAGE_SIZE, %rcx /* 2MB length */ + rep movsb + + addq $PMD_PAGE_SIZE, %r11 + addq $PMD_PAGE_SIZE, %r10 + subq $PMD_PAGE_SIZE, %r9 /* Kernel length decrement */ + jnz 1b /* Kernel length not zero? */ + + /* Restore PAT register */ + push %rdx /* Save original PAT value */ + movl $MSR_IA32_CR_PAT, %ecx + rdmsr + pop %rdx /* Restore original PAT value */ + wrmsr + + ret +.L__enc_copy_end: +ENDPROC(__enc_copy)