From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S932968AbcHVWyo (ORCPT ); Mon, 22 Aug 2016 18:54:44 -0400 Received: from mail-dm3nam03on0045.outbound.protection.outlook.com ([104.47.41.45]:39352 "EHLO NAM03-DM3-obe.outbound.protection.outlook.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S932828AbcHVWyg (ORCPT ); Mon, 22 Aug 2016 18:54:36 -0400 X-Greylist: delayed 1127 seconds by postgrey-1.27 at vger.kernel.org; Mon, 22 Aug 2016 18:54:35 EDT Authentication-Results: spf=none (sender IP is ) smtp.mailfrom=Thomas.Lendacky@amd.com; From: Tom Lendacky Subject: [RFC PATCH v2 14/20] x86: DMA support for memory encryption To: , , , , , , , , CC: Radim =?utf-8?b?S3LEjW3DocWZ?= , Arnd Bergmann , Jonathan Corbet , Matt Fleming , Joerg Roedel , "Konrad Rzeszutek Wilk" , Andrey Ryabinin , Ingo Molnar , Borislav Petkov , "Andy Lutomirski" , "H. Peter Anvin" , Paolo Bonzini , Alexander Potapenko , "Thomas Gleixner" , Dmitry Vyukov Date: Mon, 22 Aug 2016 17:38:07 -0500 Message-ID: <20160822223807.29880.69294.stgit@tlendack-t1.amdoffice.net> In-Reply-To: <20160822223529.29880.50884.stgit@tlendack-t1.amdoffice.net> References: <20160822223529.29880.50884.stgit@tlendack-t1.amdoffice.net> User-Agent: StGit/0.17.1-dirty MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: 7bit X-Originating-IP: [165.204.77.1] X-ClientProxiedBy: BN6PR17CA0011.namprd17.prod.outlook.com (10.173.147.21) To DM5PR12MB1146.namprd12.prod.outlook.com (10.168.236.141) X-MS-Office365-Filtering-Correlation-Id: ba91281e-348b-417c-3072-08d3cadd00c1 X-Microsoft-Exchange-Diagnostics: 1;DM5PR12MB1146;2:0WzW6RpLHdwh7lHyW4gj0FwRRKxMPxCd/vyibLfG82fu2QCOpvjB3H75HLQK1cyRrkkPjgSLmISZYWObAK+cgsb1P3LU2aoNzCnSUuiHEkZBh09fVJkeoC5pRL4Z0irbf35chE3KF20Y4jppBNrRO4vNNv26qfej2yUlNIYSLeWlK9yZodO7hR7MnPa2dhTr;3:8zHOAZpy+24hoAVnynXakpStW2mVtGLr4J6h4oXxxZSOa2JToZyKyI+Wzmgi2f8z2876I5RcglBcWjq/phBJVQdMJbC+SVfv62vMawrx8wwDLK8lvDywA5Ag8RixOplV;25:Q3aVihWoWZetRUa3sxZ1L5I9SXQfvDh+Ccv3ztlvUG6+EakK4p2qtCXJMIAPnrOHhgJsq+GfKMmFblYYvCEGGvwUj35nVN6IUM4gC8NUtHy1nfAovkXiy3G7g7lnwANU7lQNBA1fXmnQJ3kM/39AWbxcHoppMBlAmzfBWNXJGfoN8ZDCO2JrvTCwts0yyaZveuOhfPfpdxWAijTHDHAoWmpiuuqaMN1mErZUA3uvXUIkfvSXJsprUMsDF9zS89N5+b4RO2pINvjquBdBx/JFStWc47C/vitRZ5jABl3E4E/65SYukPpp+phuCWzTldByHUSrbSbkFqSY52NChtHkwE1JFzIF8arp5slW8aLhvaA9OTr8oB85JI95Bzgqy8JH8SMuC39gHu27hFGuAZU3fHQM4a0ENIIdOHb54LNMKhY= X-Microsoft-Antispam: UriScan:;BCL:0;PCL:0;RULEID:;SRVR:DM5PR12MB1146; X-Microsoft-Exchange-Diagnostics: 1;DM5PR12MB1146;31:iMQD3wpkKkbrMIERJQw0fGCEuwHimaiRyEAZY2G6nI9+Lr6Gd2ZLHNLEgW60EL/AH7PZ090CYleDi+Z5KNx1oUR+CVZq0xxJj/bCaYaDzQTfcOmYAahhMoq0ORJcJ1+Ja2/J32TmYuSfdrxnNAmEGbhDIueZJ+30UWFfdoJ327kRntziWsjqKEl4F4UpfiA2z9jl+ykQ62Fe6QQ3wLSPIU3O7/nCIbpmHV/LsnhXOtE=;20:dbQdTy8K0wN+uqVWeGqqARkTGBZYo4X1NGTqxSVvJ6nJNdb9/pmAwmNPpPTwq1WzoCVJ1edsM7rATzzkStIrritAFw2OCF9QkThK46b7JE8WezZgIPnXffRAonADqooEN4N9F7KeOy6x6RY6BoBgwFgBattDvYWPUohYryGc4ZEQdsBEBBFVBhjkWzTulxCx3naD5yylBeuDNzh6x6H1H7V/T4BxiTEMhsrc7f5B69l/xnBBC7oUGt7g0NyuwfBwEEhu39zbm7dcauy6oWEIJNGdank4hsv7uTewlMl7vrscLZiDsewnThOtZwYTEXN3/FWrxzGLSOVyllxpOJIqnWxFRzMVQ/FViVJrIh+N4HpAljfL841hIO+BwKNyc47ACdCDfmhH5N2N9PZ1oRB6mCFHQbchotA0FkeTtRanyVYuS5IeT6OQkJZnaFL2OThd4z8anZNDLmslzQk13oLbaGl59IQ3xagDrSClAmASj8Kx/0St8ZeH1lAR7LFvNDGs X-Microsoft-Antispam-PRVS: X-Exchange-Antispam-Report-Test: UriScan:(767451399110); X-Exchange-Antispam-Report-CFA-Test: BCL:0;PCL:0;RULEID:(6040176)(601004)(2401047)(5005006)(8121501046)(3002001)(10201501046)(6055026);SRVR:DM5PR12MB1146;BCL:0;PCL:0;RULEID:;SRVR:DM5PR12MB1146; X-Microsoft-Exchange-Diagnostics: 1;DM5PR12MB1146;4:0/drkYWIAhDe14g94J7rqHaR7MB2UTcLsAP7k40GOgoT2clrauP4L3ziqwQmMrWJPUsJxWQuI81seGwuplmyssrpLbDUH9HkJ2YOXOmyWe5JAvf4sZ7Ne4NcVC4CCqr6p8DHyuNy+9HitOU0Q8rpDDUQLtFrPBsX/MhEadUeBwvMEl4ZdnAwNsgtT27Pg32XeLwh1kMIQUYLS/ZUBxVxOYtmOTOGCP8N9bQGscWmM4KjorcSWEtU6QJldae8LPpfniNp4NcSVvS52H6UQ0PswqpI2WvRRFzeHLzrbIPBEYF6iruj2mbbi8xx6/Tlvp3MGX7CHPCsfxss2wlpediWWT32hPH1DilFLMvd2hSglGVDybPkXiPxzy8FAi2MCG0Wgms1Cylj6aSdZJZTS9fcU6tcdPOl94rnilDAddpiUtZh8cYX7K6eFsbzUhxHoW6h X-Forefront-PRVS: 00429279BA X-Forefront-Antispam-Report: SFV:NSPM;SFS:(10009020)(4630300001)(6009001)(7916002)(199003)(189002)(77096005)(229853001)(106356001)(101416001)(97736004)(103116003)(47776003)(2950100001)(5001770100001)(586003)(97746001)(92566002)(54356999)(50986999)(76176999)(8676002)(189998001)(5660300001)(33646002)(230700001)(4326007)(7416002)(4001350100001)(3846002)(81156014)(68736007)(81166006)(66066001)(50466002)(7846002)(9686002)(1076002)(6116002)(2906002)(69596002)(19580405001)(83506001)(105586002)(305945005)(575784001)(2201001)(42186005)(53416004)(86362001)(23676002)(19580395003)(7736002)(71626007)(217873001);DIR:OUT;SFP:1101;SCL:1;SRVR:DM5PR12MB1146;H:tlendack-t1.amdoffice.net;FPR:;SPF:None;PTR:InfoNoRecords;A:1;MX:1;LANG:en; X-Microsoft-Exchange-Diagnostics: =?utf-8?B?MTtETTVQUjEyTUIxMTQ2OzIzOkZ4bHVTcWhjRWZLdlMwSUY1bERUZEtzaW1H?= =?utf-8?B?V01UR3MybUk3M3JDS1ZKTnkvYXNCUU9jaFVPVDBlV25KNktHaHdPTi9Ob2lF?= =?utf-8?B?d0pqWnc4MW5jUUxjd0JZdVQ0UWlKYlZYZ3g1aEk3dnZybEpqdUtsc2dHQ3py?= =?utf-8?B?U2tWQi8zQUliQnRzL0t4M0tCdkVmci9uL3c1c2xJemk2UVpSUndsbHM0NndI?= =?utf-8?B?R3JjczlXd0tnY21FUEpvZ3dZVHUrTnBCT1h2REx2NnVUdU1DR3RrcUpDL1Fh?= =?utf-8?B?YlF1MWFJRXloUVV6RTI0KzBHYUhmM28waUdNZzFEZDhuVE80NnJNeUt6Sjg0?= =?utf-8?B?b3d6T0pWbU1oTjJST01zTzRIVEFUN0s3aHNVcHFwdEh6TDN0NzQvbkVrNnZK?= =?utf-8?B?R0VpN0RNZnMzbmVuS0drYTJLWGxabXJXNHdvbkRWNTI3N1pJU0p5M1VjSllh?= =?utf-8?B?M0hXQWFHbGxjaTRDc3ZjZ050akcyVmRHSXBIcGsxWG1uU3NZOThOS3FvcHZ1?= =?utf-8?B?cUJBb2pOaGk2OHpEVlBFbnZTa1VWcnJRNnp4VHpPcWxhK2UzWHNrWVh0V3ky?= =?utf-8?B?UEw4a1ZNR2JZamY5a3YrcHQrTm9DaDJSWjN4U3g2ejBpbEZ5RFg1V2tHaVl3?= =?utf-8?B?Rmd5U3dQQ1NPcVZLbWVzQUwxQnlrVmU4WDBVa01NcjkzQkI5U1FNMjJxZGk3?= =?utf-8?B?UnhzWC9IZHBtZmJBOVh5Mnl6eVl2aWplUDAzNEZhbHl0dFVnbzdZQzFldVc0?= =?utf-8?B?dDltd1VlZEtTamI4SzhHeEZFYU1jRStjcUdaMlZ0SVhNVnoxaVZTYmFJNk1L?= =?utf-8?B?R012eFZ6WXQ3djk4SVBzN3E1M21weTFkbEY2VlpJY3grUEszQ2RrV3FWU0xV?= =?utf-8?B?RG1rSVJGanlibzdTc2R0YkF6bHZkWDVQcVE1OHlnSUNTdlRyREZYVko0bW14?= =?utf-8?B?ckFnOTRwMUdQdXpmVEpTU3RHdjVLSjBPWmQ1L1FHNjNJZjZ3OEdZOFZwWTJx?= =?utf-8?B?bi9mMi9kSkpWZEpKUkhOVDVGKzhrZXplVy9WbnRzZGcvelV2UlE2KzJqbTFU?= =?utf-8?B?bjBrcnlubXFYNkFsZVFkNWpWVC9yeEhmTjFwOFY3aDNvVW01OVloYTBxWitu?= =?utf-8?B?MTg4R0c3ajJEN3o4S25pajg5MEEyMThvajFaenBPSFo3NVpqRmRucWRGUWRD?= =?utf-8?B?ZVd3SmZCWGF0SUs2SFRZYmhHUm1jMmJxM0ZoWmxjQXF1UzRnSzg0YmgrWHJN?= =?utf-8?B?VlF3bkVhZ1dsaEdSeXUrbGovTHpRVzhZa2dqeVVwZkRQNVZUa0Znd2RGQ1lK?= =?utf-8?B?YzdwSGtST3Z2eEp2SDduVmMzSUVvcy9rUmNHNVZTSWdkeVdMVlVKVGFTem9q?= =?utf-8?B?ZXRGY2QzUkZYN3ZuOEVSNVR5SCt2SGlWY1V2ZFpEUTE0OEh6OWNRZXo5ZU1Y?= =?utf-8?B?bEh1NGFYcW5SWG80aldOOG9GRUVkNHAvK0F3ZzB5dlU0RXVmcThZUjZ3Z3lo?= =?utf-8?B?VUVabWI2bUtJbVozeXBKVzJYeUMxWTRDaGozNm52N0o4ZFhxanlRQnI3dm5s?= =?utf-8?B?blhDSWxYTHRBT1NTdW9FQjhibERFRG5JQzJ4L1hiZGR0NTFHQktNMWVFbUxY?= =?utf-8?B?dzczV3E2OVlEOUpvMjZ3c0ZhZjh2dGl4cFFZUlBvOVhwNWU5QU5Sa2tQY3cy?= =?utf-8?B?UzhnRXdNUXJxOGZwRFdpZ0w2emZxS3pjeE9mdDY1bEhEZFdONzB1Y1hlemlI?= =?utf-8?B?SjJ3bWw2Q0xaOC92T0RNWmg0Skh2bjJ3RG1jTjUvUG1YcWFhNGpQWkhuQVQv?= =?utf-8?Q?5oCsSfQg9yUHp?= X-Microsoft-Exchange-Diagnostics: 1;DM5PR12MB1146;6:QJWxtz2a//FnNGgfGyOBpFm7ALwpfBJJmt/c69FDGZUeksLcydefTekYieBq0d/cOLXCn6sM34hm06f88cVLlPUTYYC5/4W27RsMRsg4U1/f1zNsB0Tw4/x5z28zTJK+X881uHt1Yy8uiU80ZDfw2hFyH0kjCGXUbvzjL0rBoPJpIXgXU8HVtdcQKO6vJTngfAnTtap0BE6KQ9i3CAmeDmj37KZCQQs/AIYqlYzmZD4N9Eg5UF6+gz/HvrBEFLYfIGXAi/BOzd3B1VOGtT8kU7Vd/9n39kQ2zUMmyGibHdwVy12QUleKrDouRNrjrJjuWP7LAF/Kfc4Gt++XBGh2MA==;5:STip7Vy+lEcQm+TLaVbiTu7a40n2DkbqPWMyKBaC8fCxxBdpwz5QftPP7rvcjzxmFMkrAutvJEpyKRZMkKlPAA66Ejr5Fp31SFO+8g9RQpDT6vz8TLDIKxS0I8nf4NrE0Mmsi7fChhlZsrLO0TeRbQ==;24:h2tBxx2Ab3p2v0eRNMKQJl/1oYSgqv314ld4CqGj54Q17YoR+v8vBZ+Crn+9vIB/1R6kyDi5U40P506+Y/F55yGwwu+U7DY7LYTD0D8Tw2M=;7:9BCa1Jc9zaMCDlqE9cU8utwa29JH+n3MgYBVHFtDcqmGHhWVUTrLvgHzvEXm0VDccLFLD2jQm4cpqDVCyfDs8nlTdpsuxLXAH8CrdlS1F5QJTAjnCUGdZw/TbTwLrYT5/LaIhfBtU2fTXdKgajYRpT5gnBOBlkinn1u582keWtPpHh17bSnEbfS7DsTHYeoTPIGJziOFttH3v+VfTiZfNWdzaCt0OpAtbr7fn4C4gkQGRx1+KX3exmPbVWp0DTpV SpamDiagnosticOutput: 1:99 SpamDiagnosticMetadata: NSPM X-Microsoft-Exchange-Diagnostics: 1;DM5PR12MB1146;20:SmXAj6I6pB9nFloUp2ZxQAKqDpy45m559QOcKcuvnh3xPpDQFZZgQwgEXaJm4/wdnlFNMFpBIlwBT7EYqzGHruJm2SGui6GYIChLkdWdKULl9mPuW9eSRNhvPiDOIWvAsSqel0Zj7hQ07tdCmeyVY8T2RlTK+2ctLIOHVmaALcGRERM0BU5Iozj5RX0BYi5ENXjRDMVGNDRoW0ACYCHGxyZztvDVhlgJIGY2ZSykkLKpBQp2BnFqfkS1RtGbL6pM X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 22 Aug 2016 22:38:11.6897 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM5PR12MB1146 Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Since DMA addresses will effectively look like 48-bit addresses when the memory encryption mask is set, SWIOTLB is needed if the DMA mask of the device performing the DMA does not support 48-bits. SWIOTLB will be initialized to create un-encrypted bounce buffers for use by these devices. Signed-off-by: Tom Lendacky --- arch/x86/include/asm/dma-mapping.h | 5 ++- arch/x86/include/asm/mem_encrypt.h | 6 +++ arch/x86/kernel/pci-dma.c | 11 ++++-- arch/x86/kernel/pci-nommu.c | 2 + arch/x86/kernel/pci-swiotlb.c | 8 +++-- arch/x86/mm/mem_encrypt.c | 22 ++++++++++++ include/linux/swiotlb.h | 1 + init/main.c | 13 +++++++ lib/swiotlb.c | 64 ++++++++++++++++++++++++++++++++---- 9 files changed, 115 insertions(+), 17 deletions(-) diff --git a/arch/x86/include/asm/dma-mapping.h b/arch/x86/include/asm/dma-mapping.h index 4446162..c9cdcae 100644 --- a/arch/x86/include/asm/dma-mapping.h +++ b/arch/x86/include/asm/dma-mapping.h @@ -12,6 +12,7 @@ #include #include #include +#include #ifdef CONFIG_ISA # define ISA_DMA_BIT_MASK DMA_BIT_MASK(24) @@ -69,12 +70,12 @@ static inline bool dma_capable(struct device *dev, dma_addr_t addr, size_t size) static inline dma_addr_t phys_to_dma(struct device *dev, phys_addr_t paddr) { - return paddr; + return paddr | sme_me_mask; } static inline phys_addr_t dma_to_phys(struct device *dev, dma_addr_t daddr) { - return daddr; + return daddr & ~sme_me_mask; } #endif /* CONFIG_X86_DMA_REMAP */ diff --git a/arch/x86/include/asm/mem_encrypt.h b/arch/x86/include/asm/mem_encrypt.h index 5616ed1..384fdfb 100644 --- a/arch/x86/include/asm/mem_encrypt.h +++ b/arch/x86/include/asm/mem_encrypt.h @@ -33,6 +33,12 @@ void __init sme_early_mem_dec(resource_size_t paddr, void __init sme_early_init(void); +/* Architecture __weak replacement functions */ +void __init mem_encrypt_init(void); + +unsigned long swiotlb_get_me_mask(void); +void swiotlb_set_mem_dec(void *vaddr, unsigned long size); + #define __sme_pa(x) (__pa((x)) | sme_me_mask) #define __sme_pa_nodebug(x) (__pa_nodebug((x)) | sme_me_mask) diff --git a/arch/x86/kernel/pci-dma.c b/arch/x86/kernel/pci-dma.c index d30c377..0ce28df 100644 --- a/arch/x86/kernel/pci-dma.c +++ b/arch/x86/kernel/pci-dma.c @@ -92,9 +92,12 @@ again: /* CMA can be used only in the context which permits sleeping */ if (gfpflags_allow_blocking(flag)) { page = dma_alloc_from_contiguous(dev, count, get_order(size)); - if (page && page_to_phys(page) + size > dma_mask) { - dma_release_from_contiguous(dev, page, count); - page = NULL; + if (page) { + addr = phys_to_dma(dev, page_to_phys(page)); + if (addr + size > dma_mask) { + dma_release_from_contiguous(dev, page, count); + page = NULL; + } } } /* fallback */ @@ -103,7 +106,7 @@ again: if (!page) return NULL; - addr = page_to_phys(page); + addr = phys_to_dma(dev, page_to_phys(page)); if (addr + size > dma_mask) { __free_pages(page, get_order(size)); diff --git a/arch/x86/kernel/pci-nommu.c b/arch/x86/kernel/pci-nommu.c index 00e71ce..922c10d 100644 --- a/arch/x86/kernel/pci-nommu.c +++ b/arch/x86/kernel/pci-nommu.c @@ -30,7 +30,7 @@ static dma_addr_t nommu_map_page(struct device *dev, struct page *page, enum dma_data_direction dir, unsigned long attrs) { - dma_addr_t bus = page_to_phys(page) + offset; + dma_addr_t bus = phys_to_dma(dev, page_to_phys(page)) + offset; WARN_ON(size == 0); if (!check_addr("map_single", dev, bus, size)) return DMA_ERROR_CODE; diff --git a/arch/x86/kernel/pci-swiotlb.c b/arch/x86/kernel/pci-swiotlb.c index b47edb8..34a9e524 100644 --- a/arch/x86/kernel/pci-swiotlb.c +++ b/arch/x86/kernel/pci-swiotlb.c @@ -12,6 +12,8 @@ #include #include #include +#include + int swiotlb __read_mostly; void *x86_swiotlb_alloc_coherent(struct device *hwdev, size_t size, @@ -64,13 +66,15 @@ static struct dma_map_ops swiotlb_dma_ops = { * pci_swiotlb_detect_override - set swiotlb to 1 if necessary * * This returns non-zero if we are forced to use swiotlb (by the boot - * option). + * option). If memory encryption is enabled then swiotlb will be set + * to 1 so that bounce buffers are allocated and used for devices that + * do not support the addressing range required for the encryption mask. */ int __init pci_swiotlb_detect_override(void) { int use_swiotlb = swiotlb | swiotlb_force; - if (swiotlb_force) + if (swiotlb_force || sme_me_mask) swiotlb = 1; return use_swiotlb; diff --git a/arch/x86/mm/mem_encrypt.c b/arch/x86/mm/mem_encrypt.c index b0f39c5..6b2e8bf 100644 --- a/arch/x86/mm/mem_encrypt.c +++ b/arch/x86/mm/mem_encrypt.c @@ -12,6 +12,8 @@ #include #include +#include +#include #include #include @@ -172,3 +174,23 @@ void __init sme_early_init(void) for (i = 0; i < ARRAY_SIZE(protection_map); i++) protection_map[i] = __pgprot(pgprot_val(protection_map[i]) | sme_me_mask); } + +/* Architecture __weak replacement functions */ +void __init mem_encrypt_init(void) +{ + if (!sme_me_mask) + return; + + /* Make SWIOTLB use an unencrypted DMA area */ + swiotlb_clear_encryption(); +} + +unsigned long swiotlb_get_me_mask(void) +{ + return sme_me_mask; +} + +void swiotlb_set_mem_dec(void *vaddr, unsigned long size) +{ + sme_set_mem_dec(vaddr, size); +} diff --git a/include/linux/swiotlb.h b/include/linux/swiotlb.h index 5f81f8a..5c909fc 100644 --- a/include/linux/swiotlb.h +++ b/include/linux/swiotlb.h @@ -29,6 +29,7 @@ int swiotlb_init_with_tbl(char *tlb, unsigned long nslabs, int verbose); extern unsigned long swiotlb_nr_tbl(void); unsigned long swiotlb_size_or_default(void); extern int swiotlb_late_init_with_tbl(char *tlb, unsigned long nslabs); +extern void __init swiotlb_clear_encryption(void); /* * Enumeration for sync targets diff --git a/init/main.c b/init/main.c index a8a58e2..82c7cd9 100644 --- a/init/main.c +++ b/init/main.c @@ -458,6 +458,10 @@ void __init __weak thread_stack_cache_init(void) } #endif +void __init __weak mem_encrypt_init(void) +{ +} + /* * Set up kernel memory allocators */ @@ -598,6 +602,15 @@ asmlinkage __visible void __init start_kernel(void) */ locking_selftest(); + /* + * This needs to be called before any devices perform DMA + * operations that might use the swiotlb bounce buffers. + * This call will mark the bounce buffers as un-encrypted so + * that the usage of them will not cause "plain-text" data + * to be decrypted when accessed. + */ + mem_encrypt_init(); + #ifdef CONFIG_BLK_DEV_INITRD if (initrd_start && !initrd_below_start_ok && page_to_pfn(virt_to_page((void *)initrd_start)) < min_low_pfn) { diff --git a/lib/swiotlb.c b/lib/swiotlb.c index 22e13a0..15d5741 100644 --- a/lib/swiotlb.c +++ b/lib/swiotlb.c @@ -131,6 +131,26 @@ unsigned long swiotlb_size_or_default(void) return size ? size : (IO_TLB_DEFAULT_SIZE); } +/* + * Support for memory encryption. If memory encryption is supported, then an + * override to these functions will be provided. + */ +unsigned long __weak swiotlb_get_me_mask(void) +{ + return 0; +} + +void __weak swiotlb_set_mem_dec(void *vaddr, unsigned long size) +{ +} + +/* For swiotlb, clear memory encryption mask from dma addresses */ +static dma_addr_t swiotlb_phys_to_dma(struct device *hwdev, + phys_addr_t address) +{ + return phys_to_dma(hwdev, address) & ~swiotlb_get_me_mask(); +} + /* Note that this doesn't work with highmem page */ static dma_addr_t swiotlb_virt_to_bus(struct device *hwdev, volatile void *address) @@ -159,6 +179,30 @@ void swiotlb_print_info(void) bytes >> 20, vstart, vend - 1); } +/* + * If memory encryption is active, the DMA address for an encrypted page may + * be beyond the range of the device. If bounce buffers are required be sure + * that they are not on an encrypted page. This should be called before the + * iotlb area is used. + */ +void __init swiotlb_clear_encryption(void) +{ + void *vaddr; + unsigned long bytes; + + if (no_iotlb_memory || !io_tlb_start || late_alloc) + return; + + vaddr = phys_to_virt(io_tlb_start); + bytes = PAGE_ALIGN(io_tlb_nslabs << IO_TLB_SHIFT); + swiotlb_set_mem_dec(vaddr, bytes); + memset(vaddr, 0, bytes); + + vaddr = phys_to_virt(io_tlb_overflow_buffer); + bytes = PAGE_ALIGN(io_tlb_overflow); + swiotlb_set_mem_dec(vaddr, bytes); +} + int __init swiotlb_init_with_tbl(char *tlb, unsigned long nslabs, int verbose) { void *v_overflow_buffer; @@ -294,6 +338,8 @@ swiotlb_late_init_with_tbl(char *tlb, unsigned long nslabs) io_tlb_start = virt_to_phys(tlb); io_tlb_end = io_tlb_start + bytes; + /* Keep TLB in unencrypted memory if memory encryption is active */ + swiotlb_set_mem_dec(tlb, bytes); memset(tlb, 0, bytes); /* @@ -304,6 +350,8 @@ swiotlb_late_init_with_tbl(char *tlb, unsigned long nslabs) if (!v_overflow_buffer) goto cleanup2; + /* Keep overflow in unencrypted memory if memory encryption is active */ + swiotlb_set_mem_dec(v_overflow_buffer, io_tlb_overflow); io_tlb_overflow_buffer = virt_to_phys(v_overflow_buffer); /* @@ -541,7 +589,7 @@ static phys_addr_t map_single(struct device *hwdev, phys_addr_t phys, size_t size, enum dma_data_direction dir) { - dma_addr_t start_dma_addr = phys_to_dma(hwdev, io_tlb_start); + dma_addr_t start_dma_addr = swiotlb_phys_to_dma(hwdev, io_tlb_start); return swiotlb_tbl_map_single(hwdev, start_dma_addr, phys, size, dir); } @@ -659,7 +707,7 @@ swiotlb_alloc_coherent(struct device *hwdev, size_t size, goto err_warn; ret = phys_to_virt(paddr); - dev_addr = phys_to_dma(hwdev, paddr); + dev_addr = swiotlb_phys_to_dma(hwdev, paddr); /* Confirm address can be DMA'd by device */ if (dev_addr + size - 1 > dma_mask) { @@ -758,15 +806,15 @@ dma_addr_t swiotlb_map_page(struct device *dev, struct page *page, map = map_single(dev, phys, size, dir); if (map == SWIOTLB_MAP_ERROR) { swiotlb_full(dev, size, dir, 1); - return phys_to_dma(dev, io_tlb_overflow_buffer); + return swiotlb_phys_to_dma(dev, io_tlb_overflow_buffer); } - dev_addr = phys_to_dma(dev, map); + dev_addr = swiotlb_phys_to_dma(dev, map); /* Ensure that the address returned is DMA'ble */ if (!dma_capable(dev, dev_addr, size)) { swiotlb_tbl_unmap_single(dev, map, size, dir); - return phys_to_dma(dev, io_tlb_overflow_buffer); + return swiotlb_phys_to_dma(dev, io_tlb_overflow_buffer); } return dev_addr; @@ -901,7 +949,7 @@ swiotlb_map_sg_attrs(struct device *hwdev, struct scatterlist *sgl, int nelems, sg_dma_len(sgl) = 0; return 0; } - sg->dma_address = phys_to_dma(hwdev, map); + sg->dma_address = swiotlb_phys_to_dma(hwdev, map); } else sg->dma_address = dev_addr; sg_dma_len(sg) = sg->length; @@ -985,7 +1033,7 @@ EXPORT_SYMBOL(swiotlb_sync_sg_for_device); int swiotlb_dma_mapping_error(struct device *hwdev, dma_addr_t dma_addr) { - return (dma_addr == phys_to_dma(hwdev, io_tlb_overflow_buffer)); + return (dma_addr == swiotlb_phys_to_dma(hwdev, io_tlb_overflow_buffer)); } EXPORT_SYMBOL(swiotlb_dma_mapping_error); @@ -998,6 +1046,6 @@ EXPORT_SYMBOL(swiotlb_dma_mapping_error); int swiotlb_dma_supported(struct device *hwdev, u64 mask) { - return phys_to_dma(hwdev, io_tlb_end - 1) <= mask; + return swiotlb_phys_to_dma(hwdev, io_tlb_end - 1) <= mask; } EXPORT_SYMBOL(swiotlb_dma_supported);