From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S932474AbcKJAxR (ORCPT ); Wed, 9 Nov 2016 19:53:17 -0500 Received: from mail-bl2nam02on0080.outbound.protection.outlook.com ([104.47.38.80]:2016 "EHLO NAM02-BL2-obe.outbound.protection.outlook.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1754599AbcKJAxH (ORCPT ); Wed, 9 Nov 2016 19:53:07 -0500 Authentication-Results: spf=none (sender IP is ) smtp.mailfrom=Thomas.Lendacky@amd.com; From: Tom Lendacky Subject: [RFC PATCH v3 13/20] x86: DMA support for memory encryption To: , , , , , , , , CC: Rik van Riel , Radim =?utf-8?b?S3LEjW3DocWZ?= , Arnd Bergmann , Jonathan Corbet , Matt Fleming , Joerg Roedel , Konrad Rzeszutek Wilk , "Paolo Bonzini" , Larry Woodman , "Ingo Molnar" , Borislav Petkov , Andy Lutomirski , "H. Peter Anvin" , Andrey Ryabinin , Alexander Potapenko , "Thomas Gleixner" , Dmitry Vyukov Date: Wed, 9 Nov 2016 18:37:23 -0600 Message-ID: <20161110003723.3280.62636.stgit@tlendack-t1.amdoffice.net> In-Reply-To: <20161110003426.3280.2999.stgit@tlendack-t1.amdoffice.net> References: <20161110003426.3280.2999.stgit@tlendack-t1.amdoffice.net> User-Agent: StGit/0.17.1-dirty MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: 7bit X-Originating-IP: [165.204.77.1] X-ClientProxiedBy: DM5PR20CA0003.namprd20.prod.outlook.com (10.173.136.141) To CY4PR12MB1141.namprd12.prod.outlook.com (10.168.163.149) X-MS-Office365-Filtering-Correlation-Id: b2c84936-9098-46dd-5e4b-08d40901bedf X-Microsoft-Exchange-Diagnostics: 1;CY4PR12MB1141;2:sCwffxpursJqLRahPzpOQD/IJlNyUIFTbEQExXSEm4R0pmGiwo5jefV//SkrG7lPm+l8BENAf3WP5u+POj4ZAcEYeCtFZEDX3MiaveAL6uE7daOFHftk1KbXGrd2T/bCzwjoyDZHD7loIvtSwAGXQpTXvT56s89TldTWaRkknozCBAqSxzYPDyFgp0vP/r3IJil5vFr8CIcFnZW7Yqb+GQ==;3:/rx7J4rQFADKVjypOsHMlHAWEBwQsC+eQamQfOzFu8xUcHb8SjBezaFfjfktb0ft8P8jBztvAfmbq4aGTxMzBU37CKHbGTQLKtdZqFBV7R21QD8/5oFmO9qu9mTZ4H2w73B0kOMQi8x3BGVw70rbRA==;25:Wqw9aOI9UwmpUWvKz1n3kXuUJZmpafd6WaixnFH6RGM7YhB9aPwq5O3jdx/ER3kMLrdf4wUdTJGKwxW9tIJNFnRpIDwuzP44UjXzXSny/ZZR6JVQBJ/+116/sh65Voh3C17HX9YzyAcDyqtDXxLZU4XY14oDf9DrEMjKUDqxJtzacr9D7+OOG8Y8+ztRc+5QTcA+62oBTHP9DFD7wXLQv3dkCTqpn9Kb/i8OFWxqtdOkK4tRTaCGERPEAMGeFl4zdu4KE79qtDz/8OLe7JPsaUbcRosMWt6qrHaIMIrWYypMN/UUVCQ9+0hWRqNQk8fge8YDvh6CCnaG3g2Q+UUczlcQioOdoSVsvY6w6oWzl1Z8nZkqcZ0VBdMhXqhVKiolg3iWvWwEFl94ZacHGPlPUV3O4GgSG3w1VGbn4aWhSASTFSsOOMW8i+O5o7d2VlqP X-Microsoft-Antispam: UriScan:;BCL:0;PCL:0;RULEID:;SRVR:CY4PR12MB1141; X-Microsoft-Exchange-Diagnostics: 1;CY4PR12MB1141;31:JsbiN/pHHmkza10WNfErvy065LsbC6qCCbEAnP6IYMbRzIQAKNkpfFFrnPVMFYaF/fL+3xTk9p2e+CAzCjweM2mo1VSY61G/HkNk7wUtv+zHnKLS+NHbl1VUzHqwC03KRuPzsSuXNla/2yorlN2qooonk3l9+X9XWM341K5zhzSYxtK4bFHwVNOfmdl3QhV1/B6EqPbpCO6fepEc1iAV7gVtsPyFjvA/ak7ltsmOgUSXeMDT77oOQKwJxIoEF+YLeh1RES7zlRUnA+k7U05DJQ==;20:88MdMeXC6xk9cev4W6S5my9PqLJpO+piPvJnBYCsCLGR3kXo2qkVAdMsLwHm4bPoMiixV8tvaKjVh2bd+53B/xRYFOnvYtZM+lJtRGByTD5GpT8j8PsjZgI/u+lKMRc1TPJOpN6lum1Dz6sMQJKbcNdKjm8bm/LVTPZqrV/wKiSWRe55aTk6ph8i8OcGrJZnC753YMMW0xDES0scJG6If7fdf0sSY6mILUQ+V5wh+lNNKLoaKEb+7j+8d7pCvkPk54WBHIiK5HaU6w5CyuB0co01IB2R3q02MbQJUeocoo8RJ7A+PWyTVJOiNrZlfuKDXJnFiHqqFDH4EhUN5x3ngHJHb/PFc/gso+mKUnVhu1t3U1507THJjsVqkr/GkSLnkZhumL5D4zXOWlPv6MM3H+igaT9PjfFykgp4hhgR10w4I0KA/Rh5/LAVVEHL3/1nDcJj5wE9HqB5O0DUxm5AvmbcKCOKlgn8godde2l6iLZm/dpnz38tGP+ce27JUR5F X-Microsoft-Antispam-PRVS: X-Exchange-Antispam-Report-Test: UriScan:(767451399110); X-Exchange-Antispam-Report-CFA-Test: BCL:0;PCL:0;RULEID:(6040176)(601004)(2401047)(8121501046)(5005006)(3002001)(10201501046)(6055026);SRVR:CY4PR12MB1141;BCL:0;PCL:0;RULEID:;SRVR:CY4PR12MB1141; X-Microsoft-Exchange-Diagnostics: 1;CY4PR12MB1141;4:8fMOFauBqtHn2NpyI0uN3gZsOhOOBSvgRM3Z8K9b9qmiv8NJhsGaY/7l92bqzvbuAp8HCBsJz0BvYbrvWxxrJb/2EEKYfxfTpy2ZxMEcnDxvHkcdVjiYMEJIOpnPduX5YyPV/J8bOjDg/emwHx5BvgD6qWsu9JtIWFv8+EJfMhwB9oDKJRhW8NqaYIJ8K2ZSxT+PLjgRoEG4BBuAvLMBzDq8ZMjIJeE5Ey1hF9knpFXuQrDddijQPe7/ASkh/FJv8aFhTJOa2sffdWD16cHEs+P4Lcx8HPYntoHVey4U/FEOP5O0JgzK2a//jQNx6T7aKLC23oCgSEl8mr1y2bksNC2qvRSh6kDPL6JV3c19vJ8NEMp9CZQpj4/tvu/eL4j7k8QH8AaiPSE9ZSMUx+hKrFEqZSsjQdezhP23+QkT0W54dbE3km2LzZhSdzoV2+vjex/hRcadgALY7rwf5EeEnQ== X-Forefront-PRVS: 01221E3973 X-Forefront-Antispam-Report: SFV:NSPM;SFS:(10009020)(4630300001)(6009001)(7916002)(199003)(189002)(103116003)(92566002)(47776003)(23676002)(8676002)(7736002)(69596002)(50466002)(189998001)(81166006)(66066001)(9686002)(305945005)(5660300001)(77096005)(575784001)(7846002)(101416001)(86362001)(2201001)(2950100002)(5001770100001)(50986999)(4001350100001)(76176999)(54356999)(97736004)(1076002)(81156014)(53416004)(105586002)(42186005)(4326007)(83506001)(7416002)(97746001)(6116002)(230700001)(3846002)(586003)(68736007)(106356001)(33646002)(2906002)(71626007)(217873001);DIR:OUT;SFP:1101;SCL:1;SRVR:CY4PR12MB1141;H:tlendack-t1.amdoffice.net;FPR:;SPF:None;PTR:InfoNoRecords;A:1;MX:1;LANG:en; X-Microsoft-Exchange-Diagnostics: =?utf-8?B?MTtDWTRQUjEyTUIxMTQxOzIzOnpwWXo5OG9ha242MFpGWDZZcHZKdDUrblRX?= =?utf-8?B?UU1vWTdYdzMyd0hzamxRRjIzS1U4QklqQm83QzZOOHk4RlVlazl4ZitzUnBm?= =?utf-8?B?OUNvcVJXWkR6RDBQS2EvQmZncHNBSFJOWENJc3I3aDRBUFRGSVUwczlmVUdS?= =?utf-8?B?S1FxeTFHdE9FODducUZwb2hTZWNoNUtsa3huTUtKTGt3RlZhMm5TellLa0hi?= =?utf-8?B?SEtQWFp5TWxRdzVLTW1MQXVUczIvczV0RG95d0VnTnJIeVMwRnlOUGJYUXUy?= =?utf-8?B?S1J6d3JsKy91QlZsRGZ6QXVMVWFsdzdBcXJKVVJsdlN1UFVoZVZYSmZMRmJu?= =?utf-8?B?SGVkOW5xVjZTWVV3M2JpSWtrd2VudEZJVEhtV1IrVlREcmYwczdSRVVSU0ZH?= =?utf-8?B?amYwZTlYU0tkUk13aHk0dkNiWW9mVU02QXZOeXZTUnc1aW5zeHFWdWM1YmNm?= =?utf-8?B?QkJOQUtIeDZmcUhGSk9mdUdwTS9QQWp0dzVPT1NzOEw3eVdtNjdTZjFSUU1D?= =?utf-8?B?MTAwNEJRMzVKcFRXTUN3dHk1amFrQ1NTTllYNmlySUZ6MXdUemN3QWxTWUZH?= =?utf-8?B?NTZyeHpqcEZWeVVQNWxrVEI5V09oNTRIa1NIM294d0craXQxMWNuQUZydFJz?= =?utf-8?B?d0NRMHBEcnJ2VkdnbkFWNHlRR0pSNERZRExCQ1ZSemlBRXVxRHZ1UEQ1NGE0?= =?utf-8?B?RzlVMm41NzQvR1djNUlab1VFV1pwb3o3b0U5cVVNTXd4ak5aNEVwVFZZUDdv?= =?utf-8?B?WlYwMi9MbDRkbzYwaHQyaG4wdTRtbklDc2lwVE5rRDlncFJGZnoycUR4MXdt?= =?utf-8?B?Y2c0bXBxdjhsY3NCNk93c0lSejlWZmwxQWpRUFJOaXBVdmI2WU9QeXR4bGs5?= =?utf-8?B?VGNZVTh5bklPR0Q5ZHdjMHpCSG1SV3RDUS9UdTloZzhzRG9SeklIVFZJUGk2?= =?utf-8?B?VC94NjM3TDgrQVZ4L3B4TVdJdytEbGZVOHdNQXhtVlo5YUlnbXFmWXJiTnh2?= =?utf-8?B?Z2FxY09kR3VtTE9YZU9WT1FrY1c4bXpmUjBjcWlJZkNXVW04N0NWcE91bzFK?= =?utf-8?B?eWtHckZ1cklhayt2dFM3WUhwSGFoRkFvTURKYnphU1gzRUs5SURwUlphZGJv?= =?utf-8?B?M3F6czlwRk5lZ094SUdudWNzOTdXREhJQm43NGU0aTF0SkFFYXhLSUhiYVRN?= =?utf-8?B?QmJKaWlaMnEwOUZGMkhuL21RbUtyamV2VDdoaC9mNE02cjRCQkVWMUZsdE5v?= =?utf-8?B?MjFvbmRVTVU1aVkvWldmSy96OURYdFMwNWp1ZTRyMWUzY3UvRHpXWXlWUC9l?= =?utf-8?B?dXA0V0lDNkVma3NFNWk4anYrenoxR2Y4SEIzMFlUMU1LUVhaQzVEcGlYUHdG?= =?utf-8?B?WE9hOTFvMDhjdTNCQ1ZwRnlPdXEvLzd6NWVlMlRCRDZ6ZS9scUxwQlAwOXJC?= =?utf-8?B?bEVpYUFnUmo3U2pXTHErSWhiemVLNjR1NTdzZjZOVEUvVWhSZ0sxK3BqRTZD?= =?utf-8?B?dmV4TjFkTVVxVmZtVVh2SkpQdDhpQWtNWlk5WEc4djA3ZHNvcExhUTJMMEU3?= =?utf-8?B?OGNCSjBXV2tic2NaVUJvRStpa3R5N2UxcDU2ZW15WHNTcTk2VC92Ky96c2ZY?= =?utf-8?B?SDBDeEs1U2pQdFVBckR2Rk9STlZQcG4zVFFmdG1EM3pUMlMxcnp4T0lzZTB1?= =?utf-8?Q?RVT39NhhriSHiMWhs8pJPacI+PFZga2KaMBIorI?= X-Microsoft-Exchange-Diagnostics: 1;CY4PR12MB1141;6:tCHKLn9J4Ra91SL9pr7kGkJDimWtkQZh0zKzmv+8Vf3r5wJcF+3PJRGYjdvbjAea16v3or5rQ3pLWlqMpIyaC7+TzP4/tblQ6Vp+dxCy25SwYzd/jR83fcPbvjmYd85y/EN7dkzuCgw6/qOAW7hfzVMwEJAcvu+HQwVn6qJDPlfZd8rYsth4mUtzJNwgmj9T9kpfEIyqB0+MYYu8NxK3MlIViiakzSx3RcQT3/K2HpVhhX+4iifl6qpGmO2C8IaNZRH81vl/FpR0AViR8R69EJuqHSCF5QHbuqHXab2C0h0AD64sxzarvKNzqGpVc6wsZ84MjPKop6XhQn5VViZH5x4wmaoatlrEOcCx+5T1v9E=;5:/lFwbYn9qgZxcCECnUewQ+1r3n//j0sjZMuNHmVi77v5oCEZS+6LPDDkAWAU+bwYe3gDnaQyu2qF0/3NA7sCxiGsiPFAP/3EIxVDuJoG53Pn69c2jbX+mCCQSYpNp1IcZpGwVE7HpUWy06jDuMkHoA==;24:kmhtqU04b0Ri41+S1oeeOnNAK6p1sSLcDPo5r7OsUHUqOmbjslC3/+IPgluzsufFOxwKDK5TMes0qF6wMs4bqhH5l5SsOLE5rWRFKmlQIYk= SpamDiagnosticOutput: 1:99 SpamDiagnosticMetadata: NSPM X-Microsoft-Exchange-Diagnostics: 1;CY4PR12MB1141;7:lCvnXg4ddKwmL74JxECgjQ6S40zu79MUZNWhfEXpt3XSwDfrzOE3W6VEZJvE7+HUKbNfsjtNebemoXVURC4rXFWruJD/yBAMF0W7iMLbL8+vNYevZRMcoN9RL8uoaTilnNEngPb4wwzLkoJ3jc1sN4tzQsp8SrnRNFJIrQRgP4Wez+y55A4JfdfZHHo8ywNsBh+LzYVKojplN6GZmcW5QMMs/0SkvPJqKauD4AFQA/FNbK7iEtWOV+Q1rwJAdhFDO61uaeEc9sXe6cObZthohGgi4WuadpezEobcDtSaqskrTuq6/QOeLDPWmoS11Zky1VAFiFGCVwFDc7IucLPKEiAebO3lGV1oLU1V8Msl0xs=;20:ZHmCnT16hW5WTGYikTqzoocE0cXDz6rST47lTJiJilAcuuZI5QClWHwEk3STmpCayKdGhDCctVBTvqI7lB76bD+nQohPLhcxDQQly4pV6h0oCanBbTv3B808nHhWqmuCVSjeh9m77O4wXgfoiBKwZsZ9fvIB670Lz76WPl/Cwzozah2OTQXEivtUUEEcUXQLwZTeeEenirIV15xEd91AP8TSkDddidbqtYNiwomYTt1enQW4nHN3LcbmzRwn7W1X X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 10 Nov 2016 00:37:25.7080 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-Transport-CrossTenantHeadersStamped: CY4PR12MB1141 Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Since DMA addresses will effectively look like 48-bit addresses when the memory encryption mask is set, SWIOTLB is needed if the DMA mask of the device performing the DMA does not support 48-bits. SWIOTLB will be initialized to create un-encrypted bounce buffers for use by these devices. Signed-off-by: Tom Lendacky --- arch/x86/include/asm/dma-mapping.h | 5 ++- arch/x86/include/asm/mem_encrypt.h | 5 +++ arch/x86/kernel/pci-dma.c | 11 ++++--- arch/x86/kernel/pci-nommu.c | 2 + arch/x86/kernel/pci-swiotlb.c | 8 ++++- arch/x86/mm/mem_encrypt.c | 17 +++++++++++ include/linux/swiotlb.h | 1 + init/main.c | 13 ++++++++ lib/swiotlb.c | 58 +++++++++++++++++++++++++++++++----- 9 files changed, 103 insertions(+), 17 deletions(-) diff --git a/arch/x86/include/asm/dma-mapping.h b/arch/x86/include/asm/dma-mapping.h index 4446162..c9cdcae 100644 --- a/arch/x86/include/asm/dma-mapping.h +++ b/arch/x86/include/asm/dma-mapping.h @@ -12,6 +12,7 @@ #include #include #include +#include #ifdef CONFIG_ISA # define ISA_DMA_BIT_MASK DMA_BIT_MASK(24) @@ -69,12 +70,12 @@ static inline bool dma_capable(struct device *dev, dma_addr_t addr, size_t size) static inline dma_addr_t phys_to_dma(struct device *dev, phys_addr_t paddr) { - return paddr; + return paddr | sme_me_mask; } static inline phys_addr_t dma_to_phys(struct device *dev, dma_addr_t daddr) { - return daddr; + return daddr & ~sme_me_mask; } #endif /* CONFIG_X86_DMA_REMAP */ diff --git a/arch/x86/include/asm/mem_encrypt.h b/arch/x86/include/asm/mem_encrypt.h index d544481..a024451 100644 --- a/arch/x86/include/asm/mem_encrypt.h +++ b/arch/x86/include/asm/mem_encrypt.h @@ -35,6 +35,11 @@ void __init sme_encrypt_ramdisk(resource_size_t paddr, void __init sme_early_init(void); +/* Architecture __weak replacement functions */ +void __init mem_encrypt_init(void); + +void swiotlb_set_mem_unenc(void *vaddr, unsigned long size); + #define __sme_pa(x) (__pa((x)) | sme_me_mask) #define __sme_pa_nodebug(x) (__pa_nodebug((x)) | sme_me_mask) diff --git a/arch/x86/kernel/pci-dma.c b/arch/x86/kernel/pci-dma.c index d30c377..0ce28df 100644 --- a/arch/x86/kernel/pci-dma.c +++ b/arch/x86/kernel/pci-dma.c @@ -92,9 +92,12 @@ again: /* CMA can be used only in the context which permits sleeping */ if (gfpflags_allow_blocking(flag)) { page = dma_alloc_from_contiguous(dev, count, get_order(size)); - if (page && page_to_phys(page) + size > dma_mask) { - dma_release_from_contiguous(dev, page, count); - page = NULL; + if (page) { + addr = phys_to_dma(dev, page_to_phys(page)); + if (addr + size > dma_mask) { + dma_release_from_contiguous(dev, page, count); + page = NULL; + } } } /* fallback */ @@ -103,7 +106,7 @@ again: if (!page) return NULL; - addr = page_to_phys(page); + addr = phys_to_dma(dev, page_to_phys(page)); if (addr + size > dma_mask) { __free_pages(page, get_order(size)); diff --git a/arch/x86/kernel/pci-nommu.c b/arch/x86/kernel/pci-nommu.c index 00e71ce..922c10d 100644 --- a/arch/x86/kernel/pci-nommu.c +++ b/arch/x86/kernel/pci-nommu.c @@ -30,7 +30,7 @@ static dma_addr_t nommu_map_page(struct device *dev, struct page *page, enum dma_data_direction dir, unsigned long attrs) { - dma_addr_t bus = page_to_phys(page) + offset; + dma_addr_t bus = phys_to_dma(dev, page_to_phys(page)) + offset; WARN_ON(size == 0); if (!check_addr("map_single", dev, bus, size)) return DMA_ERROR_CODE; diff --git a/arch/x86/kernel/pci-swiotlb.c b/arch/x86/kernel/pci-swiotlb.c index b47edb8..34a9e524 100644 --- a/arch/x86/kernel/pci-swiotlb.c +++ b/arch/x86/kernel/pci-swiotlb.c @@ -12,6 +12,8 @@ #include #include #include +#include + int swiotlb __read_mostly; void *x86_swiotlb_alloc_coherent(struct device *hwdev, size_t size, @@ -64,13 +66,15 @@ static struct dma_map_ops swiotlb_dma_ops = { * pci_swiotlb_detect_override - set swiotlb to 1 if necessary * * This returns non-zero if we are forced to use swiotlb (by the boot - * option). + * option). If memory encryption is enabled then swiotlb will be set + * to 1 so that bounce buffers are allocated and used for devices that + * do not support the addressing range required for the encryption mask. */ int __init pci_swiotlb_detect_override(void) { int use_swiotlb = swiotlb | swiotlb_force; - if (swiotlb_force) + if (swiotlb_force || sme_me_mask) swiotlb = 1; return use_swiotlb; diff --git a/arch/x86/mm/mem_encrypt.c b/arch/x86/mm/mem_encrypt.c index 41cfdf9..e351003 100644 --- a/arch/x86/mm/mem_encrypt.c +++ b/arch/x86/mm/mem_encrypt.c @@ -13,6 +13,8 @@ #include #include #include +#include +#include #include #include @@ -240,3 +242,18 @@ void __init sme_early_init(void) for (i = 0; i < ARRAY_SIZE(protection_map); i++) protection_map[i] = __pgprot(pgprot_val(protection_map[i]) | sme_me_mask); } + +/* Architecture __weak replacement functions */ +void __init mem_encrypt_init(void) +{ + if (!sme_me_mask) + return; + + /* Make SWIOTLB use an unencrypted DMA area */ + swiotlb_clear_encryption(); +} + +void swiotlb_set_mem_unenc(void *vaddr, unsigned long size) +{ + sme_set_mem_unenc(vaddr, size); +} diff --git a/include/linux/swiotlb.h b/include/linux/swiotlb.h index 5f81f8a..5c909fc 100644 --- a/include/linux/swiotlb.h +++ b/include/linux/swiotlb.h @@ -29,6 +29,7 @@ int swiotlb_init_with_tbl(char *tlb, unsigned long nslabs, int verbose); extern unsigned long swiotlb_nr_tbl(void); unsigned long swiotlb_size_or_default(void); extern int swiotlb_late_init_with_tbl(char *tlb, unsigned long nslabs); +extern void __init swiotlb_clear_encryption(void); /* * Enumeration for sync targets diff --git a/init/main.c b/init/main.c index a8a58e2..ae37f0d 100644 --- a/init/main.c +++ b/init/main.c @@ -458,6 +458,10 @@ void __init __weak thread_stack_cache_init(void) } #endif +void __init __weak mem_encrypt_init(void) +{ +} + /* * Set up kernel memory allocators */ @@ -598,6 +602,15 @@ asmlinkage __visible void __init start_kernel(void) */ locking_selftest(); + /* + * This needs to be called before any devices perform DMA + * operations that might use the swiotlb bounce buffers. + * This call will mark the bounce buffers as un-encrypted so + * that their usage will not cause "plain-text" data to be + * decrypted when accessed. + */ + mem_encrypt_init(); + #ifdef CONFIG_BLK_DEV_INITRD if (initrd_start && !initrd_below_start_ok && page_to_pfn(virt_to_page((void *)initrd_start)) < min_low_pfn) { diff --git a/lib/swiotlb.c b/lib/swiotlb.c index 22e13a0..638e99c 100644 --- a/lib/swiotlb.c +++ b/lib/swiotlb.c @@ -30,6 +30,7 @@ #include #include #include +#include #include #include @@ -131,6 +132,17 @@ unsigned long swiotlb_size_or_default(void) return size ? size : (IO_TLB_DEFAULT_SIZE); } +void __weak swiotlb_set_mem_unenc(void *vaddr, unsigned long size) +{ +} + +/* For swiotlb, clear memory encryption mask from dma addresses */ +static dma_addr_t swiotlb_phys_to_dma(struct device *hwdev, + phys_addr_t address) +{ + return phys_to_dma(hwdev, address) & ~sme_me_mask; +} + /* Note that this doesn't work with highmem page */ static dma_addr_t swiotlb_virt_to_bus(struct device *hwdev, volatile void *address) @@ -159,6 +171,31 @@ void swiotlb_print_info(void) bytes >> 20, vstart, vend - 1); } +/* + * If memory encryption is active, the DMA address for an encrypted page may + * be beyond the range of the device. If bounce buffers are required be sure + * that they are not on an encrypted page. This should be called before the + * iotlb area is used. + */ +void __init swiotlb_clear_encryption(void) +{ + void *vaddr; + unsigned long bytes; + + if (no_iotlb_memory || !io_tlb_start || late_alloc) + return; + + vaddr = phys_to_virt(io_tlb_start); + bytes = PAGE_ALIGN(io_tlb_nslabs << IO_TLB_SHIFT); + swiotlb_set_mem_unenc(vaddr, bytes); + memset(vaddr, 0, bytes); + + vaddr = phys_to_virt(io_tlb_overflow_buffer); + bytes = PAGE_ALIGN(io_tlb_overflow); + swiotlb_set_mem_unenc(vaddr, bytes); + memset(vaddr, 0, bytes); +} + int __init swiotlb_init_with_tbl(char *tlb, unsigned long nslabs, int verbose) { void *v_overflow_buffer; @@ -294,6 +331,8 @@ swiotlb_late_init_with_tbl(char *tlb, unsigned long nslabs) io_tlb_start = virt_to_phys(tlb); io_tlb_end = io_tlb_start + bytes; + /* Keep TLB in unencrypted memory if memory encryption is active */ + swiotlb_set_mem_unenc(tlb, bytes); memset(tlb, 0, bytes); /* @@ -304,6 +343,9 @@ swiotlb_late_init_with_tbl(char *tlb, unsigned long nslabs) if (!v_overflow_buffer) goto cleanup2; + /* Keep overflow in unencrypted memory if memory encryption is active */ + swiotlb_set_mem_unenc(v_overflow_buffer, io_tlb_overflow); + memset(v_overflow_buffer, 0, io_tlb_overflow); io_tlb_overflow_buffer = virt_to_phys(v_overflow_buffer); /* @@ -541,7 +583,7 @@ static phys_addr_t map_single(struct device *hwdev, phys_addr_t phys, size_t size, enum dma_data_direction dir) { - dma_addr_t start_dma_addr = phys_to_dma(hwdev, io_tlb_start); + dma_addr_t start_dma_addr = swiotlb_phys_to_dma(hwdev, io_tlb_start); return swiotlb_tbl_map_single(hwdev, start_dma_addr, phys, size, dir); } @@ -659,7 +701,7 @@ swiotlb_alloc_coherent(struct device *hwdev, size_t size, goto err_warn; ret = phys_to_virt(paddr); - dev_addr = phys_to_dma(hwdev, paddr); + dev_addr = swiotlb_phys_to_dma(hwdev, paddr); /* Confirm address can be DMA'd by device */ if (dev_addr + size - 1 > dma_mask) { @@ -758,15 +800,15 @@ dma_addr_t swiotlb_map_page(struct device *dev, struct page *page, map = map_single(dev, phys, size, dir); if (map == SWIOTLB_MAP_ERROR) { swiotlb_full(dev, size, dir, 1); - return phys_to_dma(dev, io_tlb_overflow_buffer); + return swiotlb_phys_to_dma(dev, io_tlb_overflow_buffer); } - dev_addr = phys_to_dma(dev, map); + dev_addr = swiotlb_phys_to_dma(dev, map); /* Ensure that the address returned is DMA'ble */ if (!dma_capable(dev, dev_addr, size)) { swiotlb_tbl_unmap_single(dev, map, size, dir); - return phys_to_dma(dev, io_tlb_overflow_buffer); + return swiotlb_phys_to_dma(dev, io_tlb_overflow_buffer); } return dev_addr; @@ -901,7 +943,7 @@ swiotlb_map_sg_attrs(struct device *hwdev, struct scatterlist *sgl, int nelems, sg_dma_len(sgl) = 0; return 0; } - sg->dma_address = phys_to_dma(hwdev, map); + sg->dma_address = swiotlb_phys_to_dma(hwdev, map); } else sg->dma_address = dev_addr; sg_dma_len(sg) = sg->length; @@ -985,7 +1027,7 @@ EXPORT_SYMBOL(swiotlb_sync_sg_for_device); int swiotlb_dma_mapping_error(struct device *hwdev, dma_addr_t dma_addr) { - return (dma_addr == phys_to_dma(hwdev, io_tlb_overflow_buffer)); + return (dma_addr == swiotlb_phys_to_dma(hwdev, io_tlb_overflow_buffer)); } EXPORT_SYMBOL(swiotlb_dma_mapping_error); @@ -998,6 +1040,6 @@ EXPORT_SYMBOL(swiotlb_dma_mapping_error); int swiotlb_dma_supported(struct device *hwdev, u64 mask) { - return phys_to_dma(hwdev, io_tlb_end - 1) <= mask; + return swiotlb_phys_to_dma(hwdev, io_tlb_end - 1) <= mask; } EXPORT_SYMBOL(swiotlb_dma_supported); From mboxrd@z Thu Jan 1 00:00:00 1970 From: Tom Lendacky Subject: [RFC PATCH v3 13/20] x86: DMA support for memory encryption Date: Wed, 9 Nov 2016 18:37:23 -0600 Message-ID: <20161110003723.3280.62636.stgit@tlendack-t1.amdoffice.net> References: <20161110003426.3280.2999.stgit@tlendack-t1.amdoffice.net> Mime-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: 7bit Return-path: In-Reply-To: <20161110003426.3280.2999.stgit@tlendack-t1.amdoffice.net> Sender: owner-linux-mm@kvack.org To: linux-arch@vger.kernel.org, linux-efi@vger.kernel.org, kvm@vger.kernel.org, linux-doc@vger.kernel.org, x86@kernel.org, linux-kernel@vger.kernel.org, kasan-dev@googlegroups.com, linux-mm@kvack.org, iommu@lists.linux-foundation.org Cc: Rik van Riel , Radim =?utf-8?b?S3LEjW3DocWZ?= , Arnd Bergmann , Jonathan Corbet , Matt Fleming , Joerg Roedel , Konrad Rzeszutek Wilk , Paolo Bonzini , Larry Woodman , Ingo Molnar , Borislav Petkov , Andy Lutomirski , "H. Peter Anvin" , Andrey Ryabinin , Alexander Potapenko , Thomas Gleixner , Dmitry Vyukov List-Id: linux-efi@vger.kernel.org Since DMA addresses will effectively look like 48-bit addresses when the memory encryption mask is set, SWIOTLB is needed if the DMA mask of the device performing the DMA does not support 48-bits. SWIOTLB will be initialized to create un-encrypted bounce buffers for use by these devices. Signed-off-by: Tom Lendacky --- arch/x86/include/asm/dma-mapping.h | 5 ++- arch/x86/include/asm/mem_encrypt.h | 5 +++ arch/x86/kernel/pci-dma.c | 11 ++++--- arch/x86/kernel/pci-nommu.c | 2 + arch/x86/kernel/pci-swiotlb.c | 8 ++++- arch/x86/mm/mem_encrypt.c | 17 +++++++++++ include/linux/swiotlb.h | 1 + init/main.c | 13 ++++++++ lib/swiotlb.c | 58 +++++++++++++++++++++++++++++++----- 9 files changed, 103 insertions(+), 17 deletions(-) diff --git a/arch/x86/include/asm/dma-mapping.h b/arch/x86/include/asm/dma-mapping.h index 4446162..c9cdcae 100644 --- a/arch/x86/include/asm/dma-mapping.h +++ b/arch/x86/include/asm/dma-mapping.h @@ -12,6 +12,7 @@ #include #include #include +#include #ifdef CONFIG_ISA # define ISA_DMA_BIT_MASK DMA_BIT_MASK(24) @@ -69,12 +70,12 @@ static inline bool dma_capable(struct device *dev, dma_addr_t addr, size_t size) static inline dma_addr_t phys_to_dma(struct device *dev, phys_addr_t paddr) { - return paddr; + return paddr | sme_me_mask; } static inline phys_addr_t dma_to_phys(struct device *dev, dma_addr_t daddr) { - return daddr; + return daddr & ~sme_me_mask; } #endif /* CONFIG_X86_DMA_REMAP */ diff --git a/arch/x86/include/asm/mem_encrypt.h b/arch/x86/include/asm/mem_encrypt.h index d544481..a024451 100644 --- a/arch/x86/include/asm/mem_encrypt.h +++ b/arch/x86/include/asm/mem_encrypt.h @@ -35,6 +35,11 @@ void __init sme_encrypt_ramdisk(resource_size_t paddr, void __init sme_early_init(void); +/* Architecture __weak replacement functions */ +void __init mem_encrypt_init(void); + +void swiotlb_set_mem_unenc(void *vaddr, unsigned long size); + #define __sme_pa(x) (__pa((x)) | sme_me_mask) #define __sme_pa_nodebug(x) (__pa_nodebug((x)) | sme_me_mask) diff --git a/arch/x86/kernel/pci-dma.c b/arch/x86/kernel/pci-dma.c index d30c377..0ce28df 100644 --- a/arch/x86/kernel/pci-dma.c +++ b/arch/x86/kernel/pci-dma.c @@ -92,9 +92,12 @@ again: /* CMA can be used only in the context which permits sleeping */ if (gfpflags_allow_blocking(flag)) { page = dma_alloc_from_contiguous(dev, count, get_order(size)); - if (page && page_to_phys(page) + size > dma_mask) { - dma_release_from_contiguous(dev, page, count); - page = NULL; + if (page) { + addr = phys_to_dma(dev, page_to_phys(page)); + if (addr + size > dma_mask) { + dma_release_from_contiguous(dev, page, count); + page = NULL; + } } } /* fallback */ @@ -103,7 +106,7 @@ again: if (!page) return NULL; - addr = page_to_phys(page); + addr = phys_to_dma(dev, page_to_phys(page)); if (addr + size > dma_mask) { __free_pages(page, get_order(size)); diff --git a/arch/x86/kernel/pci-nommu.c b/arch/x86/kernel/pci-nommu.c index 00e71ce..922c10d 100644 --- a/arch/x86/kernel/pci-nommu.c +++ b/arch/x86/kernel/pci-nommu.c @@ -30,7 +30,7 @@ static dma_addr_t nommu_map_page(struct device *dev, struct page *page, enum dma_data_direction dir, unsigned long attrs) { - dma_addr_t bus = page_to_phys(page) + offset; + dma_addr_t bus = phys_to_dma(dev, page_to_phys(page)) + offset; WARN_ON(size == 0); if (!check_addr("map_single", dev, bus, size)) return DMA_ERROR_CODE; diff --git a/arch/x86/kernel/pci-swiotlb.c b/arch/x86/kernel/pci-swiotlb.c index b47edb8..34a9e524 100644 --- a/arch/x86/kernel/pci-swiotlb.c +++ b/arch/x86/kernel/pci-swiotlb.c @@ -12,6 +12,8 @@ #include #include #include +#include + int swiotlb __read_mostly; void *x86_swiotlb_alloc_coherent(struct device *hwdev, size_t size, @@ -64,13 +66,15 @@ static struct dma_map_ops swiotlb_dma_ops = { * pci_swiotlb_detect_override - set swiotlb to 1 if necessary * * This returns non-zero if we are forced to use swiotlb (by the boot - * option). + * option). If memory encryption is enabled then swiotlb will be set + * to 1 so that bounce buffers are allocated and used for devices that + * do not support the addressing range required for the encryption mask. */ int __init pci_swiotlb_detect_override(void) { int use_swiotlb = swiotlb | swiotlb_force; - if (swiotlb_force) + if (swiotlb_force || sme_me_mask) swiotlb = 1; return use_swiotlb; diff --git a/arch/x86/mm/mem_encrypt.c b/arch/x86/mm/mem_encrypt.c index 41cfdf9..e351003 100644 --- a/arch/x86/mm/mem_encrypt.c +++ b/arch/x86/mm/mem_encrypt.c @@ -13,6 +13,8 @@ #include #include #include +#include +#include #include #include @@ -240,3 +242,18 @@ void __init sme_early_init(void) for (i = 0; i < ARRAY_SIZE(protection_map); i++) protection_map[i] = __pgprot(pgprot_val(protection_map[i]) | sme_me_mask); } + +/* Architecture __weak replacement functions */ +void __init mem_encrypt_init(void) +{ + if (!sme_me_mask) + return; + + /* Make SWIOTLB use an unencrypted DMA area */ + swiotlb_clear_encryption(); +} + +void swiotlb_set_mem_unenc(void *vaddr, unsigned long size) +{ + sme_set_mem_unenc(vaddr, size); +} diff --git a/include/linux/swiotlb.h b/include/linux/swiotlb.h index 5f81f8a..5c909fc 100644 --- a/include/linux/swiotlb.h +++ b/include/linux/swiotlb.h @@ -29,6 +29,7 @@ int swiotlb_init_with_tbl(char *tlb, unsigned long nslabs, int verbose); extern unsigned long swiotlb_nr_tbl(void); unsigned long swiotlb_size_or_default(void); extern int swiotlb_late_init_with_tbl(char *tlb, unsigned long nslabs); +extern void __init swiotlb_clear_encryption(void); /* * Enumeration for sync targets diff --git a/init/main.c b/init/main.c index a8a58e2..ae37f0d 100644 --- a/init/main.c +++ b/init/main.c @@ -458,6 +458,10 @@ void __init __weak thread_stack_cache_init(void) } #endif +void __init __weak mem_encrypt_init(void) +{ +} + /* * Set up kernel memory allocators */ @@ -598,6 +602,15 @@ asmlinkage __visible void __init start_kernel(void) */ locking_selftest(); + /* + * This needs to be called before any devices perform DMA + * operations that might use the swiotlb bounce buffers. + * This call will mark the bounce buffers as un-encrypted so + * that their usage will not cause "plain-text" data to be + * decrypted when accessed. + */ + mem_encrypt_init(); + #ifdef CONFIG_BLK_DEV_INITRD if (initrd_start && !initrd_below_start_ok && page_to_pfn(virt_to_page((void *)initrd_start)) < min_low_pfn) { diff --git a/lib/swiotlb.c b/lib/swiotlb.c index 22e13a0..638e99c 100644 --- a/lib/swiotlb.c +++ b/lib/swiotlb.c @@ -30,6 +30,7 @@ #include #include #include +#include #include #include @@ -131,6 +132,17 @@ unsigned long swiotlb_size_or_default(void) return size ? size : (IO_TLB_DEFAULT_SIZE); } +void __weak swiotlb_set_mem_unenc(void *vaddr, unsigned long size) +{ +} + +/* For swiotlb, clear memory encryption mask from dma addresses */ +static dma_addr_t swiotlb_phys_to_dma(struct device *hwdev, + phys_addr_t address) +{ + return phys_to_dma(hwdev, address) & ~sme_me_mask; +} + /* Note that this doesn't work with highmem page */ static dma_addr_t swiotlb_virt_to_bus(struct device *hwdev, volatile void *address) @@ -159,6 +171,31 @@ void swiotlb_print_info(void) bytes >> 20, vstart, vend - 1); } +/* + * If memory encryption is active, the DMA address for an encrypted page may + * be beyond the range of the device. If bounce buffers are required be sure + * that they are not on an encrypted page. This should be called before the + * iotlb area is used. + */ +void __init swiotlb_clear_encryption(void) +{ + void *vaddr; + unsigned long bytes; + + if (no_iotlb_memory || !io_tlb_start || late_alloc) + return; + + vaddr = phys_to_virt(io_tlb_start); + bytes = PAGE_ALIGN(io_tlb_nslabs << IO_TLB_SHIFT); + swiotlb_set_mem_unenc(vaddr, bytes); + memset(vaddr, 0, bytes); + + vaddr = phys_to_virt(io_tlb_overflow_buffer); + bytes = PAGE_ALIGN(io_tlb_overflow); + swiotlb_set_mem_unenc(vaddr, bytes); + memset(vaddr, 0, bytes); +} + int __init swiotlb_init_with_tbl(char *tlb, unsigned long nslabs, int verbose) { void *v_overflow_buffer; @@ -294,6 +331,8 @@ swiotlb_late_init_with_tbl(char *tlb, unsigned long nslabs) io_tlb_start = virt_to_phys(tlb); io_tlb_end = io_tlb_start + bytes; + /* Keep TLB in unencrypted memory if memory encryption is active */ + swiotlb_set_mem_unenc(tlb, bytes); memset(tlb, 0, bytes); /* @@ -304,6 +343,9 @@ swiotlb_late_init_with_tbl(char *tlb, unsigned long nslabs) if (!v_overflow_buffer) goto cleanup2; + /* Keep overflow in unencrypted memory if memory encryption is active */ + swiotlb_set_mem_unenc(v_overflow_buffer, io_tlb_overflow); + memset(v_overflow_buffer, 0, io_tlb_overflow); io_tlb_overflow_buffer = virt_to_phys(v_overflow_buffer); /* @@ -541,7 +583,7 @@ static phys_addr_t map_single(struct device *hwdev, phys_addr_t phys, size_t size, enum dma_data_direction dir) { - dma_addr_t start_dma_addr = phys_to_dma(hwdev, io_tlb_start); + dma_addr_t start_dma_addr = swiotlb_phys_to_dma(hwdev, io_tlb_start); return swiotlb_tbl_map_single(hwdev, start_dma_addr, phys, size, dir); } @@ -659,7 +701,7 @@ swiotlb_alloc_coherent(struct device *hwdev, size_t size, goto err_warn; ret = phys_to_virt(paddr); - dev_addr = phys_to_dma(hwdev, paddr); + dev_addr = swiotlb_phys_to_dma(hwdev, paddr); /* Confirm address can be DMA'd by device */ if (dev_addr + size - 1 > dma_mask) { @@ -758,15 +800,15 @@ dma_addr_t swiotlb_map_page(struct device *dev, struct page *page, map = map_single(dev, phys, size, dir); if (map == SWIOTLB_MAP_ERROR) { swiotlb_full(dev, size, dir, 1); - return phys_to_dma(dev, io_tlb_overflow_buffer); + return swiotlb_phys_to_dma(dev, io_tlb_overflow_buffer); } - dev_addr = phys_to_dma(dev, map); + dev_addr = swiotlb_phys_to_dma(dev, map); /* Ensure that the address returned is DMA'ble */ if (!dma_capable(dev, dev_addr, size)) { swiotlb_tbl_unmap_single(dev, map, size, dir); - return phys_to_dma(dev, io_tlb_overflow_buffer); + return swiotlb_phys_to_dma(dev, io_tlb_overflow_buffer); } return dev_addr; @@ -901,7 +943,7 @@ swiotlb_map_sg_attrs(struct device *hwdev, struct scatterlist *sgl, int nelems, sg_dma_len(sgl) = 0; return 0; } - sg->dma_address = phys_to_dma(hwdev, map); + sg->dma_address = swiotlb_phys_to_dma(hwdev, map); } else sg->dma_address = dev_addr; sg_dma_len(sg) = sg->length; @@ -985,7 +1027,7 @@ EXPORT_SYMBOL(swiotlb_sync_sg_for_device); int swiotlb_dma_mapping_error(struct device *hwdev, dma_addr_t dma_addr) { - return (dma_addr == phys_to_dma(hwdev, io_tlb_overflow_buffer)); + return (dma_addr == swiotlb_phys_to_dma(hwdev, io_tlb_overflow_buffer)); } EXPORT_SYMBOL(swiotlb_dma_mapping_error); @@ -998,6 +1040,6 @@ EXPORT_SYMBOL(swiotlb_dma_mapping_error); int swiotlb_dma_supported(struct device *hwdev, u64 mask) { - return phys_to_dma(hwdev, io_tlb_end - 1) <= mask; + return swiotlb_phys_to_dma(hwdev, io_tlb_end - 1) <= mask; } EXPORT_SYMBOL(swiotlb_dma_supported); -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: email@kvack.org From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-bl2nam02on0080.outbound.protection.outlook.com ([104.47.38.80]:2016 "EHLO NAM02-BL2-obe.outbound.protection.outlook.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1754599AbcKJAxH (ORCPT ); Wed, 9 Nov 2016 19:53:07 -0500 From: Tom Lendacky Subject: [RFC PATCH v3 13/20] x86: DMA support for memory encryption Date: Wed, 9 Nov 2016 18:37:23 -0600 Message-ID: <20161110003723.3280.62636.stgit@tlendack-t1.amdoffice.net> In-Reply-To: <20161110003426.3280.2999.stgit@tlendack-t1.amdoffice.net> References: <20161110003426.3280.2999.stgit@tlendack-t1.amdoffice.net> MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: 7bit Sender: linux-arch-owner@vger.kernel.org List-ID: To: linux-arch@vger.kernel.org, linux-efi@vger.kernel.org, kvm@vger.kernel.org, linux-doc@vger.kernel.org, x86@kernel.org, linux-kernel@vger.kernel.org, kasan-dev@googlegroups.com, linux-mm@kvack.org, iommu@lists.linux-foundation.org Cc: Rik van Riel , Radim =?utf-8?b?S3LEjW3DocWZ?= , Arnd Bergmann , Jonathan Corbet , Matt Fleming , Joerg Roedel , Konrad Rzeszutek Wilk , Paolo Bonzini , Larry Woodman , Ingo Molnar , Borislav Petkov , Andy Lutomirski , "H. Peter Anvin" , Andrey Ryabinin , Alexander Potapenko , Thomas Gleixner , Dmitry Vyukov Message-ID: <20161110003723.UpWeQpfTaR76cfnYIVBtXHhvCiNQEf4Ig3uLY_76rXM@z> Since DMA addresses will effectively look like 48-bit addresses when the memory encryption mask is set, SWIOTLB is needed if the DMA mask of the device performing the DMA does not support 48-bits. SWIOTLB will be initialized to create un-encrypted bounce buffers for use by these devices. Signed-off-by: Tom Lendacky --- arch/x86/include/asm/dma-mapping.h | 5 ++- arch/x86/include/asm/mem_encrypt.h | 5 +++ arch/x86/kernel/pci-dma.c | 11 ++++--- arch/x86/kernel/pci-nommu.c | 2 + arch/x86/kernel/pci-swiotlb.c | 8 ++++- arch/x86/mm/mem_encrypt.c | 17 +++++++++++ include/linux/swiotlb.h | 1 + init/main.c | 13 ++++++++ lib/swiotlb.c | 58 +++++++++++++++++++++++++++++++----- 9 files changed, 103 insertions(+), 17 deletions(-) diff --git a/arch/x86/include/asm/dma-mapping.h b/arch/x86/include/asm/dma-mapping.h index 4446162..c9cdcae 100644 --- a/arch/x86/include/asm/dma-mapping.h +++ b/arch/x86/include/asm/dma-mapping.h @@ -12,6 +12,7 @@ #include #include #include +#include #ifdef CONFIG_ISA # define ISA_DMA_BIT_MASK DMA_BIT_MASK(24) @@ -69,12 +70,12 @@ static inline bool dma_capable(struct device *dev, dma_addr_t addr, size_t size) static inline dma_addr_t phys_to_dma(struct device *dev, phys_addr_t paddr) { - return paddr; + return paddr | sme_me_mask; } static inline phys_addr_t dma_to_phys(struct device *dev, dma_addr_t daddr) { - return daddr; + return daddr & ~sme_me_mask; } #endif /* CONFIG_X86_DMA_REMAP */ diff --git a/arch/x86/include/asm/mem_encrypt.h b/arch/x86/include/asm/mem_encrypt.h index d544481..a024451 100644 --- a/arch/x86/include/asm/mem_encrypt.h +++ b/arch/x86/include/asm/mem_encrypt.h @@ -35,6 +35,11 @@ void __init sme_encrypt_ramdisk(resource_size_t paddr, void __init sme_early_init(void); +/* Architecture __weak replacement functions */ +void __init mem_encrypt_init(void); + +void swiotlb_set_mem_unenc(void *vaddr, unsigned long size); + #define __sme_pa(x) (__pa((x)) | sme_me_mask) #define __sme_pa_nodebug(x) (__pa_nodebug((x)) | sme_me_mask) diff --git a/arch/x86/kernel/pci-dma.c b/arch/x86/kernel/pci-dma.c index d30c377..0ce28df 100644 --- a/arch/x86/kernel/pci-dma.c +++ b/arch/x86/kernel/pci-dma.c @@ -92,9 +92,12 @@ again: /* CMA can be used only in the context which permits sleeping */ if (gfpflags_allow_blocking(flag)) { page = dma_alloc_from_contiguous(dev, count, get_order(size)); - if (page && page_to_phys(page) + size > dma_mask) { - dma_release_from_contiguous(dev, page, count); - page = NULL; + if (page) { + addr = phys_to_dma(dev, page_to_phys(page)); + if (addr + size > dma_mask) { + dma_release_from_contiguous(dev, page, count); + page = NULL; + } } } /* fallback */ @@ -103,7 +106,7 @@ again: if (!page) return NULL; - addr = page_to_phys(page); + addr = phys_to_dma(dev, page_to_phys(page)); if (addr + size > dma_mask) { __free_pages(page, get_order(size)); diff --git a/arch/x86/kernel/pci-nommu.c b/arch/x86/kernel/pci-nommu.c index 00e71ce..922c10d 100644 --- a/arch/x86/kernel/pci-nommu.c +++ b/arch/x86/kernel/pci-nommu.c @@ -30,7 +30,7 @@ static dma_addr_t nommu_map_page(struct device *dev, struct page *page, enum dma_data_direction dir, unsigned long attrs) { - dma_addr_t bus = page_to_phys(page) + offset; + dma_addr_t bus = phys_to_dma(dev, page_to_phys(page)) + offset; WARN_ON(size == 0); if (!check_addr("map_single", dev, bus, size)) return DMA_ERROR_CODE; diff --git a/arch/x86/kernel/pci-swiotlb.c b/arch/x86/kernel/pci-swiotlb.c index b47edb8..34a9e524 100644 --- a/arch/x86/kernel/pci-swiotlb.c +++ b/arch/x86/kernel/pci-swiotlb.c @@ -12,6 +12,8 @@ #include #include #include +#include + int swiotlb __read_mostly; void *x86_swiotlb_alloc_coherent(struct device *hwdev, size_t size, @@ -64,13 +66,15 @@ static struct dma_map_ops swiotlb_dma_ops = { * pci_swiotlb_detect_override - set swiotlb to 1 if necessary * * This returns non-zero if we are forced to use swiotlb (by the boot - * option). + * option). If memory encryption is enabled then swiotlb will be set + * to 1 so that bounce buffers are allocated and used for devices that + * do not support the addressing range required for the encryption mask. */ int __init pci_swiotlb_detect_override(void) { int use_swiotlb = swiotlb | swiotlb_force; - if (swiotlb_force) + if (swiotlb_force || sme_me_mask) swiotlb = 1; return use_swiotlb; diff --git a/arch/x86/mm/mem_encrypt.c b/arch/x86/mm/mem_encrypt.c index 41cfdf9..e351003 100644 --- a/arch/x86/mm/mem_encrypt.c +++ b/arch/x86/mm/mem_encrypt.c @@ -13,6 +13,8 @@ #include #include #include +#include +#include #include #include @@ -240,3 +242,18 @@ void __init sme_early_init(void) for (i = 0; i < ARRAY_SIZE(protection_map); i++) protection_map[i] = __pgprot(pgprot_val(protection_map[i]) | sme_me_mask); } + +/* Architecture __weak replacement functions */ +void __init mem_encrypt_init(void) +{ + if (!sme_me_mask) + return; + + /* Make SWIOTLB use an unencrypted DMA area */ + swiotlb_clear_encryption(); +} + +void swiotlb_set_mem_unenc(void *vaddr, unsigned long size) +{ + sme_set_mem_unenc(vaddr, size); +} diff --git a/include/linux/swiotlb.h b/include/linux/swiotlb.h index 5f81f8a..5c909fc 100644 --- a/include/linux/swiotlb.h +++ b/include/linux/swiotlb.h @@ -29,6 +29,7 @@ int swiotlb_init_with_tbl(char *tlb, unsigned long nslabs, int verbose); extern unsigned long swiotlb_nr_tbl(void); unsigned long swiotlb_size_or_default(void); extern int swiotlb_late_init_with_tbl(char *tlb, unsigned long nslabs); +extern void __init swiotlb_clear_encryption(void); /* * Enumeration for sync targets diff --git a/init/main.c b/init/main.c index a8a58e2..ae37f0d 100644 --- a/init/main.c +++ b/init/main.c @@ -458,6 +458,10 @@ void __init __weak thread_stack_cache_init(void) } #endif +void __init __weak mem_encrypt_init(void) +{ +} + /* * Set up kernel memory allocators */ @@ -598,6 +602,15 @@ asmlinkage __visible void __init start_kernel(void) */ locking_selftest(); + /* + * This needs to be called before any devices perform DMA + * operations that might use the swiotlb bounce buffers. + * This call will mark the bounce buffers as un-encrypted so + * that their usage will not cause "plain-text" data to be + * decrypted when accessed. + */ + mem_encrypt_init(); + #ifdef CONFIG_BLK_DEV_INITRD if (initrd_start && !initrd_below_start_ok && page_to_pfn(virt_to_page((void *)initrd_start)) < min_low_pfn) { diff --git a/lib/swiotlb.c b/lib/swiotlb.c index 22e13a0..638e99c 100644 --- a/lib/swiotlb.c +++ b/lib/swiotlb.c @@ -30,6 +30,7 @@ #include #include #include +#include #include #include @@ -131,6 +132,17 @@ unsigned long swiotlb_size_or_default(void) return size ? size : (IO_TLB_DEFAULT_SIZE); } +void __weak swiotlb_set_mem_unenc(void *vaddr, unsigned long size) +{ +} + +/* For swiotlb, clear memory encryption mask from dma addresses */ +static dma_addr_t swiotlb_phys_to_dma(struct device *hwdev, + phys_addr_t address) +{ + return phys_to_dma(hwdev, address) & ~sme_me_mask; +} + /* Note that this doesn't work with highmem page */ static dma_addr_t swiotlb_virt_to_bus(struct device *hwdev, volatile void *address) @@ -159,6 +171,31 @@ void swiotlb_print_info(void) bytes >> 20, vstart, vend - 1); } +/* + * If memory encryption is active, the DMA address for an encrypted page may + * be beyond the range of the device. If bounce buffers are required be sure + * that they are not on an encrypted page. This should be called before the + * iotlb area is used. + */ +void __init swiotlb_clear_encryption(void) +{ + void *vaddr; + unsigned long bytes; + + if (no_iotlb_memory || !io_tlb_start || late_alloc) + return; + + vaddr = phys_to_virt(io_tlb_start); + bytes = PAGE_ALIGN(io_tlb_nslabs << IO_TLB_SHIFT); + swiotlb_set_mem_unenc(vaddr, bytes); + memset(vaddr, 0, bytes); + + vaddr = phys_to_virt(io_tlb_overflow_buffer); + bytes = PAGE_ALIGN(io_tlb_overflow); + swiotlb_set_mem_unenc(vaddr, bytes); + memset(vaddr, 0, bytes); +} + int __init swiotlb_init_with_tbl(char *tlb, unsigned long nslabs, int verbose) { void *v_overflow_buffer; @@ -294,6 +331,8 @@ swiotlb_late_init_with_tbl(char *tlb, unsigned long nslabs) io_tlb_start = virt_to_phys(tlb); io_tlb_end = io_tlb_start + bytes; + /* Keep TLB in unencrypted memory if memory encryption is active */ + swiotlb_set_mem_unenc(tlb, bytes); memset(tlb, 0, bytes); /* @@ -304,6 +343,9 @@ swiotlb_late_init_with_tbl(char *tlb, unsigned long nslabs) if (!v_overflow_buffer) goto cleanup2; + /* Keep overflow in unencrypted memory if memory encryption is active */ + swiotlb_set_mem_unenc(v_overflow_buffer, io_tlb_overflow); + memset(v_overflow_buffer, 0, io_tlb_overflow); io_tlb_overflow_buffer = virt_to_phys(v_overflow_buffer); /* @@ -541,7 +583,7 @@ static phys_addr_t map_single(struct device *hwdev, phys_addr_t phys, size_t size, enum dma_data_direction dir) { - dma_addr_t start_dma_addr = phys_to_dma(hwdev, io_tlb_start); + dma_addr_t start_dma_addr = swiotlb_phys_to_dma(hwdev, io_tlb_start); return swiotlb_tbl_map_single(hwdev, start_dma_addr, phys, size, dir); } @@ -659,7 +701,7 @@ swiotlb_alloc_coherent(struct device *hwdev, size_t size, goto err_warn; ret = phys_to_virt(paddr); - dev_addr = phys_to_dma(hwdev, paddr); + dev_addr = swiotlb_phys_to_dma(hwdev, paddr); /* Confirm address can be DMA'd by device */ if (dev_addr + size - 1 > dma_mask) { @@ -758,15 +800,15 @@ dma_addr_t swiotlb_map_page(struct device *dev, struct page *page, map = map_single(dev, phys, size, dir); if (map == SWIOTLB_MAP_ERROR) { swiotlb_full(dev, size, dir, 1); - return phys_to_dma(dev, io_tlb_overflow_buffer); + return swiotlb_phys_to_dma(dev, io_tlb_overflow_buffer); } - dev_addr = phys_to_dma(dev, map); + dev_addr = swiotlb_phys_to_dma(dev, map); /* Ensure that the address returned is DMA'ble */ if (!dma_capable(dev, dev_addr, size)) { swiotlb_tbl_unmap_single(dev, map, size, dir); - return phys_to_dma(dev, io_tlb_overflow_buffer); + return swiotlb_phys_to_dma(dev, io_tlb_overflow_buffer); } return dev_addr; @@ -901,7 +943,7 @@ swiotlb_map_sg_attrs(struct device *hwdev, struct scatterlist *sgl, int nelems, sg_dma_len(sgl) = 0; return 0; } - sg->dma_address = phys_to_dma(hwdev, map); + sg->dma_address = swiotlb_phys_to_dma(hwdev, map); } else sg->dma_address = dev_addr; sg_dma_len(sg) = sg->length; @@ -985,7 +1027,7 @@ EXPORT_SYMBOL(swiotlb_sync_sg_for_device); int swiotlb_dma_mapping_error(struct device *hwdev, dma_addr_t dma_addr) { - return (dma_addr == phys_to_dma(hwdev, io_tlb_overflow_buffer)); + return (dma_addr == swiotlb_phys_to_dma(hwdev, io_tlb_overflow_buffer)); } EXPORT_SYMBOL(swiotlb_dma_mapping_error); @@ -998,6 +1040,6 @@ EXPORT_SYMBOL(swiotlb_dma_mapping_error); int swiotlb_dma_supported(struct device *hwdev, u64 mask) { - return phys_to_dma(hwdev, io_tlb_end - 1) <= mask; + return swiotlb_phys_to_dma(hwdev, io_tlb_end - 1) <= mask; } EXPORT_SYMBOL(swiotlb_dma_supported); From mboxrd@z Thu Jan 1 00:00:00 1970 From: Tom Lendacky Subject: [RFC PATCH v3 13/20] x86: DMA support for memory encryption Date: Wed, 9 Nov 2016 18:37:23 -0600 Message-ID: <20161110003723.3280.62636.stgit@tlendack-t1.amdoffice.net> References: <20161110003426.3280.2999.stgit@tlendack-t1.amdoffice.net> Mime-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: 7bit Cc: Rik van Riel , Radim =?utf-8?b?S3LEjW3DocWZ?= , Arnd Bergmann , Jonathan Corbet , Matt Fleming , Joerg Roedel , Konrad Rzeszutek Wilk , "Paolo Bonzini" , Larry Woodman , "Ingo Molnar" , Borislav Petkov , Andy Lutomirski , "H. Peter Anvin" , Andrey Ryabinin , Alexander Potapenko , "Thomas Gleixner" , Dmitry Vyukov To: , , , , , , , , Return-path: In-Reply-To: <20161110003426.3280.2999.stgit@tlendack-t1.amdoffice.net> Sender: owner-linux-mm@kvack.org List-Id: kvm.vger.kernel.org Since DMA addresses will effectively look like 48-bit addresses when the memory encryption mask is set, SWIOTLB is needed if the DMA mask of the device performing the DMA does not support 48-bits. SWIOTLB will be initialized to create un-encrypted bounce buffers for use by these devices. Signed-off-by: Tom Lendacky --- arch/x86/include/asm/dma-mapping.h | 5 ++- arch/x86/include/asm/mem_encrypt.h | 5 +++ arch/x86/kernel/pci-dma.c | 11 ++++--- arch/x86/kernel/pci-nommu.c | 2 + arch/x86/kernel/pci-swiotlb.c | 8 ++++- arch/x86/mm/mem_encrypt.c | 17 +++++++++++ include/linux/swiotlb.h | 1 + init/main.c | 13 ++++++++ lib/swiotlb.c | 58 +++++++++++++++++++++++++++++++----- 9 files changed, 103 insertions(+), 17 deletions(-) diff --git a/arch/x86/include/asm/dma-mapping.h b/arch/x86/include/asm/dma-mapping.h index 4446162..c9cdcae 100644 --- a/arch/x86/include/asm/dma-mapping.h +++ b/arch/x86/include/asm/dma-mapping.h @@ -12,6 +12,7 @@ #include #include #include +#include #ifdef CONFIG_ISA # define ISA_DMA_BIT_MASK DMA_BIT_MASK(24) @@ -69,12 +70,12 @@ static inline bool dma_capable(struct device *dev, dma_addr_t addr, size_t size) static inline dma_addr_t phys_to_dma(struct device *dev, phys_addr_t paddr) { - return paddr; + return paddr | sme_me_mask; } static inline phys_addr_t dma_to_phys(struct device *dev, dma_addr_t daddr) { - return daddr; + return daddr & ~sme_me_mask; } #endif /* CONFIG_X86_DMA_REMAP */ diff --git a/arch/x86/include/asm/mem_encrypt.h b/arch/x86/include/asm/mem_encrypt.h index d544481..a024451 100644 --- a/arch/x86/include/asm/mem_encrypt.h +++ b/arch/x86/include/asm/mem_encrypt.h @@ -35,6 +35,11 @@ void __init sme_encrypt_ramdisk(resource_size_t paddr, void __init sme_early_init(void); +/* Architecture __weak replacement functions */ +void __init mem_encrypt_init(void); + +void swiotlb_set_mem_unenc(void *vaddr, unsigned long size); + #define __sme_pa(x) (__pa((x)) | sme_me_mask) #define __sme_pa_nodebug(x) (__pa_nodebug((x)) | sme_me_mask) diff --git a/arch/x86/kernel/pci-dma.c b/arch/x86/kernel/pci-dma.c index d30c377..0ce28df 100644 --- a/arch/x86/kernel/pci-dma.c +++ b/arch/x86/kernel/pci-dma.c @@ -92,9 +92,12 @@ again: /* CMA can be used only in the context which permits sleeping */ if (gfpflags_allow_blocking(flag)) { page = dma_alloc_from_contiguous(dev, count, get_order(size)); - if (page && page_to_phys(page) + size > dma_mask) { - dma_release_from_contiguous(dev, page, count); - page = NULL; + if (page) { + addr = phys_to_dma(dev, page_to_phys(page)); + if (addr + size > dma_mask) { + dma_release_from_contiguous(dev, page, count); + page = NULL; + } } } /* fallback */ @@ -103,7 +106,7 @@ again: if (!page) return NULL; - addr = page_to_phys(page); + addr = phys_to_dma(dev, page_to_phys(page)); if (addr + size > dma_mask) { __free_pages(page, get_order(size)); diff --git a/arch/x86/kernel/pci-nommu.c b/arch/x86/kernel/pci-nommu.c index 00e71ce..922c10d 100644 --- a/arch/x86/kernel/pci-nommu.c +++ b/arch/x86/kernel/pci-nommu.c @@ -30,7 +30,7 @@ static dma_addr_t nommu_map_page(struct device *dev, struct page *page, enum dma_data_direction dir, unsigned long attrs) { - dma_addr_t bus = page_to_phys(page) + offset; + dma_addr_t bus = phys_to_dma(dev, page_to_phys(page)) + offset; WARN_ON(size == 0); if (!check_addr("map_single", dev, bus, size)) return DMA_ERROR_CODE; diff --git a/arch/x86/kernel/pci-swiotlb.c b/arch/x86/kernel/pci-swiotlb.c index b47edb8..34a9e524 100644 --- a/arch/x86/kernel/pci-swiotlb.c +++ b/arch/x86/kernel/pci-swiotlb.c @@ -12,6 +12,8 @@ #include #include #include +#include + int swiotlb __read_mostly; void *x86_swiotlb_alloc_coherent(struct device *hwdev, size_t size, @@ -64,13 +66,15 @@ static struct dma_map_ops swiotlb_dma_ops = { * pci_swiotlb_detect_override - set swiotlb to 1 if necessary * * This returns non-zero if we are forced to use swiotlb (by the boot - * option). + * option). If memory encryption is enabled then swiotlb will be set + * to 1 so that bounce buffers are allocated and used for devices that + * do not support the addressing range required for the encryption mask. */ int __init pci_swiotlb_detect_override(void) { int use_swiotlb = swiotlb | swiotlb_force; - if (swiotlb_force) + if (swiotlb_force || sme_me_mask) swiotlb = 1; return use_swiotlb; diff --git a/arch/x86/mm/mem_encrypt.c b/arch/x86/mm/mem_encrypt.c index 41cfdf9..e351003 100644 --- a/arch/x86/mm/mem_encrypt.c +++ b/arch/x86/mm/mem_encrypt.c @@ -13,6 +13,8 @@ #include #include #include +#include +#include #include #include @@ -240,3 +242,18 @@ void __init sme_early_init(void) for (i = 0; i < ARRAY_SIZE(protection_map); i++) protection_map[i] = __pgprot(pgprot_val(protection_map[i]) | sme_me_mask); } + +/* Architecture __weak replacement functions */ +void __init mem_encrypt_init(void) +{ + if (!sme_me_mask) + return; + + /* Make SWIOTLB use an unencrypted DMA area */ + swiotlb_clear_encryption(); +} + +void swiotlb_set_mem_unenc(void *vaddr, unsigned long size) +{ + sme_set_mem_unenc(vaddr, size); +} diff --git a/include/linux/swiotlb.h b/include/linux/swiotlb.h index 5f81f8a..5c909fc 100644 --- a/include/linux/swiotlb.h +++ b/include/linux/swiotlb.h @@ -29,6 +29,7 @@ int swiotlb_init_with_tbl(char *tlb, unsigned long nslabs, int verbose); extern unsigned long swiotlb_nr_tbl(void); unsigned long swiotlb_size_or_default(void); extern int swiotlb_late_init_with_tbl(char *tlb, unsigned long nslabs); +extern void __init swiotlb_clear_encryption(void); /* * Enumeration for sync targets diff --git a/init/main.c b/init/main.c index a8a58e2..ae37f0d 100644 --- a/init/main.c +++ b/init/main.c @@ -458,6 +458,10 @@ void __init __weak thread_stack_cache_init(void) } #endif +void __init __weak mem_encrypt_init(void) +{ +} + /* * Set up kernel memory allocators */ @@ -598,6 +602,15 @@ asmlinkage __visible void __init start_kernel(void) */ locking_selftest(); + /* + * This needs to be called before any devices perform DMA + * operations that might use the swiotlb bounce buffers. + * This call will mark the bounce buffers as un-encrypted so + * that their usage will not cause "plain-text" data to be + * decrypted when accessed. + */ + mem_encrypt_init(); + #ifdef CONFIG_BLK_DEV_INITRD if (initrd_start && !initrd_below_start_ok && page_to_pfn(virt_to_page((void *)initrd_start)) < min_low_pfn) { diff --git a/lib/swiotlb.c b/lib/swiotlb.c index 22e13a0..638e99c 100644 --- a/lib/swiotlb.c +++ b/lib/swiotlb.c @@ -30,6 +30,7 @@ #include #include #include +#include #include #include @@ -131,6 +132,17 @@ unsigned long swiotlb_size_or_default(void) return size ? size : (IO_TLB_DEFAULT_SIZE); } +void __weak swiotlb_set_mem_unenc(void *vaddr, unsigned long size) +{ +} + +/* For swiotlb, clear memory encryption mask from dma addresses */ +static dma_addr_t swiotlb_phys_to_dma(struct device *hwdev, + phys_addr_t address) +{ + return phys_to_dma(hwdev, address) & ~sme_me_mask; +} + /* Note that this doesn't work with highmem page */ static dma_addr_t swiotlb_virt_to_bus(struct device *hwdev, volatile void *address) @@ -159,6 +171,31 @@ void swiotlb_print_info(void) bytes >> 20, vstart, vend - 1); } +/* + * If memory encryption is active, the DMA address for an encrypted page may + * be beyond the range of the device. If bounce buffers are required be sure + * that they are not on an encrypted page. This should be called before the + * iotlb area is used. + */ +void __init swiotlb_clear_encryption(void) +{ + void *vaddr; + unsigned long bytes; + + if (no_iotlb_memory || !io_tlb_start || late_alloc) + return; + + vaddr = phys_to_virt(io_tlb_start); + bytes = PAGE_ALIGN(io_tlb_nslabs << IO_TLB_SHIFT); + swiotlb_set_mem_unenc(vaddr, bytes); + memset(vaddr, 0, bytes); + + vaddr = phys_to_virt(io_tlb_overflow_buffer); + bytes = PAGE_ALIGN(io_tlb_overflow); + swiotlb_set_mem_unenc(vaddr, bytes); + memset(vaddr, 0, bytes); +} + int __init swiotlb_init_with_tbl(char *tlb, unsigned long nslabs, int verbose) { void *v_overflow_buffer; @@ -294,6 +331,8 @@ swiotlb_late_init_with_tbl(char *tlb, unsigned long nslabs) io_tlb_start = virt_to_phys(tlb); io_tlb_end = io_tlb_start + bytes; + /* Keep TLB in unencrypted memory if memory encryption is active */ + swiotlb_set_mem_unenc(tlb, bytes); memset(tlb, 0, bytes); /* @@ -304,6 +343,9 @@ swiotlb_late_init_with_tbl(char *tlb, unsigned long nslabs) if (!v_overflow_buffer) goto cleanup2; + /* Keep overflow in unencrypted memory if memory encryption is active */ + swiotlb_set_mem_unenc(v_overflow_buffer, io_tlb_overflow); + memset(v_overflow_buffer, 0, io_tlb_overflow); io_tlb_overflow_buffer = virt_to_phys(v_overflow_buffer); /* @@ -541,7 +583,7 @@ static phys_addr_t map_single(struct device *hwdev, phys_addr_t phys, size_t size, enum dma_data_direction dir) { - dma_addr_t start_dma_addr = phys_to_dma(hwdev, io_tlb_start); + dma_addr_t start_dma_addr = swiotlb_phys_to_dma(hwdev, io_tlb_start); return swiotlb_tbl_map_single(hwdev, start_dma_addr, phys, size, dir); } @@ -659,7 +701,7 @@ swiotlb_alloc_coherent(struct device *hwdev, size_t size, goto err_warn; ret = phys_to_virt(paddr); - dev_addr = phys_to_dma(hwdev, paddr); + dev_addr = swiotlb_phys_to_dma(hwdev, paddr); /* Confirm address can be DMA'd by device */ if (dev_addr + size - 1 > dma_mask) { @@ -758,15 +800,15 @@ dma_addr_t swiotlb_map_page(struct device *dev, struct page *page, map = map_single(dev, phys, size, dir); if (map == SWIOTLB_MAP_ERROR) { swiotlb_full(dev, size, dir, 1); - return phys_to_dma(dev, io_tlb_overflow_buffer); + return swiotlb_phys_to_dma(dev, io_tlb_overflow_buffer); } - dev_addr = phys_to_dma(dev, map); + dev_addr = swiotlb_phys_to_dma(dev, map); /* Ensure that the address returned is DMA'ble */ if (!dma_capable(dev, dev_addr, size)) { swiotlb_tbl_unmap_single(dev, map, size, dir); - return phys_to_dma(dev, io_tlb_overflow_buffer); + return swiotlb_phys_to_dma(dev, io_tlb_overflow_buffer); } return dev_addr; @@ -901,7 +943,7 @@ swiotlb_map_sg_attrs(struct device *hwdev, struct scatterlist *sgl, int nelems, sg_dma_len(sgl) = 0; return 0; } - sg->dma_address = phys_to_dma(hwdev, map); + sg->dma_address = swiotlb_phys_to_dma(hwdev, map); } else sg->dma_address = dev_addr; sg_dma_len(sg) = sg->length; @@ -985,7 +1027,7 @@ EXPORT_SYMBOL(swiotlb_sync_sg_for_device); int swiotlb_dma_mapping_error(struct device *hwdev, dma_addr_t dma_addr) { - return (dma_addr == phys_to_dma(hwdev, io_tlb_overflow_buffer)); + return (dma_addr == swiotlb_phys_to_dma(hwdev, io_tlb_overflow_buffer)); } EXPORT_SYMBOL(swiotlb_dma_mapping_error); @@ -998,6 +1040,6 @@ EXPORT_SYMBOL(swiotlb_dma_mapping_error); int swiotlb_dma_supported(struct device *hwdev, u64 mask) { - return phys_to_dma(hwdev, io_tlb_end - 1) <= mask; + return swiotlb_phys_to_dma(hwdev, io_tlb_end - 1) <= mask; } EXPORT_SYMBOL(swiotlb_dma_supported); -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: email@kvack.org