From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1753751AbcDZW6b (ORCPT ); Tue, 26 Apr 2016 18:58:31 -0400 Received: from mail-bl2on0077.outbound.protection.outlook.com ([65.55.169.77]:39240 "EHLO na01-bl2-obe.outbound.protection.outlook.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1753169AbcDZW6Y (ORCPT ); Tue, 26 Apr 2016 18:58:24 -0400 Authentication-Results: vger.kernel.org; dkim=none (message not signed) header.d=none;vger.kernel.org; dmarc=none action=none header.from=amd.com; From: Tom Lendacky Subject: [RFC PATCH v1 13/18] x86: DMA support for memory encryption To: , , , , , , , , CC: Radim =?utf-8?b?S3LEjW3DocWZ?= , Arnd Bergmann , Jonathan Corbet , Matt Fleming , Joerg Roedel , "Konrad Rzeszutek Wilk" , Paolo Bonzini , "Ingo Molnar" , Borislav Petkov , "H. Peter Anvin" , Andrey Ryabinin , "Alexander Potapenko" , Thomas Gleixner , "Dmitry Vyukov" Date: Tue, 26 Apr 2016 17:58:12 -0500 Message-ID: <20160426225812.13567.91220.stgit@tlendack-t1.amdoffice.net> In-Reply-To: <20160426225553.13567.19459.stgit@tlendack-t1.amdoffice.net> References: <20160426225553.13567.19459.stgit@tlendack-t1.amdoffice.net> User-Agent: StGit/0.17.1-dirty MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: 7bit X-Originating-IP: [165.204.77.1] X-ClientProxiedBy: BLUPR05CA0043.namprd05.prod.outlook.com (10.141.20.13) To DM3PR1201MB1117.namprd12.prod.outlook.com (10.164.198.17) X-MS-Office365-Filtering-Correlation-Id: 2b4f3151-2b52-46de-1973-08d36e264273 X-Microsoft-Exchange-Diagnostics: 1;DM3PR1201MB1117;2:Dqa1ed390gqiH80u8KrZ+kFLV1mkyZeSQTXflXjJaIsN12XALUD6+SO81NphzV2CUKG9PUsAv6UBWQYsk2h+ltCJJIZcAnjw9/02R0bSzF/b3IqGBOHggA1Segb24WMk2DuH+CT+9lkAfT98ejSt/eQ/URgCVR/86W5EM3/+vJ6v5AFr7NboW+renR+a5Ofc;3:d6Z1Y4aN4ME9y938g/fbw64BG5WNmKWMBOruZSHDIfyN+tEawQUeUAgiA/cJOiqbeCaWjPGIZceMrEhQDb+KUwzjo0Kw2aqQ30DC+dVfeDfavNy8eH+YbtqgtgV4AstB X-Microsoft-Antispam: UriScan:;BCL:0;PCL:0;RULEID:;SRVR:DM3PR1201MB1117; X-Microsoft-Exchange-Diagnostics: 1;DM3PR1201MB1117;25:776x0vZi0hbDVBFktQCtQ/3d9LuZ271X7wtqr+Nr7Z5TMAmHUU7Fbg8qjHqbOYiOpo3lyGjguiRZQnwT3vDU3AUjFIp2eWKI/O4mHNfEqnXFoWH6s0tVr52EwCSJX0LaIfTgZcjmYKd/MikEnfZILnxviAIoeS3crKAnVkFszaoHtc9+kY/FpzXdHJWPKkX2lNmOnPSBAfQ5KUfsIhRYMCIGl8i3S7z7V2X3dWLekZX3TA2hLFr63N5UL/3YtW0DdsnCnV7s8Q6BWt6V/rmd7ppZ/dJ2IJa80rcnh8KTb8wI0DRsWuc3YQ+Xg7hS5FB/tOlVaCHnEZ9uKyXqBtSHTYdyXpi3c6lhj2VyFril7wC5Q94LnHlff7CnxTR8s5O5mRS1bsiMu8J0WiDwz7AEoA1XbgVL4lAvnBRGweZTJMwLXMxaU/hkFsP7/XkhdncsWj2+JT8UkRXx2GZN4PSsdRh0duF/MsAXSoafS3KHiCWhUBwU981o3Gme8sbwwV1oudA5IjSoRimqsysr/Co8AgnyM1ucvWSIIUD5Gh1wUjU1zl7SUMWkCj/+hLeSbP8+/fsgkTQttVyKwffpRZl1nW/Tyzndj4jqPL2ZobroUXdT74P4oWxKvYXiDVcS1P1lUSK6sVpJKHzkA/MzYYZm7g== X-Microsoft-Exchange-Diagnostics: 1;DM3PR1201MB1117;20:QGgKtEG2iDSOXylyjoWjTVdNeI4BZ5LXIsRdYaaf506t+0jUXzjMmqQCCxp22ALC1r2dt2yv4x2XMQtB9GUG+42no+B+zbmBqhREO1z8nut8MGCdhOJfl32z3aNU7SLSBgExcWNYI2BUAHVoFOGV5DVbnH87gHBt13OW8BB0wzzePBsqsC9IXUNNNcf8TbqmWEGIkbqq1QJigjCWy/TWbMDFuW+cjj9kiEfISr5EfotX+xPV6EssrFmxWktNj10wpu2BBJiUcCSk6gyXJE5/Ve7RrErwIphD6jC8SApLppkmTz/+NvLjkeZwgB4ua3LOmOTsFLxqatmw55TBjRhSfvcjDanL3F7kyFnATNyiGtzKLKGKRAfJQrVMDTIoPuCeBE4MdWQnpsf4spoyQ0koXBa9R0rK2YrupEH9ZQxbxn9URyIbqUmKb/wPemSJ05pIZYJxbKH//elbsftWGmF8llq+8QIFrk1gWMfVhy+VPsKyww6a+ecDn51w13CDFZTc;4:G8OARZwyiK7sbZdCKM+c3OB/6EzmLKMRO4gc+B9Us6Bduhlk+eA7jFfVPWJXBXuk9JXaaQ+YJYLXXU3z+PlhJfqu6b9bUK16C5AUVLy0WIiHE4gq1Hr4pt+F1+7JLmsFh4Z6TozB3RTzyOqa34lfR0FcuKrlxy9RMVaC+En+r4Thu5UYHmOWYoJCXaXLFI1SVgkGsv9s+oBJ8vGcDur/dngmSslVkn263vw/+OLwNpcsluTh7nEAP7O42MlDbvr9yl4MbC63DCAm4w+3dgAWDy5fkTVOi2FzuGq6C07LoygB/7RKAVjZEXBNbEQn0f55MLc401JLf8vti++u1Q1i2lwF9IR9f6fv7y8eNEe9/WsekcYzBo2fYNdjxreUr2Y2h9v03awbhQssBpCAT0me3AFEy/0LgC9DjhNl0cgPaIo= X-Microsoft-Antispam-PRVS: X-Exchange-Antispam-Report-Test: UriScan:; X-Exchange-Antispam-Report-CFA-Test: BCL:0;PCL:0;RULEID:(9101521072)(601004)(2401047)(5005006)(8121501046)(10201501046)(3002001)(6055026);SRVR:DM3PR1201MB1117;BCL:0;PCL:0;RULEID:;SRVR:DM3PR1201MB1117; X-Forefront-PRVS: 0924C6A0D5 X-Forefront-Antispam-Report: SFV:NSPM;SFS:(10009020)(4630300001)(6009001)(5008740100001)(50986999)(54356999)(42186005)(76176999)(5004730100002)(53416004)(103116003)(97746001)(230700001)(81166005)(9686002)(2950100001)(77096005)(2201001)(92566002)(66066001)(19580405001)(19580395003)(33646002)(229853001)(4001350100001)(86362001)(47776003)(1076002)(50466002)(5001770100001)(4326007)(6116002)(189998001)(2906002)(23676002)(1096002)(586003)(71626007)(217873001)(473944003);DIR:OUT;SFP:1101;SCL:1;SRVR:DM3PR1201MB1117;H:tlendack-t1.amdoffice.net;FPR:;SPF:None;MLV:sfv;LANG:en; X-Microsoft-Exchange-Diagnostics: =?utf-8?B?MTtETTNQUjEyMDFNQjExMTc7MjM6Q3FyeWZuWEc1L1d0YUIza3ZWQml2dUxh?= =?utf-8?B?dGdHK2M2RWs2N3Q5aHVsS2dKa25xQW95ZHJpTnlBN080enJvbGR0OWcxQXhm?= =?utf-8?B?cldTcnMzOStPeVg5allwanUzeEZPUnZBVk81WmdnaUhpM2RJTEN4WGFmYksw?= =?utf-8?B?cW4wRDhWOG80dFhvY0xQUUpaem5vcVhTdGRYcEh4cW54SUNCVGZWcDFwZVFi?= =?utf-8?B?QXNhZ2poWVgwZ0puVkgraENzajVSQllwc0dxMnVncU1EdkNRN2VjZXJVb0Fs?= =?utf-8?B?cFdpT3ZMOXMyUHRpenF5blYxTC82bkc5Yms2SjVaQnVqTmtBYVBRQndEcUZH?= =?utf-8?B?VHlrOFhhcTZ1a0JyNWFBMXlhcmhVenE0Zm5CaXZPZ3JaeC9XWEpRMUpXaGF0?= =?utf-8?B?S2FZYUw3WVQxcUZMK21kZlQ4YVVzTUIrbjdJeTB4c3ZESmJNM0N5OFBJYUxR?= =?utf-8?B?eC9JdXF4YzZLUHpXU2ZxZ25WVklaSjNZdk5Hczd4ZzREYzNaaWxqRDJzdUYv?= =?utf-8?B?TEo0cUJqcEZkbHcwNzhnT0EwYkcyNldYTXIwUFNCam4wQ0hGNG01ZFZNWm9Y?= =?utf-8?B?d0dMRm5VUGttbVRqVDVPdjc0YTNpbGZaZTNCS0E5TjBJdkpJOEZEMC90eXU3?= =?utf-8?B?OVBIVzNmcG9FY0YvYjBaM056c2RjM29UeDMyNDNxMkdCeUl4bnVUYXI1SDNF?= =?utf-8?B?Sk1RbUR3alBpV3pVZkVwOGVQQ0pZbDkvMU9xVVBxanpBM2crNUVSa3Q4YzZW?= =?utf-8?B?OHlsZHFlT3VTMGFNTjdsNDZUem11R2I1c1N5SmlEc1F5a0plemIrYjVtdXJ4?= =?utf-8?B?R0tBK2JvTTRTTXZJMGlNODNIUUdKbUhlV1I4M0M1RGp2VVVKaEFEbkt4MGlo?= =?utf-8?B?SUVXRjZmR2ZnYmJxL0RzSnozR3Z1L21aYjNUWG9tQjJQM2dmUWszMDYrWTlB?= =?utf-8?B?SFRiSHc5R1V5YVFYL09CbFdFUFFLdi8xRE1takczM3ZSWXJJMHRwQnQzeGlu?= =?utf-8?B?K3ZWeW11MjJSWGVSKzNVeWhpUktmRHUxa1VhVUI3dWJLeEN3VmxoTllJS1hu?= =?utf-8?B?RmFPdVl6MWk4R1Q0MG1UTkljaExwUUk3cnEvcnVpRzNPZjdFb1h5RjZNQzdB?= =?utf-8?B?V3NEbzQrMFFMLzZ0Ri96Y0F3SzZQM2ZLVXN2MkNhRDZTWWkwMmRmN3FhdTlt?= =?utf-8?B?aVl4TWZKZE1TMWxKYmplS3JXNjlkUXJPKzh3WHduNTgwUUVwVHFnck9QUWc4?= =?utf-8?B?c1hvNko1VEVGaGZ6S0gzanFQTDlrRk5oQWIwNHViUmlqS3RFVzhidUNLeWtP?= =?utf-8?B?Szh3TjJjWi8zK2kxMVN4SGh3Zktxd0ZWZU84VFlRQTZCeTNzNUJqcTdVVElz?= =?utf-8?B?bWtqYTNkMVlvTTBycFQ5UC9aTnpJYnRVVW9tR1NVZGp1Q3c5UDFDTTJCdjVP?= =?utf-8?Q?pgV99lzo=3D?= X-Microsoft-Exchange-Diagnostics: 1;DM3PR1201MB1117;5:OwskswXaY3jigbWSm4uvbKkfHN/eAdk8zYmRKsLsuPZWP0tyMW3TOQpPt7BPMbBwMayRvD56hvuq71jKWRDq3xWswvmoLedj8dFyzvgRb5XyqWOsJFjJlSLdWbc74tQZF9mGnyDNxsdEKYenNnDBnQ==;24:HKrMBbdqKi8x7VbkDErHni4wsNZS4TrWgP73KV4u3w8YfA3BAp4sahyO4hQO3IYck3msZp/sC0Zh+jwl+rKXYsOZHx+DEVrTuMLEXE5HZjo=;7:JzY1r/d4E6ITiQ8ZSlq2EGlQNL678udVWmeY+VPAvU62BOGa8UgTfjTVLOu0QEQOWA8swVBDMmDN1O/F1R9RkJHuodniB4s6ExvSt0I7qxxOaR1mkk5IHqGxUEbpkw6D2uOwb++JlDYkAYClVQ+RO2d9O/0KrVjA8pZuBfvAeKQ=;20:/ibC6HJCKPoBTkLe81v3nuaiwIORDi81NVqQ4JJEvnjL/hkBQiJ+qzLtqmpd1eqMlx8kJkY1rQzUS35t5YTXQm+2TuMGy1Ywn9Rl4rgiueHvPDm7YdScP+1AoIinHuXDfBshCEQTRIsvNO1sVgRBqbNCgzCIk+rdH7o14YnT9sVZq6npZMlMIrHMjOFXwXAAMIWQlU5bT62Qea6CqiDTAUhVFicmRbHQWDgCwUuEsTzuX9y6RFxDeXgAQHHB35E5 X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 26 Apr 2016 22:58:17.1710 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM3PR1201MB1117 Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Since DMA addresses will effectively look like 48-bit addresses when the memory encryption mask is set, SWIOTLB is needed if the DMA mask of the device performing the DMA does not support 48-bits. SWIOTLB will be initialized to create un-encrypted bounce buffers for use by these devices. Signed-off-by: Tom Lendacky --- arch/x86/include/asm/dma-mapping.h | 5 ++- arch/x86/include/asm/mem_encrypt.h | 5 +++ arch/x86/kernel/pci-dma.c | 11 ++++-- arch/x86/kernel/pci-nommu.c | 2 + arch/x86/kernel/pci-swiotlb.c | 8 +++-- arch/x86/mm/mem_encrypt.c | 21 ++++++++++++ include/linux/swiotlb.h | 1 + init/main.c | 6 +++ lib/swiotlb.c | 64 ++++++++++++++++++++++++++++++++---- 9 files changed, 106 insertions(+), 17 deletions(-) diff --git a/arch/x86/include/asm/dma-mapping.h b/arch/x86/include/asm/dma-mapping.h index 3a27b93..33a4f6d 100644 --- a/arch/x86/include/asm/dma-mapping.h +++ b/arch/x86/include/asm/dma-mapping.h @@ -13,6 +13,7 @@ #include #include #include +#include #ifdef CONFIG_ISA # define ISA_DMA_BIT_MASK DMA_BIT_MASK(24) @@ -70,12 +71,12 @@ static inline bool dma_capable(struct device *dev, dma_addr_t addr, size_t size) static inline dma_addr_t phys_to_dma(struct device *dev, phys_addr_t paddr) { - return paddr; + return paddr | sme_me_mask; } static inline phys_addr_t dma_to_phys(struct device *dev, dma_addr_t daddr) { - return daddr; + return daddr & ~sme_me_mask; } #endif /* CONFIG_X86_DMA_REMAP */ diff --git a/arch/x86/include/asm/mem_encrypt.h b/arch/x86/include/asm/mem_encrypt.h index 42868f5..d17d8cf 100644 --- a/arch/x86/include/asm/mem_encrypt.h +++ b/arch/x86/include/asm/mem_encrypt.h @@ -37,6 +37,11 @@ void __init *sme_early_memremap(resource_size_t paddr, void __init sme_early_init(void); /* Architecture __weak replacement functions */ +void __init mem_encrypt_init(void); + +unsigned long swiotlb_get_me_mask(void); +void swiotlb_set_mem_dec(void *vaddr, unsigned long size); + void __init *efi_me_early_memremap(resource_size_t paddr, unsigned long size); diff --git a/arch/x86/kernel/pci-dma.c b/arch/x86/kernel/pci-dma.c index 6ba014c..bd1daae 100644 --- a/arch/x86/kernel/pci-dma.c +++ b/arch/x86/kernel/pci-dma.c @@ -92,9 +92,12 @@ again: /* CMA can be used only in the context which permits sleeping */ if (gfpflags_allow_blocking(flag)) { page = dma_alloc_from_contiguous(dev, count, get_order(size)); - if (page && page_to_phys(page) + size > dma_mask) { - dma_release_from_contiguous(dev, page, count); - page = NULL; + if (page) { + addr = phys_to_dma(dev, page_to_phys(page)); + if (addr + size > dma_mask) { + dma_release_from_contiguous(dev, page, count); + page = NULL; + } } } /* fallback */ @@ -103,7 +106,7 @@ again: if (!page) return NULL; - addr = page_to_phys(page); + addr = phys_to_dma(dev, page_to_phys(page)); if (addr + size > dma_mask) { __free_pages(page, get_order(size)); diff --git a/arch/x86/kernel/pci-nommu.c b/arch/x86/kernel/pci-nommu.c index da15918..ca2b820 100644 --- a/arch/x86/kernel/pci-nommu.c +++ b/arch/x86/kernel/pci-nommu.c @@ -30,7 +30,7 @@ static dma_addr_t nommu_map_page(struct device *dev, struct page *page, enum dma_data_direction dir, struct dma_attrs *attrs) { - dma_addr_t bus = page_to_phys(page) + offset; + dma_addr_t bus = phys_to_dma(dev, page_to_phys(page)) + offset; WARN_ON(size == 0); if (!check_addr("map_single", dev, bus, size)) return DMA_ERROR_CODE; diff --git a/arch/x86/kernel/pci-swiotlb.c b/arch/x86/kernel/pci-swiotlb.c index 7c577a1..0ae083d 100644 --- a/arch/x86/kernel/pci-swiotlb.c +++ b/arch/x86/kernel/pci-swiotlb.c @@ -12,6 +12,8 @@ #include #include #include +#include + int swiotlb __read_mostly; void *x86_swiotlb_alloc_coherent(struct device *hwdev, size_t size, @@ -64,13 +66,15 @@ static struct dma_map_ops swiotlb_dma_ops = { * pci_swiotlb_detect_override - set swiotlb to 1 if necessary * * This returns non-zero if we are forced to use swiotlb (by the boot - * option). + * option). If memory encryption is enabled then swiotlb will be set + * to 1 so that bounce buffers are allocated and used for devices that + * do not support the addressing range required for the encryption mask. */ int __init pci_swiotlb_detect_override(void) { int use_swiotlb = swiotlb | swiotlb_force; - if (swiotlb_force) + if (swiotlb_force || sme_me_mask) swiotlb = 1; return use_swiotlb; diff --git a/arch/x86/mm/mem_encrypt.c b/arch/x86/mm/mem_encrypt.c index 7d56d1b..594dc65 100644 --- a/arch/x86/mm/mem_encrypt.c +++ b/arch/x86/mm/mem_encrypt.c @@ -12,6 +12,8 @@ #include #include +#include +#include #include #include @@ -168,6 +170,25 @@ void __init sme_early_init(void) } /* Architecture __weak replacement functions */ +void __init mem_encrypt_init(void) +{ + if (!sme_me_mask) + return; + + /* Make SWIOTLB use an unencrypted DMA area */ + swiotlb_clear_encryption(); +} + +unsigned long swiotlb_get_me_mask(void) +{ + return sme_me_mask; +} + +void swiotlb_set_mem_dec(void *vaddr, unsigned long size) +{ + sme_set_mem_dec(vaddr, size); +} + void __init *efi_me_early_memremap(resource_size_t paddr, unsigned long size) { diff --git a/include/linux/swiotlb.h b/include/linux/swiotlb.h index 017fced..121b9de 100644 --- a/include/linux/swiotlb.h +++ b/include/linux/swiotlb.h @@ -30,6 +30,7 @@ int swiotlb_init_with_tbl(char *tlb, unsigned long nslabs, int verbose); extern unsigned long swiotlb_nr_tbl(void); unsigned long swiotlb_size_or_default(void); extern int swiotlb_late_init_with_tbl(char *tlb, unsigned long nslabs); +extern void __init swiotlb_clear_encryption(void); /* * Enumeration for sync targets diff --git a/init/main.c b/init/main.c index b3c6e36..1013d1c 100644 --- a/init/main.c +++ b/init/main.c @@ -458,6 +458,10 @@ void __init __weak thread_info_cache_init(void) } #endif +void __init __weak mem_encrypt_init(void) +{ +} + /* * Set up kernel memory allocators */ @@ -597,6 +601,8 @@ asmlinkage __visible void __init start_kernel(void) */ locking_selftest(); + mem_encrypt_init(); + #ifdef CONFIG_BLK_DEV_INITRD if (initrd_start && !initrd_below_start_ok && page_to_pfn(virt_to_page((void *)initrd_start)) < min_low_pfn) { diff --git a/lib/swiotlb.c b/lib/swiotlb.c index 76f29ec..339ffdc 100644 --- a/lib/swiotlb.c +++ b/lib/swiotlb.c @@ -131,6 +131,26 @@ unsigned long swiotlb_size_or_default(void) return size ? size : (IO_TLB_DEFAULT_SIZE); } +/* + * Support for memory encryption. If memory encryption is supported, then an + * override to these functions will be provided. + */ +unsigned long __weak swiotlb_get_me_mask(void) +{ + return 0; +} + +void __weak swiotlb_set_mem_dec(void *vaddr, unsigned long size) +{ +} + +/* For swiotlb, clear memory encryption mask from dma addresses */ +static dma_addr_t swiotlb_phys_to_dma(struct device *hwdev, + phys_addr_t address) +{ + return phys_to_dma(hwdev, address) & ~swiotlb_get_me_mask(); +} + /* Note that this doesn't work with highmem page */ static dma_addr_t swiotlb_virt_to_bus(struct device *hwdev, volatile void *address) @@ -159,6 +179,30 @@ void swiotlb_print_info(void) bytes >> 20, vstart, vend - 1); } +/* + * If memory encryption is active, the DMA address for an encrypted page may + * be beyond the range of the device. If bounce buffers are required be sure + * that they are not on an encrypted page. This should be called before the + * iotlb area is used. + */ +void __init swiotlb_clear_encryption(void) +{ + void *vaddr; + unsigned long bytes; + + if (no_iotlb_memory || !io_tlb_start || late_alloc) + return; + + vaddr = phys_to_virt(io_tlb_start); + bytes = PAGE_ALIGN(io_tlb_nslabs << IO_TLB_SHIFT); + swiotlb_set_mem_dec(vaddr, bytes); + memset(vaddr, 0, bytes); + + vaddr = phys_to_virt(io_tlb_overflow_buffer); + bytes = PAGE_ALIGN(io_tlb_overflow); + swiotlb_set_mem_dec(vaddr, bytes); +} + int __init swiotlb_init_with_tbl(char *tlb, unsigned long nslabs, int verbose) { void *v_overflow_buffer; @@ -294,6 +338,8 @@ swiotlb_late_init_with_tbl(char *tlb, unsigned long nslabs) io_tlb_start = virt_to_phys(tlb); io_tlb_end = io_tlb_start + bytes; + /* Keep TLB in unencrypted memory if memory encryption is active */ + swiotlb_set_mem_dec(tlb, bytes); memset(tlb, 0, bytes); /* @@ -304,6 +350,8 @@ swiotlb_late_init_with_tbl(char *tlb, unsigned long nslabs) if (!v_overflow_buffer) goto cleanup2; + /* Keep overflow in unencrypted memory if memory encryption is active */ + swiotlb_set_mem_dec(v_overflow_buffer, io_tlb_overflow); io_tlb_overflow_buffer = virt_to_phys(v_overflow_buffer); /* @@ -541,7 +589,7 @@ static phys_addr_t map_single(struct device *hwdev, phys_addr_t phys, size_t size, enum dma_data_direction dir) { - dma_addr_t start_dma_addr = phys_to_dma(hwdev, io_tlb_start); + dma_addr_t start_dma_addr = swiotlb_phys_to_dma(hwdev, io_tlb_start); return swiotlb_tbl_map_single(hwdev, start_dma_addr, phys, size, dir); } @@ -659,7 +707,7 @@ swiotlb_alloc_coherent(struct device *hwdev, size_t size, goto err_warn; ret = phys_to_virt(paddr); - dev_addr = phys_to_dma(hwdev, paddr); + dev_addr = swiotlb_phys_to_dma(hwdev, paddr); /* Confirm address can be DMA'd by device */ if (dev_addr + size - 1 > dma_mask) { @@ -758,15 +806,15 @@ dma_addr_t swiotlb_map_page(struct device *dev, struct page *page, map = map_single(dev, phys, size, dir); if (map == SWIOTLB_MAP_ERROR) { swiotlb_full(dev, size, dir, 1); - return phys_to_dma(dev, io_tlb_overflow_buffer); + return swiotlb_phys_to_dma(dev, io_tlb_overflow_buffer); } - dev_addr = phys_to_dma(dev, map); + dev_addr = swiotlb_phys_to_dma(dev, map); /* Ensure that the address returned is DMA'ble */ if (!dma_capable(dev, dev_addr, size)) { swiotlb_tbl_unmap_single(dev, map, size, dir); - return phys_to_dma(dev, io_tlb_overflow_buffer); + return swiotlb_phys_to_dma(dev, io_tlb_overflow_buffer); } return dev_addr; @@ -901,7 +949,7 @@ swiotlb_map_sg_attrs(struct device *hwdev, struct scatterlist *sgl, int nelems, sg_dma_len(sgl) = 0; return 0; } - sg->dma_address = phys_to_dma(hwdev, map); + sg->dma_address = swiotlb_phys_to_dma(hwdev, map); } else sg->dma_address = dev_addr; sg_dma_len(sg) = sg->length; @@ -984,7 +1032,7 @@ EXPORT_SYMBOL(swiotlb_sync_sg_for_device); int swiotlb_dma_mapping_error(struct device *hwdev, dma_addr_t dma_addr) { - return (dma_addr == phys_to_dma(hwdev, io_tlb_overflow_buffer)); + return (dma_addr == swiotlb_phys_to_dma(hwdev, io_tlb_overflow_buffer)); } EXPORT_SYMBOL(swiotlb_dma_mapping_error); @@ -997,6 +1045,6 @@ EXPORT_SYMBOL(swiotlb_dma_mapping_error); int swiotlb_dma_supported(struct device *hwdev, u64 mask) { - return phys_to_dma(hwdev, io_tlb_end - 1) <= mask; + return swiotlb_phys_to_dma(hwdev, io_tlb_end - 1) <= mask; } EXPORT_SYMBOL(swiotlb_dma_supported);