[PATCH] iommu/vt-d: Implement dma_[un]map_resource()

[PATCH] iommu/vt-d: Implement dma_[un]map_resource()

[Logan Gunthorpe]

Currently the Intel IOMMU uses the default dma_[un]map_resource()
implementations does nothing and simply returns the physical address
unmodified.

However, this doesn't create the IOVA entries necessary for addresses
mapped this way to work when the IOMMU is enabled. Thus, when the
IOMMU is enabled, drivers relying on dma_map_resource() will trigger
DMAR errors. We see this when running ntb_transport with the IOMMU
enabled, DMA, and switchtec hardware.

The implementation for intel_map_resource() is nearly identical to
intel_map_page(), we just have to re-create __intel_map_single().
dma_unmap_resource() uses intel_unmap_page() directly as the
functions are identical.

Signed-off-by: Logan Gunthorpe <logang@deltatee.com>
Cc: David Woodhouse <dwmw2@infradead.org>
Cc: Joerg Roedel <joro@8bytes.org>
---
 drivers/iommu/intel-iommu.c | 25 ++++++++++++++++++++-----
 1 file changed, 20 insertions(+), 5 deletions(-)

diff --git a/drivers/iommu/intel-iommu.c b/drivers/iommu/intel-iommu.c
index 2bd9ac285c0d..81fb265f7730 100644
--- a/drivers/iommu/intel-iommu.c
+++ b/drivers/iommu/intel-iommu.c
@@ -3649,11 +3649,9 @@ static int iommu_no_mapping(struct device *dev)
 	return 0;
 }
 
-static dma_addr_t __intel_map_page(struct device *dev, struct page *page,
-				   unsigned long offset, size_t size, int dir,
-				   u64 dma_mask)
+static dma_addr_t __intel_map_single(struct device *dev, phys_addr_t paddr,
+				     size_t size, int dir, u64 dma_mask)
 {
-	phys_addr_t paddr = page_to_phys(page) + offset;
 	struct dmar_domain *domain;
 	phys_addr_t start_paddr;
 	unsigned long iova_pfn;
@@ -3710,6 +3708,14 @@ static dma_addr_t __intel_map_page(struct device *dev, struct page *page,
 	return DMA_MAPPING_ERROR;
 }
 
+static dma_addr_t __intel_map_page(struct device *dev, struct page *page,
+				   unsigned long offset, size_t size, int dir,
+				   u64 dma_mask)
+{
+	return __intel_map_single(dev, page_to_phys(page) + offset, size, dir,
+				  dma_mask);
+}
+
 static dma_addr_t intel_map_page(struct device *dev, struct page *page,
 				 unsigned long offset, size_t size,
 				 enum dma_data_direction dir,
@@ -3718,6 +3724,13 @@ static dma_addr_t intel_map_page(struct device *dev, struct page *page,
 	return __intel_map_page(dev, page, offset, size, dir, *dev->dma_mask);
 }
 
+static dma_addr_t intel_map_resource(struct device *dev, phys_addr_t phys_addr,
+				     size_t size, enum dma_data_direction dir,
+				     unsigned long attrs)
+{
+	return __intel_map_single(dev, phys_addr, size, dir, *dev->dma_mask);
+}
+
 static void intel_unmap(struct device *dev, dma_addr_t dev_addr, size_t size)
 {
 	struct dmar_domain *domain;
@@ -3924,6 +3937,8 @@ static const struct dma_map_ops intel_dma_ops = {
 	.unmap_sg = intel_unmap_sg,
 	.map_page = intel_map_page,
 	.unmap_page = intel_unmap_page,
+	.map_resource = intel_map_resource,
+	.unmap_resource = intel_unmap_page,
 	.dma_supported = dma_direct_supported,
 };
 
@@ -4080,7 +4095,7 @@ static int init_iommu_hw(void)
 				iommu_disable_protect_mem_regions(iommu);
 			continue;
 		}
-	
+
 		iommu_flush_write_buffer(iommu);
 
 		iommu_set_root_entry(iommu);
-- 
2.19.0

Re: [PATCH] iommu/vt-d: Implement dma_[un]map_resource()

[Christoph Hellwig]

On Fri, Jan 18, 2019 at 05:05:59PM -0700, Logan Gunthorpe wrote:
> However, this doesn't create the IOVA entries necessary for addresses
> mapped this way to work when the IOMMU is enabled. Thus, when the
> IOMMU is enabled, drivers relying on dma_map_resource() will trigger
> DMAR errors. We see this when running ntb_transport with the IOMMU
> enabled, DMA, and switchtec hardware.

Which resources do you plan to map?  At least for PCIe P2P adding
an address translation seems wrong to me.

> 
> The implementation for intel_map_resource() is nearly identical to
> intel_map_page(), we just have to re-create __intel_map_single().
> dma_unmap_resource() uses intel_unmap_page() directly as the
> functions are identical.

Instead of having two tiny wrappers I'd just revert
964f2311a6862f1fbcc044d0828ad90030928b7f if we need to pass a real
physical address now.

Re: [PATCH] iommu/vt-d: Implement dma_[un]map_resource()

[Logan Gunthorpe]

On 2019-01-19 2:40 a.m., Christoph Hellwig wrote:
> Which resources do you plan to map?  At least for PCIe P2P adding
> an address translation seems wrong to me.

It's mapping a PCI BAR but not for PCIe P2P. In this case, we are using
the Intel I/OAT DMA engine to copy data from a PCI BAR to/from memory. See:

http://lkml.kernel.org/r/20190119001001.13087-1-logang@deltatee.com

>>
>> The implementation for intel_map_resource() is nearly identical to
>> intel_map_page(), we just have to re-create __intel_map_single().
>> dma_unmap_resource() uses intel_unmap_page() directly as the
>> functions are identical.
> 
> Instead of having two tiny wrappers I'd just revert
> 964f2311a6862f1fbcc044d0828ad90030928b7f if we need to pass a real
> physical address now.

Ok, I can resubmit this with that cleanup. Should I do it in two commits
(1 revert + 1 new) or is one commit that just restores the original
helper fine?

Thanks,

Logan

Re: [PATCH] iommu/vt-d: Implement dma_[un]map_resource()

[Christoph Hellwig]

On Sat, Jan 19, 2019 at 11:46:22AM -0700, Logan Gunthorpe wrote:
> > Instead of having two tiny wrappers I'd just revert
> > 964f2311a6862f1fbcc044d0828ad90030928b7f if we need to pass a real
> > physical address now.
> 
> Ok, I can resubmit this with that cleanup. Should I do it in two commits
> (1 revert + 1 new) or is one commit that just restores the original
> helper fine?

I think a single commit is fine.