[PATCH] swiotlb 0.7: separation of physical and virtual address translation.

[PATCH] swiotlb 0.7: separation of physical and virtual address translation.

[Konrad Rzeszutek Wilk]

Fujita-san et al.

Attached is a set of patches that separate the address translation
(virt_to_phys, virt_to_bus, etc) from the SWIOTLB library.

Since the last posting [v6] I've done:
 - Minimized the list exported functions/variable with a prefix of: "swiotbl_tbl".
 - Made the usage of 'int dir' to be 'enum dma_data_direction'.
[v5-v6 changes:]
 - Made the exported functions/variables have the 'swiotlb_bk' prefix to be
   more uniform.
 - dropped the checkpatches/other reworks

.. and the writeup for this set set of patches:

The idea behind this set of patches is to make it possible to have separate
mechanisms for translating virtual to physical or virtual to DMA addresses
on platforms which need an SWIOTLB, and where physical != PCI bus address.

One customers of this is the pv-ops project, which can switch between
different modes of operation depending on the environment it is running in:
bare-metal or virtualized (Xen for now). Another is the Wii DMA implementation
so that the Wii USB controller can work.

On bare-metal SWIOTLB is used when there are no hardware IOMMU. In virtualized
environment it used when PCI pass-through is enabled for the guest. The problems
with PCI pass-through is that the guest's idea of PFN's is not the real thing.
To fix that, there is translation layer for PFN->machine frame number and vice-versa.
To bubble that up to the SWIOTLB layer there are two possible solutions.

One solution has been to wholesale copy the SWIOTLB, stick it in
arch/x86/xen/swiotlb.c and modify the virt_to_phys, phys_to_virt and others
to use the Xen address translation functions. Unfortunately, since the kernel can
run on bare-metal, there would be big code overlap with the real SWIOTLB.
(git://git.kernel.org/pub/scm/linux/kernel/git/jeremy/xen.git xen/dom0/swiotlb-new)

Another approach, which this set of patches explores, is to abstract the
address translation and address determination functions away from the
SWIOTLB book-keeping functions. This way the core SWIOTLB library functions
are present in one place, while the address related functions are in
a separate library that can be loaded when running under non-bare-metal platform.

The set of patches is also accessible on

git://git.kernel.org/pub/scm/linux/kernel/git/konrad/swiotlb-2.6.git swiotlb-0.7

 include/linux/swiotlb.h |   37 +++++++-
 lib/swiotlb.c           |  244 +++++++++++++++++++++++++---------------------
 2 files changed, 168 insertions(+), 113 deletions(-)

P.S.
There is also a branch utilizing the above mentioned patches: xen-swiotlb-0.7

[PATCH 1/6] swiotlb: Make internal bookkeeping functions have 'swiotlb_tbl' prefix.

[Konrad Rzeszutek Wilk]

The functions that operate on io_tlb_list/io_tlb_start/io_tlb_orig_addr
have the prefix 'swiotlb_tbl' now.

Signed-off-by: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
---
 lib/swiotlb.c |   35 +++++++++++++++++++----------------
 1 files changed, 19 insertions(+), 16 deletions(-)

diff --git a/lib/swiotlb.c b/lib/swiotlb.c
index 437eedb..1798814 100644
--- a/lib/swiotlb.c
+++ b/lib/swiotlb.c
@@ -60,8 +60,8 @@ enum dma_sync_target {
 int swiotlb_force;
 
 /*
- * Used to do a quick range check in unmap_single and
- * sync_single_*, to see if the memory was in fact allocated by this
+ * Used to do a quick range check in swiotlb_tbl_unmap_single and
+ * swiotlb_tbl_sync_single_*, to see if the memory was in fact allocated by this
  * API.
  */
 static char *io_tlb_start, *io_tlb_end;
@@ -364,7 +364,8 @@ static void swiotlb_bounce(phys_addr_t phys, char *dma_addr, size_t size,
  * Allocates bounce buffer and returns its kernel virtual address.
  */
 static void *
-map_single(struct device *hwdev, phys_addr_t phys, size_t size, int dir)
+swiotlb_tbl_map_single(struct device *hwdev, phys_addr_t phys, size_t size,
+		      int dir)
 {
 	unsigned long flags;
 	char *dma_addr;
@@ -470,7 +471,8 @@ found:
  * dma_addr is the kernel virtual address of the bounce buffer to unmap.
  */
 static void
-do_unmap_single(struct device *hwdev, char *dma_addr, size_t size, int dir)
+swiotlb_tbl_unmap_single(struct device *hwdev, char *dma_addr, size_t size,
+			int dir)
 {
 	unsigned long flags;
 	int i, count, nslots = ALIGN(size, 1 << IO_TLB_SHIFT) >> IO_TLB_SHIFT;
@@ -510,7 +512,7 @@ do_unmap_single(struct device *hwdev, char *dma_addr, size_t size, int dir)
 }
 
 static void
-sync_single(struct device *hwdev, char *dma_addr, size_t size,
+swiotlb_tbl_sync_single(struct device *hwdev, char *dma_addr, size_t size,
 	    int dir, int target)
 {
 	int index = (dma_addr - io_tlb_start) >> IO_TLB_SHIFT;
@@ -558,11 +560,11 @@ swiotlb_alloc_coherent(struct device *hwdev, size_t size,
 	}
 	if (!ret) {
 		/*
-		 * We are either out of memory or the device can't DMA
-		 * to GFP_DMA memory; fall back on map_single(), which
+		 * We are either out of memory or the device can't DMA to
+		 * GFP_DMA memory; fall back on swiotlb_tbl_map_single(), which
 		 * will grab memory from the lowest available address range.
 		 */
-		ret = map_single(hwdev, 0, size, DMA_FROM_DEVICE);
+		ret = swiotlb_tbl_map_single(hwdev, 0, size, DMA_FROM_DEVICE);
 		if (!ret)
 			return NULL;
 	}
@@ -577,7 +579,7 @@ swiotlb_alloc_coherent(struct device *hwdev, size_t size,
 		       (unsigned long long)dev_addr);
 
 		/* DMA_TO_DEVICE to avoid memcpy in unmap_single */
-		do_unmap_single(hwdev, ret, size, DMA_TO_DEVICE);
+		swiotlb_tbl_unmap_single(hwdev, ret, size, DMA_TO_DEVICE);
 		return NULL;
 	}
 	*dma_handle = dev_addr;
@@ -595,8 +597,8 @@ swiotlb_free_coherent(struct device *hwdev, size_t size, void *vaddr,
 	if (!is_swiotlb_buffer(paddr))
 		free_pages((unsigned long)vaddr, get_order(size));
 	else
-		/* DMA_TO_DEVICE to avoid memcpy in unmap_single */
-		do_unmap_single(hwdev, vaddr, size, DMA_TO_DEVICE);
+		/* DMA_TO_DEVICE to avoid memcpy in swiotlb_tbl_unmap_single */
+		swiotlb_tbl_unmap_single(hwdev, vaddr, size, DMA_TO_DEVICE);
 }
 EXPORT_SYMBOL(swiotlb_free_coherent);
 
@@ -652,7 +654,7 @@ dma_addr_t swiotlb_map_page(struct device *dev, struct page *page,
 	/*
 	 * Oh well, have to allocate and map a bounce buffer.
 	 */
-	map = map_single(dev, phys, size, dir);
+	map = swiotlb_tbl_map_single(dev, phys, size, dir);
 	if (!map) {
 		swiotlb_full(dev, size, dir, 1);
 		map = io_tlb_overflow_buffer;
@@ -686,7 +688,7 @@ static void unmap_single(struct device *hwdev, dma_addr_t dev_addr,
 	BUG_ON(dir == DMA_NONE);
 
 	if (is_swiotlb_buffer(paddr)) {
-		do_unmap_single(hwdev, phys_to_virt(paddr), size, dir);
+		swiotlb_tbl_unmap_single(hwdev, phys_to_virt(paddr), size, dir);
 		return;
 	}
 
@@ -729,7 +731,8 @@ swiotlb_sync_single(struct device *hwdev, dma_addr_t dev_addr,
 	BUG_ON(dir == DMA_NONE);
 
 	if (is_swiotlb_buffer(paddr)) {
-		sync_single(hwdev, phys_to_virt(paddr), size, dir, target);
+		swiotlb_tbl_sync_single(hwdev, phys_to_virt(paddr), size, dir,
+				       target);
 		return;
 	}
 
@@ -817,8 +820,8 @@ swiotlb_map_sg_attrs(struct device *hwdev, struct scatterlist *sgl, int nelems,
 
 		if (swiotlb_force ||
 		    !dma_capable(hwdev, dev_addr, sg->length)) {
-			void *map = map_single(hwdev, sg_phys(sg),
-					       sg->length, dir);
+			void *map = swiotlb_tbl_map_single(hwdev, sg_phys(sg),
+							   sg->length, dir);
 			if (!map) {
 				/* Don't panic here, we expect map_sg users
 				   to do proper error handling. */
-- 
1.6.2.5

[PATCH 2/6] swiotlb: swiotlb_tbl_map_single: abstract out swiotlb_virt_to_bus calls out.

[Konrad Rzeszutek Wilk]

We want to move that function out of swiotlb_tbl_map_single so that the caller
of this function does the virt->phys->bus address translation.

Signed-off-by: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
---
 lib/swiotlb.c |   22 ++++++++++++++--------
 1 files changed, 14 insertions(+), 8 deletions(-)

diff --git a/lib/swiotlb.c b/lib/swiotlb.c
index 1798814..bc516bb 100644
--- a/lib/swiotlb.c
+++ b/lib/swiotlb.c
@@ -364,21 +364,19 @@ static void swiotlb_bounce(phys_addr_t phys, char *dma_addr, size_t size,
  * Allocates bounce buffer and returns its kernel virtual address.
  */
 static void *
-swiotlb_tbl_map_single(struct device *hwdev, phys_addr_t phys, size_t size,
-		      int dir)
+swiotlb_tbl_map_single(struct device *hwdev, phys_addr_t phys,
+		      unsigned long start_dma_addr, size_t size, int dir)
 {
 	unsigned long flags;
 	char *dma_addr;
 	unsigned int nslots, stride, index, wrap;
 	int i;
-	unsigned long start_dma_addr;
 	unsigned long mask;
 	unsigned long offset_slots;
 	unsigned long max_slots;
 
 	mask = dma_get_seg_boundary(hwdev);
-	start_dma_addr = swiotlb_virt_to_bus(hwdev, io_tlb_start) & mask;
-
+	start_dma_addr = start_dma_addr & mask;
 	offset_slots = ALIGN(start_dma_addr, 1 << IO_TLB_SHIFT) >> IO_TLB_SHIFT;
 
 	/*
@@ -546,6 +544,7 @@ swiotlb_alloc_coherent(struct device *hwdev, size_t size,
 	void *ret;
 	int order = get_order(size);
 	u64 dma_mask = DMA_BIT_MASK(32);
+	unsigned long start_dma_addr;
 
 	if (hwdev && hwdev->coherent_dma_mask)
 		dma_mask = hwdev->coherent_dma_mask;
@@ -564,7 +563,9 @@ swiotlb_alloc_coherent(struct device *hwdev, size_t size,
 		 * GFP_DMA memory; fall back on swiotlb_tbl_map_single(), which
 		 * will grab memory from the lowest available address range.
 		 */
-		ret = swiotlb_tbl_map_single(hwdev, 0, size, DMA_FROM_DEVICE);
+		start_dma_addr = swiotlb_virt_to_bus(hwdev, io_tlb_start);
+		ret = swiotlb_tbl_map_single(hwdev, 0, start_dma_addr, size,
+					    DMA_FROM_DEVICE);
 		if (!ret)
 			return NULL;
 	}
@@ -638,6 +639,7 @@ dma_addr_t swiotlb_map_page(struct device *dev, struct page *page,
 			    enum dma_data_direction dir,
 			    struct dma_attrs *attrs)
 {
+	unsigned long start_dma_addr;
 	phys_addr_t phys = page_to_phys(page) + offset;
 	dma_addr_t dev_addr = phys_to_dma(dev, phys);
 	void *map;
@@ -654,7 +656,8 @@ dma_addr_t swiotlb_map_page(struct device *dev, struct page *page,
 	/*
 	 * Oh well, have to allocate and map a bounce buffer.
 	 */
-	map = swiotlb_tbl_map_single(dev, phys, size, dir);
+	start_dma_addr = swiotlb_virt_to_bus(dev, io_tlb_start);
+	map = swiotlb_tbl_map_single(dev, phys, start_dma_addr, size, dir);
 	if (!map) {
 		swiotlb_full(dev, size, dir, 1);
 		map = io_tlb_overflow_buffer;
@@ -809,11 +812,13 @@ int
 swiotlb_map_sg_attrs(struct device *hwdev, struct scatterlist *sgl, int nelems,
 		     enum dma_data_direction dir, struct dma_attrs *attrs)
 {
+	unsigned long start_dma_addr;
 	struct scatterlist *sg;
 	int i;
 
 	BUG_ON(dir == DMA_NONE);
 
+	start_dma_addr = swiotlb_virt_to_bus(hwdev, io_tlb_start);
 	for_each_sg(sgl, sg, nelems, i) {
 		phys_addr_t paddr = sg_phys(sg);
 		dma_addr_t dev_addr = phys_to_dma(hwdev, paddr);
@@ -821,7 +826,8 @@ swiotlb_map_sg_attrs(struct device *hwdev, struct scatterlist *sgl, int nelems,
 		if (swiotlb_force ||
 		    !dma_capable(hwdev, dev_addr, sg->length)) {
 			void *map = swiotlb_tbl_map_single(hwdev, sg_phys(sg),
-							   sg->length, dir);
+							  start_dma_addr,
+							  sg->length, dir);
 			if (!map) {
 				/* Don't panic here, we expect map_sg users
 				   to do proper error handling. */
-- 
1.6.2.5

[PATCH 3/6] swiotlb: Make exportable bookkeeping functions and variables have same prefix.

[Konrad Rzeszutek Wilk]

We prefix pertient book keeping functions and variables with the
'swiotlb_tbl' prefix.

Signed-off-by: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
---
 lib/swiotlb.c |  149 +++++++++++++++++++++++++++++---------------------------
 1 files changed, 77 insertions(+), 72 deletions(-)

diff --git a/lib/swiotlb.c b/lib/swiotlb.c
index bc516bb..5451517 100644
--- a/lib/swiotlb.c
+++ b/lib/swiotlb.c
@@ -64,13 +64,13 @@ int swiotlb_force;
  * swiotlb_tbl_sync_single_*, to see if the memory was in fact allocated by this
  * API.
  */
-static char *io_tlb_start, *io_tlb_end;
+static char *swiotlb_tbl_start, *io_tlb_end;
 
 /*
- * The number of IO TLB blocks (in groups of 64) betweeen io_tlb_start and
+ * The number of IO TLB blocks (in groups of 64) betweeen swiotlb_tbl_start and
  * io_tlb_end.  This is command line adjustable via setup_io_tlb_npages.
  */
-static unsigned long io_tlb_nslabs;
+static unsigned long swiotlb_tbl_nslabs;
 
 /*
  * When the IOMMU overflows we return a fallback buffer. This sets the size.
@@ -103,9 +103,9 @@ static int __init
 setup_io_tlb_npages(char *str)
 {
 	if (isdigit(*str)) {
-		io_tlb_nslabs = simple_strtoul(str, &str, 0);
+		swiotlb_tbl_nslabs = simple_strtoul(str, &str, 0);
 		/* avoid tail segment of size < IO_TLB_SEGSIZE */
-		io_tlb_nslabs = ALIGN(io_tlb_nslabs, IO_TLB_SEGSIZE);
+		swiotlb_tbl_nslabs = ALIGN(swiotlb_tbl_nslabs, IO_TLB_SEGSIZE);
 	}
 	if (*str == ',')
 		++str;
@@ -126,14 +126,14 @@ static dma_addr_t swiotlb_virt_to_bus(struct device *hwdev,
 
 void swiotlb_print_info(void)
 {
-	unsigned long bytes = io_tlb_nslabs << IO_TLB_SHIFT;
+	unsigned long bytes = swiotlb_tbl_nslabs << IO_TLB_SHIFT;
 	phys_addr_t pstart, pend;
 
-	pstart = virt_to_phys(io_tlb_start);
+	pstart = virt_to_phys(swiotlb_tbl_start);
 	pend = virt_to_phys(io_tlb_end);
 
 	printk(KERN_INFO "Placing %luMB software IO TLB between %p - %p\n",
-	       bytes >> 20, io_tlb_start, io_tlb_end);
+	       bytes >> 20, swiotlb_tbl_start, io_tlb_end);
 	printk(KERN_INFO "software IO TLB at phys %#llx - %#llx\n",
 	       (unsigned long long)pstart,
 	       (unsigned long long)pend);
@@ -148,31 +148,32 @@ swiotlb_init_with_default_size(size_t default_size, int verbose)
 {
 	unsigned long i, bytes;
 
-	if (!io_tlb_nslabs) {
-		io_tlb_nslabs = (default_size >> IO_TLB_SHIFT);
-		io_tlb_nslabs = ALIGN(io_tlb_nslabs, IO_TLB_SEGSIZE);
+	if (!swiotlb_tbl_nslabs) {
+		swiotlb_tbl_nslabs = (default_size >> IO_TLB_SHIFT);
+		swiotlb_tbl_nslabs = ALIGN(swiotlb_tbl_nslabs, IO_TLB_SEGSIZE);
 	}
 
-	bytes = io_tlb_nslabs << IO_TLB_SHIFT;
+	bytes = swiotlb_tbl_nslabs << IO_TLB_SHIFT;
 
 	/*
 	 * Get IO TLB memory from the low pages
 	 */
-	io_tlb_start = alloc_bootmem_low_pages(bytes);
-	if (!io_tlb_start)
+	swiotlb_tbl_start = alloc_bootmem_low_pages(bytes);
+	if (!swiotlb_tbl_start)
 		panic("Cannot allocate SWIOTLB buffer");
-	io_tlb_end = io_tlb_start + bytes;
+	io_tlb_end = swiotlb_tbl_start + bytes;
 
 	/*
 	 * Allocate and initialize the free list array.  This array is used
 	 * to find contiguous free memory regions of size up to IO_TLB_SEGSIZE
-	 * between io_tlb_start and io_tlb_end.
+	 * between swiotlb_tbl_start and io_tlb_end.
 	 */
-	io_tlb_list = alloc_bootmem(io_tlb_nslabs * sizeof(int));
-	for (i = 0; i < io_tlb_nslabs; i++)
- 		io_tlb_list[i] = IO_TLB_SEGSIZE - OFFSET(i, IO_TLB_SEGSIZE);
+	io_tlb_list = alloc_bootmem(swiotlb_tbl_nslabs * sizeof(int));
+	for (i = 0; i < swiotlb_tbl_nslabs; i++)
+		io_tlb_list[i] = IO_TLB_SEGSIZE - OFFSET(i, IO_TLB_SEGSIZE);
 	io_tlb_index = 0;
-	io_tlb_orig_addr = alloc_bootmem(io_tlb_nslabs * sizeof(phys_addr_t));
+	io_tlb_orig_addr = alloc_bootmem(swiotlb_tbl_nslabs *
+					     sizeof(phys_addr_t));
 
 	/*
 	 * Get the overflow emergency buffer
@@ -198,69 +199,70 @@ swiotlb_init(int verbose)
 int
 swiotlb_late_init_with_default_size(size_t default_size)
 {
-	unsigned long i, bytes, req_nslabs = io_tlb_nslabs;
+	unsigned long i, bytes, req_nslabs = swiotlb_tbl_nslabs;
 	unsigned int order;
 
-	if (!io_tlb_nslabs) {
-		io_tlb_nslabs = (default_size >> IO_TLB_SHIFT);
-		io_tlb_nslabs = ALIGN(io_tlb_nslabs, IO_TLB_SEGSIZE);
+	if (!swiotlb_tbl_nslabs) {
+		swiotlb_tbl_nslabs = (default_size >> IO_TLB_SHIFT);
+		swiotlb_tbl_nslabs = ALIGN(swiotlb_tbl_nslabs, IO_TLB_SEGSIZE);
 	}
 
 	/*
 	 * Get IO TLB memory from the low pages
 	 */
-	order = get_order(io_tlb_nslabs << IO_TLB_SHIFT);
-	io_tlb_nslabs = SLABS_PER_PAGE << order;
-	bytes = io_tlb_nslabs << IO_TLB_SHIFT;
+	order = get_order(swiotlb_tbl_nslabs << IO_TLB_SHIFT);
+	swiotlb_tbl_nslabs = SLABS_PER_PAGE << order;
+	bytes = swiotlb_tbl_nslabs << IO_TLB_SHIFT;
 
 	while ((SLABS_PER_PAGE << order) > IO_TLB_MIN_SLABS) {
-		io_tlb_start = (void *)__get_free_pages(GFP_DMA | __GFP_NOWARN,
+		swiotlb_tbl_start = (void *)__get_free_pages(GFP_DMA |
+							__GFP_NOWARN,
 							order);
-		if (io_tlb_start)
+		if (swiotlb_tbl_start)
 			break;
 		order--;
 	}
 
-	if (!io_tlb_start)
+	if (!swiotlb_tbl_start)
 		goto cleanup1;
 
 	if (order != get_order(bytes)) {
 		printk(KERN_WARNING "Warning: only able to allocate %ld MB "
 		       "for software IO TLB\n", (PAGE_SIZE << order) >> 20);
-		io_tlb_nslabs = SLABS_PER_PAGE << order;
-		bytes = io_tlb_nslabs << IO_TLB_SHIFT;
+		swiotlb_tbl_nslabs = SLABS_PER_PAGE << order;
+		bytes = swiotlb_tbl_nslabs << IO_TLB_SHIFT;
 	}
-	io_tlb_end = io_tlb_start + bytes;
-	memset(io_tlb_start, 0, bytes);
+	io_tlb_end = swiotlb_tbl_start + bytes;
+	memset(swiotlb_tbl_start, 0, bytes);
 
 	/*
 	 * Allocate and initialize the free list array.  This array is used
 	 * to find contiguous free memory regions of size up to IO_TLB_SEGSIZE
-	 * between io_tlb_start and io_tlb_end.
+	 * between swiotlb_tbl_start and io_tlb_end.
 	 */
 	io_tlb_list = (unsigned int *)__get_free_pages(GFP_KERNEL,
-	                              get_order(io_tlb_nslabs * sizeof(int)));
+				get_order(swiotlb_tbl_nslabs * sizeof(int)));
 	if (!io_tlb_list)
 		goto cleanup2;
 
-	for (i = 0; i < io_tlb_nslabs; i++)
- 		io_tlb_list[i] = IO_TLB_SEGSIZE - OFFSET(i, IO_TLB_SEGSIZE);
+	for (i = 0; i < swiotlb_tbl_nslabs; i++)
+		io_tlb_list[i] = IO_TLB_SEGSIZE - OFFSET(i, IO_TLB_SEGSIZE);
 	io_tlb_index = 0;
 
-	io_tlb_orig_addr = (phys_addr_t *)
-		__get_free_pages(GFP_KERNEL,
-				 get_order(io_tlb_nslabs *
-					   sizeof(phys_addr_t)));
+	io_tlb_orig_addr = (phys_addr_t *)__get_free_pages(GFP_KERNEL,
+						get_order(swiotlb_tbl_nslabs *
+						  sizeof(phys_addr_t)));
 	if (!io_tlb_orig_addr)
 		goto cleanup3;
 
-	memset(io_tlb_orig_addr, 0, io_tlb_nslabs * sizeof(phys_addr_t));
+	memset(io_tlb_orig_addr, 0, swiotlb_tbl_nslabs *
+					sizeof(phys_addr_t));
 
 	/*
 	 * Get the overflow emergency buffer
 	 */
 	io_tlb_overflow_buffer = (void *)__get_free_pages(GFP_DMA,
-	                                          get_order(io_tlb_overflow));
+					  get_order(io_tlb_overflow));
 	if (!io_tlb_overflow_buffer)
 		goto cleanup4;
 
@@ -272,18 +274,18 @@ swiotlb_late_init_with_default_size(size_t default_size)
 
 cleanup4:
 	free_pages((unsigned long)io_tlb_orig_addr,
-		   get_order(io_tlb_nslabs * sizeof(phys_addr_t)));
+		   get_order(swiotlb_tbl_nslabs * sizeof(phys_addr_t)));
 	io_tlb_orig_addr = NULL;
 cleanup3:
-	free_pages((unsigned long)io_tlb_list, get_order(io_tlb_nslabs *
+	free_pages((unsigned long)io_tlb_list, get_order(swiotlb_tbl_nslabs *
 	                                                 sizeof(int)));
 	io_tlb_list = NULL;
 cleanup2:
 	io_tlb_end = NULL;
-	free_pages((unsigned long)io_tlb_start, order);
-	io_tlb_start = NULL;
+	free_pages((unsigned long)swiotlb_tbl_start, order);
+	swiotlb_tbl_start = NULL;
 cleanup1:
-	io_tlb_nslabs = req_nslabs;
+	swiotlb_tbl_nslabs = req_nslabs;
 	return -ENOMEM;
 }
 
@@ -296,26 +298,26 @@ void __init swiotlb_free(void)
 		free_pages((unsigned long)io_tlb_overflow_buffer,
 			   get_order(io_tlb_overflow));
 		free_pages((unsigned long)io_tlb_orig_addr,
-			   get_order(io_tlb_nslabs * sizeof(phys_addr_t)));
-		free_pages((unsigned long)io_tlb_list, get_order(io_tlb_nslabs *
-								 sizeof(int)));
-		free_pages((unsigned long)io_tlb_start,
-			   get_order(io_tlb_nslabs << IO_TLB_SHIFT));
+			   get_order(swiotlb_tbl_nslabs * sizeof(phys_addr_t)));
+		free_pages((unsigned long)io_tlb_list,
+			   get_order(swiotlb_tbl_nslabs * sizeof(int)));
+		free_pages((unsigned long)swiotlb_tbl_start,
+			   get_order(swiotlb_tbl_nslabs << IO_TLB_SHIFT));
 	} else {
 		free_bootmem_late(__pa(io_tlb_overflow_buffer),
 				  io_tlb_overflow);
 		free_bootmem_late(__pa(io_tlb_orig_addr),
-				  io_tlb_nslabs * sizeof(phys_addr_t));
+				  swiotlb_tbl_nslabs * sizeof(phys_addr_t));
 		free_bootmem_late(__pa(io_tlb_list),
-				  io_tlb_nslabs * sizeof(int));
-		free_bootmem_late(__pa(io_tlb_start),
-				  io_tlb_nslabs << IO_TLB_SHIFT);
+				  swiotlb_tbl_nslabs * sizeof(int));
+		free_bootmem_late(__pa(swiotlb_tbl_start),
+				  swiotlb_tbl_nslabs << IO_TLB_SHIFT);
 	}
 }
 
 static int is_swiotlb_buffer(phys_addr_t paddr)
 {
-	return paddr >= virt_to_phys(io_tlb_start) &&
+	return paddr >= virt_to_phys(swiotlb_tbl_start) &&
 		paddr < virt_to_phys(io_tlb_end);
 }
 
@@ -404,7 +406,7 @@ swiotlb_tbl_map_single(struct device *hwdev, phys_addr_t phys,
 	 */
 	spin_lock_irqsave(&io_tlb_lock, flags);
 	index = ALIGN(io_tlb_index, stride);
-	if (index >= io_tlb_nslabs)
+	if (index >= swiotlb_tbl_nslabs)
 		index = 0;
 	wrap = index;
 
@@ -412,7 +414,7 @@ swiotlb_tbl_map_single(struct device *hwdev, phys_addr_t phys,
 		while (iommu_is_span_boundary(index, nslots, offset_slots,
 					      max_slots)) {
 			index += stride;
-			if (index >= io_tlb_nslabs)
+			if (index >= swiotlb_tbl_nslabs)
 				index = 0;
 			if (index == wrap)
 				goto not_found;
@@ -428,21 +430,22 @@ swiotlb_tbl_map_single(struct device *hwdev, phys_addr_t phys,
 
 			for (i = index; i < (int) (index + nslots); i++)
 				io_tlb_list[i] = 0;
-			for (i = index - 1; (OFFSET(i, IO_TLB_SEGSIZE) != IO_TLB_SEGSIZE - 1) && io_tlb_list[i]; i--)
+			for (i = index - 1; (OFFSET(i, IO_TLB_SEGSIZE)
+			     != IO_TLB_SEGSIZE - 1) && io_tlb_list[i]; i--)
 				io_tlb_list[i] = ++count;
-			dma_addr = io_tlb_start + (index << IO_TLB_SHIFT);
+			dma_addr = swiotlb_tbl_start + (index << IO_TLB_SHIFT);
 
 			/*
 			 * Update the indices to avoid searching in the next
 			 * round.
 			 */
-			io_tlb_index = ((index + nslots) < io_tlb_nslabs
+			io_tlb_index = ((index + nslots) < swiotlb_tbl_nslabs
 					? (index + nslots) : 0);
 
 			goto found;
 		}
 		index += stride;
-		if (index >= io_tlb_nslabs)
+		if (index >= swiotlb_tbl_nslabs)
 			index = 0;
 	} while (index != wrap);
 
@@ -474,7 +477,7 @@ swiotlb_tbl_unmap_single(struct device *hwdev, char *dma_addr, size_t size,
 {
 	unsigned long flags;
 	int i, count, nslots = ALIGN(size, 1 << IO_TLB_SHIFT) >> IO_TLB_SHIFT;
-	int index = (dma_addr - io_tlb_start) >> IO_TLB_SHIFT;
+	int index = (dma_addr - swiotlb_tbl_start) >> IO_TLB_SHIFT;
 	phys_addr_t phys = io_tlb_orig_addr[index];
 
 	/*
@@ -503,7 +506,8 @@ swiotlb_tbl_unmap_single(struct device *hwdev, char *dma_addr, size_t size,
 		 * Step 2: merge the returned slots with the preceding slots,
 		 * if available (non zero)
 		 */
-		for (i = index - 1; (OFFSET(i, IO_TLB_SEGSIZE) != IO_TLB_SEGSIZE -1) && io_tlb_list[i]; i--)
+		for (i = index - 1; (OFFSET(i, IO_TLB_SEGSIZE)
+		     != IO_TLB_SEGSIZE - 1) && io_tlb_list[i]; i--)
 			io_tlb_list[i] = ++count;
 	}
 	spin_unlock_irqrestore(&io_tlb_lock, flags);
@@ -513,7 +517,7 @@ static void
 swiotlb_tbl_sync_single(struct device *hwdev, char *dma_addr, size_t size,
 	    int dir, int target)
 {
-	int index = (dma_addr - io_tlb_start) >> IO_TLB_SHIFT;
+	int index = (dma_addr - swiotlb_tbl_start) >> IO_TLB_SHIFT;
 	phys_addr_t phys = io_tlb_orig_addr[index];
 
 	phys += ((unsigned long)dma_addr & ((1 << IO_TLB_SHIFT) - 1));
@@ -563,7 +567,7 @@ swiotlb_alloc_coherent(struct device *hwdev, size_t size,
 		 * GFP_DMA memory; fall back on swiotlb_tbl_map_single(), which
 		 * will grab memory from the lowest available address range.
 		 */
-		start_dma_addr = swiotlb_virt_to_bus(hwdev, io_tlb_start);
+		start_dma_addr = swiotlb_virt_to_bus(hwdev, swiotlb_tbl_start);
 		ret = swiotlb_tbl_map_single(hwdev, 0, start_dma_addr, size,
 					    DMA_FROM_DEVICE);
 		if (!ret)
@@ -656,7 +660,7 @@ dma_addr_t swiotlb_map_page(struct device *dev, struct page *page,
 	/*
 	 * Oh well, have to allocate and map a bounce buffer.
 	 */
-	start_dma_addr = swiotlb_virt_to_bus(dev, io_tlb_start);
+	start_dma_addr = swiotlb_virt_to_bus(dev, swiotlb_tbl_start);
 	map = swiotlb_tbl_map_single(dev, phys, start_dma_addr, size, dir);
 	if (!map) {
 		swiotlb_full(dev, size, dir, 1);
@@ -818,7 +822,7 @@ swiotlb_map_sg_attrs(struct device *hwdev, struct scatterlist *sgl, int nelems,
 
 	BUG_ON(dir == DMA_NONE);
 
-	start_dma_addr = swiotlb_virt_to_bus(hwdev, io_tlb_start);
+	start_dma_addr = swiotlb_virt_to_bus(hwdev, swiotlb_tbl_start);
 	for_each_sg(sgl, sg, nelems, i) {
 		phys_addr_t paddr = sg_phys(sg);
 		dma_addr_t dev_addr = phys_to_dma(hwdev, paddr);
@@ -919,7 +923,8 @@ EXPORT_SYMBOL(swiotlb_sync_sg_for_device);
 int
 swiotlb_dma_mapping_error(struct device *hwdev, dma_addr_t dma_addr)
 {
-	return (dma_addr == swiotlb_virt_to_bus(hwdev, io_tlb_overflow_buffer));
+	return (dma_addr == swiotlb_virt_to_bus(hwdev,
+						io_tlb_overflow_buffer));
 }
 EXPORT_SYMBOL(swiotlb_dma_mapping_error);
 
-- 
1.6.2.5

[PATCH 4/6] swiotlb: search and replace "int dir" with "enum dma_data_direction dir"

[Konrad Rzeszutek Wilk]

. to catch anybody doing something funky.

Signed-off-by: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
---
 include/linux/swiotlb.h |    4 ++--
 lib/swiotlb.c           |   22 ++++++++++++----------
 2 files changed, 14 insertions(+), 12 deletions(-)

diff --git a/include/linux/swiotlb.h b/include/linux/swiotlb.h
index febedcf..2a98e6a 100644
--- a/include/linux/swiotlb.h
+++ b/include/linux/swiotlb.h
@@ -42,11 +42,11 @@ extern void swiotlb_unmap_page(struct device *hwdev, dma_addr_t dev_addr,
 
 extern int
 swiotlb_map_sg(struct device *hwdev, struct scatterlist *sg, int nents,
-	       int direction);
+	       enum dma_data_direction direction);
 
 extern void
 swiotlb_unmap_sg(struct device *hwdev, struct scatterlist *sg, int nents,
-		 int direction);
+		 enum dma_data_direction direction);
 
 extern int
 swiotlb_map_sg_attrs(struct device *hwdev, struct scatterlist *sgl, int nelems,
diff --git a/lib/swiotlb.c b/lib/swiotlb.c
index 5451517..94fc749 100644
--- a/lib/swiotlb.c
+++ b/lib/swiotlb.c
@@ -367,7 +367,8 @@ static void swiotlb_bounce(phys_addr_t phys, char *dma_addr, size_t size,
  */
 static void *
 swiotlb_tbl_map_single(struct device *hwdev, phys_addr_t phys,
-		      unsigned long start_dma_addr, size_t size, int dir)
+		       unsigned long start_dma_addr, size_t size,
+		       enum dma_data_direction dir)
 {
 	unsigned long flags;
 	char *dma_addr;
@@ -473,7 +474,7 @@ found:
  */
 static void
 swiotlb_tbl_unmap_single(struct device *hwdev, char *dma_addr, size_t size,
-			int dir)
+			enum dma_data_direction dir)
 {
 	unsigned long flags;
 	int i, count, nslots = ALIGN(size, 1 << IO_TLB_SHIFT) >> IO_TLB_SHIFT;
@@ -515,7 +516,7 @@ swiotlb_tbl_unmap_single(struct device *hwdev, char *dma_addr, size_t size,
 
 static void
 swiotlb_tbl_sync_single(struct device *hwdev, char *dma_addr, size_t size,
-	    int dir, int target)
+	    enum dma_data_direction dir, int target)
 {
 	int index = (dma_addr - swiotlb_tbl_start) >> IO_TLB_SHIFT;
 	phys_addr_t phys = io_tlb_orig_addr[index];
@@ -608,7 +609,8 @@ swiotlb_free_coherent(struct device *hwdev, size_t size, void *vaddr,
 EXPORT_SYMBOL(swiotlb_free_coherent);
 
 static void
-swiotlb_full(struct device *dev, size_t size, int dir, int do_panic)
+swiotlb_full(struct device *dev, size_t size, enum dma_data_direction dir,
+	     int do_panic)
 {
 	/*
 	 * Ran out of IOMMU space for this operation. This is very bad.
@@ -688,7 +690,7 @@ EXPORT_SYMBOL_GPL(swiotlb_map_page);
  * whatever the device wrote there.
  */
 static void unmap_single(struct device *hwdev, dma_addr_t dev_addr,
-			 size_t size, int dir)
+			 size_t size, enum dma_data_direction dir)
 {
 	phys_addr_t paddr = dma_to_phys(hwdev, dev_addr);
 
@@ -731,7 +733,7 @@ EXPORT_SYMBOL_GPL(swiotlb_unmap_page);
  */
 static void
 swiotlb_sync_single(struct device *hwdev, dma_addr_t dev_addr,
-		    size_t size, int dir, int target)
+		    size_t size, enum dma_data_direction dir, int target)
 {
 	phys_addr_t paddr = dma_to_phys(hwdev, dev_addr);
 
@@ -771,7 +773,7 @@ EXPORT_SYMBOL(swiotlb_sync_single_for_device);
 static void
 swiotlb_sync_single_range(struct device *hwdev, dma_addr_t dev_addr,
 			  unsigned long offset, size_t size,
-			  int dir, int target)
+			  enum dma_data_direction dir, int target)
 {
 	swiotlb_sync_single(hwdev, dev_addr + offset, size, dir, target);
 }
@@ -852,7 +854,7 @@ EXPORT_SYMBOL(swiotlb_map_sg_attrs);
 
 int
 swiotlb_map_sg(struct device *hwdev, struct scatterlist *sgl, int nelems,
-	       int dir)
+	       enum dma_data_direction dir)
 {
 	return swiotlb_map_sg_attrs(hwdev, sgl, nelems, dir, NULL);
 }
@@ -879,7 +881,7 @@ EXPORT_SYMBOL(swiotlb_unmap_sg_attrs);
 
 void
 swiotlb_unmap_sg(struct device *hwdev, struct scatterlist *sgl, int nelems,
-		 int dir)
+		 enum dma_data_direction dir)
 {
 	return swiotlb_unmap_sg_attrs(hwdev, sgl, nelems, dir, NULL);
 }
@@ -894,7 +896,7 @@ EXPORT_SYMBOL(swiotlb_unmap_sg);
  */
 static void
 swiotlb_sync_sg(struct device *hwdev, struct scatterlist *sgl,
-		int nelems, int dir, int target)
+		int nelems, enum dma_data_direction dir, int target)
 {
 	struct scatterlist *sg;
 	int i;
-- 
1.6.2.5

[PATCH 5/6] swiotlb: Make swiotlb bookkeeping functions visible in the header file.

[Konrad Rzeszutek Wilk]

We put the init, free, and functions dealing with the operations on
the SWIOTLB buffer at the top of the header. Also we export some of the
variables that are used by the dma_ops functions.

Signed-off-by: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
---
 include/linux/swiotlb.h |   33 +++++++++++++++++++++++++++++++++
 lib/swiotlb.c           |   42 ++++++++++++++++++++----------------------
 2 files changed, 53 insertions(+), 22 deletions(-)

diff --git a/include/linux/swiotlb.h b/include/linux/swiotlb.h
index 2a98e6a..42df529 100644
--- a/include/linux/swiotlb.h
+++ b/include/linux/swiotlb.h
@@ -24,6 +24,39 @@ extern int swiotlb_force;
 
 extern void swiotlb_init(int verbose);
 
+/* Internal book-keeping functions. Must be linked against the library
+ * to take advantage of them.*/
+#ifdef CONFIG_SWIOTLB
+/*
+ * Enumeration for sync targets
+ */
+enum dma_sync_target {
+	SYNC_FOR_CPU = 0,
+	SYNC_FOR_DEVICE = 1,
+};
+extern char *swiotlb_tbl_start;
+extern unsigned long swiotlb_tbl_nslabs;
+extern int is_swiotlb_buffer(phys_addr_t paddr);
+extern void *swiotlb_tbl_map_single(struct device *hwdev, phys_addr_t phys,
+				    unsigned long start_dma_addr, size_t size,
+				    enum dma_data_direction dir);
+
+extern void swiotlb_tbl_unmap_single(struct device *hwdev, char *dma_addr,
+				     size_t size, enum dma_data_direction dir);
+
+extern void swiotlb_tbl_sync_single(struct device *hwdev, char *dma_addr,
+				    size_t size, enum dma_data_direction dir,
+				    enum dma_sync_target target);
+
+/* Accessory functions. */
+extern void swiotlb_bounce(phys_addr_t phys, char *dma_addr, size_t size,
+			   enum dma_data_direction dir);
+extern void swiotlb_full(struct device *dev, size_t size,
+			 enum dma_data_direction dir, int do_panic);
+
+#endif
+
+/* swiotlb.c: dma_ops functions. */
 extern void
 *swiotlb_alloc_coherent(struct device *hwdev, size_t size,
 			dma_addr_t *dma_handle, gfp_t flags);
diff --git a/lib/swiotlb.c b/lib/swiotlb.c
index 94fc749..5443ad5 100644
--- a/lib/swiotlb.c
+++ b/lib/swiotlb.c
@@ -49,14 +49,6 @@
  */
 #define IO_TLB_MIN_SLABS ((1<<20) >> IO_TLB_SHIFT)
 
-/*
- * Enumeration for sync targets
- */
-enum dma_sync_target {
-	SYNC_FOR_CPU = 0,
-	SYNC_FOR_DEVICE = 1,
-};
-
 int swiotlb_force;
 
 /*
@@ -64,13 +56,14 @@ int swiotlb_force;
  * swiotlb_tbl_sync_single_*, to see if the memory was in fact allocated by this
  * API.
  */
-static char *swiotlb_tbl_start, *io_tlb_end;
+char *swiotlb_tbl_start;
+static char *io_tlb_end;
 
 /*
  * The number of IO TLB blocks (in groups of 64) betweeen swiotlb_tbl_start and
  * io_tlb_end.  This is command line adjustable via setup_io_tlb_npages.
  */
-static unsigned long swiotlb_tbl_nslabs;
+unsigned long swiotlb_tbl_nslabs;
 
 /*
  * When the IOMMU overflows we return a fallback buffer. This sets the size.
@@ -315,7 +308,7 @@ void __init swiotlb_free(void)
 	}
 }
 
-static int is_swiotlb_buffer(phys_addr_t paddr)
+int is_swiotlb_buffer(phys_addr_t paddr)
 {
 	return paddr >= virt_to_phys(swiotlb_tbl_start) &&
 		paddr < virt_to_phys(io_tlb_end);
@@ -324,7 +317,7 @@ static int is_swiotlb_buffer(phys_addr_t paddr)
 /*
  * Bounce: copy the swiotlb buffer back to the original dma location
  */
-static void swiotlb_bounce(phys_addr_t phys, char *dma_addr, size_t size,
+void swiotlb_bounce(phys_addr_t phys, char *dma_addr, size_t size,
 			   enum dma_data_direction dir)
 {
 	unsigned long pfn = PFN_DOWN(phys);
@@ -365,7 +358,7 @@ static void swiotlb_bounce(phys_addr_t phys, char *dma_addr, size_t size,
 /*
  * Allocates bounce buffer and returns its kernel virtual address.
  */
-static void *
+void *
 swiotlb_tbl_map_single(struct device *hwdev, phys_addr_t phys,
 		       unsigned long start_dma_addr, size_t size,
 		       enum dma_data_direction dir)
@@ -472,9 +465,9 @@ found:
 /*
  * dma_addr is the kernel virtual address of the bounce buffer to unmap.
  */
-static void
+void
 swiotlb_tbl_unmap_single(struct device *hwdev, char *dma_addr, size_t size,
-			enum dma_data_direction dir)
+			 enum dma_data_direction dir)
 {
 	unsigned long flags;
 	int i, count, nslots = ALIGN(size, 1 << IO_TLB_SHIFT) >> IO_TLB_SHIFT;
@@ -514,9 +507,10 @@ swiotlb_tbl_unmap_single(struct device *hwdev, char *dma_addr, size_t size,
 	spin_unlock_irqrestore(&io_tlb_lock, flags);
 }
 
-static void
+void
 swiotlb_tbl_sync_single(struct device *hwdev, char *dma_addr, size_t size,
-	    enum dma_data_direction dir, int target)
+			enum dma_data_direction dir,
+			enum dma_sync_target target)
 {
 	int index = (dma_addr - swiotlb_tbl_start) >> IO_TLB_SHIFT;
 	phys_addr_t phys = io_tlb_orig_addr[index];
@@ -608,7 +602,7 @@ swiotlb_free_coherent(struct device *hwdev, size_t size, void *vaddr,
 }
 EXPORT_SYMBOL(swiotlb_free_coherent);
 
-static void
+void
 swiotlb_full(struct device *dev, size_t size, enum dma_data_direction dir,
 	     int do_panic)
 {
@@ -733,7 +727,8 @@ EXPORT_SYMBOL_GPL(swiotlb_unmap_page);
  */
 static void
 swiotlb_sync_single(struct device *hwdev, dma_addr_t dev_addr,
-		    size_t size, enum dma_data_direction dir, int target)
+		    size_t size, enum dma_data_direction dir,
+		    enum dma_sync_target target)
 {
 	phys_addr_t paddr = dma_to_phys(hwdev, dev_addr);
 
@@ -773,7 +768,8 @@ EXPORT_SYMBOL(swiotlb_sync_single_for_device);
 static void
 swiotlb_sync_single_range(struct device *hwdev, dma_addr_t dev_addr,
 			  unsigned long offset, size_t size,
-			  enum dma_data_direction dir, int target)
+			  enum dma_data_direction dir,
+			  enum dma_sync_target target)
 {
 	swiotlb_sync_single(hwdev, dev_addr + offset, size, dir, target);
 }
@@ -866,7 +862,8 @@ EXPORT_SYMBOL(swiotlb_map_sg);
  */
 void
 swiotlb_unmap_sg_attrs(struct device *hwdev, struct scatterlist *sgl,
-		       int nelems, enum dma_data_direction dir, struct dma_attrs *attrs)
+		       int nelems, enum dma_data_direction dir,
+		       struct dma_attrs *attrs)
 {
 	struct scatterlist *sg;
 	int i;
@@ -896,7 +893,8 @@ EXPORT_SYMBOL(swiotlb_unmap_sg);
  */
 static void
 swiotlb_sync_sg(struct device *hwdev, struct scatterlist *sgl,
-		int nelems, enum dma_data_direction dir, int target)
+		int nelems, enum dma_data_direction dir,
+		enum dma_sync_target target)
 {
 	struct scatterlist *sg;
 	int i;
-- 
1.6.2.5

[PATCH 6/6] swiotlb: EXPORT_SYMBOL_GPL functions + variables that are defined in the header file.

[Konrad Rzeszutek Wilk]

Make the functions and variables that are now declared in the swiotlb.h
header file visible by the linker.

Signed-off-by: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
---
 lib/swiotlb.c |    8 ++++++++
 1 files changed, 8 insertions(+), 0 deletions(-)

diff --git a/lib/swiotlb.c b/lib/swiotlb.c
index 5443ad5..48f1e94 100644
--- a/lib/swiotlb.c
+++ b/lib/swiotlb.c
@@ -58,12 +58,14 @@ int swiotlb_force;
  */
 char *swiotlb_tbl_start;
 static char *io_tlb_end;
+EXPORT_SYMBOL_GPL(swiotlb_tbl_start);
 
 /*
  * The number of IO TLB blocks (in groups of 64) betweeen swiotlb_tbl_start and
  * io_tlb_end.  This is command line adjustable via setup_io_tlb_npages.
  */
 unsigned long swiotlb_tbl_nslabs;
+EXPORT_SYMBOL_GPL(swiotlb_tbl_nslabs);
 
 /*
  * When the IOMMU overflows we return a fallback buffer. This sets the size.
@@ -313,6 +315,7 @@ int is_swiotlb_buffer(phys_addr_t paddr)
 	return paddr >= virt_to_phys(swiotlb_tbl_start) &&
 		paddr < virt_to_phys(io_tlb_end);
 }
+EXPORT_SYMBOL_GPL(is_swiotlb_buffer);
 
 /*
  * Bounce: copy the swiotlb buffer back to the original dma location
@@ -354,6 +357,7 @@ void swiotlb_bounce(phys_addr_t phys, char *dma_addr, size_t size,
 			memcpy(phys_to_virt(phys), dma_addr, size);
 	}
 }
+EXPORT_SYMBOL_GPL(swiotlb_bounce);
 
 /*
  * Allocates bounce buffer and returns its kernel virtual address.
@@ -461,6 +465,7 @@ found:
 
 	return dma_addr;
 }
+EXPORT_SYMBOL_GPL(swiotlb_tbl_map_single);
 
 /*
  * dma_addr is the kernel virtual address of the bounce buffer to unmap.
@@ -506,6 +511,7 @@ swiotlb_tbl_unmap_single(struct device *hwdev, char *dma_addr, size_t size,
 	}
 	spin_unlock_irqrestore(&io_tlb_lock, flags);
 }
+EXPORT_SYMBOL_GPL(swiotlb_tbl_unmap_single);
 
 void
 swiotlb_tbl_sync_single(struct device *hwdev, char *dma_addr, size_t size,
@@ -534,6 +540,7 @@ swiotlb_tbl_sync_single(struct device *hwdev, char *dma_addr, size_t size,
 		BUG();
 	}
 }
+EXPORT_SYMBOL_GPL(swiotlb_tbl_sync_single);
 
 void *
 swiotlb_alloc_coherent(struct device *hwdev, size_t size,
@@ -626,6 +633,7 @@ swiotlb_full(struct device *dev, size_t size, enum dma_data_direction dir,
 	if (dir == DMA_TO_DEVICE)
 		panic("DMA: Random memory could be DMA read\n");
 }
+EXPORT_SYMBOL_GPL(swiotlb_full);
 
 /*
  * Map a single buffer of the indicated size for DMA in streaming mode.  The
-- 
1.6.2.5

Re: [LKML] [PATCH] swiotlb 0.7: separation of physical and virtual address translation.

[Konrad Rzeszutek Wilk]

On Wed, Apr 07, 2010 at 04:29:24PM -0400, Konrad Rzeszutek Wilk wrote:
> Fujita-san et al.
> 
> Attached is a set of patches that separate the address translation
> (virt_to_phys, virt_to_bus, etc) from the SWIOTLB library.
> 
> Since the last posting [v6] I've done:
>  - Minimized the list exported functions/variable with a prefix of: "swiotbl_tbl".
>  - Made the usage of 'int dir' to be 'enum dma_data_direction'.

Hey Fujita-san,

I was wondering if you had the opportunity to review the patches?

Cheers,
Konrad

Re: [PATCH 2/6] swiotlb: swiotlb_tbl_map_single: abstract out swiotlb_virt_to_bus calls out.

[FUJITA Tomonori]

On Wed,  7 Apr 2010 16:29:26 -0400
Konrad Rzeszutek Wilk <konrad.wilk@oracle.com> wrote:

> We want to move that function out of swiotlb_tbl_map_single so that the caller
> of this function does the virt->phys->bus address translation.
> 
> Signed-off-by: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
> ---
>  lib/swiotlb.c |   22 ++++++++++++++--------
>  1 files changed, 14 insertions(+), 8 deletions(-)

It would be better to do something like that instead of spreading
swiotlb_virt_to_bus()?

=
From: FUJITA Tomonori <fujita.tomonori@lab.ntt.co.jp>
Subject: [PATCH] swiotlb: add swiotlb_tlb_map_single library function

swiotlb_tlb_map_single() takes the dma address of iotlb instead of
using swiotlb_virt_to_bus().

Signed-off-by: FUJITA Tomonori <fujita.tomonori@lab.ntt.co.jp>
---
 lib/swiotlb.c |   25 +++++++++++++++++--------
 1 files changed, 17 insertions(+), 8 deletions(-)

diff --git a/lib/swiotlb.c b/lib/swiotlb.c
index a37ee04..58f489a 100644
--- a/lib/swiotlb.c
+++ b/lib/swiotlb.c
@@ -373,25 +373,22 @@ static void swiotlb_bounce(phys_addr_t phys, char *dma_addr, size_t size,
 	}
 }
 
-/*
- * Allocates bounce buffer and returns its kernel virtual address.
- */
-static void *
-map_single(struct device *hwdev, phys_addr_t phys, size_t size, int dir)
+void *swiotlb_tlb_map_single(struct device *hwdev, u64 tlb_dma_addr,
+			     phys_addr_t phys, size_t size, int dir)
 {
 	unsigned long flags;
 	char *dma_addr;
 	unsigned int nslots, stride, index, wrap;
 	int i;
-	unsigned long start_dma_addr;
 	unsigned long mask;
 	unsigned long offset_slots;
 	unsigned long max_slots;
 
 	mask = dma_get_seg_boundary(hwdev);
-	start_dma_addr = swiotlb_virt_to_bus(hwdev, io_tlb_start) & mask;
 
-	offset_slots = ALIGN(start_dma_addr, 1 << IO_TLB_SHIFT) >> IO_TLB_SHIFT;
+	tlb_dma_addr &= mask;
+
+	offset_slots = ALIGN(tlb_dma_addr, 1 << IO_TLB_SHIFT) >> IO_TLB_SHIFT;
 
 	/*
  	 * Carefully handle integer overflow which can occur when mask == ~0UL.
@@ -480,6 +477,18 @@ found:
 }
 
 /*
+ * Allocates bounce buffer and returns its kernel virtual address.
+ */
+
+static void *
+map_single(struct device *hwdev, phys_addr_t phys, size_t size, int dir)
+{
+	u64 start_dma_addr = swiotlb_virt_to_bus(hwdev, io_tlb_start);
+
+	return swiotlb_tlb_map_single(hwdev, start_dma_addr, phys, size, dir);
+}
+
+/*
  * dma_addr is the kernel virtual address of the bounce buffer to unmap.
  */
 static void
-- 
1.6.5

Re: [PATCH 3/6] swiotlb: Make exportable bookkeeping functions and variables have same prefix.

[FUJITA Tomonori]

On Wed,  7 Apr 2010 16:29:27 -0400
Konrad Rzeszutek Wilk <konrad.wilk@oracle.com> wrote:

> We prefix pertient book keeping functions and variables with the
> 'swiotlb_tbl' prefix.
> 
> Signed-off-by: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
> ---
>  lib/swiotlb.c |  149 +++++++++++++++++++++++++++++---------------------------
>  1 files changed, 77 insertions(+), 72 deletions(-)

Can we create the swiotlb initialization function instead of lots of
renaming and exporting iotlb values?

=
From: FUJITA Tomonori <fujita.tomonori@lab.ntt.co.jp>
Subject: [PATCH 1/2] swiotlb: add the swiotlb initialization function
with iotlb memory

This enables the caller to initialize swiotlb with its own iotlb
memory.

Signed-off-by: FUJITA Tomonori <fujita.tomonori@lab.ntt.co.jp>
---
 include/linux/swiotlb.h |    1 +
 lib/swiotlb.c           |   48 +++++++++++++++++++++++++++++-----------------
 2 files changed, 31 insertions(+), 18 deletions(-)

diff --git a/include/linux/swiotlb.h b/include/linux/swiotlb.h
index febedcf..17b5298 100644
--- a/include/linux/swiotlb.h
+++ b/include/linux/swiotlb.h
@@ -23,6 +23,7 @@ extern int swiotlb_force;
 #define IO_TLB_SHIFT 11
 
 extern void swiotlb_init(int verbose);
+extern void swiotlb_init_with_tlb(char *tlb, unsigned long nslabs, int verbose);
 
 extern void
 *swiotlb_alloc_coherent(struct device *hwdev, size_t size,
diff --git a/lib/swiotlb.c b/lib/swiotlb.c
index 5fddf72..a37ee04 100644
--- a/lib/swiotlb.c
+++ b/lib/swiotlb.c
@@ -140,28 +140,14 @@ void swiotlb_print_info(void)
 	       (unsigned long long)pend);
 }
 
-/*
- * Statically reserve bounce buffer space and initialize bounce buffer data
- * structures for the software IO TLB used to implement the DMA API.
- */
-void __init
-swiotlb_init_with_default_size(size_t default_size, int verbose)
+void __init swiotlb_init_with_tlb(char *tlb, unsigned long nslabs, int verbose)
 {
 	unsigned long i, bytes;
 
-	if (!io_tlb_nslabs) {
-		io_tlb_nslabs = (default_size >> IO_TLB_SHIFT);
-		io_tlb_nslabs = ALIGN(io_tlb_nslabs, IO_TLB_SEGSIZE);
-	}
+	bytes = nslabs << IO_TLB_SHIFT;
 
-	bytes = io_tlb_nslabs << IO_TLB_SHIFT;
-
-	/*
-	 * Get IO TLB memory from the low pages
-	 */
-	io_tlb_start = alloc_bootmem_low_pages(bytes);
-	if (!io_tlb_start)
-		panic("Cannot allocate SWIOTLB buffer");
+	io_tlb_nslabs = nslabs;
+	io_tlb_start = tlb;
 	io_tlb_end = io_tlb_start + bytes;
 
 	/*
@@ -185,6 +171,32 @@ swiotlb_init_with_default_size(size_t default_size, int verbose)
 		swiotlb_print_info();
 }
 
+/*
+ * Statically reserve bounce buffer space and initialize bounce buffer data
+ * structures for the software IO TLB used to implement the DMA API.
+ */
+void __init
+swiotlb_init_with_default_size(size_t default_size, int verbose)
+{
+	unsigned long bytes;
+
+	if (!io_tlb_nslabs) {
+		io_tlb_nslabs = (default_size >> IO_TLB_SHIFT);
+		io_tlb_nslabs = ALIGN(io_tlb_nslabs, IO_TLB_SEGSIZE);
+	}
+
+	bytes = io_tlb_nslabs << IO_TLB_SHIFT;
+
+	/*
+	 * Get IO TLB memory from the low pages
+	 */
+	io_tlb_start = alloc_bootmem_low_pages(bytes);
+	if (!io_tlb_start)
+		panic("Cannot allocate SWIOTLB buffer");
+
+	swiotlb_init_with_tlb(io_tlb_start, io_tlb_nslabs, verbose);
+}
+
 void __init
 swiotlb_init(int verbose)
 {
-- 
1.6.5

Re: [PATCH 5/6] swiotlb: Make swiotlb bookkeeping functions visible in the header file.

[FUJITA Tomonori]

On Wed,  7 Apr 2010 16:29:29 -0400
Konrad Rzeszutek Wilk <konrad.wilk@oracle.com> wrote:

> We put the init, free, and functions dealing with the operations on
> the SWIOTLB buffer at the top of the header. Also we export some of the
> variables that are used by the dma_ops functions.
> 
> Signed-off-by: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
> ---
>  include/linux/swiotlb.h |   33 +++++++++++++++++++++++++++++++++
>  lib/swiotlb.c           |   42 ++++++++++++++++++++----------------------
>  2 files changed, 53 insertions(+), 22 deletions(-)


> +extern void swiotlb_full(struct device *dev, size_t size,
> +			 enum dma_data_direction dir, int do_panic);
> +

Please don't. You create a new IOMMU implementation with the helper
function. We don't want another IOMMU implementation that has broken
'overflow buffer' concept.

swiotlb-xen should handle DMA mapping errors properly instead of
calling swiotlb_full().

btw, if you try to push xen swiotlb stuff to mainline, please don't
put xen swiotlb stuff to include/linux/swiotlb.h.

Re: [PATCH 2/6] swiotlb: swiotlb_tbl_map_single: abstract out swiotlb_virt_to_bus calls out.

[Konrad Rzeszutek Wilk]

On Sun, May 09, 2010 at 10:41:15PM +0900, FUJITA Tomonori wrote:
> On Wed,  7 Apr 2010 16:29:26 -0400
> Konrad Rzeszutek Wilk <konrad.wilk@oracle.com> wrote:
> 
> > We want to move that function out of swiotlb_tbl_map_single so that the caller
> > of this function does the virt->phys->bus address translation.
> > 
> > Signed-off-by: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
> > ---
> >  lib/swiotlb.c |   22 ++++++++++++++--------
> >  1 files changed, 14 insertions(+), 8 deletions(-)
> 
> It would be better to do something like that instead of spreading
> swiotlb_virt_to_bus()?

Yes! Thought I am confused by the name. In the past you mentioned 'tbl',
but here it is 'tlb'. I presume 'tbl' is the right one?

> Subject: [PATCH] swiotlb: add swiotlb_tlb_map_single library function
> 
> swiotlb_tlb_map_single() takes the dma address of iotlb instead of
> using swiotlb_virt_to_bus().

Re: [PATCH 3/6] swiotlb: Make exportable bookkeeping functions and variables have same prefix.

[Konrad Rzeszutek Wilk]

On Sun, May 09, 2010 at 10:41:25PM +0900, FUJITA Tomonori wrote:
> On Wed,  7 Apr 2010 16:29:27 -0400
> Konrad Rzeszutek Wilk <konrad.wilk@oracle.com> wrote:
> 
> > We prefix pertient book keeping functions and variables with the
> > 'swiotlb_tbl' prefix.
> > 
> > Signed-off-by: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
> > ---
> >  lib/swiotlb.c |  149 +++++++++++++++++++++++++++++---------------------------
> >  1 files changed, 77 insertions(+), 72 deletions(-)
> 
> Can we create the swiotlb initialization function instead of lots of
> renaming and exporting iotlb values?

Yes, this would definitly do it! Let me rebase my tree on top of your
patches.

Thought I was thinking to modify your patches so they have the _tbl, instead
of _tlb in their name?

> 
> =
> From: FUJITA Tomonori <fujita.tomonori@lab.ntt.co.jp>
> Subject: [PATCH 1/2] swiotlb: add the swiotlb initialization function
> with iotlb memory
> 
> This enables the caller to initialize swiotlb with its own iotlb
> memory.
> 
> Signed-off-by: FUJITA Tomonori <fujita.tomonori@lab.ntt.co.jp>

Re: [PATCH 5/6] swiotlb: Make swiotlb bookkeeping functions visible in the header file.

[Konrad Rzeszutek Wilk]

> > +extern void swiotlb_full(struct device *dev, size_t size,
> > +			 enum dma_data_direction dir, int do_panic);
> > +
> 
> Please don't. You create a new IOMMU implementation with the helper
> function. We don't want another IOMMU implementation that has broken
> 'overflow buffer' concept.

Of course. Will remove that. Thanks again for your review, much
appreciated.

> 
> swiotlb-xen should handle DMA mapping errors properly instead of
> calling swiotlb_full().

<nods>
> 
> btw, if you try to push xen swiotlb stuff to mainline, please don't
> put xen swiotlb stuff to include/linux/swiotlb.h.

OK.

Re: [PATCH 2/6] swiotlb: swiotlb_tbl_map_single: abstract out swiotlb_virt_to_bus calls out.

[FUJITA Tomonori]

On Mon, 10 May 2010 15:31:18 -0400
Konrad Rzeszutek Wilk <konrad.wilk@oracle.com> wrote:

> On Sun, May 09, 2010 at 10:41:15PM +0900, FUJITA Tomonori wrote:
> > On Wed,  7 Apr 2010 16:29:26 -0400
> > Konrad Rzeszutek Wilk <konrad.wilk@oracle.com> wrote:
> > 
> > > We want to move that function out of swiotlb_tbl_map_single so that the caller
> > > of this function does the virt->phys->bus address translation.
> > > 
> > > Signed-off-by: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
> > > ---
> > >  lib/swiotlb.c |   22 ++++++++++++++--------
> > >  1 files changed, 14 insertions(+), 8 deletions(-)
> > 
> > It would be better to do something like that instead of spreading
> > swiotlb_virt_to_bus()?
> 
> Yes! Thought I am confused by the name. In the past you mentioned 'tbl',
> but here it is 'tlb'. I presume 'tbl' is the right one?

Oops, yeah. I meant 'tbl' for 'table' since Calgary and POWERPC IOMMUs
use 'iommu_table' struct to store the global info.

Please replace 'tlb' with 'tbl'.

Re: [PATCH 3/6] swiotlb: Make exportable bookkeeping functions and variables have same prefix.

[FUJITA Tomonori]

On Mon, 10 May 2010 15:33:44 -0400
Konrad Rzeszutek Wilk <konrad.wilk@oracle.com> wrote:

> On Sun, May 09, 2010 at 10:41:25PM +0900, FUJITA Tomonori wrote:
> > On Wed,  7 Apr 2010 16:29:27 -0400
> > Konrad Rzeszutek Wilk <konrad.wilk@oracle.com> wrote:
> > 
> > > We prefix pertient book keeping functions and variables with the
> > > 'swiotlb_tbl' prefix.
> > > 
> > > Signed-off-by: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
> > > ---
> > >  lib/swiotlb.c |  149 +++++++++++++++++++++++++++++---------------------------
> > >  1 files changed, 77 insertions(+), 72 deletions(-)
> > 
> > Can we create the swiotlb initialization function instead of lots of
> > renaming and exporting iotlb values?
> 
> Yes, this would definitly do it! Let me rebase my tree on top of your
> patches.
> 
> Thought I was thinking to modify your patches so they have the _tbl, instead
> of _tlb in their name?

Yeah, please replace tlb with tbl.