[PATCH 00/22] DMA-API/PCI map_peer_resource support for peer-to-peer

[PATCH 00/22] DMA-API/PCI map_peer_resource support for peer-to-peer

[Will Davis]

Hi,

This is the sixth version of a patchset to add the DMA APIs necessary to
map and unmap a PCI device's BAR to and from another PCI device's IOVA
domain. This enables PCI peer-to-peer traffic on x86 platforms where an
IOMMU is enabled (this requires that peer-to-peer traffic be supported by
the underlying chipset as well, of course).

Thanks,
Will

v6:
- added stubs for remaining x86 dma_map_ops: calgary, gart, sta2x11,
  swiotlb, xen-swiotlb
- factored part of x86 PCI nommu implementation into a generic PCI
  helper, pci_resource_to_peer()
- split x86 dma_peer_mapping_error() implementation into a separate
  patch

v5: http://www.spinics.net/lists/linux-pci/msg43975.html
- moved topology and ACS checks into a separate
  pci_peer_traffic_supported() API, which determines whether PCI peer-
  to-peer traffic may work based on language in the PCI and PCI Express
  specifications (excerpts included in comments)
- added CONFIG_HAS_DMA_P2P guards to x86 PCI no-mmu implementation of
  map_peer_resource
- modified a few patch descriptions for style

v4: http://www.spinics.net/lists/linux-pci/msg43136.html
- added HAS_DMA_P2P Kconfig option to guard new APIs, which are only
  supported on x86 with this patch series
- added a new dma_peer_addr_t type to handle future platform support
  where the dma_peer_addr_t needs to have the same width as the bus
  address, which can be wider than the dma_addr_t returned by other
  DMA APIs
- added a new dma_peer_mapping_error() and associated PCI wrapper to
  detect errors returned via dma_map_peer_resource() API and its
  PCI wrapper
- modified the interface of the new API to explicitly take the peer
  device as an argument
- renamed the (un)map_resource APIs to (un)map_peer_resource, since
  they directly receive a 'peer' argument now
- modified the new PCI interface to take the BAR index instead of the
  actual struct resource
- modified the pci-nommu implementation to perform topology and ACS
  checks before allowing
- modified both the AMD and Intel IOMMU implementations to require
  that both devices be behind the same host bridge
- exposed a couple of preexisting PCI utility functions to aid in the
  topology checks for all of the x86 implementations

v3: http://www.spinics.net/lists/linux-pci/msg41703.html
- changed dma_map_resource() to BUG() if the map_resource DMA op is not
  provided, instead of returning 0
- updated documentation to remove requirement to check for 0 return value
  as an error
- remove const keyword from struct dma_map_ops in new DMA APIs for
  consistency with other APIs

v2: http://www.spinics.net/lists/linux-pci/msg41192.html
- added documentation for the new DMA APIs
- fixed physical-to-bus address conversion in the nommu implementation

v1: http://www.spinics.net/lists/linux-pci/msg40747.html

Will Davis (22):
  lib/Kconfig: add HAS_DMA_P2P for peer-to-peer support
  linux/types.h: Add dma_peer_addr_t type
  dma-debug: add checking for map/unmap_peer_resource
  DMA-API: Introduce dma_(un)map_peer_resource
  dma-mapping: pci: add pci_(un)map_peer_resource
  DMA-API: Add peer resource mapping documentation
  PCI: Export pci_find_host_bridge()
  PCI: Add pci_find_common_upstream_dev()
  PCI: Add pci_peer_traffic_supported()
  PCI: Add pci_resource_to_peer()
  swiotlb: Add map_peer_resource stub
  x86, swiotlb: Add swiotlb_map_peer_resource() to swiotlb_dma_ops
  x86, platform: Add swiotlb_map_peer_resource() to sta2x11_dma_ops
  swiotlb-xen: add map_peer_resource stub
  pci-swiotlb-xen: add xen_swiotlb_map_peer_resource to
    xen_swiotlb_dma_ops
  iommu/amd: Implement (un)map_peer_resource
  iommu/vt-d: implement (un)map_peer_resource
  x86: add pci-nommu implementation of map_peer_resource
  x86: Calgary: Add map_peer_resource stub
  x86: gart: Add map_peer_resource stub
  x86: add dma_peer_mapping_error()
  x86: declare support for DMA P2P

 Documentation/DMA-API-HOWTO.txt          | 69 ++++++++++++++++++++++
 Documentation/DMA-API.txt                | 38 ++++++++++--
 arch/x86/Kconfig                         |  1 +
 arch/x86/include/asm/dma-mapping.h       | 18 +++++-
 arch/x86/kernel/amd_gart_64.c            | 16 ++++++
 arch/x86/kernel/pci-calgary_64.c         | 16 ++++++
 arch/x86/kernel/pci-nommu.c              | 30 ++++++++++
 arch/x86/kernel/pci-swiotlb.c            |  3 +
 arch/x86/pci/sta2x11-fixup.c             |  3 +
 arch/x86/xen/pci-swiotlb-xen.c           |  3 +
 drivers/iommu/amd_iommu.c                | 99 +++++++++++++++++++++++++++-----
 drivers/iommu/intel-iommu.c              | 38 ++++++++++++
 drivers/pci/host-bridge.c                | 19 +-----
 drivers/pci/pci.c                        | 99 ++++++++++++++++++++++++++++++++
 drivers/pci/search.c                     | 25 ++++++++
 drivers/xen/swiotlb-xen.c                | 20 +++++++
 include/asm-generic/dma-mapping-common.h | 43 ++++++++++++++
 include/asm-generic/pci-dma-compat.h     | 23 ++++++++
 include/linux/dma-debug.h                | 39 ++++++++++++-
 include/linux/dma-mapping.h              | 12 ++++
 include/linux/pci.h                      | 28 +++++++++
 include/linux/swiotlb.h                  |  8 +++
 include/linux/types.h                    | 13 ++++-
 include/xen/swiotlb-xen.h                | 11 ++++
 lib/Kconfig                              |  5 ++
 lib/dma-debug.c                          | 86 ++++++++++++++++++++++++++-
 lib/swiotlb.c                            | 17 ++++++
 27 files changed, 740 insertions(+), 42 deletions(-)

-- 
2.5.1

[PATCH 00/22] DMA-API/PCI map_peer_resource support for peer-to-peer

[Will Davis]

Hi,

This is the sixth version of a patchset to add the DMA APIs necessary to
map and unmap a PCI device's BAR to and from another PCI device's IOVA
domain. This enables PCI peer-to-peer traffic on x86 platforms where an
IOMMU is enabled (this requires that peer-to-peer traffic be supported by
the underlying chipset as well, of course).

Thanks,
Will

v6:
- added stubs for remaining x86 dma_map_ops: calgary, gart, sta2x11,
  swiotlb, xen-swiotlb
- factored part of x86 PCI nommu implementation into a generic PCI
  helper, pci_resource_to_peer()
- split x86 dma_peer_mapping_error() implementation into a separate
  patch

v5: http://www.spinics.net/lists/linux-pci/msg43975.html
- moved topology and ACS checks into a separate
  pci_peer_traffic_supported() API, which determines whether PCI peer-
  to-peer traffic may work based on language in the PCI and PCI Express
  specifications (excerpts included in comments)
- added CONFIG_HAS_DMA_P2P guards to x86 PCI no-mmu implementation of
  map_peer_resource
- modified a few patch descriptions for style

v4: http://www.spinics.net/lists/linux-pci/msg43136.html
- added HAS_DMA_P2P Kconfig option to guard new APIs, which are only
  supported on x86 with this patch series
- added a new dma_peer_addr_t type to handle future platform support
  where the dma_peer_addr_t needs to have the same width as the bus
  address, which can be wider than the dma_addr_t returned by other
  DMA APIs
- added a new dma_peer_mapping_error() and associated PCI wrapper to
  detect errors returned via dma_map_peer_resource() API and its
  PCI wrapper
- modified the interface of the new API to explicitly take the peer
  device as an argument
- renamed the (un)map_resource APIs to (un)map_peer_resource, since
  they directly receive a 'peer' argument now
- modified the new PCI interface to take the BAR index instead of the
  actual struct resource
- modified the pci-nommu implementation to perform topology and ACS
  checks before allowing
- modified both the AMD and Intel IOMMU implementations to require
  that both devices be behind the same host bridge
- exposed a couple of preexisting PCI utility functions to aid in the
  topology checks for all of the x86 implementations

v3: http://www.spinics.net/lists/linux-pci/msg41703.html
- changed dma_map_resource() to BUG() if the map_resource DMA op is not
  provided, instead of returning 0
- updated documentation to remove requirement to check for 0 return value
  as an error
- remove const keyword from struct dma_map_ops in new DMA APIs for
  consistency with other APIs

v2: http://www.spinics.net/lists/linux-pci/msg41192.html
- added documentation for the new DMA APIs
- fixed physical-to-bus address conversion in the nommu implementation

v1: http://www.spinics.net/lists/linux-pci/msg40747.html

Will Davis (22):
  lib/Kconfig: add HAS_DMA_P2P for peer-to-peer support
  linux/types.h: Add dma_peer_addr_t type
  dma-debug: add checking for map/unmap_peer_resource
  DMA-API: Introduce dma_(un)map_peer_resource
  dma-mapping: pci: add pci_(un)map_peer_resource
  DMA-API: Add peer resource mapping documentation
  PCI: Export pci_find_host_bridge()
  PCI: Add pci_find_common_upstream_dev()
  PCI: Add pci_peer_traffic_supported()
  PCI: Add pci_resource_to_peer()
  swiotlb: Add map_peer_resource stub
  x86, swiotlb: Add swiotlb_map_peer_resource() to swiotlb_dma_ops
  x86, platform: Add swiotlb_map_peer_resource() to sta2x11_dma_ops
  swiotlb-xen: add map_peer_resource stub
  pci-swiotlb-xen: add xen_swiotlb_map_peer_resource to
    xen_swiotlb_dma_ops
  iommu/amd: Implement (un)map_peer_resource
  iommu/vt-d: implement (un)map_peer_resource
  x86: add pci-nommu implementation of map_peer_resource
  x86: Calgary: Add map_peer_resource stub
  x86: gart: Add map_peer_resource stub
  x86: add dma_peer_mapping_error()
  x86: declare support for DMA P2P

 Documentation/DMA-API-HOWTO.txt          | 69 ++++++++++++++++++++++
 Documentation/DMA-API.txt                | 38 ++++++++++--
 arch/x86/Kconfig                         |  1 +
 arch/x86/include/asm/dma-mapping.h       | 18 +++++-
 arch/x86/kernel/amd_gart_64.c            | 16 ++++++
 arch/x86/kernel/pci-calgary_64.c         | 16 ++++++
 arch/x86/kernel/pci-nommu.c              | 30 ++++++++++
 arch/x86/kernel/pci-swiotlb.c            |  3 +
 arch/x86/pci/sta2x11-fixup.c             |  3 +
 arch/x86/xen/pci-swiotlb-xen.c           |  3 +
 drivers/iommu/amd_iommu.c                | 99 +++++++++++++++++++++++++++-----
 drivers/iommu/intel-iommu.c              | 38 ++++++++++++
 drivers/pci/host-bridge.c                | 19 +-----
 drivers/pci/pci.c                        | 99 ++++++++++++++++++++++++++++++++
 drivers/pci/search.c                     | 25 ++++++++
 drivers/xen/swiotlb-xen.c                | 20 +++++++
 include/asm-generic/dma-mapping-common.h | 43 ++++++++++++++
 include/asm-generic/pci-dma-compat.h     | 23 ++++++++
 include/linux/dma-debug.h                | 39 ++++++++++++-
 include/linux/dma-mapping.h              | 12 ++++
 include/linux/pci.h                      | 28 +++++++++
 include/linux/swiotlb.h                  |  8 +++
 include/linux/types.h                    | 13 ++++-
 include/xen/swiotlb-xen.h                | 11 ++++
 lib/Kconfig                              |  5 ++
 lib/dma-debug.c                          | 86 ++++++++++++++++++++++++++-
 lib/swiotlb.c                            | 17 ++++++
 27 files changed, 740 insertions(+), 42 deletions(-)

-- 
2.5.1

[PATCH 01/22] lib/Kconfig: add HAS_DMA_P2P for peer-to-peer support

[Will Davis]

This option is dependent on HAS_DMA, and indicates whether or not the DMA
API should include the peer resource (P2P) interfaces.

Signed-off-by: Will Davis <wdavis@nvidia.com>
---
 lib/Kconfig | 5 +++++
 1 file changed, 5 insertions(+)

diff --git a/lib/Kconfig b/lib/Kconfig
index 3a2ef67..0b670ab 100644
--- a/lib/Kconfig
+++ b/lib/Kconfig
@@ -392,6 +392,11 @@ config HAS_DMA
 	depends on !NO_DMA
 	default y
 
+config HAS_DMA_P2P
+	bool
+	depends on HAS_DMA
+	default n
+
 config CHECK_SIGNATURE
 	bool
 
-- 
2.5.1

[PATCH 01/22] lib/Kconfig: add HAS_DMA_P2P for peer-to-peer support

[Will Davis]

This option is dependent on HAS_DMA, and indicates whether or not the DMA
API should include the peer resource (P2P) interfaces.

Signed-off-by: Will Davis <wdavis@nvidia.com>
---
 lib/Kconfig | 5 +++++
 1 file changed, 5 insertions(+)

diff --git a/lib/Kconfig b/lib/Kconfig
index 3a2ef67..0b670ab 100644
--- a/lib/Kconfig
+++ b/lib/Kconfig
@@ -392,6 +392,11 @@ config HAS_DMA
 	depends on !NO_DMA
 	default y
 
+config HAS_DMA_P2P
+	bool
+	depends on HAS_DMA
+	default n
+
 config CHECK_SIGNATURE
 	bool
 
-- 
2.5.1

[PATCH 02/22] linux/types.h: Add dma_peer_addr_t type

[Will Davis]

On some platforms, a dma_addr_t is only 32 bits wide, but the bus addresses
can be 64 bits wide. For peer-to-peer DMA, a device needs to use the full
64-bit address. Therefore, introduce a DMA address type that is not
constrained by the CPU DMA address size.

This is only needed for peer-to-peer DMA, so guard this definition with a
CONFIG_HAS_DMA_P2P check.

Signed-off-by: Will Davis <wdavis@nvidia.com>
---
 include/linux/types.h | 13 +++++++++++--
 1 file changed, 11 insertions(+), 2 deletions(-)

diff --git a/include/linux/types.h b/include/linux/types.h
index 8715287..dd85173 100644
--- a/include/linux/types.h
+++ b/include/linux/types.h
@@ -140,13 +140,13 @@ typedef unsigned long blkcnt_t;
 #define pgoff_t unsigned long
 
 /*
- * A dma_addr_t can hold any valid DMA address, i.e., any address returned
+ * A dma_addr_t can hold any valid DMA address, i.e., most addresses returned
  * by the DMA API.
  *
  * If the DMA API only uses 32-bit addresses, dma_addr_t need only be 32
  * bits wide.  Bus addresses, e.g., PCI BARs, may be wider than 32 bits,
  * but drivers do memory-mapped I/O to ioremapped kernel virtual addresses,
- * so they don't care about the size of the actual bus addresses.
+ * so they don't care about the size of the actual bus addresses in most cases.
  */
 #ifdef CONFIG_ARCH_DMA_ADDR_T_64BIT
 typedef u64 dma_addr_t;
@@ -154,6 +154,15 @@ typedef u64 dma_addr_t;
 typedef u32 dma_addr_t;
 #endif
 
+/*
+ * Unlike dma_addr_t, a dma_peer_addr_t can hold any bus address for the
+ * platform. Drivers should only need this for peer-to-peer use cases, where
+ * the DMA target is also on the bus.
+ */
+#ifdef CONFIG_HAS_DMA_P2P
+typedef u64 dma_peer_addr_t;
+#endif
+
 typedef unsigned __bitwise__ gfp_t;
 typedef unsigned __bitwise__ fmode_t;
 typedef unsigned __bitwise__ oom_flags_t;
-- 
2.5.1

[PATCH 02/22] linux/types.h: Add dma_peer_addr_t type

[Will Davis]

On some platforms, a dma_addr_t is only 32 bits wide, but the bus addresses
can be 64 bits wide. For peer-to-peer DMA, a device needs to use the full
64-bit address. Therefore, introduce a DMA address type that is not
constrained by the CPU DMA address size.

This is only needed for peer-to-peer DMA, so guard this definition with a
CONFIG_HAS_DMA_P2P check.

Signed-off-by: Will Davis <wdavis@nvidia.com>
---
 include/linux/types.h | 13 +++++++++++--
 1 file changed, 11 insertions(+), 2 deletions(-)

diff --git a/include/linux/types.h b/include/linux/types.h
index 8715287..dd85173 100644
--- a/include/linux/types.h
+++ b/include/linux/types.h
@@ -140,13 +140,13 @@ typedef unsigned long blkcnt_t;
 #define pgoff_t unsigned long
 
 /*
- * A dma_addr_t can hold any valid DMA address, i.e., any address returned
+ * A dma_addr_t can hold any valid DMA address, i.e., most addresses returned
  * by the DMA API.
  *
  * If the DMA API only uses 32-bit addresses, dma_addr_t need only be 32
  * bits wide.  Bus addresses, e.g., PCI BARs, may be wider than 32 bits,
  * but drivers do memory-mapped I/O to ioremapped kernel virtual addresses,
- * so they don't care about the size of the actual bus addresses.
+ * so they don't care about the size of the actual bus addresses in most cases.
  */
 #ifdef CONFIG_ARCH_DMA_ADDR_T_64BIT
 typedef u64 dma_addr_t;
@@ -154,6 +154,15 @@ typedef u64 dma_addr_t;
 typedef u32 dma_addr_t;
 #endif
 
+/*
+ * Unlike dma_addr_t, a dma_peer_addr_t can hold any bus address for the
+ * platform. Drivers should only need this for peer-to-peer use cases, where
+ * the DMA target is also on the bus.
+ */
+#ifdef CONFIG_HAS_DMA_P2P
+typedef u64 dma_peer_addr_t;
+#endif
+
 typedef unsigned __bitwise__ gfp_t;
 typedef unsigned __bitwise__ fmode_t;
 typedef unsigned __bitwise__ oom_flags_t;
-- 
2.5.1

[PATCH 03/22] dma-debug: add checking for map/unmap_peer_resource

[Will Davis]

Add debug callbacks for the new dma_map_peer_resource and
dma_unmap_peer_resource functions.

Also add a callback for dma_peer_mapping_error. Most of the time this will
be identical to dma_mapping_error, except where the dma_addr_t is not as
wide as dma_peer_addr_t.

Guard these new callbacks behind a CONFIG_HAS_DMA_P2P check.

Signed-off-by: Will Davis <wdavis@nvidia.com>
Reviewed-by: Terence Ripperda <tripperda@nvidia.com>
Reviewed-by: John Hubbard <jhubbard@nvidia.com>
---
 include/linux/dma-debug.h | 39 ++++++++++++++++++++-
 lib/dma-debug.c           | 86 +++++++++++++++++++++++++++++++++++++++++++++--
 2 files changed, 122 insertions(+), 3 deletions(-)

diff --git a/include/linux/dma-debug.h b/include/linux/dma-debug.h
index fe8cb61..4ff8364 100644
--- a/include/linux/dma-debug.h
+++ b/include/linux/dma-debug.h
@@ -43,7 +43,21 @@ extern void debug_dma_mapping_error(struct device *dev, dma_addr_t dma_addr);
 
 extern void debug_dma_unmap_page(struct device *dev, dma_addr_t addr,
 				 size_t size, int direction, bool map_single);
-
+#ifdef CONFIG_HAS_DMA_P2P
+extern void debug_dma_map_peer_resource(struct device *dev,
+					struct device *peer,
+					struct resource *resource,
+					size_t offset, size_t size,
+					int direction,
+					dma_peer_addr_t dma_addr);
+
+extern void debug_dma_peer_mapping_error(struct device *dev,
+					 dma_peer_addr_t dma_addr);
+
+extern void debug_dma_unmap_peer_resource(struct device *dev,
+					  dma_peer_addr_t addr,
+					  size_t size, int direction);
+#endif
 extern void debug_dma_map_sg(struct device *dev, struct scatterlist *sg,
 			     int nents, int mapped_ents, int direction);
 
@@ -120,6 +134,29 @@ static inline void debug_dma_unmap_page(struct device *dev, dma_addr_t addr,
 {
 }
 
+#ifdef CONFIG_HAS_DMA_P2P
+static inline void debug_dma_map_peer_resource(struct device *dev,
+					       struct device *peer,
+					       struct resource *resource,
+					       size_t offset, size_t size,
+					       int direction,
+					       dma_peer_addr_t dma_addr)
+{
+}
+
+static inline void debug_dma_peer_mapping_error(struct device *dev,
+						dma_peer_addr_t dma_addr)
+{
+}
+
+static inline void debug_dma_unmap_peer_resource(struct device *dev,
+						 dma_peer_addr_t addr,
+						 size_t size,
+						 int direction)
+{
+}
+#endif
+
 static inline void debug_dma_map_sg(struct device *dev, struct scatterlist *sg,
 				    int nents, int mapped_ents, int direction)
 {
diff --git a/lib/dma-debug.c b/lib/dma-debug.c
index ae4b65e..622f569 100644
--- a/lib/dma-debug.c
+++ b/lib/dma-debug.c
@@ -43,6 +43,7 @@ enum {
 	dma_debug_page,
 	dma_debug_sg,
 	dma_debug_coherent,
+	dma_debug_peer_resource,
 };
 
 enum map_err_types {
@@ -64,7 +65,8 @@ enum map_err_types {
  * @direction: enum dma_data_direction
  * @sg_call_ents: 'nents' from dma_map_sg
  * @sg_mapped_ents: 'mapped_ents' from dma_map_sg
- * @map_err_type: track whether dma_mapping_error() was checked
+ * @map_err_type: track whether dma_mapping_error() or dma_peer_mapping_error()
+ *		  was called
  * @stacktrace: support backtraces when a violation is detected
  */
 struct dma_debug_entry {
@@ -1250,6 +1252,23 @@ out:
 	put_hash_bucket(bucket, &flags);
 }
 
+static void check_mapping_error(struct dma_debug_entry *entry, bool peer)
+{
+	if (peer && !(entry->type == dma_debug_peer_resource))
+		err_printk(entry->dev, entry, "DMA-API: device driver calls "
+				"dma_peer_mapping_error() on dma_addr_t "
+				"[device address=0x%016llx] [size=%llu bytes]\n",
+				(unsigned long long)entry->dev_addr,
+				entry->size);
+
+	if (!peer && (entry->type == dma_debug_peer_resource))
+		err_printk(entry->dev, entry, "DMA-API: device driver calls "
+				"dma_mapping_error() on dma_peer_addr_t "
+				"[device address=0x%016llx] [size=%llu bytes]\n",
+				(unsigned long long)entry->dev_addr,
+				entry->size);
+}
+
 void debug_dma_map_page(struct device *dev, struct page *page, size_t offset,
 			size_t size, int direction, dma_addr_t dma_addr,
 			bool map_single)
@@ -1289,7 +1308,9 @@ void debug_dma_map_page(struct device *dev, struct page *page, size_t offset,
 }
 EXPORT_SYMBOL(debug_dma_map_page);
 
-void debug_dma_mapping_error(struct device *dev, dma_addr_t dma_addr)
+static void debug_dma_mapping_error_common(struct device *dev,
+					   u64 dma_addr,
+					   bool peer)
 {
 	struct dma_debug_entry ref;
 	struct dma_debug_entry *entry;
@@ -1319,12 +1340,18 @@ void debug_dma_mapping_error(struct device *dev, dma_addr_t dma_addr)
 		 */
 		if (entry->map_err_type == MAP_ERR_NOT_CHECKED) {
 			entry->map_err_type = MAP_ERR_CHECKED;
+			check_mapping_error(entry, peer);
 			break;
 		}
 	}
 
 	put_hash_bucket(bucket, &flags);
 }
+
+void debug_dma_mapping_error(struct device *dev, dma_addr_t dma_addr)
+{
+	debug_dma_mapping_error_common(dev, (u64)dma_addr, false);
+}
 EXPORT_SYMBOL(debug_dma_mapping_error);
 
 void debug_dma_unmap_page(struct device *dev, dma_addr_t addr,
@@ -1348,6 +1375,61 @@ void debug_dma_unmap_page(struct device *dev, dma_addr_t addr,
 }
 EXPORT_SYMBOL(debug_dma_unmap_page);
 
+#ifdef CONFIG_HAS_DMA_P2P
+void debug_dma_map_peer_resource(struct device *dev, struct device *peer,
+				 struct resource *resource, size_t offset,
+				 size_t size, int direction,
+				 dma_peer_addr_t dma_addr)
+{
+	struct dma_debug_entry *entry;
+
+	if (unlikely(dma_debug_disabled()))
+		return;
+
+	if (dma_mapping_error(dev, dma_addr))
+		return;
+
+	entry = dma_entry_alloc();
+	if (!entry)
+		return;
+
+	entry->dev       = dev;
+	entry->type      = dma_debug_peer_resource;
+	entry->pfn       = resource->start >> PAGE_SHIFT;
+	entry->offset    = offset,
+	entry->dev_addr  = dma_addr;
+	entry->size      = size;
+	entry->direction = direction;
+
+	add_dma_entry(entry);
+}
+EXPORT_SYMBOL(debug_dma_map_peer_resource);
+
+void debug_dma_peer_mapping_error(struct device *dev, dma_peer_addr_t dma_addr)
+{
+	debug_dma_mapping_error_common(dev, (u64)dma_addr, true);
+}
+EXPORT_SYMBOL(debug_dma_peer_mapping_error);
+
+void debug_dma_unmap_peer_resource(struct device *dev, dma_peer_addr_t addr,
+				   size_t size, int direction)
+{
+	struct dma_debug_entry ref = {
+		.type           = dma_debug_peer_resource,
+		.dev            = dev,
+		.dev_addr       = addr,
+		.size           = size,
+		.direction      = direction,
+	};
+
+	if (unlikely(dma_debug_disabled()))
+		return;
+
+	check_unmap(&ref);
+}
+EXPORT_SYMBOL(debug_dma_unmap_peer_resource);
+#endif
+
 void debug_dma_map_sg(struct device *dev, struct scatterlist *sg,
 		      int nents, int mapped_ents, int direction)
 {
-- 
2.5.1

[PATCH 03/22] dma-debug: add checking for map/unmap_peer_resource

[Will Davis]

Add debug callbacks for the new dma_map_peer_resource and
dma_unmap_peer_resource functions.

Also add a callback for dma_peer_mapping_error. Most of the time this will
be identical to dma_mapping_error, except where the dma_addr_t is not as
wide as dma_peer_addr_t.

Guard these new callbacks behind a CONFIG_HAS_DMA_P2P check.

Signed-off-by: Will Davis <wdavis@nvidia.com>
Reviewed-by: Terence Ripperda <tripperda@nvidia.com>
Reviewed-by: John Hubbard <jhubbard@nvidia.com>
---
 include/linux/dma-debug.h | 39 ++++++++++++++++++++-
 lib/dma-debug.c           | 86 +++++++++++++++++++++++++++++++++++++++++++++--
 2 files changed, 122 insertions(+), 3 deletions(-)

diff --git a/include/linux/dma-debug.h b/include/linux/dma-debug.h
index fe8cb61..4ff8364 100644
--- a/include/linux/dma-debug.h
+++ b/include/linux/dma-debug.h
@@ -43,7 +43,21 @@ extern void debug_dma_mapping_error(struct device *dev, dma_addr_t dma_addr);
 
 extern void debug_dma_unmap_page(struct device *dev, dma_addr_t addr,
 				 size_t size, int direction, bool map_single);
-
+#ifdef CONFIG_HAS_DMA_P2P
+extern void debug_dma_map_peer_resource(struct device *dev,
+					struct device *peer,
+					struct resource *resource,
+					size_t offset, size_t size,
+					int direction,
+					dma_peer_addr_t dma_addr);
+
+extern void debug_dma_peer_mapping_error(struct device *dev,
+					 dma_peer_addr_t dma_addr);
+
+extern void debug_dma_unmap_peer_resource(struct device *dev,
+					  dma_peer_addr_t addr,
+					  size_t size, int direction);
+#endif
 extern void debug_dma_map_sg(struct device *dev, struct scatterlist *sg,
 			     int nents, int mapped_ents, int direction);
 
@@ -120,6 +134,29 @@ static inline void debug_dma_unmap_page(struct device *dev, dma_addr_t addr,
 {
 }
 
+#ifdef CONFIG_HAS_DMA_P2P
+static inline void debug_dma_map_peer_resource(struct device *dev,
+					       struct device *peer,
+					       struct resource *resource,
+					       size_t offset, size_t size,
+					       int direction,
+					       dma_peer_addr_t dma_addr)
+{
+}
+
+static inline void debug_dma_peer_mapping_error(struct device *dev,
+						dma_peer_addr_t dma_addr)
+{
+}
+
+static inline void debug_dma_unmap_peer_resource(struct device *dev,
+						 dma_peer_addr_t addr,
+						 size_t size,
+						 int direction)
+{
+}
+#endif
+
 static inline void debug_dma_map_sg(struct device *dev, struct scatterlist *sg,
 				    int nents, int mapped_ents, int direction)
 {
diff --git a/lib/dma-debug.c b/lib/dma-debug.c
index ae4b65e..622f569 100644
--- a/lib/dma-debug.c
+++ b/lib/dma-debug.c
@@ -43,6 +43,7 @@ enum {
 	dma_debug_page,
 	dma_debug_sg,
 	dma_debug_coherent,
+	dma_debug_peer_resource,
 };
 
 enum map_err_types {
@@ -64,7 +65,8 @@ enum map_err_types {
  * @direction: enum dma_data_direction
  * @sg_call_ents: 'nents' from dma_map_sg
  * @sg_mapped_ents: 'mapped_ents' from dma_map_sg
- * @map_err_type: track whether dma_mapping_error() was checked
+ * @map_err_type: track whether dma_mapping_error() or dma_peer_mapping_error()
+ *		  was called
  * @stacktrace: support backtraces when a violation is detected
  */
 struct dma_debug_entry {
@@ -1250,6 +1252,23 @@ out:
 	put_hash_bucket(bucket, &flags);
 }
 
+static void check_mapping_error(struct dma_debug_entry *entry, bool peer)
+{
+	if (peer && !(entry->type == dma_debug_peer_resource))
+		err_printk(entry->dev, entry, "DMA-API: device driver calls "
+				"dma_peer_mapping_error() on dma_addr_t "
+				"[device address=0x%016llx] [size=%llu bytes]\n",
+				(unsigned long long)entry->dev_addr,
+				entry->size);
+
+	if (!peer && (entry->type == dma_debug_peer_resource))
+		err_printk(entry->dev, entry, "DMA-API: device driver calls "
+				"dma_mapping_error() on dma_peer_addr_t "
+				"[device address=0x%016llx] [size=%llu bytes]\n",
+				(unsigned long long)entry->dev_addr,
+				entry->size);
+}
+
 void debug_dma_map_page(struct device *dev, struct page *page, size_t offset,
 			size_t size, int direction, dma_addr_t dma_addr,
 			bool map_single)
@@ -1289,7 +1308,9 @@ void debug_dma_map_page(struct device *dev, struct page *page, size_t offset,
 }
 EXPORT_SYMBOL(debug_dma_map_page);
 
-void debug_dma_mapping_error(struct device *dev, dma_addr_t dma_addr)
+static void debug_dma_mapping_error_common(struct device *dev,
+					   u64 dma_addr,
+					   bool peer)
 {
 	struct dma_debug_entry ref;
 	struct dma_debug_entry *entry;
@@ -1319,12 +1340,18 @@ void debug_dma_mapping_error(struct device *dev, dma_addr_t dma_addr)
 		 */
 		if (entry->map_err_type == MAP_ERR_NOT_CHECKED) {
 			entry->map_err_type = MAP_ERR_CHECKED;
+			check_mapping_error(entry, peer);
 			break;
 		}
 	}
 
 	put_hash_bucket(bucket, &flags);
 }
+
+void debug_dma_mapping_error(struct device *dev, dma_addr_t dma_addr)
+{
+	debug_dma_mapping_error_common(dev, (u64)dma_addr, false);
+}
 EXPORT_SYMBOL(debug_dma_mapping_error);
 
 void debug_dma_unmap_page(struct device *dev, dma_addr_t addr,
@@ -1348,6 +1375,61 @@ void debug_dma_unmap_page(struct device *dev, dma_addr_t addr,
 }
 EXPORT_SYMBOL(debug_dma_unmap_page);
 
+#ifdef CONFIG_HAS_DMA_P2P
+void debug_dma_map_peer_resource(struct device *dev, struct device *peer,
+				 struct resource *resource, size_t offset,
+				 size_t size, int direction,
+				 dma_peer_addr_t dma_addr)
+{
+	struct dma_debug_entry *entry;
+
+	if (unlikely(dma_debug_disabled()))
+		return;
+
+	if (dma_mapping_error(dev, dma_addr))
+		return;
+
+	entry = dma_entry_alloc();
+	if (!entry)
+		return;
+
+	entry->dev       = dev;
+	entry->type      = dma_debug_peer_resource;
+	entry->pfn       = resource->start >> PAGE_SHIFT;
+	entry->offset    = offset,
+	entry->dev_addr  = dma_addr;
+	entry->size      = size;
+	entry->direction = direction;
+
+	add_dma_entry(entry);
+}
+EXPORT_SYMBOL(debug_dma_map_peer_resource);
+
+void debug_dma_peer_mapping_error(struct device *dev, dma_peer_addr_t dma_addr)
+{
+	debug_dma_mapping_error_common(dev, (u64)dma_addr, true);
+}
+EXPORT_SYMBOL(debug_dma_peer_mapping_error);
+
+void debug_dma_unmap_peer_resource(struct device *dev, dma_peer_addr_t addr,
+				   size_t size, int direction)
+{
+	struct dma_debug_entry ref = {
+		.type           = dma_debug_peer_resource,
+		.dev            = dev,
+		.dev_addr       = addr,
+		.size           = size,
+		.direction      = direction,
+	};
+
+	if (unlikely(dma_debug_disabled()))
+		return;
+
+	check_unmap(&ref);
+}
+EXPORT_SYMBOL(debug_dma_unmap_peer_resource);
+#endif
+
 void debug_dma_map_sg(struct device *dev, struct scatterlist *sg,
 		      int nents, int mapped_ents, int direction)
 {
-- 
2.5.1

[PATCH 04/22] DMA-API: Introduce dma_(un)map_peer_resource

[Will Davis]

Add functions to DMA-map and -unmap a peer device's resource for a given
device. This will allow devices to DMA-map, for example, another device's
BAR region on PCI to enable peer-to-peer transactions.

Guard these new functions behind CONFIG_HAS_DMA_P2P.

Signed-off-by: Will Davis <wdavis@nvidia.com>
---
 include/asm-generic/dma-mapping-common.h | 43 ++++++++++++++++++++++++++++++++
 include/linux/dma-mapping.h              | 12 +++++++++
 2 files changed, 55 insertions(+)

diff --git a/include/asm-generic/dma-mapping-common.h b/include/asm-generic/dma-mapping-common.h
index 940d5ec..45eec17 100644
--- a/include/asm-generic/dma-mapping-common.h
+++ b/include/asm-generic/dma-mapping-common.h
@@ -73,6 +73,42 @@ static inline void dma_unmap_sg_attrs(struct device *dev, struct scatterlist *sg
 		ops->unmap_sg(dev, sg, nents, dir, attrs);
 }
 
+#ifdef CONFIG_HAS_DMA_P2P
+static inline dma_peer_addr_t dma_map_peer_resource_attrs(struct device *dev,
+							  struct device *peer,
+							  struct resource *res,
+							  size_t offset,
+							  size_t size,
+							  enum dma_data_direction dir,
+							  struct dma_attrs *attrs)
+{
+	struct dma_map_ops *ops = get_dma_ops(dev);
+	dma_peer_addr_t addr;
+
+	BUG_ON(!valid_dma_direction(dir));
+	BUG_ON(ops->map_peer_resource == NULL);
+	addr = ops->map_peer_resource(dev, peer, res, offset, size, dir,
+				      attrs);
+	debug_dma_map_peer_resource(dev, peer, res, offset, size, dir, addr);
+
+	return addr;
+}
+
+static inline void dma_unmap_peer_resource_attrs(struct device *dev,
+						 dma_peer_addr_t addr,
+						 size_t size,
+						 enum dma_data_direction dir,
+						 struct dma_attrs *attrs)
+{
+	struct dma_map_ops *ops = get_dma_ops(dev);
+
+	BUG_ON(!valid_dma_direction(dir));
+	if (ops->unmap_peer_resource)
+		ops->unmap_peer_resource(dev, addr, size, dir, attrs);
+	debug_dma_unmap_peer_resource(dev, addr, size, dir);
+}
+#endif
+
 static inline dma_addr_t dma_map_page(struct device *dev, struct page *page,
 				      size_t offset, size_t size,
 				      enum dma_data_direction dir)
@@ -181,6 +217,13 @@ dma_sync_sg_for_device(struct device *dev, struct scatterlist *sg,
 #define dma_map_sg(d, s, n, r) dma_map_sg_attrs(d, s, n, r, NULL)
 #define dma_unmap_sg(d, s, n, r) dma_unmap_sg_attrs(d, s, n, r, NULL)
 
+#ifdef CONFIG_HAS_DMA_P2P
+#define dma_map_peer_resource(d, p, e, o, s, r) \
+	dma_map_peer_resource_attrs(d, p, e, o, s, r, NULL)
+#define dma_unmap_peer_resource(d, a, s, r) \
+	dma_unmap_peer_resource_attrs(d, a, s, r, NULL)
+#endif
+
 extern int dma_common_mmap(struct device *dev, struct vm_area_struct *vma,
 			   void *cpu_addr, dma_addr_t dma_addr, size_t size);
 
diff --git a/include/linux/dma-mapping.h b/include/linux/dma-mapping.h
index ac07ff0..7b8fddc 100644
--- a/include/linux/dma-mapping.h
+++ b/include/linux/dma-mapping.h
@@ -34,6 +34,18 @@ struct dma_map_ops {
 	void (*unmap_page)(struct device *dev, dma_addr_t dma_handle,
 			   size_t size, enum dma_data_direction dir,
 			   struct dma_attrs *attrs);
+#ifdef CONFIG_HAS_DMA_P2P
+	dma_peer_addr_t (*map_peer_resource)(struct device *dev,
+					     struct device *peer,
+					     struct resource *res,
+					     unsigned long offset, size_t size,
+					     enum dma_data_direction dir,
+					     struct dma_attrs *attrs);
+	void (*unmap_peer_resource)(struct device *dev,
+				    dma_peer_addr_t dma_handle,
+				    size_t size, enum dma_data_direction dir,
+				    struct dma_attrs *attrs);
+#endif
 	/*
 	 * map_sg returns 0 on error and a value > 0 on success.
 	 * It should never return a value < 0.
-- 
2.5.1

[PATCH 04/22] DMA-API: Introduce dma_(un)map_peer_resource

[Will Davis]

Add functions to DMA-map and -unmap a peer device's resource for a given
device. This will allow devices to DMA-map, for example, another device's
BAR region on PCI to enable peer-to-peer transactions.

Guard these new functions behind CONFIG_HAS_DMA_P2P.

Signed-off-by: Will Davis <wdavis-DDmLM1+adcrQT0dZR+AlfA@public.gmane.org>
---
 include/asm-generic/dma-mapping-common.h | 43 ++++++++++++++++++++++++++++++++
 include/linux/dma-mapping.h              | 12 +++++++++
 2 files changed, 55 insertions(+)

diff --git a/include/asm-generic/dma-mapping-common.h b/include/asm-generic/dma-mapping-common.h
index 940d5ec..45eec17 100644
--- a/include/asm-generic/dma-mapping-common.h
+++ b/include/asm-generic/dma-mapping-common.h
@@ -73,6 +73,42 @@ static inline void dma_unmap_sg_attrs(struct device *dev, struct scatterlist *sg
 		ops->unmap_sg(dev, sg, nents, dir, attrs);
 }
 
+#ifdef CONFIG_HAS_DMA_P2P
+static inline dma_peer_addr_t dma_map_peer_resource_attrs(struct device *dev,
+							  struct device *peer,
+							  struct resource *res,
+							  size_t offset,
+							  size_t size,
+							  enum dma_data_direction dir,
+							  struct dma_attrs *attrs)
+{
+	struct dma_map_ops *ops = get_dma_ops(dev);
+	dma_peer_addr_t addr;
+
+	BUG_ON(!valid_dma_direction(dir));
+	BUG_ON(ops->map_peer_resource == NULL);
+	addr = ops->map_peer_resource(dev, peer, res, offset, size, dir,
+				      attrs);
+	debug_dma_map_peer_resource(dev, peer, res, offset, size, dir, addr);
+
+	return addr;
+}
+
+static inline void dma_unmap_peer_resource_attrs(struct device *dev,
+						 dma_peer_addr_t addr,
+						 size_t size,
+						 enum dma_data_direction dir,
+						 struct dma_attrs *attrs)
+{
+	struct dma_map_ops *ops = get_dma_ops(dev);
+
+	BUG_ON(!valid_dma_direction(dir));
+	if (ops->unmap_peer_resource)
+		ops->unmap_peer_resource(dev, addr, size, dir, attrs);
+	debug_dma_unmap_peer_resource(dev, addr, size, dir);
+}
+#endif
+
 static inline dma_addr_t dma_map_page(struct device *dev, struct page *page,
 				      size_t offset, size_t size,
 				      enum dma_data_direction dir)
@@ -181,6 +217,13 @@ dma_sync_sg_for_device(struct device *dev, struct scatterlist *sg,
 #define dma_map_sg(d, s, n, r) dma_map_sg_attrs(d, s, n, r, NULL)
 #define dma_unmap_sg(d, s, n, r) dma_unmap_sg_attrs(d, s, n, r, NULL)
 
+#ifdef CONFIG_HAS_DMA_P2P
+#define dma_map_peer_resource(d, p, e, o, s, r) \
+	dma_map_peer_resource_attrs(d, p, e, o, s, r, NULL)
+#define dma_unmap_peer_resource(d, a, s, r) \
+	dma_unmap_peer_resource_attrs(d, a, s, r, NULL)
+#endif
+
 extern int dma_common_mmap(struct device *dev, struct vm_area_struct *vma,
 			   void *cpu_addr, dma_addr_t dma_addr, size_t size);
 
diff --git a/include/linux/dma-mapping.h b/include/linux/dma-mapping.h
index ac07ff0..7b8fddc 100644
--- a/include/linux/dma-mapping.h
+++ b/include/linux/dma-mapping.h
@@ -34,6 +34,18 @@ struct dma_map_ops {
 	void (*unmap_page)(struct device *dev, dma_addr_t dma_handle,
 			   size_t size, enum dma_data_direction dir,
 			   struct dma_attrs *attrs);
+#ifdef CONFIG_HAS_DMA_P2P
+	dma_peer_addr_t (*map_peer_resource)(struct device *dev,
+					     struct device *peer,
+					     struct resource *res,
+					     unsigned long offset, size_t size,
+					     enum dma_data_direction dir,
+					     struct dma_attrs *attrs);
+	void (*unmap_peer_resource)(struct device *dev,
+				    dma_peer_addr_t dma_handle,
+				    size_t size, enum dma_data_direction dir,
+				    struct dma_attrs *attrs);
+#endif
 	/*
 	 * map_sg returns 0 on error and a value > 0 on success.
 	 * It should never return a value < 0.
-- 
2.5.1

[PATCH 05/22] dma-mapping: pci: add pci_(un)map_peer_resource

[Will Davis]

Simply route these through to the new dma_(un)map_peer_resource APIs.

Also route pci_peer_mapping_error through to the new dma_peer_mapping_error
API.

Guard these new functions behind CONFIG_HAS_DMA_P2P.

Signed-off-by: Will Davis <wdavis@nvidia.com>
Reviewed-by: Terence Ripperda <tripperda@nvidia.com>
Reviewed-by: John Hubbard <jhubbard@nvidia.com>
---
 include/asm-generic/pci-dma-compat.h | 23 +++++++++++++++++++++++
 1 file changed, 23 insertions(+)

diff --git a/include/asm-generic/pci-dma-compat.h b/include/asm-generic/pci-dma-compat.h
index c110843..7baf09e 100644
--- a/include/asm-generic/pci-dma-compat.h
+++ b/include/asm-generic/pci-dma-compat.h
@@ -61,6 +61,29 @@ pci_unmap_page(struct pci_dev *hwdev, dma_addr_t dma_address,
 	dma_unmap_page(hwdev == NULL ? NULL : &hwdev->dev, dma_address, size, (enum dma_data_direction)direction);
 }
 
+#ifdef CONFIG_HAS_DMA_P2P
+static inline dma_peer_addr_t
+pci_map_peer_resource(struct pci_dev *hwdev, struct pci_dev *peer,
+		      int res, unsigned long offset, size_t size,
+		      int direction)
+{
+	return dma_map_peer_resource(hwdev == NULL ? NULL : &hwdev->dev, peer == NULL ? NULL : &peer->dev, peer == NULL ? NULL : &peer->resource[res], offset, size, (enum dma_data_direction)direction);
+}
+
+static inline int
+pci_dma_peer_mapping_error(struct pci_dev *pdev, dma_peer_addr_t dma_addr)
+{
+	return dma_peer_mapping_error(&pdev->dev, dma_addr);
+}
+
+static inline void
+pci_unmap_peer_resource(struct pci_dev *hwdev, dma_peer_addr_t dma_address,
+			size_t size, int direction)
+{
+	dma_unmap_peer_resource(hwdev == NULL ? NULL : &hwdev->dev, dma_address, size, (enum dma_data_direction)direction);
+}
+#endif
+
 static inline int
 pci_map_sg(struct pci_dev *hwdev, struct scatterlist *sg,
 	   int nents, int direction)
-- 
2.5.1

[PATCH 05/22] dma-mapping: pci: add pci_(un)map_peer_resource

[Will Davis]

Simply route these through to the new dma_(un)map_peer_resource APIs.

Also route pci_peer_mapping_error through to the new dma_peer_mapping_error
API.

Guard these new functions behind CONFIG_HAS_DMA_P2P.

Signed-off-by: Will Davis <wdavis-DDmLM1+adcrQT0dZR+AlfA@public.gmane.org>
Reviewed-by: Terence Ripperda <tripperda-DDmLM1+adcrQT0dZR+AlfA@public.gmane.org>
Reviewed-by: John Hubbard <jhubbard-DDmLM1+adcrQT0dZR+AlfA@public.gmane.org>
---
 include/asm-generic/pci-dma-compat.h | 23 +++++++++++++++++++++++
 1 file changed, 23 insertions(+)

diff --git a/include/asm-generic/pci-dma-compat.h b/include/asm-generic/pci-dma-compat.h
index c110843..7baf09e 100644
--- a/include/asm-generic/pci-dma-compat.h
+++ b/include/asm-generic/pci-dma-compat.h
@@ -61,6 +61,29 @@ pci_unmap_page(struct pci_dev *hwdev, dma_addr_t dma_address,
 	dma_unmap_page(hwdev == NULL ? NULL : &hwdev->dev, dma_address, size, (enum dma_data_direction)direction);
 }
 
+#ifdef CONFIG_HAS_DMA_P2P
+static inline dma_peer_addr_t
+pci_map_peer_resource(struct pci_dev *hwdev, struct pci_dev *peer,
+		      int res, unsigned long offset, size_t size,
+		      int direction)
+{
+	return dma_map_peer_resource(hwdev == NULL ? NULL : &hwdev->dev, peer == NULL ? NULL : &peer->dev, peer == NULL ? NULL : &peer->resource[res], offset, size, (enum dma_data_direction)direction);
+}
+
+static inline int
+pci_dma_peer_mapping_error(struct pci_dev *pdev, dma_peer_addr_t dma_addr)
+{
+	return dma_peer_mapping_error(&pdev->dev, dma_addr);
+}
+
+static inline void
+pci_unmap_peer_resource(struct pci_dev *hwdev, dma_peer_addr_t dma_address,
+			size_t size, int direction)
+{
+	dma_unmap_peer_resource(hwdev == NULL ? NULL : &hwdev->dev, dma_address, size, (enum dma_data_direction)direction);
+}
+#endif
+
 static inline int
 pci_map_sg(struct pci_dev *hwdev, struct scatterlist *sg,
 	   int nents, int direction)
-- 
2.5.1

[PATCH 06/22] DMA-API: Add peer resource mapping documentation

[Will Davis]

Add references to both the general API documentation, as well as the HOWTO,
about the dma_(un)map_peer_resource() and dma_peer_mapping_error() APIs.

Signed-off-by: Will Davis <wdavis@nvidia.com>
---
 Documentation/DMA-API-HOWTO.txt | 69 +++++++++++++++++++++++++++++++++++++++++
 Documentation/DMA-API.txt       | 38 ++++++++++++++++++++---
 2 files changed, 102 insertions(+), 5 deletions(-)

diff --git a/Documentation/DMA-API-HOWTO.txt b/Documentation/DMA-API-HOWTO.txt
index 55b70b9..053a18f 100644
--- a/Documentation/DMA-API-HOWTO.txt
+++ b/Documentation/DMA-API-HOWTO.txt
@@ -143,6 +143,10 @@ What about block I/O and networking buffers?  The block I/O and
 networking subsystems make sure that the buffers they use are valid
 for you to DMA from/to.
 
+In some systems, it may also be possible to DMA to and/or from a peer
+device's MMIO region, as described by a 'struct resource'. This is
+referred to as a peer-to-peer mapping.
+
 			DMA addressing limitations
 
 Does your device have any DMA addressing limitations?  For example, is
@@ -653,6 +657,45 @@ Every dma_map_{single,sg}() call should have its dma_unmap_{single,sg}()
 counterpart, because the DMA address space is a shared resource and
 you could render the machine unusable by consuming all DMA addresses.
 
+Peer-to-peer DMA mappings can be obtained using dma_map_peer_resource() to
+map another device's MMIO region for the given device:
+
+	dma_peer_addr_t dma_handle = dma_map_peer_resource(dev, peer, res,
+							   offset, size,
+							   direction);
+	if (dma_peer_mapping_error(dev, dma_handle))
+	{
+		/*
+		 * reduce current DMA mapping usage,
+		 * delay and try again later or
+		 * reset driver. P2P DMA may not be
+		 * supported between the devices.
+		 */
+		goto map_error_handling;
+	}
+
+	...
+
+	dma_unmap_peer_resource(dev, dma_handle, size, direction);
+
+Here, "res" is the struct resource * of the MMIO region, and "offset" means
+byte offset within the given resource.
+
+You should call dma_peer_mapping_error() as dma_map_peer_resource() could fail
+and return error.
+
+Not all implementations support dma_map_peer_resource(), but those that do
+should also implement dma_peer_mapping_error(). dma_map_peer_resource() may
+fail for various reasons depending on the implementation, but generally
+may fail for the same reasons as those outlined under the dma_map_single()
+discussion. In addition, the success of dma_map_peer_resource() may depend
+on the physical topology between the two devices. For example, in the x86
+implementations, peer-to-peer mappings will fail if the devices are behind
+different host bridges.
+
+You should call dma_unmap_peer_resource() when DMA activity is finished,
+e.g., from the interrupt which told you that the DMA transfer is done.
+
 If you need to use the same streaming DMA region multiple times and touch
 the data in between the DMA transfers, the buffer needs to be synced
 properly in order for the CPU and device to see the most up-to-date and
@@ -785,6 +828,23 @@ failure can be determined by:
 		goto map_error_handling;
 	}
 
+- checking the dma_peer_addr_t returned from dma_map_peer_resource() by using
+  dma_peer_mapping_error():
+
+  	dma_peer_addr_t dma_handle;
+
+	dma_handle = dma_map_peer_resource(dev, peer, res, offset, size,
+					   direction);
+	if (dma_peer_mapping_error(dev, dma_handle)) {
+		/*
+		 * reduce current DMA mapping usage,
+		 * delay and try again later or
+		 * reset driver. P2P DMA may not be
+		 * supported between the devices.
+		 */
+		goto map_error_handling;
+	}
+
 - unmap pages that are already mapped, when mapping error occurs in the middle
   of a multiple page mapping attempt. These example are applicable to
   dma_map_page() as well.
@@ -956,6 +1016,15 @@ to "Closing".
    support multiple types of IOMMUs in a single system, the example of
    x86 or powerpc helps.
 
+4) Peer-to-peer requirements.
+
+   If your platform can support peer-to-peer DMA, enable HAS_DMA_P2P to
+   include the peer-resource mapping interfaces in the DMA API, and
+   provide an implementation of dma_peer_mapping_error(). The latter part
+   is necessary because on some platforms, the dma_addr_t is not as wide
+   as the full bus address, and therefore the existing dma_mapping_error()
+   API may not be sufficient.
+
 			   Closing
 
 This document, and the API itself, would not be in its current
diff --git a/Documentation/DMA-API.txt b/Documentation/DMA-API.txt
index 7eba542..c6f11a1 100644
--- a/Documentation/DMA-API.txt
+++ b/Documentation/DMA-API.txt
@@ -283,14 +283,42 @@ and <size> parameters are provided to do partial page mapping, it is
 recommended that you never use these unless you really know what the
 cache width is.
 
+dma_peer_addr_t
+dma_map_peer_resource(struct device *dev, struct device *peer,
+		      struct resource *res, unsigned long offset,
+		      size_t size, enum dma_data_direction_direction)
+
+API for mapping resources. This API allows a driver to map a peer
+device's resource for DMA. All the notes and warnings for the other
+APIs apply here. Also, the success of this API does not validate or
+guarantee that peer-to-peer transactions between the device and its
+peer will be functional. They only grant access so that if such
+transactions are possible, an IOMMU will not prevent them from
+succeeding. However, a failure of this API may indicate that peer-to-peer
+transactions are not expected to succeed across the underlying topology
+between the devices.
+
+void
+dma_unmap_peer_resource(struct device *dev, dma_peer_addr_t dma_address,
+			size_t size, enum dma_data_direction direction)
+
+Unmaps the resource previously mapped. All the parameters passed in
+must be identical to those passed in to (and returned by) the mapping
+API.
+
 int
 dma_mapping_error(struct device *dev, dma_addr_t dma_addr)
 
-In some circumstances dma_map_single() and dma_map_page() will fail to create
-a mapping. A driver can check for these errors by testing the returned
-DMA address with dma_mapping_error(). A non-zero return value means the mapping
-could not be created and the driver should take appropriate action (e.g.
-reduce current DMA mapping usage or delay and try again later).
+int
+dma_peer_mapping_error(struct device *dev, dma_peer_addr_t dma_addr)
+
+In some circumstances dma_map_single(), dma_map_page() and
+dma_map_peer_resource() will fail to create a mapping. A driver can check for
+these errors by testing the returned DMA address with dma_mapping_error()
+(or dma_peer_mapping_error() for dma_map_peer_resource()). A non-zero return
+value means the mapping could not be created and the driver should take
+appropriate action (e.g. reduce current DMA mapping usage or delay and try
+again later).
 
 	int
 	dma_map_sg(struct device *dev, struct scatterlist *sg,
-- 
2.5.1

[PATCH 06/22] DMA-API: Add peer resource mapping documentation

[Will Davis]

Add references to both the general API documentation, as well as the HOWTO,
about the dma_(un)map_peer_resource() and dma_peer_mapping_error() APIs.

Signed-off-by: Will Davis <wdavis-DDmLM1+adcrQT0dZR+AlfA@public.gmane.org>
---
 Documentation/DMA-API-HOWTO.txt | 69 +++++++++++++++++++++++++++++++++++++++++
 Documentation/DMA-API.txt       | 38 ++++++++++++++++++++---
 2 files changed, 102 insertions(+), 5 deletions(-)

diff --git a/Documentation/DMA-API-HOWTO.txt b/Documentation/DMA-API-HOWTO.txt
index 55b70b9..053a18f 100644
--- a/Documentation/DMA-API-HOWTO.txt
+++ b/Documentation/DMA-API-HOWTO.txt
@@ -143,6 +143,10 @@ What about block I/O and networking buffers?  The block I/O and
 networking subsystems make sure that the buffers they use are valid
 for you to DMA from/to.
 
+In some systems, it may also be possible to DMA to and/or from a peer
+device's MMIO region, as described by a 'struct resource'. This is
+referred to as a peer-to-peer mapping.
+
 			DMA addressing limitations
 
 Does your device have any DMA addressing limitations?  For example, is
@@ -653,6 +657,45 @@ Every dma_map_{single,sg}() call should have its dma_unmap_{single,sg}()
 counterpart, because the DMA address space is a shared resource and
 you could render the machine unusable by consuming all DMA addresses.
 
+Peer-to-peer DMA mappings can be obtained using dma_map_peer_resource() to
+map another device's MMIO region for the given device:
+
+	dma_peer_addr_t dma_handle = dma_map_peer_resource(dev, peer, res,
+							   offset, size,
+							   direction);
+	if (dma_peer_mapping_error(dev, dma_handle))
+	{
+		/*
+		 * reduce current DMA mapping usage,
+		 * delay and try again later or
+		 * reset driver. P2P DMA may not be
+		 * supported between the devices.
+		 */
+		goto map_error_handling;
+	}
+
+	...
+
+	dma_unmap_peer_resource(dev, dma_handle, size, direction);
+
+Here, "res" is the struct resource * of the MMIO region, and "offset" means
+byte offset within the given resource.
+
+You should call dma_peer_mapping_error() as dma_map_peer_resource() could fail
+and return error.
+
+Not all implementations support dma_map_peer_resource(), but those that do
+should also implement dma_peer_mapping_error(). dma_map_peer_resource() may
+fail for various reasons depending on the implementation, but generally
+may fail for the same reasons as those outlined under the dma_map_single()
+discussion. In addition, the success of dma_map_peer_resource() may depend
+on the physical topology between the two devices. For example, in the x86
+implementations, peer-to-peer mappings will fail if the devices are behind
+different host bridges.
+
+You should call dma_unmap_peer_resource() when DMA activity is finished,
+e.g., from the interrupt which told you that the DMA transfer is done.
+
 If you need to use the same streaming DMA region multiple times and touch
 the data in between the DMA transfers, the buffer needs to be synced
 properly in order for the CPU and device to see the most up-to-date and
@@ -785,6 +828,23 @@ failure can be determined by:
 		goto map_error_handling;
 	}
 
+- checking the dma_peer_addr_t returned from dma_map_peer_resource() by using
+  dma_peer_mapping_error():
+
+  	dma_peer_addr_t dma_handle;
+
+	dma_handle = dma_map_peer_resource(dev, peer, res, offset, size,
+					   direction);
+	if (dma_peer_mapping_error(dev, dma_handle)) {
+		/*
+		 * reduce current DMA mapping usage,
+		 * delay and try again later or
+		 * reset driver. P2P DMA may not be
+		 * supported between the devices.
+		 */
+		goto map_error_handling;
+	}
+
 - unmap pages that are already mapped, when mapping error occurs in the middle
   of a multiple page mapping attempt. These example are applicable to
   dma_map_page() as well.
@@ -956,6 +1016,15 @@ to "Closing".
    support multiple types of IOMMUs in a single system, the example of
    x86 or powerpc helps.
 
+4) Peer-to-peer requirements.
+
+   If your platform can support peer-to-peer DMA, enable HAS_DMA_P2P to
+   include the peer-resource mapping interfaces in the DMA API, and
+   provide an implementation of dma_peer_mapping_error(). The latter part
+   is necessary because on some platforms, the dma_addr_t is not as wide
+   as the full bus address, and therefore the existing dma_mapping_error()
+   API may not be sufficient.
+
 			   Closing
 
 This document, and the API itself, would not be in its current
diff --git a/Documentation/DMA-API.txt b/Documentation/DMA-API.txt
index 7eba542..c6f11a1 100644
--- a/Documentation/DMA-API.txt
+++ b/Documentation/DMA-API.txt
@@ -283,14 +283,42 @@ and <size> parameters are provided to do partial page mapping, it is
 recommended that you never use these unless you really know what the
 cache width is.
 
+dma_peer_addr_t
+dma_map_peer_resource(struct device *dev, struct device *peer,
+		      struct resource *res, unsigned long offset,
+		      size_t size, enum dma_data_direction_direction)
+
+API for mapping resources. This API allows a driver to map a peer
+device's resource for DMA. All the notes and warnings for the other
+APIs apply here. Also, the success of this API does not validate or
+guarantee that peer-to-peer transactions between the device and its
+peer will be functional. They only grant access so that if such
+transactions are possible, an IOMMU will not prevent them from
+succeeding. However, a failure of this API may indicate that peer-to-peer
+transactions are not expected to succeed across the underlying topology
+between the devices.
+
+void
+dma_unmap_peer_resource(struct device *dev, dma_peer_addr_t dma_address,
+			size_t size, enum dma_data_direction direction)
+
+Unmaps the resource previously mapped. All the parameters passed in
+must be identical to those passed in to (and returned by) the mapping
+API.
+
 int
 dma_mapping_error(struct device *dev, dma_addr_t dma_addr)
 
-In some circumstances dma_map_single() and dma_map_page() will fail to create
-a mapping. A driver can check for these errors by testing the returned
-DMA address with dma_mapping_error(). A non-zero return value means the mapping
-could not be created and the driver should take appropriate action (e.g.
-reduce current DMA mapping usage or delay and try again later).
+int
+dma_peer_mapping_error(struct device *dev, dma_peer_addr_t dma_addr)
+
+In some circumstances dma_map_single(), dma_map_page() and
+dma_map_peer_resource() will fail to create a mapping. A driver can check for
+these errors by testing the returned DMA address with dma_mapping_error()
+(or dma_peer_mapping_error() for dma_map_peer_resource()). A non-zero return
+value means the mapping could not be created and the driver should take
+appropriate action (e.g. reduce current DMA mapping usage or delay and try
+again later).
 
 	int
 	dma_map_sg(struct device *dev, struct scatterlist *sg,
-- 
2.5.1

[PATCH 07/22] PCI: Export pci_find_host_bridge()

[Will Davis]

Return the struct pci_host_bridge corresponding to the given
struct pci_bus.

Signed-off-by: Will Davis <wdavis@nvidia.com>
---
 drivers/pci/host-bridge.c | 19 ++-----------------
 include/linux/pci.h       |  7 +++++++
 2 files changed, 9 insertions(+), 17 deletions(-)

diff --git a/drivers/pci/host-bridge.c b/drivers/pci/host-bridge.c
index 5f4a2e0..6c2c82a 100644
--- a/drivers/pci/host-bridge.c
+++ b/drivers/pci/host-bridge.c
@@ -8,25 +8,10 @@
 
 #include "pci.h"
 
-static struct pci_bus *find_pci_root_bus(struct pci_bus *bus)
-{
-	while (bus->parent)
-		bus = bus->parent;
-
-	return bus;
-}
-
-struct pci_host_bridge *pci_find_host_bridge(struct pci_bus *bus)
-{
-	struct pci_bus *root_bus = find_pci_root_bus(bus);
-
-	return to_pci_host_bridge(root_bus->bridge);
-}
-
 struct device *pci_get_host_bridge_device(struct pci_dev *dev)
 {
-	struct pci_bus *root_bus = find_pci_root_bus(dev->bus);
-	struct device *bridge = root_bus->bridge;
+	struct pci_host_bridge *host_bridge = pci_find_host_bridge(dev->bus);
+	struct device *bridge = &host_bridge->dev;
 
 	kobject_get(&bridge->kobj);
 	return bridge;
diff --git a/include/linux/pci.h b/include/linux/pci.h
index 8a0321a..d58063e 100644
--- a/include/linux/pci.h
+++ b/include/linux/pci.h
@@ -512,6 +512,13 @@ static inline struct pci_dev *pci_upstream_bridge(struct pci_dev *dev)
 	return dev->bus->self;
 }
 
+static inline struct pci_host_bridge *pci_find_host_bridge(struct pci_bus *bus)
+{
+	while (bus->parent)
+		bus = bus->parent;
+	return to_pci_host_bridge(bus->bridge);
+}
+
 struct device *pci_get_host_bridge_device(struct pci_dev *dev);
 void pci_put_host_bridge_device(struct device *dev);
 
-- 
2.5.1

[PATCH 07/22] PCI: Export pci_find_host_bridge()

[Will Davis]

Return the struct pci_host_bridge corresponding to the given
struct pci_bus.

Signed-off-by: Will Davis <wdavis@nvidia.com>
---
 drivers/pci/host-bridge.c | 19 ++-----------------
 include/linux/pci.h       |  7 +++++++
 2 files changed, 9 insertions(+), 17 deletions(-)

diff --git a/drivers/pci/host-bridge.c b/drivers/pci/host-bridge.c
index 5f4a2e0..6c2c82a 100644
--- a/drivers/pci/host-bridge.c
+++ b/drivers/pci/host-bridge.c
@@ -8,25 +8,10 @@
 
 #include "pci.h"
 
-static struct pci_bus *find_pci_root_bus(struct pci_bus *bus)
-{
-	while (bus->parent)
-		bus = bus->parent;
-
-	return bus;
-}
-
-struct pci_host_bridge *pci_find_host_bridge(struct pci_bus *bus)
-{
-	struct pci_bus *root_bus = find_pci_root_bus(bus);
-
-	return to_pci_host_bridge(root_bus->bridge);
-}
-
 struct device *pci_get_host_bridge_device(struct pci_dev *dev)
 {
-	struct pci_bus *root_bus = find_pci_root_bus(dev->bus);
-	struct device *bridge = root_bus->bridge;
+	struct pci_host_bridge *host_bridge = pci_find_host_bridge(dev->bus);
+	struct device *bridge = &host_bridge->dev;
 
 	kobject_get(&bridge->kobj);
 	return bridge;
diff --git a/include/linux/pci.h b/include/linux/pci.h
index 8a0321a..d58063e 100644
--- a/include/linux/pci.h
+++ b/include/linux/pci.h
@@ -512,6 +512,13 @@ static inline struct pci_dev *pci_upstream_bridge(struct pci_dev *dev)
 	return dev->bus->self;
 }
 
+static inline struct pci_host_bridge *pci_find_host_bridge(struct pci_bus *bus)
+{
+	while (bus->parent)
+		bus = bus->parent;
+	return to_pci_host_bridge(bus->bridge);
+}
+
 struct device *pci_get_host_bridge_device(struct pci_dev *dev);
 void pci_put_host_bridge_device(struct device *dev);
 
-- 
2.5.1

[PATCH 08/22] PCI: Add pci_find_common_upstream_dev()

[Will Davis]

Add an interface to find the first device which is upstream of both
devices.

Signed-off-by: Will Davis <wdavis@nvidia.com>
---
 drivers/pci/search.c | 25 +++++++++++++++++++++++++
 include/linux/pci.h  |  2 ++
 2 files changed, 27 insertions(+)

diff --git a/drivers/pci/search.c b/drivers/pci/search.c
index a20ce7d..b6ed554 100644
--- a/drivers/pci/search.c
+++ b/drivers/pci/search.c
@@ -384,3 +384,28 @@ int pci_dev_present(const struct pci_device_id *ids)
 	return 0;
 }
 EXPORT_SYMBOL(pci_dev_present);
+
+/**
+ * pci_find_common_upstream_dev - Returns the first common upstream device
+ * @dev: the PCI device from which the bus hierarchy walk will start
+ * @other: the PCI device whose bus number will be used for the search
+ *
+ * Walks up the bus hierarchy from the @dev device, looking for the first bus
+ * which the @other device is downstream of. Returns %NULL if the devices do
+ * not share any upstream devices.
+ */
+struct pci_dev *pci_find_common_upstream_dev(struct pci_dev *dev,
+					     struct pci_dev *other)
+{
+	struct pci_dev *pdev = dev;
+
+	while (pdev != NULL) {
+		if ((other->bus->number >= pdev->bus->busn_res.start) &&
+		    (other->bus->number <= pdev->bus->busn_res.end))
+			return pdev;
+		pdev = pci_upstream_bridge(pdev);
+	}
+
+	return NULL;
+}
+EXPORT_SYMBOL(pci_find_common_upstream_dev);
diff --git a/include/linux/pci.h b/include/linux/pci.h
index d58063e..8262b9e 100644
--- a/include/linux/pci.h
+++ b/include/linux/pci.h
@@ -860,6 +860,8 @@ static inline struct pci_dev *pci_get_bus_and_slot(unsigned int bus,
 }
 struct pci_dev *pci_get_class(unsigned int class, struct pci_dev *from);
 int pci_dev_present(const struct pci_device_id *ids);
+struct pci_dev *pci_find_common_upstream_dev(struct pci_dev *from,
+					     struct pci_dev *to);
 
 int pci_bus_read_config_byte(struct pci_bus *bus, unsigned int devfn,
 			     int where, u8 *val);
-- 
2.5.1

[PATCH 08/22] PCI: Add pci_find_common_upstream_dev()

[Will Davis]

Add an interface to find the first device which is upstream of both
devices.

Signed-off-by: Will Davis <wdavis@nvidia.com>
---
 drivers/pci/search.c | 25 +++++++++++++++++++++++++
 include/linux/pci.h  |  2 ++
 2 files changed, 27 insertions(+)

diff --git a/drivers/pci/search.c b/drivers/pci/search.c
index a20ce7d..b6ed554 100644
--- a/drivers/pci/search.c
+++ b/drivers/pci/search.c
@@ -384,3 +384,28 @@ int pci_dev_present(const struct pci_device_id *ids)
 	return 0;
 }
 EXPORT_SYMBOL(pci_dev_present);
+
+/**
+ * pci_find_common_upstream_dev - Returns the first common upstream device
+ * @dev: the PCI device from which the bus hierarchy walk will start
+ * @other: the PCI device whose bus number will be used for the search
+ *
+ * Walks up the bus hierarchy from the @dev device, looking for the first bus
+ * which the @other device is downstream of. Returns %NULL if the devices do
+ * not share any upstream devices.
+ */
+struct pci_dev *pci_find_common_upstream_dev(struct pci_dev *dev,
+					     struct pci_dev *other)
+{
+	struct pci_dev *pdev = dev;
+
+	while (pdev != NULL) {
+		if ((other->bus->number >= pdev->bus->busn_res.start) &&
+		    (other->bus->number <= pdev->bus->busn_res.end))
+			return pdev;
+		pdev = pci_upstream_bridge(pdev);
+	}
+
+	return NULL;
+}
+EXPORT_SYMBOL(pci_find_common_upstream_dev);
diff --git a/include/linux/pci.h b/include/linux/pci.h
index d58063e..8262b9e 100644
--- a/include/linux/pci.h
+++ b/include/linux/pci.h
@@ -860,6 +860,8 @@ static inline struct pci_dev *pci_get_bus_and_slot(unsigned int bus,
 }
 struct pci_dev *pci_get_class(unsigned int class, struct pci_dev *from);
 int pci_dev_present(const struct pci_device_id *ids);
+struct pci_dev *pci_find_common_upstream_dev(struct pci_dev *from,
+					     struct pci_dev *to);
 
 int pci_bus_read_config_byte(struct pci_bus *bus, unsigned int devfn,
 			     int where, u8 *val);
-- 
2.5.1

[PATCH 09/22] PCI: Add pci_peer_traffic_supported()

[Will Davis]

Add checks for topology and ACS configuration to determine whether or not
peer traffic should be supported between two PCI devices.

Signed-off-by: Will Davis <wdavis@nvidia.com>
---
 drivers/pci/pci.c   | 99 +++++++++++++++++++++++++++++++++++++++++++++++++++++
 include/linux/pci.h |  3 ++
 2 files changed, 102 insertions(+)

diff --git a/drivers/pci/pci.c b/drivers/pci/pci.c
index 0008c95..b8ba0f0 100644
--- a/drivers/pci/pci.c
+++ b/drivers/pci/pci.c
@@ -25,6 +25,7 @@
 #include <linux/device.h>
 #include <linux/pm_runtime.h>
 #include <linux/pci_hotplug.h>
+#include <linux/iommu.h>
 #include <asm-generic/pci-bridge.h>
 #include <asm/setup.h>
 #include "pci.h"
@@ -4302,6 +4303,104 @@ void pci_ignore_hotplug(struct pci_dev *dev)
 }
 EXPORT_SYMBOL_GPL(pci_ignore_hotplug);
 
+bool pci_peer_traffic_supported(struct pci_dev *dev, struct pci_dev *peer)
+{
+	struct pci_host_bridge *dev_host_bridge;
+	struct pci_host_bridge *peer_host_bridge;
+
+	/*
+	 * Disallow the peer-to-peer traffic if the devices do not share a
+	 * host bridge. The PCI specifications does not make any guarantees
+	 * about P2P capabilities between devices under separate domains.
+	 *
+	 * PCI Local Bus Specification Revision 3.0, section 3.10:
+	 *    "Peer-to-peer transactions crossing multiple host bridges
+	 *     PCI host bridges may, but are not required to, support PCI
+	 *     peer-to-peer transactions that traverse multiple PCI host
+	 *     bridges."
+	 */
+	dev_host_bridge = pci_find_host_bridge(dev->bus);
+	peer_host_bridge = pci_find_host_bridge(peer->bus);
+	if (dev_host_bridge != peer_host_bridge)
+		return false;
+
+	if (pci_is_pcie(dev) && pci_is_pcie(peer)) {
+		/*
+		 * Access Control Services (ACS) Checks
+		 *
+		 * ACS has a capability bit for P2P Request Redirects (RR),
+		 * but unfortunately it doesn't tell us much about the real
+		 * capabilities of the hardware.
+		 *
+		 * PCI Express Base Specification Revision 3.0, section
+		 * 6.12.1.1:
+		 *    "ACS P2P Request Redirect: must be implemented by Root
+		 *     Ports that support peer-to-peer traffic with other
+		 *     Root Ports; [80]"
+		 * but
+		 *    "[80] Root Port indication of ACS P2P Request Redirect
+		 *     or ACS P2P Completion Redirect support does not imply
+		 *     any particular level of peer-to-peer support by the
+		 *     Root Complex, or that peer-to-peer traffic is
+		 *     supported at all"
+		 */
+		struct pci_dev *rpdev = dev->bus->self;
+		struct pci_dev *rppeer = peer->bus->self;
+		struct pci_dev *common_upstream;
+		int pos;
+		u16 cap;
+
+		while ((rpdev) && (pci_is_pcie(rpdev)) &&
+		       (pci_pcie_type(rpdev) != PCI_EXP_TYPE_ROOT_PORT))
+			rpdev = rpdev->bus->self;
+
+		while ((rppeer) && (pci_is_pcie(rppeer)) &&
+		       (pci_pcie_type(rppeer) != PCI_EXP_TYPE_ROOT_PORT))
+			rppeer = rppeer->bus->self;
+
+		common_upstream = pci_find_common_upstream_dev(dev, peer);
+
+		/*
+		 * If ACS is not implemented, we have no idea about P2P
+		 * support. Optimistically allow this if there is a common
+		 * upstream device.
+		 */
+		pos = pci_find_ext_capability(rpdev, PCI_EXT_CAP_ID_ACS);
+		if (!pos)
+			return common_upstream != NULL;
+
+		/*
+		 * If the devices are under the same root port and have a common
+		 * upstream device, allow if the root port is further upstream
+		 * from the common upstream device and the common upstream
+		 * device has Upstream Forwarding disabled, or if the root port
+		 * is the common upstream device and ACS is not implemented.
+		 */
+		pci_read_config_word(rpdev, pos + PCI_ACS_CAP, &cap);
+		if ((rpdev == rppeer && common_upstream) &&
+		    (((common_upstream != rpdev) &&
+		      !pci_acs_enabled(common_upstream, PCI_ACS_UF)) ||
+		     ((common_upstream == rpdev) && ((cap & PCI_ACS_RR) == 0))))
+			return true;
+
+		/*
+		 * If ACS RR is implemented and disabled, allow only if the
+		 * devices are under the same root port.
+		 */
+		if (cap & PCI_ACS_RR && !pci_acs_enabled(rpdev, PCI_ACS_RR))
+			return rpdev == rppeer;
+
+		/*
+		 * If ACS RR is not implemented, or is implemented and enabled,
+		 * only allow if there's a translation agent enabled to do the
+		 * redirect.
+		 */
+		return iommu_present(&pci_bus_type);
+	}
+
+	return false;
+}
+
 #define RESOURCE_ALIGNMENT_PARAM_SIZE COMMAND_LINE_SIZE
 static char resource_alignment_param[RESOURCE_ALIGNMENT_PARAM_SIZE] = {0};
 static DEFINE_SPINLOCK(resource_alignment_lock);
diff --git a/include/linux/pci.h b/include/linux/pci.h
index 8262b9e..db0cb51 100644
--- a/include/linux/pci.h
+++ b/include/linux/pci.h
@@ -824,6 +824,8 @@ void pci_stop_root_bus(struct pci_bus *bus);
 void pci_remove_root_bus(struct pci_bus *bus);
 void pci_setup_cardbus(struct pci_bus *bus);
 void pci_sort_breadthfirst(void);
+bool pci_peer_traffic_supported(struct pci_dev *dev, struct pci_dev *peer);
+
 #define dev_is_pci(d) ((d)->bus == &pci_bus_type)
 #define dev_is_pf(d) ((dev_is_pci(d) ? to_pci_dev(d)->is_physfn : false))
 #define dev_num_vf(d) ((dev_is_pci(d) ? pci_num_vf(to_pci_dev(d)) : 0))
@@ -1914,4 +1916,5 @@ static inline bool pci_ari_enabled(struct pci_bus *bus)
 {
 	return bus->self && bus->self->ari_enabled;
 }
+
 #endif /* LINUX_PCI_H */
-- 
2.5.1

[PATCH 09/22] PCI: Add pci_peer_traffic_supported()

[Will Davis]

Add checks for topology and ACS configuration to determine whether or not
peer traffic should be supported between two PCI devices.

Signed-off-by: Will Davis <wdavis@nvidia.com>
---
 drivers/pci/pci.c   | 99 +++++++++++++++++++++++++++++++++++++++++++++++++++++
 include/linux/pci.h |  3 ++
 2 files changed, 102 insertions(+)

diff --git a/drivers/pci/pci.c b/drivers/pci/pci.c
index 0008c95..b8ba0f0 100644
--- a/drivers/pci/pci.c
+++ b/drivers/pci/pci.c
@@ -25,6 +25,7 @@
 #include <linux/device.h>
 #include <linux/pm_runtime.h>
 #include <linux/pci_hotplug.h>
+#include <linux/iommu.h>
 #include <asm-generic/pci-bridge.h>
 #include <asm/setup.h>
 #include "pci.h"
@@ -4302,6 +4303,104 @@ void pci_ignore_hotplug(struct pci_dev *dev)
 }
 EXPORT_SYMBOL_GPL(pci_ignore_hotplug);
 
+bool pci_peer_traffic_supported(struct pci_dev *dev, struct pci_dev *peer)
+{
+	struct pci_host_bridge *dev_host_bridge;
+	struct pci_host_bridge *peer_host_bridge;
+
+	/*
+	 * Disallow the peer-to-peer traffic if the devices do not share a
+	 * host bridge. The PCI specifications does not make any guarantees
+	 * about P2P capabilities between devices under separate domains.
+	 *
+	 * PCI Local Bus Specification Revision 3.0, section 3.10:
+	 *    "Peer-to-peer transactions crossing multiple host bridges
+	 *     PCI host bridges may, but are not required to, support PCI
+	 *     peer-to-peer transactions that traverse multiple PCI host
+	 *     bridges."
+	 */
+	dev_host_bridge = pci_find_host_bridge(dev->bus);
+	peer_host_bridge = pci_find_host_bridge(peer->bus);
+	if (dev_host_bridge != peer_host_bridge)
+		return false;
+
+	if (pci_is_pcie(dev) && pci_is_pcie(peer)) {
+		/*
+		 * Access Control Services (ACS) Checks
+		 *
+		 * ACS has a capability bit for P2P Request Redirects (RR),
+		 * but unfortunately it doesn't tell us much about the real
+		 * capabilities of the hardware.
+		 *
+		 * PCI Express Base Specification Revision 3.0, section
+		 * 6.12.1.1:
+		 *    "ACS P2P Request Redirect: must be implemented by Root
+		 *     Ports that support peer-to-peer traffic with other
+		 *     Root Ports; [80]"
+		 * but
+		 *    "[80] Root Port indication of ACS P2P Request Redirect
+		 *     or ACS P2P Completion Redirect support does not imply
+		 *     any particular level of peer-to-peer support by the
+		 *     Root Complex, or that peer-to-peer traffic is
+		 *     supported at all"
+		 */
+		struct pci_dev *rpdev = dev->bus->self;
+		struct pci_dev *rppeer = peer->bus->self;
+		struct pci_dev *common_upstream;
+		int pos;
+		u16 cap;
+
+		while ((rpdev) && (pci_is_pcie(rpdev)) &&
+		       (pci_pcie_type(rpdev) != PCI_EXP_TYPE_ROOT_PORT))
+			rpdev = rpdev->bus->self;
+
+		while ((rppeer) && (pci_is_pcie(rppeer)) &&
+		       (pci_pcie_type(rppeer) != PCI_EXP_TYPE_ROOT_PORT))
+			rppeer = rppeer->bus->self;
+
+		common_upstream = pci_find_common_upstream_dev(dev, peer);
+
+		/*
+		 * If ACS is not implemented, we have no idea about P2P
+		 * support. Optimistically allow this if there is a common
+		 * upstream device.
+		 */
+		pos = pci_find_ext_capability(rpdev, PCI_EXT_CAP_ID_ACS);
+		if (!pos)
+			return common_upstream != NULL;
+
+		/*
+		 * If the devices are under the same root port and have a common
+		 * upstream device, allow if the root port is further upstream
+		 * from the common upstream device and the common upstream
+		 * device has Upstream Forwarding disabled, or if the root port
+		 * is the common upstream device and ACS is not implemented.
+		 */
+		pci_read_config_word(rpdev, pos + PCI_ACS_CAP, &cap);
+		if ((rpdev == rppeer && common_upstream) &&
+		    (((common_upstream != rpdev) &&
+		      !pci_acs_enabled(common_upstream, PCI_ACS_UF)) ||
+		     ((common_upstream == rpdev) && ((cap & PCI_ACS_RR) == 0))))
+			return true;
+
+		/*
+		 * If ACS RR is implemented and disabled, allow only if the
+		 * devices are under the same root port.
+		 */
+		if (cap & PCI_ACS_RR && !pci_acs_enabled(rpdev, PCI_ACS_RR))
+			return rpdev == rppeer;
+
+		/*
+		 * If ACS RR is not implemented, or is implemented and enabled,
+		 * only allow if there's a translation agent enabled to do the
+		 * redirect.
+		 */
+		return iommu_present(&pci_bus_type);
+	}
+
+	return false;
+}
+
 #define RESOURCE_ALIGNMENT_PARAM_SIZE COMMAND_LINE_SIZE
 static char resource_alignment_param[RESOURCE_ALIGNMENT_PARAM_SIZE] = {0};
 static DEFINE_SPINLOCK(resource_alignment_lock);
diff --git a/include/linux/pci.h b/include/linux/pci.h
index 8262b9e..db0cb51 100644
--- a/include/linux/pci.h
+++ b/include/linux/pci.h
@@ -824,6 +824,8 @@ void pci_stop_root_bus(struct pci_bus *bus);
 void pci_remove_root_bus(struct pci_bus *bus);
 void pci_setup_cardbus(struct pci_bus *bus);
 void pci_sort_breadthfirst(void);
+bool pci_peer_traffic_supported(struct pci_dev *dev, struct pci_dev *peer);
+
 #define dev_is_pci(d) ((d)->bus == &pci_bus_type)
 #define dev_is_pf(d) ((dev_is_pci(d) ? to_pci_dev(d)->is_physfn : false))
 #define dev_num_vf(d) ((dev_is_pci(d) ? pci_num_vf(to_pci_dev(d)) : 0))
@@ -1914,4 +1916,5 @@ static inline bool pci_ari_enabled(struct pci_bus *bus)
 {
 	return bus->self && bus->self->ari_enabled;
 }
+
 #endif /* LINUX_PCI_H */
-- 
2.5.1

[PATCH 10/22] PCI: Add pci_resource_to_peer()

[Will Davis]

Add helper to convert a struct resource to a peer DMA address.

Signed-off-by: Will Davis <wdavis@nvidia.com>
---
 include/linux/pci.h | 16 ++++++++++++++++
 1 file changed, 16 insertions(+)

diff --git a/include/linux/pci.h b/include/linux/pci.h
index db0cb51..2a9deff 100644
--- a/include/linux/pci.h
+++ b/include/linux/pci.h
@@ -1508,6 +1508,22 @@ static inline void pci_resource_to_user(const struct pci_dev *dev, int bar,
 }
 #endif /* HAVE_ARCH_PCI_RESOURCE_TO_USER */
 
+#ifdef CONFIG_HAS_DMA_P2P
+static inline bool pci_resource_to_peer(struct pci_dev *dev,
+					struct pci_dev *peer,
+					struct resource *res,
+					dma_peer_addr_t *peer_addr) {
+	if (pci_peer_traffic_supported(dev, peer)) {
+		struct pci_bus_region region;
+
+		pcibios_resource_to_bus(dev->bus, &region, res);
+		*peer_addr = region.start;
+		return true;
+	}
+
+	return false;
+}
+#endif /* CONFIG_HAS_DMA_P2P */
 
 /*
  *  The world is not perfect and supplies us with broken PCI devices.
-- 
2.5.1

[PATCH 10/22] PCI: Add pci_resource_to_peer()

[Will Davis]

Add helper to convert a struct resource to a peer DMA address.

Signed-off-by: Will Davis <wdavis@nvidia.com>
---
 include/linux/pci.h | 16 ++++++++++++++++
 1 file changed, 16 insertions(+)

diff --git a/include/linux/pci.h b/include/linux/pci.h
index db0cb51..2a9deff 100644
--- a/include/linux/pci.h
+++ b/include/linux/pci.h
@@ -1508,6 +1508,22 @@ static inline void pci_resource_to_user(const struct pci_dev *dev, int bar,
 }
 #endif /* HAVE_ARCH_PCI_RESOURCE_TO_USER */
 
+#ifdef CONFIG_HAS_DMA_P2P
+static inline bool pci_resource_to_peer(struct pci_dev *dev,
+					struct pci_dev *peer,
+					struct resource *res,
+					dma_peer_addr_t *peer_addr) {
+	if (pci_peer_traffic_supported(dev, peer)) {
+		struct pci_bus_region region;
+
+		pcibios_resource_to_bus(dev->bus, &region, res);
+		*peer_addr = region.start;
+		return true;
+	}
+
+	return false;
+}
+#endif /* CONFIG_HAS_DMA_P2P */
 
 /*
  *  The world is not perfect and supplies us with broken PCI devices.
-- 
2.5.1

[PATCH 11/22] swiotlb: Add map_peer_resource stub

[Will Davis]

Add swiotlb stub of the 'map_peer_resource' DMA operation.

Signed-off-by: Will Davis <wdavis@nvidia.com>
---
 include/linux/swiotlb.h |  8 ++++++++
 lib/swiotlb.c           | 17 +++++++++++++++++
 2 files changed, 25 insertions(+)

diff --git a/include/linux/swiotlb.h b/include/linux/swiotlb.h
index e7a018e..872463a 100644
--- a/include/linux/swiotlb.h
+++ b/include/linux/swiotlb.h
@@ -87,6 +87,14 @@ swiotlb_unmap_sg_attrs(struct device *hwdev, struct scatterlist *sgl,
 		       int nelems, enum dma_data_direction dir,
 		       struct dma_attrs *attrs);
 
+#ifdef CONFIG_HAS_DMA_P2P
+extern dma_peer_addr_t
+swiotlb_map_peer_resource(struct device *hwdev, struct device *hwpeer,
+			  struct resource *res, unsigned long offset,
+			  size_t size, enum dma_data_direction dir,
+			  struct dma_attrs *attrs);
+#endif
+
 extern void
 swiotlb_sync_single_for_cpu(struct device *hwdev, dma_addr_t dev_addr,
 			    size_t size, enum dma_data_direction dir);
diff --git a/lib/swiotlb.c b/lib/swiotlb.c
index 76f29ec..61522a7 100644
--- a/lib/swiotlb.c
+++ b/lib/swiotlb.c
@@ -945,6 +945,23 @@ swiotlb_unmap_sg(struct device *hwdev, struct scatterlist *sgl, int nelems,
 }
 EXPORT_SYMBOL(swiotlb_unmap_sg);
 
+#ifdef CONFIG_HAS_DMA_P2P
+/*
+ * Maps a single region from a peer device's resource for DMA streaming. This
+ * is currently unimplemented because there's no bounce buffer abstraction for
+ * peer mappings.
+ */
+dma_peer_addr_t
+swiotlb_map_peer_resource(struct device *hwdev, struct device *hwpeer,
+			  struct resource *res, unsigned long offset,
+			  size_t size, enum dma_data_direction dir,
+			  struct dma_attrs *attrs)
+{
+	return phys_to_dma(hwdev, io_tlb_overflow_buffer);
+}
+EXPORT_SYMBOL(swiotlb_map_peer_resource);
+#endif
+
 /*
  * Make physical memory consistent for a set of streaming mode DMA translations
  * after a transfer.
-- 
2.5.1

[PATCH 11/22] swiotlb: Add map_peer_resource stub

[Will Davis]

Add swiotlb stub of the 'map_peer_resource' DMA operation.

Signed-off-by: Will Davis <wdavis@nvidia.com>
---
 include/linux/swiotlb.h |  8 ++++++++
 lib/swiotlb.c           | 17 +++++++++++++++++
 2 files changed, 25 insertions(+)

diff --git a/include/linux/swiotlb.h b/include/linux/swiotlb.h
index e7a018e..872463a 100644
--- a/include/linux/swiotlb.h
+++ b/include/linux/swiotlb.h
@@ -87,6 +87,14 @@ swiotlb_unmap_sg_attrs(struct device *hwdev, struct scatterlist *sgl,
 		       int nelems, enum dma_data_direction dir,
 		       struct dma_attrs *attrs);
 
+#ifdef CONFIG_HAS_DMA_P2P
+extern dma_peer_addr_t
+swiotlb_map_peer_resource(struct device *hwdev, struct device *hwpeer,
+			  struct resource *res, unsigned long offset,
+			  size_t size, enum dma_data_direction dir,
+			  struct dma_attrs *attrs);
+#endif
+
 extern void
 swiotlb_sync_single_for_cpu(struct device *hwdev, dma_addr_t dev_addr,
 			    size_t size, enum dma_data_direction dir);
diff --git a/lib/swiotlb.c b/lib/swiotlb.c
index 76f29ec..61522a7 100644
--- a/lib/swiotlb.c
+++ b/lib/swiotlb.c
@@ -945,6 +945,23 @@ swiotlb_unmap_sg(struct device *hwdev, struct scatterlist *sgl, int nelems,
 }
 EXPORT_SYMBOL(swiotlb_unmap_sg);
 
+#ifdef CONFIG_HAS_DMA_P2P
+/*
+ * Maps a single region from a peer device's resource for DMA streaming. This
+ * is currently unimplemented because there's no bounce buffer abstraction for
+ * peer mappings.
+ */
+dma_peer_addr_t
+swiotlb_map_peer_resource(struct device *hwdev, struct device *hwpeer,
+			  struct resource *res, unsigned long offset,
+			  size_t size, enum dma_data_direction dir,
+			  struct dma_attrs *attrs)
+{
+	return phys_to_dma(hwdev, io_tlb_overflow_buffer);
+}
+EXPORT_SYMBOL(swiotlb_map_peer_resource);
+#endif
+
 /*
  * Make physical memory consistent for a set of streaming mode DMA translations
  * after a transfer.
-- 
2.5.1

[PATCH 12/22] x86, swiotlb: Add swiotlb_map_peer_resource() to swiotlb_dma_ops

[Will Davis]

Signed-off-by: Will Davis <wdavis@nvidia.com>
---
 arch/x86/kernel/pci-swiotlb.c | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/arch/x86/kernel/pci-swiotlb.c b/arch/x86/kernel/pci-swiotlb.c
index adf0392..844b89b 100644
--- a/arch/x86/kernel/pci-swiotlb.c
+++ b/arch/x86/kernel/pci-swiotlb.c
@@ -57,6 +57,9 @@ static struct dma_map_ops swiotlb_dma_ops = {
 	.unmap_sg = swiotlb_unmap_sg_attrs,
 	.map_page = swiotlb_map_page,
 	.unmap_page = swiotlb_unmap_page,
+#ifdef CONFIG_HAS_DMA_P2P
+	.map_peer_resource = swiotlb_map_peer_resource,
+#endif
 	.dma_supported = NULL,
 };
 
-- 
2.5.1

[PATCH 12/22] x86, swiotlb: Add swiotlb_map_peer_resource() to swiotlb_dma_ops

[Will Davis]

Signed-off-by: Will Davis <wdavis@nvidia.com>
---
 arch/x86/kernel/pci-swiotlb.c | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/arch/x86/kernel/pci-swiotlb.c b/arch/x86/kernel/pci-swiotlb.c
index adf0392..844b89b 100644
--- a/arch/x86/kernel/pci-swiotlb.c
+++ b/arch/x86/kernel/pci-swiotlb.c
@@ -57,6 +57,9 @@ static struct dma_map_ops swiotlb_dma_ops = {
 	.unmap_sg = swiotlb_unmap_sg_attrs,
 	.map_page = swiotlb_map_page,
 	.unmap_page = swiotlb_unmap_page,
+#ifdef CONFIG_HAS_DMA_P2P
+	.map_peer_resource = swiotlb_map_peer_resource,
+#endif
 	.dma_supported = NULL,
 };
 
-- 
2.5.1

[PATCH 13/22] x86, platform: Add swiotlb_map_peer_resource() to sta2x11_dma_ops

[Will Davis]

Signed-off-by: Will Davis <wdavis@nvidia.com>
---
 arch/x86/pci/sta2x11-fixup.c | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/arch/x86/pci/sta2x11-fixup.c b/arch/x86/pci/sta2x11-fixup.c
index 5ceda85..4b8be04 100644
--- a/arch/x86/pci/sta2x11-fixup.c
+++ b/arch/x86/pci/sta2x11-fixup.c
@@ -186,6 +186,9 @@ static struct dma_map_ops sta2x11_dma_ops = {
 	.unmap_page = swiotlb_unmap_page,
 	.map_sg = swiotlb_map_sg_attrs,
 	.unmap_sg = swiotlb_unmap_sg_attrs,
+#ifdef CONFIG_HAS_DMA_P2P
+	.map_peer_resource = swiotlb_map_peer_resource,
+#endif
 	.sync_single_for_cpu = swiotlb_sync_single_for_cpu,
 	.sync_single_for_device = swiotlb_sync_single_for_device,
 	.sync_sg_for_cpu = swiotlb_sync_sg_for_cpu,
-- 
2.5.1

[PATCH 13/22] x86, platform: Add swiotlb_map_peer_resource() to sta2x11_dma_ops

[Will Davis]

Signed-off-by: Will Davis <wdavis-DDmLM1+adcrQT0dZR+AlfA@public.gmane.org>
---
 arch/x86/pci/sta2x11-fixup.c | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/arch/x86/pci/sta2x11-fixup.c b/arch/x86/pci/sta2x11-fixup.c
index 5ceda85..4b8be04 100644
--- a/arch/x86/pci/sta2x11-fixup.c
+++ b/arch/x86/pci/sta2x11-fixup.c
@@ -186,6 +186,9 @@ static struct dma_map_ops sta2x11_dma_ops = {
 	.unmap_page = swiotlb_unmap_page,
 	.map_sg = swiotlb_map_sg_attrs,
 	.unmap_sg = swiotlb_unmap_sg_attrs,
+#ifdef CONFIG_HAS_DMA_P2P
+	.map_peer_resource = swiotlb_map_peer_resource,
+#endif
 	.sync_single_for_cpu = swiotlb_sync_single_for_cpu,
 	.sync_single_for_device = swiotlb_sync_single_for_device,
 	.sync_sg_for_cpu = swiotlb_sync_sg_for_cpu,
-- 
2.5.1

[PATCH 14/22] swiotlb-xen: add map_peer_resource stub

[Will Davis]

Add Xen stub of the map_peer_resource DMA API. Return an error to
indicate no bounce-buffer implementation is available yet.

Signed-off-by: Will Davis <wdavis@nvidia.com>
---
 drivers/xen/swiotlb-xen.c | 20 ++++++++++++++++++++
 include/xen/swiotlb-xen.h | 11 +++++++++++
 2 files changed, 31 insertions(+)

diff --git a/drivers/xen/swiotlb-xen.c b/drivers/xen/swiotlb-xen.c
index 4c54932..b9879dc 100644
--- a/drivers/xen/swiotlb-xen.c
+++ b/drivers/xen/swiotlb-xen.c
@@ -475,6 +475,26 @@ void xen_swiotlb_unmap_page(struct device *hwdev, dma_addr_t dev_addr,
 }
 EXPORT_SYMBOL_GPL(xen_swiotlb_unmap_page);
 
+#ifdef CONFIG_HAS_DMA_P2P
+/*
+ * Maps a single region from a peer device's resource for DMA streaming. This is
+ * currently unimplemented for Xen because the only peer DMA mappings we're
+ * capable of creating are direct mappings with no explicit synchronization,
+ * e.g., there's no bounce buffer abstraction for peer mappings.
+ */
+dma_peer_addr_t xen_swiotlb_map_peer_resource(struct device *hwdev,
+					      struct device *hwpeer,
+					      struct resource *res,
+					      unsigned long offset,
+					      size_t size,
+					      enum dma_data_direction dir,
+					      struct dma_attrs *attrs)
+{
+	return DMA_ERROR_CODE;
+}
+EXPORT_SYMBOL_GPL(xen_swiotlb_map_peer_resource);
+#endif
+
 /*
  * Make physical memory consistent for a single streaming mode DMA translation
  * after a transfer.
diff --git a/include/xen/swiotlb-xen.h b/include/xen/swiotlb-xen.h
index 8b2eb93..dc08a4f 100644
--- a/include/xen/swiotlb-xen.h
+++ b/include/xen/swiotlb-xen.h
@@ -24,6 +24,17 @@ extern dma_addr_t xen_swiotlb_map_page(struct device *dev, struct page *page,
 extern void xen_swiotlb_unmap_page(struct device *hwdev, dma_addr_t dev_addr,
 				   size_t size, enum dma_data_direction dir,
 				   struct dma_attrs *attrs);
+
+#ifdef CONFIG_HAS_DMA_P2P
+extern dma_peer_addr_t xen_swiotlb_map_peer(struct device *hwdev,
+					    struct device *hwpeer,
+					    struct resource *res,
+					    unsigned long offset,
+					    size_t size,
+					    enum dma_data_direction dir,
+					    struct dma_attrs *attrs);
+#endif
+
 extern int
 xen_swiotlb_map_sg_attrs(struct device *hwdev, struct scatterlist *sgl,
 			 int nelems, enum dma_data_direction dir,
-- 
2.5.1

[PATCH 14/22] swiotlb-xen: add map_peer_resource stub

[Will Davis]

Add Xen stub of the map_peer_resource DMA API. Return an error to
indicate no bounce-buffer implementation is available yet.

Signed-off-by: Will Davis <wdavis-DDmLM1+adcrQT0dZR+AlfA@public.gmane.org>
---
 drivers/xen/swiotlb-xen.c | 20 ++++++++++++++++++++
 include/xen/swiotlb-xen.h | 11 +++++++++++
 2 files changed, 31 insertions(+)

diff --git a/drivers/xen/swiotlb-xen.c b/drivers/xen/swiotlb-xen.c
index 4c54932..b9879dc 100644
--- a/drivers/xen/swiotlb-xen.c
+++ b/drivers/xen/swiotlb-xen.c
@@ -475,6 +475,26 @@ void xen_swiotlb_unmap_page(struct device *hwdev, dma_addr_t dev_addr,
 }
 EXPORT_SYMBOL_GPL(xen_swiotlb_unmap_page);
 
+#ifdef CONFIG_HAS_DMA_P2P
+/*
+ * Maps a single region from a peer device's resource for DMA streaming. This is
+ * currently unimplemented for Xen because the only peer DMA mappings we're
+ * capable of creating are direct mappings with no explicit synchronization,
+ * e.g., there's no bounce buffer abstraction for peer mappings.
+ */
+dma_peer_addr_t xen_swiotlb_map_peer_resource(struct device *hwdev,
+					      struct device *hwpeer,
+					      struct resource *res,
+					      unsigned long offset,
+					      size_t size,
+					      enum dma_data_direction dir,
+					      struct dma_attrs *attrs)
+{
+	return DMA_ERROR_CODE;
+}
+EXPORT_SYMBOL_GPL(xen_swiotlb_map_peer_resource);
+#endif
+
 /*
  * Make physical memory consistent for a single streaming mode DMA translation
  * after a transfer.
diff --git a/include/xen/swiotlb-xen.h b/include/xen/swiotlb-xen.h
index 8b2eb93..dc08a4f 100644
--- a/include/xen/swiotlb-xen.h
+++ b/include/xen/swiotlb-xen.h
@@ -24,6 +24,17 @@ extern dma_addr_t xen_swiotlb_map_page(struct device *dev, struct page *page,
 extern void xen_swiotlb_unmap_page(struct device *hwdev, dma_addr_t dev_addr,
 				   size_t size, enum dma_data_direction dir,
 				   struct dma_attrs *attrs);
+
+#ifdef CONFIG_HAS_DMA_P2P
+extern dma_peer_addr_t xen_swiotlb_map_peer(struct device *hwdev,
+					    struct device *hwpeer,
+					    struct resource *res,
+					    unsigned long offset,
+					    size_t size,
+					    enum dma_data_direction dir,
+					    struct dma_attrs *attrs);
+#endif
+
 extern int
 xen_swiotlb_map_sg_attrs(struct device *hwdev, struct scatterlist *sgl,
 			 int nelems, enum dma_data_direction dir,
-- 
2.5.1

[PATCH 15/22] pci-swiotlb-xen: add xen_swiotlb_map_peer_resource to xen_swiotlb_dma_ops

[Will Davis]

Only available if CONFIG_HAS_DMA_P2P is set.

Signed-off-by: Will Davis <wdavis@nvidia.com>
---
 arch/x86/xen/pci-swiotlb-xen.c | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/arch/x86/xen/pci-swiotlb-xen.c b/arch/x86/xen/pci-swiotlb-xen.c
index 0e98e5d..84a9285 100644
--- a/arch/x86/xen/pci-swiotlb-xen.c
+++ b/arch/x86/xen/pci-swiotlb-xen.c
@@ -30,6 +30,9 @@ static struct dma_map_ops xen_swiotlb_dma_ops = {
 	.unmap_sg = xen_swiotlb_unmap_sg_attrs,
 	.map_page = xen_swiotlb_map_page,
 	.unmap_page = xen_swiotlb_unmap_page,
+#ifdef CONFIG_HAS_DMA_P2P
+	.map_peer_resource = xen_swiotlb_map_peer_resource,
+#endif
 	.dma_supported = xen_swiotlb_dma_supported,
 };
 
-- 
2.5.1

[PATCH 15/22] pci-swiotlb-xen: add xen_swiotlb_map_peer_resource to xen_swiotlb_dma_ops

[Will Davis]

Only available if CONFIG_HAS_DMA_P2P is set.

Signed-off-by: Will Davis <wdavis-DDmLM1+adcrQT0dZR+AlfA@public.gmane.org>
---
 arch/x86/xen/pci-swiotlb-xen.c | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/arch/x86/xen/pci-swiotlb-xen.c b/arch/x86/xen/pci-swiotlb-xen.c
index 0e98e5d..84a9285 100644
--- a/arch/x86/xen/pci-swiotlb-xen.c
+++ b/arch/x86/xen/pci-swiotlb-xen.c
@@ -30,6 +30,9 @@ static struct dma_map_ops xen_swiotlb_dma_ops = {
 	.unmap_sg = xen_swiotlb_unmap_sg_attrs,
 	.map_page = xen_swiotlb_map_page,
 	.unmap_page = xen_swiotlb_unmap_page,
+#ifdef CONFIG_HAS_DMA_P2P
+	.map_peer_resource = xen_swiotlb_map_peer_resource,
+#endif
 	.dma_supported = xen_swiotlb_dma_supported,
 };
 
-- 
2.5.1

[PATCH 16/22] iommu/amd: Implement (un)map_peer_resource

[Will Davis]

Implement 'map_peer_resource' for the AMD IOMMU driver. Generalize the
existing map_page implementation to operate on a physical address, and
make both map_page and map_resource wrappers around that helper (and
similarly, for unmap_page and unmap_resource).

This allows a device to map another's resource, to enable peer-to-peer
transactions.

Add behind CONFIG_HAS_DMA_P2P guards, since the dma_map_ops members are
behind them as well.

Signed-off-by: Will Davis <wdavis@nvidia.com>
Reviewed-by: Terence Ripperda <tripperda@nvidia.com>
Reviewed-by: John Hubbard <jhubbard@nvidia.com>
---
 drivers/iommu/amd_iommu.c | 99 ++++++++++++++++++++++++++++++++++++++++-------
 1 file changed, 86 insertions(+), 13 deletions(-)

diff --git a/drivers/iommu/amd_iommu.c b/drivers/iommu/amd_iommu.c
index a57e9b7..13a47f283 100644
--- a/drivers/iommu/amd_iommu.c
+++ b/drivers/iommu/amd_iommu.c
@@ -471,6 +471,10 @@ DECLARE_STATS_COUNTER(cnt_map_single);
 DECLARE_STATS_COUNTER(cnt_unmap_single);
 DECLARE_STATS_COUNTER(cnt_map_sg);
 DECLARE_STATS_COUNTER(cnt_unmap_sg);
+#ifdef CONFIG_HAS_DMA_P2P
+DECLARE_STATS_COUNTER(cnt_map_peer_resource);
+DECLARE_STATS_COUNTER(cnt_unmap_peer_resource);
+#endif
 DECLARE_STATS_COUNTER(cnt_alloc_coherent);
 DECLARE_STATS_COUNTER(cnt_free_coherent);
 DECLARE_STATS_COUNTER(cross_page);
@@ -509,6 +513,10 @@ static void amd_iommu_stats_init(void)
 	amd_iommu_stats_add(&cnt_unmap_single);
 	amd_iommu_stats_add(&cnt_map_sg);
 	amd_iommu_stats_add(&cnt_unmap_sg);
+#ifdef CONFIG_HAS_DMA_P2P
+	amd_iommu_stats_add(&cnt_map_peer_resource);
+	amd_iommu_stats_add(&cnt_unmap_peer_resource);
+#endif
 	amd_iommu_stats_add(&cnt_alloc_coherent);
 	amd_iommu_stats_add(&cnt_free_coherent);
 	amd_iommu_stats_add(&cross_page);
@@ -2585,20 +2593,16 @@ static void __unmap_single(struct dma_ops_domain *dma_dom,
 }
 
 /*
- * The exported map_single function for dma_ops.
+ * Wrapper function that contains code common to mapping a physical address
+ * range from a page or a resource.
  */
-static dma_addr_t map_page(struct device *dev, struct page *page,
-			   unsigned long offset, size_t size,
-			   enum dma_data_direction dir,
-			   struct dma_attrs *attrs)
+static dma_addr_t __map_phys(struct device *dev, phys_addr_t paddr,
+			     size_t size, enum dma_data_direction dir)
 {
 	unsigned long flags;
 	struct protection_domain *domain;
 	dma_addr_t addr;
 	u64 dma_mask;
-	phys_addr_t paddr = page_to_phys(page) + offset;
-
-	INC_STATS_COUNTER(cnt_map_single);
 
 	domain = get_domain(dev);
 	if (PTR_ERR(domain) == -EINVAL)
@@ -2624,16 +2628,15 @@ out:
 }
 
 /*
- * The exported unmap_single function for dma_ops.
+ * Wrapper function that contains code common to unmapping a physical address
+ * range from a page or a resource.
  */
-static void unmap_page(struct device *dev, dma_addr_t dma_addr, size_t size,
-		       enum dma_data_direction dir, struct dma_attrs *attrs)
+static void __unmap_phys(struct device *dev, dma_addr_t dma_addr, size_t size,
+			 enum dma_data_direction dir)
 {
 	unsigned long flags;
 	struct protection_domain *domain;
 
-	INC_STATS_COUNTER(cnt_unmap_single);
-
 	domain = get_domain(dev);
 	if (IS_ERR(domain))
 		return;
@@ -2707,6 +2710,72 @@ unmap:
 }
 
 /*
+ * The exported map_single function for dma_ops.
+ */
+static dma_addr_t map_page(struct device *dev, struct page *page,
+			   unsigned long offset, size_t size,
+			   enum dma_data_direction dir,
+			   struct dma_attrs *attrs)
+{
+	INC_STATS_COUNTER(cnt_map_single);
+
+	return __map_phys(dev, page_to_phys(page) + offset, size, dir);
+}
+
+/*
+ * The exported unmap_single function for dma_ops.
+ */
+static void unmap_page(struct device *dev, dma_addr_t dma_addr, size_t size,
+		       enum dma_data_direction dir, struct dma_attrs *attrs)
+{
+	INC_STATS_COUNTER(cnt_unmap_single);
+
+	__unmap_phys(dev, dma_addr, size, dir);
+}
+
+#ifdef CONFIG_HAS_DMA_P2P
+/*
+ * The exported map_peer_resource function for dma_ops.
+ */
+static dma_peer_addr_t map_peer_resource(struct device *dev,
+					 struct device *peer,
+					 struct resource *res,
+					 unsigned long offset,
+					 size_t size,
+					 enum dma_data_direction dir,
+					 struct dma_attrs *attrs)
+{
+	struct pci_dev *pdev;
+	struct pci_dev *ppeer;
+
+	INC_STATS_COUNTER(cnt_map_peer_resource);
+
+	if (!dev_is_pci(dev) || !dev_is_pci(peer))
+		return DMA_ERROR_CODE;
+
+	pdev = to_pci_dev(dev);
+	ppeer = to_pci_dev(peer);
+
+	if (!pci_peer_traffic_supported(pdev, ppeer))
+		return DMA_ERROR_CODE;
+
+	return __map_phys(dev, res->start + offset, size, dir);
+}
+
+/*
+ * The exported unmap_peer_resource function for dma_ops.
+ */
+static void unmap_peer_resource(struct device *dev, dma_peer_addr_t dma_addr,
+				size_t size, enum dma_data_direction dir,
+				struct dma_attrs *attrs)
+{
+	INC_STATS_COUNTER(cnt_unmap_peer_resource);
+
+	__unmap_phys(dev, dma_addr, size, dir);
+}
+#endif
+
+/*
  * The exported map_sg function for dma_ops (handles scatter-gather
  * lists).
  */
@@ -2852,6 +2921,10 @@ static struct dma_map_ops amd_iommu_dma_ops = {
 	.unmap_page = unmap_page,
 	.map_sg = map_sg,
 	.unmap_sg = unmap_sg,
+#ifdef CONFIG_HAS_DMA_P2P
+	.map_peer_resource = map_peer_resource,
+	.unmap_peer_resource = unmap_peer_resource,
+#endif
 	.dma_supported = amd_iommu_dma_supported,
 };
 
-- 
2.5.1

[PATCH 16/22] iommu/amd: Implement (un)map_peer_resource

[Will Davis]

Implement 'map_peer_resource' for the AMD IOMMU driver. Generalize the
existing map_page implementation to operate on a physical address, and
make both map_page and map_resource wrappers around that helper (and
similarly, for unmap_page and unmap_resource).

This allows a device to map another's resource, to enable peer-to-peer
transactions.

Add behind CONFIG_HAS_DMA_P2P guards, since the dma_map_ops members are
behind them as well.

Signed-off-by: Will Davis <wdavis@nvidia.com>
Reviewed-by: Terence Ripperda <tripperda@nvidia.com>
Reviewed-by: John Hubbard <jhubbard@nvidia.com>
---
 drivers/iommu/amd_iommu.c | 99 ++++++++++++++++++++++++++++++++++++++++-------
 1 file changed, 86 insertions(+), 13 deletions(-)

diff --git a/drivers/iommu/amd_iommu.c b/drivers/iommu/amd_iommu.c
index a57e9b7..13a47f283 100644
--- a/drivers/iommu/amd_iommu.c
+++ b/drivers/iommu/amd_iommu.c
@@ -471,6 +471,10 @@ DECLARE_STATS_COUNTER(cnt_map_single);
 DECLARE_STATS_COUNTER(cnt_unmap_single);
 DECLARE_STATS_COUNTER(cnt_map_sg);
 DECLARE_STATS_COUNTER(cnt_unmap_sg);
+#ifdef CONFIG_HAS_DMA_P2P
+DECLARE_STATS_COUNTER(cnt_map_peer_resource);
+DECLARE_STATS_COUNTER(cnt_unmap_peer_resource);
+#endif
 DECLARE_STATS_COUNTER(cnt_alloc_coherent);
 DECLARE_STATS_COUNTER(cnt_free_coherent);
 DECLARE_STATS_COUNTER(cross_page);
@@ -509,6 +513,10 @@ static void amd_iommu_stats_init(void)
 	amd_iommu_stats_add(&cnt_unmap_single);
 	amd_iommu_stats_add(&cnt_map_sg);
 	amd_iommu_stats_add(&cnt_unmap_sg);
+#ifdef CONFIG_HAS_DMA_P2P
+	amd_iommu_stats_add(&cnt_map_peer_resource);
+	amd_iommu_stats_add(&cnt_unmap_peer_resource);
+#endif
 	amd_iommu_stats_add(&cnt_alloc_coherent);
 	amd_iommu_stats_add(&cnt_free_coherent);
 	amd_iommu_stats_add(&cross_page);
@@ -2585,20 +2593,16 @@ static void __unmap_single(struct dma_ops_domain *dma_dom,
 }
 
 /*
- * The exported map_single function for dma_ops.
+ * Wrapper function that contains code common to mapping a physical address
+ * range from a page or a resource.
  */
-static dma_addr_t map_page(struct device *dev, struct page *page,
-			   unsigned long offset, size_t size,
-			   enum dma_data_direction dir,
-			   struct dma_attrs *attrs)
+static dma_addr_t __map_phys(struct device *dev, phys_addr_t paddr,
+			     size_t size, enum dma_data_direction dir)
 {
 	unsigned long flags;
 	struct protection_domain *domain;
 	dma_addr_t addr;
 	u64 dma_mask;
-	phys_addr_t paddr = page_to_phys(page) + offset;
-
-	INC_STATS_COUNTER(cnt_map_single);
 
 	domain = get_domain(dev);
 	if (PTR_ERR(domain) == -EINVAL)
@@ -2624,16 +2628,15 @@ out:
 }
 
 /*
- * The exported unmap_single function for dma_ops.
+ * Wrapper function that contains code common to unmapping a physical address
+ * range from a page or a resource.
  */
-static void unmap_page(struct device *dev, dma_addr_t dma_addr, size_t size,
-		       enum dma_data_direction dir, struct dma_attrs *attrs)
+static void __unmap_phys(struct device *dev, dma_addr_t dma_addr, size_t size,
+			 enum dma_data_direction dir)
 {
 	unsigned long flags;
 	struct protection_domain *domain;
 
-	INC_STATS_COUNTER(cnt_unmap_single);
-
 	domain = get_domain(dev);
 	if (IS_ERR(domain))
 		return;
@@ -2707,6 +2710,72 @@ unmap:
 }
 
 /*
+ * The exported map_single function for dma_ops.
+ */
+static dma_addr_t map_page(struct device *dev, struct page *page,
+			   unsigned long offset, size_t size,
+			   enum dma_data_direction dir,
+			   struct dma_attrs *attrs)
+{
+	INC_STATS_COUNTER(cnt_map_single);
+
+	return __map_phys(dev, page_to_phys(page) + offset, size, dir);
+}
+
+/*
+ * The exported unmap_single function for dma_ops.
+ */
+static void unmap_page(struct device *dev, dma_addr_t dma_addr, size_t size,
+		       enum dma_data_direction dir, struct dma_attrs *attrs)
+{
+	INC_STATS_COUNTER(cnt_unmap_single);
+
+	__unmap_phys(dev, dma_addr, size, dir);
+}
+
+#ifdef CONFIG_HAS_DMA_P2P
+/*
+ * The exported map_peer_resource function for dma_ops.
+ */
+static dma_peer_addr_t map_peer_resource(struct device *dev,
+					 struct device *peer,
+					 struct resource *res,
+					 unsigned long offset,
+					 size_t size,
+					 enum dma_data_direction dir,
+					 struct dma_attrs *attrs)
+{
+	struct pci_dev *pdev;
+	struct pci_dev *ppeer;
+
+	INC_STATS_COUNTER(cnt_map_peer_resource);
+
+	if (!dev_is_pci(dev) || !dev_is_pci(peer))
+		return DMA_ERROR_CODE;
+
+	pdev = to_pci_dev(dev);
+	ppeer = to_pci_dev(peer);
+
+	if (!pci_peer_traffic_supported(pdev, ppeer))
+		return DMA_ERROR_CODE;
+
+	return __map_phys(dev, res->start + offset, size, dir);
+}
+
+/*
+ * The exported unmap_peer_resource function for dma_ops.
+ */
+static void unmap_peer_resource(struct device *dev, dma_peer_addr_t dma_addr,
+				size_t size, enum dma_data_direction dir,
+				struct dma_attrs *attrs)
+{
+	INC_STATS_COUNTER(cnt_unmap_peer_resource);
+
+	__unmap_phys(dev, dma_addr, size, dir);
+}
+#endif
+
+/*
  * The exported map_sg function for dma_ops (handles scatter-gather
  * lists).
  */
@@ -2852,6 +2921,10 @@ static struct dma_map_ops amd_iommu_dma_ops = {
 	.unmap_page = unmap_page,
 	.map_sg = map_sg,
 	.unmap_sg = unmap_sg,
+#ifdef CONFIG_HAS_DMA_P2P
+	.map_peer_resource = map_peer_resource,
+	.unmap_peer_resource = unmap_peer_resource,
+#endif
 	.dma_supported = amd_iommu_dma_supported,
 };
 
-- 
2.5.1

[PATCH 17/22] iommu/vt-d: implement (un)map_peer_resource

[Will Davis]

Implement 'map_peer_resource' for the Intel IOMMU driver. Simply translate
the resource to a physical address and route it to the same handlers used
by the 'map_page' API.

This allows a device to map another's resource, to enable peer-to-peer
transactions.

Add behind CONFIG_HAS_DMA_P2P guards, since the dma_map_ops members are
behind them as well.

Signed-off-by: Will Davis <wdavis@nvidia.com>
Reviewed-by: Terence Ripperda <tripperda@nvidia.com>
Reviewed-by: John Hubbard <jhubbard@nvidia.com>
---
 drivers/iommu/intel-iommu.c | 38 ++++++++++++++++++++++++++++++++++++++
 1 file changed, 38 insertions(+)

diff --git a/drivers/iommu/intel-iommu.c b/drivers/iommu/intel-iommu.c
index a98a7b2..0d3746f 100644
--- a/drivers/iommu/intel-iommu.c
+++ b/drivers/iommu/intel-iommu.c
@@ -3544,6 +3544,40 @@ static void intel_unmap_page(struct device *dev, dma_addr_t dev_addr,
 	intel_unmap(dev, dev_addr);
 }
 
+#ifdef CONFIG_HAS_DMA_P2P
+static dma_peer_addr_t intel_map_peer_resource(struct device *dev,
+					       struct device *peer,
+					       struct resource *res,
+					       unsigned long offset,
+					       size_t size,
+					       enum dma_data_direction dir,
+					       struct dma_attrs *attrs)
+{
+	struct pci_dev *pdev;
+	struct pci_dev *ppeer;
+
+	if (!dev_is_pci(dev) || !dev_is_pci(peer))
+		return 0;
+
+	pdev = to_pci_dev(dev);
+	ppeer = to_pci_dev(peer);
+
+	if (!pci_peer_traffic_supported(pdev, ppeer))
+		return 0;
+
+	return __intel_map_single(dev, res->start + offset, size,
+				  dir, *dev->dma_mask);
+}
+
+static void intel_unmap_peer_resource(struct device *dev,
+				      dma_peer_addr_t dev_addr,
+				      size_t size, enum dma_data_direction dir,
+				      struct dma_attrs *attrs)
+{
+	intel_unmap(dev, dev_addr);
+}
+#endif
+
 static void *intel_alloc_coherent(struct device *dev, size_t size,
 				  dma_addr_t *dma_handle, gfp_t flags,
 				  struct dma_attrs *attrs)
@@ -3700,6 +3734,10 @@ struct dma_map_ops intel_dma_ops = {
 	.unmap_sg = intel_unmap_sg,
 	.map_page = intel_map_page,
 	.unmap_page = intel_unmap_page,
+#ifdef CONFIG_HAS_DMA_P2P
+	.map_peer_resource = intel_map_peer_resource,
+	.unmap_peer_resource = intel_unmap_peer_resource,
+#endif
 	.mapping_error = intel_mapping_error,
 };
 
-- 
2.5.1

[PATCH 17/22] iommu/vt-d: implement (un)map_peer_resource

[Will Davis]

Implement 'map_peer_resource' for the Intel IOMMU driver. Simply translate
the resource to a physical address and route it to the same handlers used
by the 'map_page' API.

This allows a device to map another's resource, to enable peer-to-peer
transactions.

Add behind CONFIG_HAS_DMA_P2P guards, since the dma_map_ops members are
behind them as well.

Signed-off-by: Will Davis <wdavis@nvidia.com>
Reviewed-by: Terence Ripperda <tripperda@nvidia.com>
Reviewed-by: John Hubbard <jhubbard@nvidia.com>
---
 drivers/iommu/intel-iommu.c | 38 ++++++++++++++++++++++++++++++++++++++
 1 file changed, 38 insertions(+)

diff --git a/drivers/iommu/intel-iommu.c b/drivers/iommu/intel-iommu.c
index a98a7b2..0d3746f 100644
--- a/drivers/iommu/intel-iommu.c
+++ b/drivers/iommu/intel-iommu.c
@@ -3544,6 +3544,40 @@ static void intel_unmap_page(struct device *dev, dma_addr_t dev_addr,
 	intel_unmap(dev, dev_addr);
 }
 
+#ifdef CONFIG_HAS_DMA_P2P
+static dma_peer_addr_t intel_map_peer_resource(struct device *dev,
+					       struct device *peer,
+					       struct resource *res,
+					       unsigned long offset,
+					       size_t size,
+					       enum dma_data_direction dir,
+					       struct dma_attrs *attrs)
+{
+	struct pci_dev *pdev;
+	struct pci_dev *ppeer;
+
+	if (!dev_is_pci(dev) || !dev_is_pci(peer))
+		return 0;
+
+	pdev = to_pci_dev(dev);
+	ppeer = to_pci_dev(peer);
+
+	if (!pci_peer_traffic_supported(pdev, ppeer))
+		return 0;
+
+	return __intel_map_single(dev, res->start + offset, size,
+				  dir, *dev->dma_mask);
+}
+
+static void intel_unmap_peer_resource(struct device *dev,
+				      dma_peer_addr_t dev_addr,
+				      size_t size, enum dma_data_direction dir,
+				      struct dma_attrs *attrs)
+{
+	intel_unmap(dev, dev_addr);
+}
+#endif
+
 static void *intel_alloc_coherent(struct device *dev, size_t size,
 				  dma_addr_t *dma_handle, gfp_t flags,
 				  struct dma_attrs *attrs)
@@ -3700,6 +3734,10 @@ struct dma_map_ops intel_dma_ops = {
 	.unmap_sg = intel_unmap_sg,
 	.map_page = intel_map_page,
 	.unmap_page = intel_unmap_page,
+#ifdef CONFIG_HAS_DMA_P2P
+	.map_peer_resource = intel_map_peer_resource,
+	.unmap_peer_resource = intel_unmap_peer_resource,
+#endif
 	.mapping_error = intel_mapping_error,
 };
 
-- 
2.5.1

[PATCH 18/22] x86: add pci-nommu implementation of map_peer_resource

[Will Davis]

Add Intel nommu implementation of 'map_peer_resource'.

Add behind CONFIG_HAS_DMA_P2P guards, since the dma_map_ops members are
behind them as well.

Signed-off-by: Will Davis <wdavis@nvidia.com>
---
 arch/x86/kernel/pci-nommu.c | 30 ++++++++++++++++++++++++++++++
 1 file changed, 30 insertions(+)

diff --git a/arch/x86/kernel/pci-nommu.c b/arch/x86/kernel/pci-nommu.c
index da15918..135ed1b 100644
--- a/arch/x86/kernel/pci-nommu.c
+++ b/arch/x86/kernel/pci-nommu.c
@@ -38,6 +38,33 @@ static dma_addr_t nommu_map_page(struct device *dev, struct page *page,
 	return bus;
 }
 
+#ifdef CONFIG_HAS_DMA_P2P
+static dma_peer_addr_t nommu_map_peer_resource(struct device *dev,
+					       struct device *peer,
+					       struct resource *res,
+					       unsigned long offset,
+					       size_t size,
+					       enum dma_data_direction dir,
+					       struct dma_attrs *attrs)
+{
+	dma_peer_addr_t dma_address;
+
+	if (!dev_is_pci(dev) || !dev_is_pci(peer))
+		return DMA_ERROR_CODE;
+
+	if (!pci_resource_to_peer(to_pci_dev(dev), to_pci_dev(peer), res,
+				  &dma_address))
+		return DMA_ERROR_CODE;
+
+	dma_address += offset;
+	WARN_ON(size == 0);
+	if (!check_addr("map_peer_resource", dev, dma_address, size))
+		return DMA_ERROR_CODE;
+	flush_write_buffers();
+	return dma_address;
+}
+#endif
+
 /* Map a set of buffers described by scatterlist in streaming
  * mode for DMA.  This is the scatter-gather version of the
  * above pci_map_single interface.  Here the scatter gather list
@@ -93,6 +120,9 @@ struct dma_map_ops nommu_dma_ops = {
 	.free			= dma_generic_free_coherent,
 	.map_sg			= nommu_map_sg,
 	.map_page		= nommu_map_page,
+#ifdef CONFIG_HAS_DMA_P2P
+	.map_peer_resource = nommu_map_peer_resource,
+#endif
 	.sync_single_for_device = nommu_sync_single_for_device,
 	.sync_sg_for_device	= nommu_sync_sg_for_device,
 	.is_phys		= 1,
-- 
2.5.1

[PATCH 18/22] x86: add pci-nommu implementation of map_peer_resource

[Will Davis]

Add Intel nommu implementation of 'map_peer_resource'.

Add behind CONFIG_HAS_DMA_P2P guards, since the dma_map_ops members are
behind them as well.

Signed-off-by: Will Davis <wdavis@nvidia.com>
---
 arch/x86/kernel/pci-nommu.c | 30 ++++++++++++++++++++++++++++++
 1 file changed, 30 insertions(+)

diff --git a/arch/x86/kernel/pci-nommu.c b/arch/x86/kernel/pci-nommu.c
index da15918..135ed1b 100644
--- a/arch/x86/kernel/pci-nommu.c
+++ b/arch/x86/kernel/pci-nommu.c
@@ -38,6 +38,33 @@ static dma_addr_t nommu_map_page(struct device *dev, struct page *page,
 	return bus;
 }
 
+#ifdef CONFIG_HAS_DMA_P2P
+static dma_peer_addr_t nommu_map_peer_resource(struct device *dev,
+					       struct device *peer,
+					       struct resource *res,
+					       unsigned long offset,
+					       size_t size,
+					       enum dma_data_direction dir,
+					       struct dma_attrs *attrs)
+{
+	dma_peer_addr_t dma_address;
+
+	if (!dev_is_pci(dev) || !dev_is_pci(peer))
+		return DMA_ERROR_CODE;
+
+	if (!pci_resource_to_peer(to_pci_dev(dev), to_pci_dev(peer), res,
+				  &dma_address))
+		return DMA_ERROR_CODE;
+
+	dma_address += offset;
+	WARN_ON(size == 0);
+	if (!check_addr("map_peer_resource", dev, dma_address, size))
+		return DMA_ERROR_CODE;
+	flush_write_buffers();
+	return dma_address;
+}
+#endif
+
 /* Map a set of buffers described by scatterlist in streaming
  * mode for DMA.  This is the scatter-gather version of the
  * above pci_map_single interface.  Here the scatter gather list
@@ -93,6 +120,9 @@ struct dma_map_ops nommu_dma_ops = {
 	.free			= dma_generic_free_coherent,
 	.map_sg			= nommu_map_sg,
 	.map_page		= nommu_map_page,
+#ifdef CONFIG_HAS_DMA_P2P
+	.map_peer_resource = nommu_map_peer_resource,
+#endif
 	.sync_single_for_device = nommu_sync_single_for_device,
 	.sync_sg_for_device	= nommu_sync_sg_for_device,
 	.is_phys		= 1,
-- 
2.5.1

[PATCH 19/22] x86: Calgary: Add map_peer_resource stub

[Will Davis]

Return DMA_ERROR_CODE to indicate no support.

Signed-off-by: Will Davis <wdavis@nvidia.com>
---
 arch/x86/kernel/pci-calgary_64.c | 16 ++++++++++++++++
 1 file changed, 16 insertions(+)

diff --git a/arch/x86/kernel/pci-calgary_64.c b/arch/x86/kernel/pci-calgary_64.c
index 0497f71..ec3d998 100644
--- a/arch/x86/kernel/pci-calgary_64.c
+++ b/arch/x86/kernel/pci-calgary_64.c
@@ -431,6 +431,19 @@ static void calgary_unmap_page(struct device *dev, dma_addr_t dma_addr,
 	iommu_free(tbl, dma_addr, npages);
 }
 
+#ifdef CONFIG_HAS_DMA_P2P
+static dma_peer_addr_t calgary_map_peer_resource(struct device *dev,
+						 struct device *peer,
+						 struct resource *res,
+						 unsigned long offset,
+						 size_t size,
+						 enum dma_data_direction dir,
+						 struct dma_attrs *attrs)
+{
+	return DMA_ERROR_CODE;
+}
+#endif
+
 static void* calgary_alloc_coherent(struct device *dev, size_t size,
 	dma_addr_t *dma_handle, gfp_t flag, struct dma_attrs *attrs)
 {
@@ -485,6 +498,9 @@ static struct dma_map_ops calgary_dma_ops = {
 	.unmap_sg = calgary_unmap_sg,
 	.map_page = calgary_map_page,
 	.unmap_page = calgary_unmap_page,
+#ifdef CONFIG_HAS_DMA_P2P
+	.map_peer_resource = calgary_map_peer_resource,
+#endif
 };
 
 static inline void __iomem * busno_to_bbar(unsigned char num)
-- 
2.5.1

[PATCH 19/22] x86: Calgary: Add map_peer_resource stub

[Will Davis]

Return DMA_ERROR_CODE to indicate no support.

Signed-off-by: Will Davis <wdavis@nvidia.com>
---
 arch/x86/kernel/pci-calgary_64.c | 16 ++++++++++++++++
 1 file changed, 16 insertions(+)

diff --git a/arch/x86/kernel/pci-calgary_64.c b/arch/x86/kernel/pci-calgary_64.c
index 0497f71..ec3d998 100644
--- a/arch/x86/kernel/pci-calgary_64.c
+++ b/arch/x86/kernel/pci-calgary_64.c
@@ -431,6 +431,19 @@ static void calgary_unmap_page(struct device *dev, dma_addr_t dma_addr,
 	iommu_free(tbl, dma_addr, npages);
 }
 
+#ifdef CONFIG_HAS_DMA_P2P
+static dma_peer_addr_t calgary_map_peer_resource(struct device *dev,
+						 struct device *peer,
+						 struct resource *res,
+						 unsigned long offset,
+						 size_t size,
+						 enum dma_data_direction dir,
+						 struct dma_attrs *attrs)
+{
+	return DMA_ERROR_CODE;
+}
+#endif
+
 static void* calgary_alloc_coherent(struct device *dev, size_t size,
 	dma_addr_t *dma_handle, gfp_t flag, struct dma_attrs *attrs)
 {
@@ -485,6 +498,9 @@ static struct dma_map_ops calgary_dma_ops = {
 	.unmap_sg = calgary_unmap_sg,
 	.map_page = calgary_map_page,
 	.unmap_page = calgary_unmap_page,
+#ifdef CONFIG_HAS_DMA_P2P
+	.map_peer_resource = calgary_map_peer_resource,
+#endif
 };
 
 static inline void __iomem * busno_to_bbar(unsigned char num)
-- 
2.5.1

[PATCH 20/22] x86: gart: Add map_peer_resource stub

[Will Davis]

Return error code to indicate no support.

Signed-off-by: Will Davis <wdavis@nvidia.com>
---
 arch/x86/kernel/amd_gart_64.c | 16 ++++++++++++++++
 1 file changed, 16 insertions(+)

diff --git a/arch/x86/kernel/amd_gart_64.c b/arch/x86/kernel/amd_gart_64.c
index 8e3842f..6ba62e6 100644
--- a/arch/x86/kernel/amd_gart_64.c
+++ b/arch/x86/kernel/amd_gart_64.c
@@ -474,6 +474,19 @@ error:
 	return 0;
 }
 
+#ifdef CONFIG_HAS_DMA_P2P
+static dma_peer_addr_t gart_map_peer_resource(struct device *dev,
+					      struct device *peer,
+					      struct resource *res,
+					      unsigned long offset,
+					      size_t size,
+					      enum dma_data_direction dir,
+					      struct dma_attrs *attrs)
+{
+	return bad_dma_addr;
+}
+#endif
+
 /* allocate and map a coherent mapping */
 static void *
 gart_alloc_coherent(struct device *dev, size_t size, dma_addr_t *dma_addr,
@@ -704,6 +717,9 @@ static struct dma_map_ops gart_dma_ops = {
 	.alloc				= gart_alloc_coherent,
 	.free				= gart_free_coherent,
 	.mapping_error			= gart_mapping_error,
+#ifdef CONFIG_HAS_DMA_P2P
+	.map_peer_resource		= gart_map_peer_resource,
+#endif
 };
 
 static void gart_iommu_shutdown(void)
-- 
2.5.1

[PATCH 20/22] x86: gart: Add map_peer_resource stub

[Will Davis]

Return error code to indicate no support.

Signed-off-by: Will Davis <wdavis-DDmLM1+adcrQT0dZR+AlfA@public.gmane.org>
---
 arch/x86/kernel/amd_gart_64.c | 16 ++++++++++++++++
 1 file changed, 16 insertions(+)

diff --git a/arch/x86/kernel/amd_gart_64.c b/arch/x86/kernel/amd_gart_64.c
index 8e3842f..6ba62e6 100644
--- a/arch/x86/kernel/amd_gart_64.c
+++ b/arch/x86/kernel/amd_gart_64.c
@@ -474,6 +474,19 @@ error:
 	return 0;
 }
 
+#ifdef CONFIG_HAS_DMA_P2P
+static dma_peer_addr_t gart_map_peer_resource(struct device *dev,
+					      struct device *peer,
+					      struct resource *res,
+					      unsigned long offset,
+					      size_t size,
+					      enum dma_data_direction dir,
+					      struct dma_attrs *attrs)
+{
+	return bad_dma_addr;
+}
+#endif
+
 /* allocate and map a coherent mapping */
 static void *
 gart_alloc_coherent(struct device *dev, size_t size, dma_addr_t *dma_addr,
@@ -704,6 +717,9 @@ static struct dma_map_ops gart_dma_ops = {
 	.alloc				= gart_alloc_coherent,
 	.free				= gart_free_coherent,
 	.mapping_error			= gart_mapping_error,
+#ifdef CONFIG_HAS_DMA_P2P
+	.map_peer_resource		= gart_map_peer_resource,
+#endif
 };
 
 static void gart_iommu_shutdown(void)
-- 
2.5.1

[PATCH 21/22] x86: add dma_peer_mapping_error()

[Will Davis]

Identical implementation to dma_mapping_error(), but accepts values
returned from the new dma_map_peer_resource() API.

Signed-off-by: Will Davis <wdavis@nvidia.com>
---
 arch/x86/include/asm/dma-mapping.h | 18 ++++++++++++++++--
 1 file changed, 16 insertions(+), 2 deletions(-)

diff --git a/arch/x86/include/asm/dma-mapping.h b/arch/x86/include/asm/dma-mapping.h
index 1f5b728..64472d8 100644
--- a/arch/x86/include/asm/dma-mapping.h
+++ b/arch/x86/include/asm/dma-mapping.h
@@ -44,16 +44,30 @@ static inline struct dma_map_ops *get_dma_ops(struct device *dev)
 #include <asm-generic/dma-mapping-common.h>
 
 /* Make sure we keep the same behaviour */
-static inline int dma_mapping_error(struct device *dev, dma_addr_t dma_addr)
+static inline int dma_mapping_error_common(struct device *dev, u64 dma_addr)
 {
 	struct dma_map_ops *ops = get_dma_ops(dev);
-	debug_dma_mapping_error(dev, dma_addr);
 	if (ops->mapping_error)
 		return ops->mapping_error(dev, dma_addr);
 
 	return (dma_addr == DMA_ERROR_CODE);
 }
 
+static inline int dma_mapping_error(struct device *dev, dma_addr_t dma_addr)
+{
+	debug_dma_mapping_error(dev, dma_addr);
+	return dma_mapping_error_common(dev, (u64)dma_addr);
+}
+
+#ifdef CONFIG_HAS_DMA_P2P
+static inline int dma_peer_mapping_error(struct device *dev,
+					 dma_peer_addr_t dma_addr)
+{
+	debug_dma_peer_mapping_error(dev, dma_addr);
+	return dma_mapping_error_common(dev, (u64)dma_addr);
+}
+#endif
+
 #define dma_alloc_noncoherent(d, s, h, f) dma_alloc_coherent(d, s, h, f)
 #define dma_free_noncoherent(d, s, v, h) dma_free_coherent(d, s, v, h)
 
-- 
2.5.1

[PATCH 21/22] x86: add dma_peer_mapping_error()

[Will Davis]

Identical implementation to dma_mapping_error(), but accepts values
returned from the new dma_map_peer_resource() API.

Signed-off-by: Will Davis <wdavis@nvidia.com>
---
 arch/x86/include/asm/dma-mapping.h | 18 ++++++++++++++++--
 1 file changed, 16 insertions(+), 2 deletions(-)

diff --git a/arch/x86/include/asm/dma-mapping.h b/arch/x86/include/asm/dma-mapping.h
index 1f5b728..64472d8 100644
--- a/arch/x86/include/asm/dma-mapping.h
+++ b/arch/x86/include/asm/dma-mapping.h
@@ -44,16 +44,30 @@ static inline struct dma_map_ops *get_dma_ops(struct device *dev)
 #include <asm-generic/dma-mapping-common.h>
 
 /* Make sure we keep the same behaviour */
-static inline int dma_mapping_error(struct device *dev, dma_addr_t dma_addr)
+static inline int dma_mapping_error_common(struct device *dev, u64 dma_addr)
 {
 	struct dma_map_ops *ops = get_dma_ops(dev);
-	debug_dma_mapping_error(dev, dma_addr);
 	if (ops->mapping_error)
 		return ops->mapping_error(dev, dma_addr);
 
 	return (dma_addr == DMA_ERROR_CODE);
 }
 
+static inline int dma_mapping_error(struct device *dev, dma_addr_t dma_addr)
+{
+	debug_dma_mapping_error(dev, dma_addr);
+	return dma_mapping_error_common(dev, (u64)dma_addr);
+}
+
+#ifdef CONFIG_HAS_DMA_P2P
+static inline int dma_peer_mapping_error(struct device *dev,
+					 dma_peer_addr_t dma_addr)
+{
+	debug_dma_peer_mapping_error(dev, dma_addr);
+	return dma_mapping_error_common(dev, (u64)dma_addr);
+}
+#endif
+
 #define dma_alloc_noncoherent(d, s, h, f) dma_alloc_coherent(d, s, h, f)
 #define dma_free_noncoherent(d, s, v, h) dma_free_coherent(d, s, v, h)
 
-- 
2.5.1

[PATCH 22/22] x86: declare support for DMA P2P

[Will Davis]

Use CONFIG_HAS_DMA_P2P to declare that we want the DMA peer resource APIs
enabled on x86.

Signed-off-by: Will Davis <wdavis@nvidia.com>
---
 arch/x86/Kconfig | 1 +
 1 file changed, 1 insertion(+)

diff --git a/arch/x86/Kconfig b/arch/x86/Kconfig
index 3dbb7e7..581d1ad 100644
--- a/arch/x86/Kconfig
+++ b/arch/x86/Kconfig
@@ -149,6 +149,7 @@ config X86
 	select VIRT_TO_BUS
 	select X86_DEV_DMA_OPS			if X86_64
 	select X86_FEATURE_NAMES		if PROC_FS
+	select HAS_DMA_P2P
 
 config INSTRUCTION_DECODER
 	def_bool y
-- 
2.5.1

[PATCH 22/22] x86: declare support for DMA P2P

[Will Davis]

Use CONFIG_HAS_DMA_P2P to declare that we want the DMA peer resource APIs
enabled on x86.

Signed-off-by: Will Davis <wdavis@nvidia.com>
---
 arch/x86/Kconfig | 1 +
 1 file changed, 1 insertion(+)

diff --git a/arch/x86/Kconfig b/arch/x86/Kconfig
index 3dbb7e7..581d1ad 100644
--- a/arch/x86/Kconfig
+++ b/arch/x86/Kconfig
@@ -149,6 +149,7 @@ config X86
 	select VIRT_TO_BUS
 	select X86_DEV_DMA_OPS			if X86_64
 	select X86_FEATURE_NAMES		if PROC_FS
+	select HAS_DMA_P2P
 
 config INSTRUCTION_DECODER
 	def_bool y
-- 
2.5.1

Re: [PATCH 11/22] swiotlb: Add map_peer_resource stub

[Konrad Rzeszutek Wilk]

On Tue, Sep 15, 2015 at 12:10:56PM -0500, Will Davis wrote:
> Add swiotlb stub of the 'map_peer_resource' DMA operation.
> 
> Signed-off-by: Will Davis <wdavis@nvidia.com>
> ---
>  include/linux/swiotlb.h |  8 ++++++++
>  lib/swiotlb.c           | 17 +++++++++++++++++
>  2 files changed, 25 insertions(+)
> 
> diff --git a/include/linux/swiotlb.h b/include/linux/swiotlb.h
> index e7a018e..872463a 100644
> --- a/include/linux/swiotlb.h
> +++ b/include/linux/swiotlb.h
> @@ -87,6 +87,14 @@ swiotlb_unmap_sg_attrs(struct device *hwdev, struct scatterlist *sgl,
>  		       int nelems, enum dma_data_direction dir,
>  		       struct dma_attrs *attrs);
>  
> +#ifdef CONFIG_HAS_DMA_P2P
> +extern dma_peer_addr_t
> +swiotlb_map_peer_resource(struct device *hwdev, struct device *hwpeer,
> +			  struct resource *res, unsigned long offset,
> +			  size_t size, enum dma_data_direction dir,
> +			  struct dma_attrs *attrs);
> +#endif
> +
>  extern void
>  swiotlb_sync_single_for_cpu(struct device *hwdev, dma_addr_t dev_addr,
>  			    size_t size, enum dma_data_direction dir);
> diff --git a/lib/swiotlb.c b/lib/swiotlb.c
> index 76f29ec..61522a7 100644
> --- a/lib/swiotlb.c
> +++ b/lib/swiotlb.c
> @@ -945,6 +945,23 @@ swiotlb_unmap_sg(struct device *hwdev, struct scatterlist *sgl, int nelems,
>  }
>  EXPORT_SYMBOL(swiotlb_unmap_sg);
>  
> +#ifdef CONFIG_HAS_DMA_P2P
> +/*
> + * Maps a single region from a peer device's resource for DMA streaming. This
> + * is currently unimplemented because there's no bounce buffer abstraction for
> + * peer mappings.
> + */
> +dma_peer_addr_t
> +swiotlb_map_peer_resource(struct device *hwdev, struct device *hwpeer,
> +			  struct resource *res, unsigned long offset,
> +			  size_t size, enum dma_data_direction dir,
> +			  struct dma_attrs *attrs)
> +{
> +	return phys_to_dma(hwdev, io_tlb_overflow_buffer);

That seems to be implemented :-)

Did you mean to return DMA_ERROR_CODE instead?
> +}
> +EXPORT_SYMBOL(swiotlb_map_peer_resource);
> +#endif
> +
>  /*
>   * Make physical memory consistent for a set of streaming mode DMA translations
>   * after a transfer.
> -- 
> 2.5.1
> 

RE: [PATCH 11/22] swiotlb: Add map_peer_resource stub

[William Davis]

> -----Original Message-----
> From: Konrad Rzeszutek Wilk [mailto:konrad.wilk@oracle.com]
> Sent: Tuesday, September 15, 2015 2:40 PM
> To: William Davis
> Cc: Bjorn Helgaas; Alex Williamson; Joerg Roedel; iommu@lists.linux-foundation.org; linux-
> pci@vger.kernel.org; Mark Hounschell; David S. Miller; Jonathan Corbet; Terence Ripperda; John
> Hubbard; Jerome Glisse
> Subject: Re: [PATCH 11/22] swiotlb: Add map_peer_resource stub
> 
> On Tue, Sep 15, 2015 at 12:10:56PM -0500, Will Davis wrote:
> > Add swiotlb stub of the 'map_peer_resource' DMA operation.
> >
> > Signed-off-by: Will Davis <wdavis@nvidia.com>
> > ---
> >  include/linux/swiotlb.h |  8 ++++++++
> >  lib/swiotlb.c           | 17 +++++++++++++++++
> >  2 files changed, 25 insertions(+)
> >
> > diff --git a/include/linux/swiotlb.h b/include/linux/swiotlb.h
> > index e7a018e..872463a 100644
> > --- a/include/linux/swiotlb.h
> > +++ b/include/linux/swiotlb.h
> > @@ -87,6 +87,14 @@ swiotlb_unmap_sg_attrs(struct device *hwdev, struct scatterlist *sgl,
> >  		       int nelems, enum dma_data_direction dir,
> >  		       struct dma_attrs *attrs);
> >
> > +#ifdef CONFIG_HAS_DMA_P2P
> > +extern dma_peer_addr_t
> > +swiotlb_map_peer_resource(struct device *hwdev, struct device *hwpeer,
> > +			  struct resource *res, unsigned long offset,
> > +			  size_t size, enum dma_data_direction dir,
> > +			  struct dma_attrs *attrs);
> > +#endif
> > +
> >  extern void
> >  swiotlb_sync_single_for_cpu(struct device *hwdev, dma_addr_t dev_addr,
> >  			    size_t size, enum dma_data_direction dir);
> > diff --git a/lib/swiotlb.c b/lib/swiotlb.c
> > index 76f29ec..61522a7 100644
> > --- a/lib/swiotlb.c
> > +++ b/lib/swiotlb.c
> > @@ -945,6 +945,23 @@ swiotlb_unmap_sg(struct device *hwdev, struct scatterlist *sgl, int nelems,
> >  }
> >  EXPORT_SYMBOL(swiotlb_unmap_sg);
> >
> > +#ifdef CONFIG_HAS_DMA_P2P
> > +/*
> > + * Maps a single region from a peer device's resource for DMA streaming. This
> > + * is currently unimplemented because there's no bounce buffer abstraction for
> > + * peer mappings.
> > + */
> > +dma_peer_addr_t
> > +swiotlb_map_peer_resource(struct device *hwdev, struct device *hwpeer,
> > +			  struct resource *res, unsigned long offset,
> > +			  size_t size, enum dma_data_direction dir,
> > +			  struct dma_attrs *attrs)
> > +{
> > +	return phys_to_dma(hwdev, io_tlb_overflow_buffer);
> 
> That seems to be implemented :-)
> 
> Did you mean to return DMA_ERROR_CODE instead?

I didn't see DMA_ERROR_CODE used in lib/swiotlb.c, and
swiotlb_dma_mapping_error is implemented as:

	return (dma_addr == phys_to_dma(hwdev, io_tlb_overflow_buffer));

So I figured that phys_to_dma(hwdev, io_tlb_overflow_buffer) was intended
to be used as DMA_ERROR_CODE instead.

Thanks,
Will

> > +}
> > +EXPORT_SYMBOL(swiotlb_map_peer_resource);
> > +#endif
> > +
> >  /*
> >   * Make physical memory consistent for a set of streaming mode DMA translations
> >   * after a transfer.
> > --
> > 2.5.1
> >

--
nvpublic

Re: [PATCH 00/22] DMA-API/PCI map_peer_resource support for peer-to-peer

[Bjorn Helgaas]

Hi Will,

On Tue, Sep 15, 2015 at 12:10:45PM -0500, Will Davis wrote:
> Hi,
> 
> This is the sixth version of a patchset to add the DMA APIs necessary to
> map and unmap a PCI device's BAR to and from another PCI device's IOVA
> domain. This enables PCI peer-to-peer traffic on x86 platforms where an
> IOMMU is enabled (this requires that peer-to-peer traffic be supported by
> the underlying chipset as well, of course).

It's not essential, but it makes my life a little easier if you
include the version in the subject, e.g., "[PATCH v6 0/22]".  

Checkpatch complains about a few things.  You should look at all of
them, but I think you can ignore most of them.  There is at least one
whitespace error (space before tab or similar) that should be fixed.

Many of your changelogs don't make sense unless you include the
subject line.  Can you update them so the body is self-contained?
It's OK if that means you have to repeat the subject line.

This doesn't include any actual users of pci_map_peer_resource(); do
you plan to include one?  Usually we don't merge things with no
in-tree users.

Bjorn

> v6:
> - added stubs for remaining x86 dma_map_ops: calgary, gart, sta2x11,
>   swiotlb, xen-swiotlb
> - factored part of x86 PCI nommu implementation into a generic PCI
>   helper, pci_resource_to_peer()
> - split x86 dma_peer_mapping_error() implementation into a separate
>   patch
> 
> v5: http://www.spinics.net/lists/linux-pci/msg43975.html
> - moved topology and ACS checks into a separate
>   pci_peer_traffic_supported() API, which determines whether PCI peer-
>   to-peer traffic may work based on language in the PCI and PCI Express
>   specifications (excerpts included in comments)
> - added CONFIG_HAS_DMA_P2P guards to x86 PCI no-mmu implementation of
>   map_peer_resource
> - modified a few patch descriptions for style
> 
> v4: http://www.spinics.net/lists/linux-pci/msg43136.html
> - added HAS_DMA_P2P Kconfig option to guard new APIs, which are only
>   supported on x86 with this patch series
> - added a new dma_peer_addr_t type to handle future platform support
>   where the dma_peer_addr_t needs to have the same width as the bus
>   address, which can be wider than the dma_addr_t returned by other
>   DMA APIs
> - added a new dma_peer_mapping_error() and associated PCI wrapper to
>   detect errors returned via dma_map_peer_resource() API and its
>   PCI wrapper
> - modified the interface of the new API to explicitly take the peer
>   device as an argument
> - renamed the (un)map_resource APIs to (un)map_peer_resource, since
>   they directly receive a 'peer' argument now
> - modified the new PCI interface to take the BAR index instead of the
>   actual struct resource
> - modified the pci-nommu implementation to perform topology and ACS
>   checks before allowing
> - modified both the AMD and Intel IOMMU implementations to require
>   that both devices be behind the same host bridge
> - exposed a couple of preexisting PCI utility functions to aid in the
>   topology checks for all of the x86 implementations
> 
> v3: http://www.spinics.net/lists/linux-pci/msg41703.html
> - changed dma_map_resource() to BUG() if the map_resource DMA op is not
>   provided, instead of returning 0
> - updated documentation to remove requirement to check for 0 return value
>   as an error
> - remove const keyword from struct dma_map_ops in new DMA APIs for
>   consistency with other APIs
> 
> v2: http://www.spinics.net/lists/linux-pci/msg41192.html
> - added documentation for the new DMA APIs
> - fixed physical-to-bus address conversion in the nommu implementation
> 
> v1: http://www.spinics.net/lists/linux-pci/msg40747.html
> 
> Will Davis (22):
>   lib/Kconfig: add HAS_DMA_P2P for peer-to-peer support
>   linux/types.h: Add dma_peer_addr_t type
>   dma-debug: add checking for map/unmap_peer_resource
>   DMA-API: Introduce dma_(un)map_peer_resource
>   dma-mapping: pci: add pci_(un)map_peer_resource
>   DMA-API: Add peer resource mapping documentation
>   PCI: Export pci_find_host_bridge()
>   PCI: Add pci_find_common_upstream_dev()
>   PCI: Add pci_peer_traffic_supported()
>   PCI: Add pci_resource_to_peer()
>   swiotlb: Add map_peer_resource stub
>   x86, swiotlb: Add swiotlb_map_peer_resource() to swiotlb_dma_ops
>   x86, platform: Add swiotlb_map_peer_resource() to sta2x11_dma_ops
>   swiotlb-xen: add map_peer_resource stub
>   pci-swiotlb-xen: add xen_swiotlb_map_peer_resource to
>     xen_swiotlb_dma_ops
>   iommu/amd: Implement (un)map_peer_resource
>   iommu/vt-d: implement (un)map_peer_resource
>   x86: add pci-nommu implementation of map_peer_resource
>   x86: Calgary: Add map_peer_resource stub
>   x86: gart: Add map_peer_resource stub
>   x86: add dma_peer_mapping_error()
>   x86: declare support for DMA P2P
> 
>  Documentation/DMA-API-HOWTO.txt          | 69 ++++++++++++++++++++++
>  Documentation/DMA-API.txt                | 38 ++++++++++--
>  arch/x86/Kconfig                         |  1 +
>  arch/x86/include/asm/dma-mapping.h       | 18 +++++-
>  arch/x86/kernel/amd_gart_64.c            | 16 ++++++
>  arch/x86/kernel/pci-calgary_64.c         | 16 ++++++
>  arch/x86/kernel/pci-nommu.c              | 30 ++++++++++
>  arch/x86/kernel/pci-swiotlb.c            |  3 +
>  arch/x86/pci/sta2x11-fixup.c             |  3 +
>  arch/x86/xen/pci-swiotlb-xen.c           |  3 +
>  drivers/iommu/amd_iommu.c                | 99 +++++++++++++++++++++++++++-----
>  drivers/iommu/intel-iommu.c              | 38 ++++++++++++
>  drivers/pci/host-bridge.c                | 19 +-----
>  drivers/pci/pci.c                        | 99 ++++++++++++++++++++++++++++++++
>  drivers/pci/search.c                     | 25 ++++++++
>  drivers/xen/swiotlb-xen.c                | 20 +++++++
>  include/asm-generic/dma-mapping-common.h | 43 ++++++++++++++
>  include/asm-generic/pci-dma-compat.h     | 23 ++++++++
>  include/linux/dma-debug.h                | 39 ++++++++++++-
>  include/linux/dma-mapping.h              | 12 ++++
>  include/linux/pci.h                      | 28 +++++++++
>  include/linux/swiotlb.h                  |  8 +++
>  include/linux/types.h                    | 13 ++++-
>  include/xen/swiotlb-xen.h                | 11 ++++
>  lib/Kconfig                              |  5 ++
>  lib/dma-debug.c                          | 86 ++++++++++++++++++++++++++-
>  lib/swiotlb.c                            | 17 ++++++
>  27 files changed, 740 insertions(+), 42 deletions(-)
> 
> -- 
> 2.5.1
> 
> --
> To unsubscribe from this list: send the line "unsubscribe linux-pci" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH 10/22] PCI: Add pci_resource_to_peer()

[Bjorn Helgaas]

On Tue, Sep 15, 2015 at 12:10:55PM -0500, Will Davis wrote:
> Add helper to convert a struct resource to a peer DMA address.
> 
> Signed-off-by: Will Davis <wdavis@nvidia.com>
> ---
>  include/linux/pci.h | 16 ++++++++++++++++
>  1 file changed, 16 insertions(+)
> 
> diff --git a/include/linux/pci.h b/include/linux/pci.h
> index db0cb51..2a9deff 100644
> --- a/include/linux/pci.h
> +++ b/include/linux/pci.h
> @@ -1508,6 +1508,22 @@ static inline void pci_resource_to_user(const struct pci_dev *dev, int bar,
>  }
>  #endif /* HAVE_ARCH_PCI_RESOURCE_TO_USER */
>  
> +#ifdef CONFIG_HAS_DMA_P2P
> +static inline bool pci_resource_to_peer(struct pci_dev *dev,

I'm not really a fan of functions that return bool unless the
function name is obviously a predicate.  So in this case, I'd
return 0 or a negative errno.

> +					struct pci_dev *peer,
> +					struct resource *res,
> +					dma_peer_addr_t *peer_addr) {
> +	if (pci_peer_traffic_supported(dev, peer)) {
> +		struct pci_bus_region region;
> +
> +		pcibios_resource_to_bus(dev->bus, &region, res);
> +		*peer_addr = region.start;
> +		return true;
> +	}
> +
> +	return false;

It's trivial, but I'd structure it like this, which is sort of the
typical "return errors early" pattern, and reduces indentation of the
mainline path:

  if (!pci_peer_traffic_supported(dev, peer))
    return -EINVAL;

  pcibios_resource_to_bus(dev->bus, &region, res);
  *peer_addr = region.start;
  return 0;

> +}
> +#endif /* CONFIG_HAS_DMA_P2P */
>  
>  /*
>   *  The world is not perfect and supplies us with broken PCI devices.
> -- 
> 2.5.1
> 
> --
> To unsubscribe from this list: send the line "unsubscribe linux-pci" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH 10/22] PCI: Add pci_resource_to_peer()

[Bjorn Helgaas]

On Tue, Sep 15, 2015 at 12:10:55PM -0500, Will Davis wrote:
> Add helper to convert a struct resource to a peer DMA address.
> 
> Signed-off-by: Will Davis <wdavis-DDmLM1+adcrQT0dZR+AlfA@public.gmane.org>
> ---
>  include/linux/pci.h | 16 ++++++++++++++++
>  1 file changed, 16 insertions(+)
> 
> diff --git a/include/linux/pci.h b/include/linux/pci.h
> index db0cb51..2a9deff 100644
> --- a/include/linux/pci.h
> +++ b/include/linux/pci.h
> @@ -1508,6 +1508,22 @@ static inline void pci_resource_to_user(const struct pci_dev *dev, int bar,
>  }
>  #endif /* HAVE_ARCH_PCI_RESOURCE_TO_USER */
>  
> +#ifdef CONFIG_HAS_DMA_P2P
> +static inline bool pci_resource_to_peer(struct pci_dev *dev,

I'm not really a fan of functions that return bool unless the
function name is obviously a predicate.  So in this case, I'd
return 0 or a negative errno.

> +					struct pci_dev *peer,
> +					struct resource *res,
> +					dma_peer_addr_t *peer_addr) {
> +	if (pci_peer_traffic_supported(dev, peer)) {
> +		struct pci_bus_region region;
> +
> +		pcibios_resource_to_bus(dev->bus, &region, res);
> +		*peer_addr = region.start;
> +		return true;
> +	}
> +
> +	return false;

It's trivial, but I'd structure it like this, which is sort of the
typical "return errors early" pattern, and reduces indentation of the
mainline path:

  if (!pci_peer_traffic_supported(dev, peer))
    return -EINVAL;

  pcibios_resource_to_bus(dev->bus, &region, res);
  *peer_addr = region.start;
  return 0;

> +}
> +#endif /* CONFIG_HAS_DMA_P2P */
>  
>  /*
>   *  The world is not perfect and supplies us with broken PCI devices.
> -- 
> 2.5.1
> 
> --
> To unsubscribe from this list: send the line "unsubscribe linux-pci" in
> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH 09/22] PCI: Add pci_peer_traffic_supported()

[Bjorn Helgaas]

On Tue, Sep 15, 2015 at 12:10:54PM -0500, Will Davis wrote:
> Add checks for topology and ACS configuration to determine whether or not
> peer traffic should be supported between two PCI devices.
> 
> Signed-off-by: Will Davis <wdavis@nvidia.com>
> ---
>  drivers/pci/pci.c   | 99 +++++++++++++++++++++++++++++++++++++++++++++++++++++
>  include/linux/pci.h |  3 ++
>  2 files changed, 102 insertions(+)
> 
> diff --git a/drivers/pci/pci.c b/drivers/pci/pci.c
> index 0008c95..b8ba0f0 100644
> --- a/drivers/pci/pci.c
> +++ b/drivers/pci/pci.c
> @@ -25,6 +25,7 @@
>  #include <linux/device.h>
>  #include <linux/pm_runtime.h>
>  #include <linux/pci_hotplug.h>
> +#include <linux/iommu.h>
>  #include <asm-generic/pci-bridge.h>
>  #include <asm/setup.h>
>  #include "pci.h"
> @@ -4302,6 +4303,104 @@ void pci_ignore_hotplug(struct pci_dev *dev)
>  }
>  EXPORT_SYMBOL_GPL(pci_ignore_hotplug);
>  
> +bool pci_peer_traffic_supported(struct pci_dev *dev, struct pci_dev *peer)
> +{
> +	struct pci_host_bridge *dev_host_bridge;
> +	struct pci_host_bridge *peer_host_bridge;
> +
> +	/*
> +	 * Disallow the peer-to-peer traffic if the devices do not share a
> +	 * host bridge. The PCI specifications does not make any guarantees
> +	 * about P2P capabilities between devices under separate domains.
> +	 *
> +	 * PCI Local Bus Specification Revision 3.0, section 3.10:
> +	 *    "Peer-to-peer transactions crossing multiple host bridges
> +	 *     PCI host bridges may, but are not required to, support PCI
> +	 *     peer-to-peer transactions that traverse multiple PCI host
> +	 *     bridges."
> +	 */
> +	dev_host_bridge = pci_find_host_bridge(dev->bus);
> +	peer_host_bridge = pci_find_host_bridge(peer->bus);
> +	if (dev_host_bridge != peer_host_bridge)
> +		return false;
> +
> +	if (pci_is_pcie(dev) && pci_is_pcie(peer)) {

Can you use the same "return errors early" trick here to unindent
everything below, e.g.,

  if (!pci_is_pcie(dev) || !pci_is_pcie(peer))
    return false;

> +		/*
> +		 * Access Control Services (ACS) Checks
> +		 *
> +		 * ACS has a capability bit for P2P Request Redirects (RR),
> +		 * but unfortunately it doesn't tell us much about the real
> +		 * capabilities of the hardware.
> +		 *
> +		 * PCI Express Base Specification Revision 3.0, section
> +		 * 6.12.1.1:
> +		 *    "ACS P2P Request Redirect: must be implemented by Root
> +		 *     Ports that support peer-to-peer traffic with other
> +		 *     Root Ports; [80]"
> +		 * but
> +		 *    "[80] Root Port indication of ACS P2P Request Redirect
> +		 *     or ACS P2P Completion Redirect support does not imply
> +		 *     any particular level of peer-to-peer support by the
> +		 *     Root Complex, or that peer-to-peer traffic is
> +		 *     supported at all"
> +		 */
> +		struct pci_dev *rpdev = dev->bus->self;
> +		struct pci_dev *rppeer = peer->bus->self;

This will oops if rpdev or rppeer is a VF on a "virtual" bus (bus->self
is NULL in that case); see the comment near pci_is_root_bus().

> +		struct pci_dev *common_upstream;
> +		int pos;
> +		u16 cap;
> +
> +		while ((rpdev) && (pci_is_pcie(rpdev)) &&
> +		       (pci_pcie_type(rpdev) != PCI_EXP_TYPE_ROOT_PORT))
> +			rpdev = rpdev->bus->self;

I think we might need some sort of pcie_root_port() function.  We already
have a static pcie_find_root_port() that is basically the same.  And
dmar_find_matched_atsr_unit() does something very similar.  And
ixgbe_io_error_detected().  And myri10ge_enable_ecrc().

> +
> +		while ((rppeer) && (pci_is_pcie(rppeer)) &&
> +		       (pci_pcie_type(rppeer) != PCI_EXP_TYPE_ROOT_PORT))
> +			rppeer = rppeer->bus->self;
> +
> +		common_upstream = pci_find_common_upstream_dev(dev, peer);

This (and the ACS check) logically belongs above the rpdev/rppeer stuff.

> +		/*
> +		 * If ACS is not implemented, we have no idea about P2P
> +		 * support. Optimistically allow this if there is a common
> +		 * upstream device.
> +		 */
> +		pos = pci_find_ext_capability(rpdev, PCI_EXT_CAP_ID_ACS);
> +		if (!pos)
> +			return common_upstream != NULL;
> +
> +		/*
> +		 * If the devices are under the same root port and have a common
> +		 * upstream device, allow if the root port is further upstream
> +		 * from the common upstream device and the common upstream
> +		 * device has Upstream Forwarding disabled, or if the root port
> +		 * is the common upstream device and ACS is not implemented.
> +		 */
> +		pci_read_config_word(rpdev, pos + PCI_ACS_CAP, &cap);
> +		if ((rpdev == rppeer && common_upstream) &&
> +		    (((common_upstream != rpdev) &&
> +		      !pci_acs_enabled(common_upstream, PCI_ACS_UF)) ||
> +		     ((common_upstream == rpdev) && ((cap & PCI_ACS_RR) == 0))))
> +			return true;
> +
> +		/*
> +		 * If ACS RR is implemented and disabled, allow only if the
> +		 * devices are under the same root port.
> +		 */
> +		if (cap & PCI_ACS_RR && !pci_acs_enabled(rpdev, PCI_ACS_RR))
> +			return rpdev == rppeer;
> +
> +		/*
> +		 * If ACS RR is not implemented, or is implemented and enabled,
> +		 * only allow if there's a translation agent enabled to do the
> +		 * redirect.
> +		 */
> +		return iommu_present(&pci_bus_type);
> +	}
> +
> +	return false;
> +}
> +
>  #define RESOURCE_ALIGNMENT_PARAM_SIZE COMMAND_LINE_SIZE
>  static char resource_alignment_param[RESOURCE_ALIGNMENT_PARAM_SIZE] = {0};
>  static DEFINE_SPINLOCK(resource_alignment_lock);
> diff --git a/include/linux/pci.h b/include/linux/pci.h
> index 8262b9e..db0cb51 100644
> --- a/include/linux/pci.h
> +++ b/include/linux/pci.h
> @@ -824,6 +824,8 @@ void pci_stop_root_bus(struct pci_bus *bus);
>  void pci_remove_root_bus(struct pci_bus *bus);
>  void pci_setup_cardbus(struct pci_bus *bus);
>  void pci_sort_breadthfirst(void);
> +bool pci_peer_traffic_supported(struct pci_dev *dev, struct pci_dev *peer);
> +
>  #define dev_is_pci(d) ((d)->bus == &pci_bus_type)
>  #define dev_is_pf(d) ((dev_is_pci(d) ? to_pci_dev(d)->is_physfn : false))
>  #define dev_num_vf(d) ((dev_is_pci(d) ? pci_num_vf(to_pci_dev(d)) : 0))
> @@ -1914,4 +1916,5 @@ static inline bool pci_ari_enabled(struct pci_bus *bus)
>  {
>  	return bus->self && bus->self->ari_enabled;
>  }
> +
>  #endif /* LINUX_PCI_H */
> -- 
> 2.5.1
> 
> --
> To unsubscribe from this list: send the line "unsubscribe linux-pci" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH 09/22] PCI: Add pci_peer_traffic_supported()

[Benjamin Herrenschmidt]

On Tue, 2015-09-15 at 12:10 -0500, Will Davis wrote:

> +bool pci_peer_traffic_supported(struct pci_dev *dev, struct pci_dev
> *peer)
> +{
> +> 	> struct pci_host_bridge *dev_host_bridge;
> +> 	> struct pci_host_bridge *peer_host_bridge;
> +
> +> 	> /*
> +> 	>  * Disallow the peer-to-peer traffic if the devices do not share a
> +> 	>  * host bridge. The PCI specifications does not make any guarantees
> +> 	>  * about P2P capabilities between devices under separate domains.
> +> 	>  *
> +> 	>  * PCI Local Bus Specification Revision 3.0, section 3.10:
> +> 	>  *    "Peer-to-peer transactions crossing multiple host bridges
> +> 	>  *     PCI host bridges may, but are not required to, support PCI
> +> 	>  *     peer-to-peer transactions that traverse multiple PCI host
> +> 	>  *     bridges."
> +> 	>  */
> +	dev_host_bridge = pci_find_host_bridge(dev->bus);
> +> 	> peer_host_bridge = pci_find_host_bridge(peer->bus);
> +> 	> if (dev_host_bridge != peer_host_bridge)
> +> 	> 	> return false;

This needs to be platform specific. Some architectures will allow
routing between multiple bridges, some won't.

> +		/*
> +> 	> 	>  * Access Control Services (ACS) Checks
> +> 	> 	>  *
> +> 	> 	>  * ACS has a capability bit for P2P Request Redirects (RR),
> +> 	> 	>  * but unfortunately it doesn't tell us much about the real
> +> 	> 	>  * capabilities of the hardware.
> +> 	> 	>  *
> +> 	> 	>  * PCI Express Base Specification Revision 3.0, section
> +> 	> 	>  * 6.12.1.1:
> +> 	> 	>  *    "ACS P2P Request Redirect: must be implemented by Root
> +> 	> 	>  *     Ports that support peer-to-peer traffic with other
> +> 	> 	>  *     Root Ports; [80]"
> +> 	> 	>  * but
> +> 	> 	>  *    "[80] Root Port indication of ACS P2P Request Redirect
> +> 	> 	>  *     or ACS P2P Completion Redirect support does not imply
> +> 	> 	>  *     any particular level of peer-to-peer support by the
> +> 	> 	>  *     Root Complex, or that peer-to-peer traffic is
> +> 	> 	>  *     supported at all"
> +> 	> 	>  */
> +> 	> 	> struct pci_dev *rpdev = dev->bus->self;
> +> 	> 	> struct pci_dev *rppeer = peer->bus->self;
> +> 	> 	> struct pci_dev *common_upstream;
> +> 	> 	> int pos;
> +> 	> 	> u16 cap;
> +
> +> 	> 	> while ((rpdev) && (pci_is_pcie(rpdev)) &&
> +> 	> 	>        (pci_pcie_type(rpdev) != PCI_EXP_TYPE_ROOT_PORT))
> +> 	> 	> 	> rpdev = rpdev->bus->self;
> +
> +> 	> 	> while ((rppeer) && (pci_is_pcie(rppeer)) &&
> +> 	> 	>        (pci_pcie_type(rppeer) != PCI_EXP_TYPE_ROOT_PORT))
> +> 	> 	> 	> rppeer = rppeer->bus->self;
> +
> +> 	> 	> common_upstream = pci_find_common_upstream_dev(dev, peer);
> +
> +> 	> 	> /*
> +> 	> 	>  * If ACS is not implemented, we have no idea about P2P
> +> 	> 	>  * support. Optimistically allow this if there is a common
> +> 	> 	>  * upstream device.
> +> 	> 	>  */
> +> 	> 	> pos = pci_find_ext_capability(rpdev, PCI_EXT_CAP_ID_ACS);
> +> 	> 	> if (!pos)
> +> 	> 	> 	> return common_upstream != NULL;

We might need a hook as well here. PLX switch may or may not allow it
depending on some configuration bits.

> +		/*
> +> 	> 	>  * If the devices are under the same root port and have a common
> +> 	> 	>  * upstream device, allow if the root port is further upstream
> +> 	> 	>  * from the common upstream device and the common upstream
> +> 	> 	>  * device has Upstream Forwarding disabled, or if the root port
> +> 	> 	>  * is the common upstream device and ACS is not implemented.
> +> 	> 	>  */
> +> 	> 	> pci_read_config_word(rpdev, pos + PCI_ACS_CAP, &cap);
> +> 	> 	> if ((rpdev == rppeer && common_upstream) &&
> +> 	> 	>     (((common_upstream != rpdev) &&
> +> 	> 	>       !pci_acs_enabled(common_upstream, PCI_ACS_UF)) ||
> +> 	> 	>      ((common_upstream == rpdev) && ((cap & PCI_ACS_RR) == 0))))
> +> 	> 	> 	> return true;
> +
> +> 	> 	> /*
> +> 	> 	>  * If ACS RR is implemented and disabled, allow only if the
> +> 	> 	>  * devices are under the same root port.
> +> 	> 	>  */
> +> 	> 	> if (cap & PCI_ACS_RR && !pci_acs_enabled(rpdev, PCI_ACS_RR))
> +> 	> 	> 	> return rpdev == rppeer;
> +
> +> 	> 	> /*
> +> 	> 	>  * If ACS RR is not implemented, or is implemented and enabled,
> +> 	> 	>  * only allow if there's a translation agent enabled to do the
> +> 	> 	>  * redirect.
> +> 	> 	>  */
> +> 	> 	> return iommu_present(&pci_bus_type);
> +> 	> }
> +
> +> 	> return false;
> +}
> +
>  #define RESOURCE_ALIGNMENT_PARAM_SIZE COMMAND_LINE_SIZE
>  static char resource_alignment_param[RESOURCE_ALIGNMENT_PARAM_SIZE] = {0};
>  static DEFINE_SPINLOCK(resource_alignment_lock);
> diff --git a/include/linux/pci.h b/include/linux/pci.h
> index 8262b9e..db0cb51 100644
> --- a/include/linux/pci.h
> +++ b/include/linux/pci.h
> @@ -824,6 +824,8 @@ void pci_stop_root_bus(struct pci_bus *bus);
>  void pci_remove_root_bus(struct pci_bus *bus);
>  void pci_setup_cardbus(struct pci_bus *bus);
>  void pci_sort_breadthfirst(void);
> +bool pci_peer_traffic_supported(struct pci_dev *dev, struct pci_dev *peer);
> +
>  #define dev_is_pci(d) ((d)->bus == &pci_bus_type)
>  #define dev_is_pf(d) ((dev_is_pci(d) ? to_pci_dev(d)->is_physfn : false))
>  #define dev_num_vf(d) ((dev_is_pci(d) ? pci_num_vf(to_pci_dev(d)) : 0))
> @@ -1914,4 +1916,5 @@ static inline bool pci_ari_enabled(struct pci_bus *bus)
>  {
>  > 	> return bus->self && bus->self->ari_enabled;
>  }
> +
>  #endif /* LINUX_PCI_H */