intel-gfx.lists.freedesktop.org archive mirror
 help / color / mirror / Atom feed
* [Intel-gfx] P2P for DMA-buf
@ 2020-03-11 13:51 Christian König
  2020-03-11 13:51 ` [Intel-gfx] [PATCH 1/6] lib/scatterlist: add sg_set_dma_addr() function Christian König
                   ` (7 more replies)
  0 siblings, 8 replies; 25+ messages in thread
From: Christian König @ 2020-03-11 13:51 UTC (permalink / raw)
  To: David1.Zhou, hch, jgg, daniel, dri-devel, linaro-mm-sig,
	linux-media, intel-gfx

This is the third and final part of my series to start supporting P2P with DMA-buf.

The implementation is straight forward, apart from a helper to aid constructing scatterlists without having struct pages we only add a new flag indicating that an DMA-buf importer can handle peer2peer.

The exporter can then check if P2P is general possible using the pci_p2pdma_distance_many() function and if necessary can also clear the flag.

The rest is an example how to implementing the necessary functionality into the amdgpu driver to setup scatterlists pointing to device memory.

Please review and comment,
Christian.


_______________________________________________
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/intel-gfx

^ permalink raw reply	[flat|nested] 25+ messages in thread

* [Intel-gfx] [PATCH 1/6] lib/scatterlist: add sg_set_dma_addr() function
  2020-03-11 13:51 [Intel-gfx] P2P for DMA-buf Christian König
@ 2020-03-11 13:51 ` Christian König
  2020-03-11 15:28   ` Christoph Hellwig
  2020-03-11 13:51 ` [Intel-gfx] [PATCH 2/6] dma-buf: add peer2peer flag Christian König
                   ` (6 subsequent siblings)
  7 siblings, 1 reply; 25+ messages in thread
From: Christian König @ 2020-03-11 13:51 UTC (permalink / raw)
  To: David1.Zhou, hch, jgg, daniel, dri-devel, linaro-mm-sig,
	linux-media, intel-gfx

This can be used by drivers to setup P2P DMA between device
memory which is not backed by struct pages.

The drivers of the involved devices are responsible for
setting up and tearing down DMA addresses as necessary
using dma_map_resource().

The page pointer is set to NULL and only the DMA address,
length and offset values are valid.

Signed-off-by: Christian König <christian.koenig@amd.com>
---
 include/linux/scatterlist.h | 23 +++++++++++++++++++++++
 1 file changed, 23 insertions(+)

diff --git a/include/linux/scatterlist.h b/include/linux/scatterlist.h
index 6eec50fb36c8..28a477bf0bdf 100644
--- a/include/linux/scatterlist.h
+++ b/include/linux/scatterlist.h
@@ -145,6 +145,29 @@ static inline void sg_set_buf(struct scatterlist *sg, const void *buf,
 	sg_set_page(sg, virt_to_page(buf), buflen, offset_in_page(buf));
 }
 
+/**
+ * sg_set_dma_addr - Set sg entry to point at specified dma address
+ * @sg:		 SG entry
+ * @address:	 DMA address to set
+ * @len:	 Length of data
+ * @offset:	 Offset into page
+ *
+ * Description:
+ *   Use this function to set an sg entry to point to device resources mapped
+ *   using dma_map_resource(). The page pointer is set to NULL and only the DMA
+ *   address, length and offset values are valid.
+ *
+ **/
+static inline void sg_set_dma_addr(struct scatterlist *sg, dma_addr_t address,
+				   unsigned int len, unsigned int offset)
+{
+	sg_set_page(sg, NULL, len, offset);
+	sg->dma_address = address;
+#ifdef CONFIG_NEED_SG_DMA_LENGTH
+	sg->dma_length = len;
+#endif
+}
+
 /*
  * Loop over each sg element, following the pointer to a new list if necessary
  */
-- 
2.17.1

_______________________________________________
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/intel-gfx

^ permalink raw reply	[flat|nested] 25+ messages in thread

* [Intel-gfx] [PATCH 2/6] dma-buf: add peer2peer flag
  2020-03-11 13:51 [Intel-gfx] P2P for DMA-buf Christian König
  2020-03-11 13:51 ` [Intel-gfx] [PATCH 1/6] lib/scatterlist: add sg_set_dma_addr() function Christian König
@ 2020-03-11 13:51 ` Christian König
  2020-03-11 13:51 ` [Intel-gfx] [PATCH 3/6] drm/amdgpu: note that we can handle peer2peer DMA-buf Christian König
                   ` (5 subsequent siblings)
  7 siblings, 0 replies; 25+ messages in thread
From: Christian König @ 2020-03-11 13:51 UTC (permalink / raw)
  To: David1.Zhou, hch, jgg, daniel, dri-devel, linaro-mm-sig,
	linux-media, intel-gfx

Add a peer2peer flag noting that the importer can deal with device
resources which are not backed by pages.

Signed-off-by: Christian König <christian.koenig@amd.com>
---
 drivers/dma-buf/dma-buf.c |  2 ++
 include/linux/dma-buf.h   | 10 ++++++++++
 2 files changed, 12 insertions(+)

diff --git a/drivers/dma-buf/dma-buf.c b/drivers/dma-buf/dma-buf.c
index f4ace9af2191..f9220928ec90 100644
--- a/drivers/dma-buf/dma-buf.c
+++ b/drivers/dma-buf/dma-buf.c
@@ -689,6 +689,8 @@ dma_buf_dynamic_attach(struct dma_buf *dmabuf, struct device *dev,
 
 	attach->dev = dev;
 	attach->dmabuf = dmabuf;
+	if (importer_ops)
+		attach->peer2peer = importer_ops->allow_peer2peer;
 	attach->importer_ops = importer_ops;
 	attach->importer_priv = importer_priv;
 
diff --git a/include/linux/dma-buf.h b/include/linux/dma-buf.h
index 1ade486fc2bb..82e0a4a64601 100644
--- a/include/linux/dma-buf.h
+++ b/include/linux/dma-buf.h
@@ -334,6 +334,14 @@ struct dma_buf {
  * Attachment operations implemented by the importer.
  */
 struct dma_buf_attach_ops {
+	/**
+	 * @allow_peer2peer:
+	 *
+	 * If this is set to true the importer must be able to handle peer
+	 * resources without struct pages.
+	 */
+	bool allow_peer2peer;
+
 	/**
 	 * @move_notify
 	 *
@@ -362,6 +370,7 @@ struct dma_buf_attach_ops {
  * @node: list of dma_buf_attachment, protected by dma_resv lock of the dmabuf.
  * @sgt: cached mapping.
  * @dir: direction of cached mapping.
+ * @peer2peer: true if the importer can handle peer resources without pages.
  * @priv: exporter specific attachment data.
  * @importer_ops: importer operations for this attachment, if provided
  * dma_buf_map/unmap_attachment() must be called with the dma_resv lock held.
@@ -382,6 +391,7 @@ struct dma_buf_attachment {
 	struct list_head node;
 	struct sg_table *sgt;
 	enum dma_data_direction dir;
+	bool peer2peer;
 	const struct dma_buf_attach_ops *importer_ops;
 	void *importer_priv;
 	void *priv;
-- 
2.17.1

_______________________________________________
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/intel-gfx

^ permalink raw reply	[flat|nested] 25+ messages in thread

* [Intel-gfx] [PATCH 3/6] drm/amdgpu: note that we can handle peer2peer DMA-buf
  2020-03-11 13:51 [Intel-gfx] P2P for DMA-buf Christian König
  2020-03-11 13:51 ` [Intel-gfx] [PATCH 1/6] lib/scatterlist: add sg_set_dma_addr() function Christian König
  2020-03-11 13:51 ` [Intel-gfx] [PATCH 2/6] dma-buf: add peer2peer flag Christian König
@ 2020-03-11 13:51 ` Christian König
  2020-03-11 13:51 ` [Intel-gfx] [PATCH 4/6] drm/amdgpu: add checks if DMA-buf P2P is supported Christian König
                   ` (4 subsequent siblings)
  7 siblings, 0 replies; 25+ messages in thread
From: Christian König @ 2020-03-11 13:51 UTC (permalink / raw)
  To: David1.Zhou, hch, jgg, daniel, dri-devel, linaro-mm-sig,
	linux-media, intel-gfx

Importing should work out of the box.

Signed-off-by: Christian König <christian.koenig@amd.com>
---
 drivers/gpu/drm/amd/amdgpu/amdgpu_dma_buf.c | 1 +
 1 file changed, 1 insertion(+)

diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_dma_buf.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_dma_buf.c
index ffeb20f11c07..aef12ee2f1e3 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_dma_buf.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_dma_buf.c
@@ -514,6 +514,7 @@ amdgpu_dma_buf_move_notify(struct dma_buf_attachment *attach)
 }
 
 static const struct dma_buf_attach_ops amdgpu_dma_buf_attach_ops = {
+	.allow_peer2peer = true,
 	.move_notify = amdgpu_dma_buf_move_notify
 };
 
-- 
2.17.1

_______________________________________________
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/intel-gfx

^ permalink raw reply	[flat|nested] 25+ messages in thread

* [Intel-gfx] [PATCH 4/6] drm/amdgpu: add checks if DMA-buf P2P is supported
  2020-03-11 13:51 [Intel-gfx] P2P for DMA-buf Christian König
                   ` (2 preceding siblings ...)
  2020-03-11 13:51 ` [Intel-gfx] [PATCH 3/6] drm/amdgpu: note that we can handle peer2peer DMA-buf Christian König
@ 2020-03-11 13:51 ` Christian König
       [not found]   ` <20200311140415.GB31668@ziepe.ca>
  2020-03-11 13:51 ` [Intel-gfx] [PATCH 5/6] drm/amdgpu: add support for exporting VRAM using DMA-buf v2 Christian König
                   ` (3 subsequent siblings)
  7 siblings, 1 reply; 25+ messages in thread
From: Christian König @ 2020-03-11 13:51 UTC (permalink / raw)
  To: David1.Zhou, hch, jgg, daniel, dri-devel, linaro-mm-sig,
	linux-media, intel-gfx

Check if we can do peer2peer on the PCIe bus.

Signed-off-by: Christian König <christian.koenig@amd.com>
---
 drivers/gpu/drm/amd/amdgpu/amdgpu_dma_buf.c | 4 ++++
 1 file changed, 4 insertions(+)

diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_dma_buf.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_dma_buf.c
index aef12ee2f1e3..bbf67800c8a6 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_dma_buf.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_dma_buf.c
@@ -38,6 +38,7 @@
 #include <drm/amdgpu_drm.h>
 #include <linux/dma-buf.h>
 #include <linux/dma-fence-array.h>
+#include <linux/pci-p2pdma.h>
 
 /**
  * amdgpu_gem_prime_vmap - &dma_buf_ops.vmap implementation
@@ -179,6 +180,9 @@ static int amdgpu_dma_buf_attach(struct dma_buf *dmabuf,
 	struct amdgpu_device *adev = amdgpu_ttm_adev(bo->tbo.bdev);
 	int r;
 
+	if (pci_p2pdma_distance_many(adev->pdev, &attach->dev, 1, true) < 0)
+		attach->peer2peer = false;
+
 	if (attach->dev->driver == adev->dev->driver)
 		return 0;
 
-- 
2.17.1

_______________________________________________
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/intel-gfx

^ permalink raw reply	[flat|nested] 25+ messages in thread

* [Intel-gfx] [PATCH 5/6] drm/amdgpu: add support for exporting VRAM using DMA-buf v2
  2020-03-11 13:51 [Intel-gfx] P2P for DMA-buf Christian König
                   ` (3 preceding siblings ...)
  2020-03-11 13:51 ` [Intel-gfx] [PATCH 4/6] drm/amdgpu: add checks if DMA-buf P2P is supported Christian König
@ 2020-03-11 13:51 ` Christian König
  2020-03-11 15:08   ` Alex Deucher
  2020-03-11 13:51 ` [Intel-gfx] [PATCH 6/6] drm/amdgpu: improve amdgpu_gem_info debugfs file Christian König
                   ` (2 subsequent siblings)
  7 siblings, 1 reply; 25+ messages in thread
From: Christian König @ 2020-03-11 13:51 UTC (permalink / raw)
  To: David1.Zhou, hch, jgg, daniel, dri-devel, linaro-mm-sig,
	linux-media, intel-gfx

We should be able to do this now after checking all the prerequisites.

v2: fix entrie count in the sgt

Signed-off-by: Christian König <christian.koenig@amd.com>
---
 drivers/gpu/drm/amd/amdgpu/amdgpu_dma_buf.c  | 56 ++++++++---
 drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.h      | 12 ++-
 drivers/gpu/drm/amd/amdgpu/amdgpu_vram_mgr.c | 97 ++++++++++++++++++++
 3 files changed, 151 insertions(+), 14 deletions(-)

diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_dma_buf.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_dma_buf.c
index bbf67800c8a6..43d8ed7dbd00 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_dma_buf.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_dma_buf.c
@@ -276,14 +276,21 @@ static struct sg_table *amdgpu_dma_buf_map(struct dma_buf_attachment *attach,
 	struct dma_buf *dma_buf = attach->dmabuf;
 	struct drm_gem_object *obj = dma_buf->priv;
 	struct amdgpu_bo *bo = gem_to_amdgpu_bo(obj);
+	struct amdgpu_device *adev = amdgpu_ttm_adev(bo->tbo.bdev);
 	struct sg_table *sgt;
 	long r;
 
 	if (!bo->pin_count) {
-		/* move buffer into GTT */
+		/* move buffer into GTT or VRAM */
 		struct ttm_operation_ctx ctx = { false, false };
+		unsigned domains = AMDGPU_GEM_DOMAIN_GTT;
 
-		amdgpu_bo_placement_from_domain(bo, AMDGPU_GEM_DOMAIN_GTT);
+		if (bo->preferred_domains & AMDGPU_GEM_DOMAIN_VRAM &&
+		    attach->peer2peer) {
+			bo->flags |= AMDGPU_GEM_CREATE_CPU_ACCESS_REQUIRED;
+			domains |= AMDGPU_GEM_DOMAIN_VRAM;
+		}
+		amdgpu_bo_placement_from_domain(bo, domains);
 		r = ttm_bo_validate(&bo->tbo, &bo->placement, &ctx);
 		if (r)
 			return ERR_PTR(r);
@@ -293,20 +300,34 @@ static struct sg_table *amdgpu_dma_buf_map(struct dma_buf_attachment *attach,
 		return ERR_PTR(-EBUSY);
 	}
 
-	sgt = drm_prime_pages_to_sg(bo->tbo.ttm->pages, bo->tbo.num_pages);
-	if (IS_ERR(sgt))
-		return sgt;
-
-	if (!dma_map_sg_attrs(attach->dev, sgt->sgl, sgt->nents, dir,
-			      DMA_ATTR_SKIP_CPU_SYNC))
-		goto error_free;
+	switch (bo->tbo.mem.mem_type) {
+	case TTM_PL_TT:
+		sgt = drm_prime_pages_to_sg(bo->tbo.ttm->pages,
+					    bo->tbo.num_pages);
+		if (IS_ERR(sgt))
+			return sgt;
+
+		if (!dma_map_sg_attrs(attach->dev, sgt->sgl, sgt->nents, dir,
+				      DMA_ATTR_SKIP_CPU_SYNC))
+			goto error_free;
+		break;
+
+	case TTM_PL_VRAM:
+		r = amdgpu_vram_mgr_alloc_sgt(adev, &bo->tbo.mem, attach->dev,
+					      dir, &sgt);
+		if (r)
+			return ERR_PTR(r);
+		break;
+	default:
+		return ERR_PTR(-EINVAL);
+	}
 
 	return sgt;
 
 error_free:
 	sg_free_table(sgt);
 	kfree(sgt);
-	return ERR_PTR(-ENOMEM);
+	return ERR_PTR(-EBUSY);
 }
 
 /**
@@ -322,9 +343,18 @@ static void amdgpu_dma_buf_unmap(struct dma_buf_attachment *attach,
 				 struct sg_table *sgt,
 				 enum dma_data_direction dir)
 {
-	dma_unmap_sg(attach->dev, sgt->sgl, sgt->nents, dir);
-	sg_free_table(sgt);
-	kfree(sgt);
+	struct dma_buf *dma_buf = attach->dmabuf;
+	struct drm_gem_object *obj = dma_buf->priv;
+	struct amdgpu_bo *bo = gem_to_amdgpu_bo(obj);
+	struct amdgpu_device *adev = amdgpu_ttm_adev(bo->tbo.bdev);
+
+	if (sgt->sgl->page_link) {
+		dma_unmap_sg(attach->dev, sgt->sgl, sgt->nents, dir);
+		sg_free_table(sgt);
+		kfree(sgt);
+	} else {
+		amdgpu_vram_mgr_free_sgt(adev, attach->dev, dir, sgt);
+	}
 }
 
 /**
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.h b/drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.h
index 7551f3729445..a99d813b23a5 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.h
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.h
@@ -24,8 +24,9 @@
 #ifndef __AMDGPU_TTM_H__
 #define __AMDGPU_TTM_H__
 
-#include "amdgpu.h"
+#include <linux/dma-direction.h>
 #include <drm/gpu_scheduler.h>
+#include "amdgpu.h"
 
 #define AMDGPU_PL_GDS		(TTM_PL_PRIV + 0)
 #define AMDGPU_PL_GWS		(TTM_PL_PRIV + 1)
@@ -74,6 +75,15 @@ uint64_t amdgpu_gtt_mgr_usage(struct ttm_mem_type_manager *man);
 int amdgpu_gtt_mgr_recover(struct ttm_mem_type_manager *man);
 
 u64 amdgpu_vram_mgr_bo_visible_size(struct amdgpu_bo *bo);
+int amdgpu_vram_mgr_alloc_sgt(struct amdgpu_device *adev,
+			      struct ttm_mem_reg *mem,
+			      struct device *dev,
+			      enum dma_data_direction dir,
+			      struct sg_table **sgt);
+void amdgpu_vram_mgr_free_sgt(struct amdgpu_device *adev,
+			      struct device *dev,
+			      enum dma_data_direction dir,
+			      struct sg_table *sgt);
 uint64_t amdgpu_vram_mgr_usage(struct ttm_mem_type_manager *man);
 uint64_t amdgpu_vram_mgr_vis_usage(struct ttm_mem_type_manager *man);
 
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_vram_mgr.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_vram_mgr.c
index 82a3299e53c0..c6e7f00c5b21 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_vram_mgr.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_vram_mgr.c
@@ -22,6 +22,7 @@
  * Authors: Christian König
  */
 
+#include <linux/dma-mapping.h>
 #include "amdgpu.h"
 #include "amdgpu_vm.h"
 #include "amdgpu_atomfirmware.h"
@@ -458,6 +459,102 @@ static void amdgpu_vram_mgr_del(struct ttm_mem_type_manager *man,
 	mem->mm_node = NULL;
 }
 
+/**
+ * amdgpu_vram_mgr_alloc_sgt - allocate and fill a sg table
+ *
+ * @adev: amdgpu device pointer
+ * @mem: TTM memory object
+ * @dev: the other device
+ * @dir: dma direction
+ * @sgt: resulting sg table
+ *
+ * Allocate and fill a sg table from a VRAM allocation.
+ */
+int amdgpu_vram_mgr_alloc_sgt(struct amdgpu_device *adev,
+			      struct ttm_mem_reg *mem,
+			      struct device *dev,
+			      enum dma_data_direction dir,
+			      struct sg_table **sgt)
+{
+	struct drm_mm_node *node;
+	struct scatterlist *sg;
+	int num_entries = 0;
+	unsigned int pages;
+	int i, r;
+
+	*sgt = kmalloc(sizeof(*sg), GFP_KERNEL);
+	if (!*sgt)
+		return -ENOMEM;
+
+	for (pages = mem->num_pages, node = mem->mm_node;
+	     pages; pages -= node->size, ++node)
+		++num_entries;
+
+	r = sg_alloc_table(*sgt, num_entries, GFP_KERNEL);
+	if (r)
+		goto error_free;
+
+	for_each_sg((*sgt)->sgl, sg, num_entries, i)
+		sg->length = 0;
+
+	node = mem->mm_node;
+	for_each_sg((*sgt)->sgl, sg, num_entries, i) {
+		phys_addr_t phys = (node->start << PAGE_SHIFT) +
+			adev->gmc.aper_base;
+		size_t size = node->size << PAGE_SHIFT;
+		dma_addr_t addr;
+
+		++node;
+		addr = dma_map_resource(dev, phys, size, dir,
+					DMA_ATTR_SKIP_CPU_SYNC);
+		r = dma_mapping_error(dev, addr);
+		if (r)
+			goto error_unmap;
+
+		sg_set_dma_addr(sg, addr, size, 0);
+	}
+	return 0;
+
+error_unmap:
+	for_each_sg((*sgt)->sgl, sg, num_entries, i) {
+		if (!sg->length)
+			continue;
+
+		dma_unmap_resource(dev, sg->dma_address,
+				   sg->length, dir,
+				   DMA_ATTR_SKIP_CPU_SYNC);
+	}
+	sg_free_table(*sgt);
+
+error_free:
+	kfree(*sgt);
+	return r;
+}
+
+/**
+ * amdgpu_vram_mgr_alloc_sgt - allocate and fill a sg table
+ *
+ * @adev: amdgpu device pointer
+ * @sgt: sg table to free
+ *
+ * Free a previously allocate sg table.
+ */
+void amdgpu_vram_mgr_free_sgt(struct amdgpu_device *adev,
+			      struct device *dev,
+			      enum dma_data_direction dir,
+			      struct sg_table *sgt)
+{
+	struct scatterlist *sg;
+	int i;
+
+	for_each_sg(sgt->sgl, sg, sgt->nents, i)
+		dma_unmap_resource(dev, sg->dma_address,
+				   sg->length, dir,
+				   DMA_ATTR_SKIP_CPU_SYNC);
+	sg_free_table(sgt);
+	kfree(sgt);
+}
+
 /**
  * amdgpu_vram_mgr_usage - how many bytes are used in this domain
  *
-- 
2.17.1

_______________________________________________
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/intel-gfx

^ permalink raw reply	[flat|nested] 25+ messages in thread

* [Intel-gfx] [PATCH 6/6] drm/amdgpu: improve amdgpu_gem_info debugfs file
  2020-03-11 13:51 [Intel-gfx] P2P for DMA-buf Christian König
                   ` (4 preceding siblings ...)
  2020-03-11 13:51 ` [Intel-gfx] [PATCH 5/6] drm/amdgpu: add support for exporting VRAM using DMA-buf v2 Christian König
@ 2020-03-11 13:51 ` Christian König
  2020-03-11 18:09 ` [Intel-gfx] ✗ Fi.CI.CHECKPATCH: warning for series starting with [1/6] lib/scatterlist: add sg_set_dma_addr() function Patchwork
  2020-03-11 18:34 ` [Intel-gfx] ✗ Fi.CI.BAT: failure " Patchwork
  7 siblings, 0 replies; 25+ messages in thread
From: Christian König @ 2020-03-11 13:51 UTC (permalink / raw)
  To: David1.Zhou, hch, jgg, daniel, dri-devel, linaro-mm-sig,
	linux-media, intel-gfx

Note if a buffer was imported using peer2peer.

Signed-off-by: Christian König <christian.koenig@amd.com>
---
 drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c
index 4277125a79ee..e42608115c99 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c
@@ -29,6 +29,7 @@
 #include <linux/module.h>
 #include <linux/pagemap.h>
 #include <linux/pci.h>
+#include <linux/dma-buf.h>
 
 #include <drm/amdgpu_drm.h>
 #include <drm/drm_debugfs.h>
@@ -854,7 +855,8 @@ static int amdgpu_debugfs_gem_bo_info(int id, void *ptr, void *data)
 	attachment = READ_ONCE(bo->tbo.base.import_attach);
 
 	if (attachment)
-		seq_printf(m, " imported from %p", dma_buf);
+		seq_printf(m, " imported from %p%s", dma_buf,
+			   attachment->peer2peer ? " P2P" : "");
 	else if (dma_buf)
 		seq_printf(m, " exported as %p", dma_buf);
 
-- 
2.17.1

_______________________________________________
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/intel-gfx

^ permalink raw reply	[flat|nested] 25+ messages in thread

* Re: [Intel-gfx] [PATCH 4/6] drm/amdgpu: add checks if DMA-buf P2P is supported
       [not found]   ` <20200311140415.GB31668@ziepe.ca>
@ 2020-03-11 14:33     ` Christian König
       [not found]       ` <20200311143835.GD31668@ziepe.ca>
  0 siblings, 1 reply; 25+ messages in thread
From: Christian König @ 2020-03-11 14:33 UTC (permalink / raw)
  To: Jason Gunthorpe
  Cc: David1.Zhou, Logan Gunthorpe, intel-gfx, dri-devel, hch,
	linaro-mm-sig, linux-media

Am 11.03.20 um 15:04 schrieb Jason Gunthorpe:
> On Wed, Mar 11, 2020 at 02:51:56PM +0100, Christian König wrote:
>> Check if we can do peer2peer on the PCIe bus.
>>
>> Signed-off-by: Christian König <christian.koenig@amd.com>
>>   drivers/gpu/drm/amd/amdgpu/amdgpu_dma_buf.c | 4 ++++
>>   1 file changed, 4 insertions(+)
>>
>> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_dma_buf.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_dma_buf.c
>> index aef12ee2f1e3..bbf67800c8a6 100644
>> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_dma_buf.c
>> @@ -38,6 +38,7 @@
>>   #include <drm/amdgpu_drm.h>
>>   #include <linux/dma-buf.h>
>>   #include <linux/dma-fence-array.h>
>> +#include <linux/pci-p2pdma.h>
>>   
>>   /**
>>    * amdgpu_gem_prime_vmap - &dma_buf_ops.vmap implementation
>> @@ -179,6 +180,9 @@ static int amdgpu_dma_buf_attach(struct dma_buf *dmabuf,
>>   	struct amdgpu_device *adev = amdgpu_ttm_adev(bo->tbo.bdev);
>>   	int r;
>>   
>> +	if (pci_p2pdma_distance_many(adev->pdev, &attach->dev, 1, true) < 0)
>> +		attach->peer2peer = false;
>> +
> Are there other related patches than this series?
>
> p2p dma mapping needs to be done in common code, in p2pdma.c - ie this
> open coding is missing the bus_offset stuff, at least.

Yeah, I'm aware of this. But I couldn't find a better way for now.

> I really do not want to see drivers open code this stuff.
>
> We already have a p2pdma API for handling the struct page case, so I
> suggest adding some new p2pdma API to handle this for non-struct page
> cases.
>
> ie some thing like:
>
> int 'p2pdma map bar'(
>     struct pci_device *source,
>     unsigned int source_bar_number,
>     struct pci_device *dest,
>     physaddr&len *array_of_offsets & length pairs into source bar,
>     struct scatterlist *output_sgl)

Well that's exactly what I have to avoid since I don't have the array of 
offsets around and want to avoid constructing it.

Similar problem for dma_map_resource(). My example does this on demand, 
but essentially we also have use cases where this is done only once.

Ideally we would have some function to create an sgl based on some 
arbitrary collection of offsets and length inside a BAR.

Regards,
Christian.

>
> Jason

_______________________________________________
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/intel-gfx

^ permalink raw reply	[flat|nested] 25+ messages in thread

* Re: [Intel-gfx] [PATCH 4/6] drm/amdgpu: add checks if DMA-buf P2P is supported
       [not found]       ` <20200311143835.GD31668@ziepe.ca>
@ 2020-03-11 14:43         ` Christian König
  0 siblings, 0 replies; 25+ messages in thread
From: Christian König @ 2020-03-11 14:43 UTC (permalink / raw)
  To: Jason Gunthorpe
  Cc: David1.Zhou, Logan Gunthorpe, intel-gfx, dri-devel, hch,
	linaro-mm-sig, linux-media

Am 11.03.20 um 15:38 schrieb Jason Gunthorpe:
> On Wed, Mar 11, 2020 at 03:33:01PM +0100, Christian König wrote:
>> Am 11.03.20 um 15:04 schrieb Jason Gunthorpe:
>>> On Wed, Mar 11, 2020 at 02:51:56PM +0100, Christian König wrote:
>>>> Check if we can do peer2peer on the PCIe bus.
>>>>
>>>> Signed-off-by: Christian König <christian.koenig@amd.com>
>>>>    drivers/gpu/drm/amd/amdgpu/amdgpu_dma_buf.c | 4 ++++
>>>>    1 file changed, 4 insertions(+)
>>>>
>>>> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_dma_buf.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_dma_buf.c
>>>> index aef12ee2f1e3..bbf67800c8a6 100644
>>>> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_dma_buf.c
>>>> @@ -38,6 +38,7 @@
>>>>    #include <drm/amdgpu_drm.h>
>>>>    #include <linux/dma-buf.h>
>>>>    #include <linux/dma-fence-array.h>
>>>> +#include <linux/pci-p2pdma.h>
>>>>    /**
>>>>     * amdgpu_gem_prime_vmap - &dma_buf_ops.vmap implementation
>>>> @@ -179,6 +180,9 @@ static int amdgpu_dma_buf_attach(struct dma_buf *dmabuf,
>>>>    	struct amdgpu_device *adev = amdgpu_ttm_adev(bo->tbo.bdev);
>>>>    	int r;
>>>> +	if (pci_p2pdma_distance_many(adev->pdev, &attach->dev, 1, true) < 0)
>>>> +		attach->peer2peer = false;
>>>> +
>>> Are there other related patches than this series?
>>>
>>> p2p dma mapping needs to be done in common code, in p2pdma.c - ie this
>>> open coding is missing the bus_offset stuff, at least.
>> Yeah, I'm aware of this. But I couldn't find a better way for now.
> Well, it isn't optional :)
>   
>>> I really do not want to see drivers open code this stuff.
>>>
>>> We already have a p2pdma API for handling the struct page case, so I
>>> suggest adding some new p2pdma API to handle this for non-struct page
>>> cases.
>>>
>>> ie some thing like:
>>>
>>> int 'p2pdma map bar'(
>>>      struct pci_device *source,
>>>      unsigned int source_bar_number,
>>>      struct pci_device *dest,
>>>      physaddr&len *array_of_offsets & length pairs into source bar,
>>>      struct scatterlist *output_sgl)
>> Well that's exactly what I have to avoid since I don't have the array of
>> offsets around and want to avoid constructing it.
> Maybe it doesn't need an array of offsets - just a single offset and
> callers can iterate the API?

Yes, that would of course work as well.

But I was assuming that p2pdma_map_bar() needs some state between those 
calls.

>
>> Similar problem for dma_map_resource(). My example does this on demand, but
>> essentially we also have use cases where this is done only once.
> I'm not sure if this is portable. Does any IOMMU HW need to know P2P
> is happening to setup successfully? We currently support such a narrow
> scope of HW for P2P..

On the AMD hardware I'm testing this calling dma_map_resource() already 
seems to work with IOMMU enabled. (Well at least it seemed so 6month ago 
when I last tested this).

>> Ideally we would have some function to create an sgl based on some arbitrary
>> collection of offsets and length inside a BAR.
> Isn't that what I just proposed above ?

Yes, just didn't thought that this would easily possible. I will double 
check the p2pdma code again.

Thanks,
Christian.

>
> Jason

_______________________________________________
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/intel-gfx

^ permalink raw reply	[flat|nested] 25+ messages in thread

* Re: [Intel-gfx] [PATCH 5/6] drm/amdgpu: add support for exporting VRAM using DMA-buf v2
  2020-03-11 13:51 ` [Intel-gfx] [PATCH 5/6] drm/amdgpu: add support for exporting VRAM using DMA-buf v2 Christian König
@ 2020-03-11 15:08   ` Alex Deucher
  0 siblings, 0 replies; 25+ messages in thread
From: Alex Deucher @ 2020-03-11 15:08 UTC (permalink / raw)
  To: Christian König
  Cc: Chunming Zhou, Intel Graphics Development,
	Maling list - DRI developers, Christoph Hellwig, Jason Gunthorpe,
	moderated list:DMA BUFFER SHARING FRAMEWORK, linux-media

On Wed, Mar 11, 2020 at 9:52 AM Christian König
<ckoenig.leichtzumerken@gmail.com> wrote:
>
> We should be able to do this now after checking all the prerequisites.
>
> v2: fix entrie count in the sgt
>
> Signed-off-by: Christian König <christian.koenig@amd.com>
> ---
>  drivers/gpu/drm/amd/amdgpu/amdgpu_dma_buf.c  | 56 ++++++++---
>  drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.h      | 12 ++-
>  drivers/gpu/drm/amd/amdgpu/amdgpu_vram_mgr.c | 97 ++++++++++++++++++++
>  3 files changed, 151 insertions(+), 14 deletions(-)
>
> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_dma_buf.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_dma_buf.c
> index bbf67800c8a6..43d8ed7dbd00 100644
> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_dma_buf.c
> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_dma_buf.c
> @@ -276,14 +276,21 @@ static struct sg_table *amdgpu_dma_buf_map(struct dma_buf_attachment *attach,
>         struct dma_buf *dma_buf = attach->dmabuf;
>         struct drm_gem_object *obj = dma_buf->priv;
>         struct amdgpu_bo *bo = gem_to_amdgpu_bo(obj);
> +       struct amdgpu_device *adev = amdgpu_ttm_adev(bo->tbo.bdev);
>         struct sg_table *sgt;
>         long r;
>
>         if (!bo->pin_count) {
> -               /* move buffer into GTT */
> +               /* move buffer into GTT or VRAM */
>                 struct ttm_operation_ctx ctx = { false, false };
> +               unsigned domains = AMDGPU_GEM_DOMAIN_GTT;
>
> -               amdgpu_bo_placement_from_domain(bo, AMDGPU_GEM_DOMAIN_GTT);
> +               if (bo->preferred_domains & AMDGPU_GEM_DOMAIN_VRAM &&
> +                   attach->peer2peer) {
> +                       bo->flags |= AMDGPU_GEM_CREATE_CPU_ACCESS_REQUIRED;
> +                       domains |= AMDGPU_GEM_DOMAIN_VRAM;
> +               }
> +               amdgpu_bo_placement_from_domain(bo, domains);
>                 r = ttm_bo_validate(&bo->tbo, &bo->placement, &ctx);
>                 if (r)
>                         return ERR_PTR(r);
> @@ -293,20 +300,34 @@ static struct sg_table *amdgpu_dma_buf_map(struct dma_buf_attachment *attach,
>                 return ERR_PTR(-EBUSY);
>         }
>
> -       sgt = drm_prime_pages_to_sg(bo->tbo.ttm->pages, bo->tbo.num_pages);
> -       if (IS_ERR(sgt))
> -               return sgt;
> -
> -       if (!dma_map_sg_attrs(attach->dev, sgt->sgl, sgt->nents, dir,
> -                             DMA_ATTR_SKIP_CPU_SYNC))
> -               goto error_free;
> +       switch (bo->tbo.mem.mem_type) {
> +       case TTM_PL_TT:
> +               sgt = drm_prime_pages_to_sg(bo->tbo.ttm->pages,
> +                                           bo->tbo.num_pages);
> +               if (IS_ERR(sgt))
> +                       return sgt;
> +
> +               if (!dma_map_sg_attrs(attach->dev, sgt->sgl, sgt->nents, dir,
> +                                     DMA_ATTR_SKIP_CPU_SYNC))
> +                       goto error_free;
> +               break;
> +
> +       case TTM_PL_VRAM:
> +               r = amdgpu_vram_mgr_alloc_sgt(adev, &bo->tbo.mem, attach->dev,
> +                                             dir, &sgt);
> +               if (r)
> +                       return ERR_PTR(r);
> +               break;
> +       default:
> +               return ERR_PTR(-EINVAL);
> +       }
>
>         return sgt;
>
>  error_free:
>         sg_free_table(sgt);
>         kfree(sgt);
> -       return ERR_PTR(-ENOMEM);
> +       return ERR_PTR(-EBUSY);
>  }
>
>  /**
> @@ -322,9 +343,18 @@ static void amdgpu_dma_buf_unmap(struct dma_buf_attachment *attach,
>                                  struct sg_table *sgt,
>                                  enum dma_data_direction dir)
>  {
> -       dma_unmap_sg(attach->dev, sgt->sgl, sgt->nents, dir);
> -       sg_free_table(sgt);
> -       kfree(sgt);
> +       struct dma_buf *dma_buf = attach->dmabuf;
> +       struct drm_gem_object *obj = dma_buf->priv;
> +       struct amdgpu_bo *bo = gem_to_amdgpu_bo(obj);
> +       struct amdgpu_device *adev = amdgpu_ttm_adev(bo->tbo.bdev);
> +
> +       if (sgt->sgl->page_link) {
> +               dma_unmap_sg(attach->dev, sgt->sgl, sgt->nents, dir);
> +               sg_free_table(sgt);
> +               kfree(sgt);
> +       } else {
> +               amdgpu_vram_mgr_free_sgt(adev, attach->dev, dir, sgt);
> +       }
>  }
>
>  /**
> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.h b/drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.h
> index 7551f3729445..a99d813b23a5 100644
> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.h
> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.h
> @@ -24,8 +24,9 @@
>  #ifndef __AMDGPU_TTM_H__
>  #define __AMDGPU_TTM_H__
>
> -#include "amdgpu.h"
> +#include <linux/dma-direction.h>
>  #include <drm/gpu_scheduler.h>
> +#include "amdgpu.h"
>
>  #define AMDGPU_PL_GDS          (TTM_PL_PRIV + 0)
>  #define AMDGPU_PL_GWS          (TTM_PL_PRIV + 1)
> @@ -74,6 +75,15 @@ uint64_t amdgpu_gtt_mgr_usage(struct ttm_mem_type_manager *man);
>  int amdgpu_gtt_mgr_recover(struct ttm_mem_type_manager *man);
>
>  u64 amdgpu_vram_mgr_bo_visible_size(struct amdgpu_bo *bo);
> +int amdgpu_vram_mgr_alloc_sgt(struct amdgpu_device *adev,
> +                             struct ttm_mem_reg *mem,
> +                             struct device *dev,
> +                             enum dma_data_direction dir,
> +                             struct sg_table **sgt);
> +void amdgpu_vram_mgr_free_sgt(struct amdgpu_device *adev,
> +                             struct device *dev,
> +                             enum dma_data_direction dir,
> +                             struct sg_table *sgt);
>  uint64_t amdgpu_vram_mgr_usage(struct ttm_mem_type_manager *man);
>  uint64_t amdgpu_vram_mgr_vis_usage(struct ttm_mem_type_manager *man);
>
> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_vram_mgr.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_vram_mgr.c
> index 82a3299e53c0..c6e7f00c5b21 100644
> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_vram_mgr.c
> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_vram_mgr.c
> @@ -22,6 +22,7 @@
>   * Authors: Christian König
>   */
>
> +#include <linux/dma-mapping.h>
>  #include "amdgpu.h"
>  #include "amdgpu_vm.h"
>  #include "amdgpu_atomfirmware.h"
> @@ -458,6 +459,102 @@ static void amdgpu_vram_mgr_del(struct ttm_mem_type_manager *man,
>         mem->mm_node = NULL;
>  }
>
> +/**
> + * amdgpu_vram_mgr_alloc_sgt - allocate and fill a sg table
> + *
> + * @adev: amdgpu device pointer
> + * @mem: TTM memory object
> + * @dev: the other device
> + * @dir: dma direction
> + * @sgt: resulting sg table
> + *
> + * Allocate and fill a sg table from a VRAM allocation.
> + */
> +int amdgpu_vram_mgr_alloc_sgt(struct amdgpu_device *adev,
> +                             struct ttm_mem_reg *mem,
> +                             struct device *dev,
> +                             enum dma_data_direction dir,
> +                             struct sg_table **sgt)
> +{
> +       struct drm_mm_node *node;
> +       struct scatterlist *sg;
> +       int num_entries = 0;
> +       unsigned int pages;
> +       int i, r;
> +
> +       *sgt = kmalloc(sizeof(*sg), GFP_KERNEL);
> +       if (!*sgt)
> +               return -ENOMEM;
> +
> +       for (pages = mem->num_pages, node = mem->mm_node;
> +            pages; pages -= node->size, ++node)
> +               ++num_entries;
> +
> +       r = sg_alloc_table(*sgt, num_entries, GFP_KERNEL);
> +       if (r)
> +               goto error_free;
> +
> +       for_each_sg((*sgt)->sgl, sg, num_entries, i)
> +               sg->length = 0;
> +
> +       node = mem->mm_node;
> +       for_each_sg((*sgt)->sgl, sg, num_entries, i) {
> +               phys_addr_t phys = (node->start << PAGE_SHIFT) +
> +                       adev->gmc.aper_base;
> +               size_t size = node->size << PAGE_SHIFT;
> +               dma_addr_t addr;
> +
> +               ++node;
> +               addr = dma_map_resource(dev, phys, size, dir,
> +                                       DMA_ATTR_SKIP_CPU_SYNC);
> +               r = dma_mapping_error(dev, addr);
> +               if (r)
> +                       goto error_unmap;
> +
> +               sg_set_dma_addr(sg, addr, size, 0);
> +       }
> +       return 0;
> +
> +error_unmap:
> +       for_each_sg((*sgt)->sgl, sg, num_entries, i) {
> +               if (!sg->length)
> +                       continue;
> +
> +               dma_unmap_resource(dev, sg->dma_address,
> +                                  sg->length, dir,
> +                                  DMA_ATTR_SKIP_CPU_SYNC);
> +       }
> +       sg_free_table(*sgt);
> +
> +error_free:
> +       kfree(*sgt);
> +       return r;
> +}
> +
> +/**
> + * amdgpu_vram_mgr_alloc_sgt - allocate and fill a sg table

This should be:
amdgpu_vram_mgr_free_sgt - unmap and free an sg table


> + *
> + * @adev: amdgpu device pointer
> + * @sgt: sg table to free
> + *
> + * Free a previously allocate sg table.
> + */
> +void amdgpu_vram_mgr_free_sgt(struct amdgpu_device *adev,
> +                             struct device *dev,
> +                             enum dma_data_direction dir,
> +                             struct sg_table *sgt)
> +{
> +       struct scatterlist *sg;
> +       int i;
> +
> +       for_each_sg(sgt->sgl, sg, sgt->nents, i)
> +               dma_unmap_resource(dev, sg->dma_address,
> +                                  sg->length, dir,
> +                                  DMA_ATTR_SKIP_CPU_SYNC);
> +       sg_free_table(sgt);
> +       kfree(sgt);
> +}
> +
>  /**
>   * amdgpu_vram_mgr_usage - how many bytes are used in this domain
>   *
> --
> 2.17.1
>
> _______________________________________________
> dri-devel mailing list
> dri-devel@lists.freedesktop.org
> https://lists.freedesktop.org/mailman/listinfo/dri-devel
_______________________________________________
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/intel-gfx

^ permalink raw reply	[flat|nested] 25+ messages in thread

* Re: [Intel-gfx] [PATCH 1/6] lib/scatterlist: add sg_set_dma_addr() function
  2020-03-11 13:51 ` [Intel-gfx] [PATCH 1/6] lib/scatterlist: add sg_set_dma_addr() function Christian König
@ 2020-03-11 15:28   ` Christoph Hellwig
  2020-03-12 10:14     ` Christian König
  0 siblings, 1 reply; 25+ messages in thread
From: Christoph Hellwig @ 2020-03-11 15:28 UTC (permalink / raw)
  To: Christian König
  Cc: David1.Zhou, intel-gfx, dri-devel, hch, jgg, linaro-mm-sig, linux-media

On Wed, Mar 11, 2020 at 02:51:53PM +0100, Christian König wrote:
> This can be used by drivers to setup P2P DMA between device
> memory which is not backed by struct pages.
> 
> The drivers of the involved devices are responsible for
> setting up and tearing down DMA addresses as necessary
> using dma_map_resource().
> 
> The page pointer is set to NULL and only the DMA address,
> length and offset values are valid.

NAK.  The only valid way to fill DMA address in scatterlists is
dma_map_sg / dma_map_sg_attr.
_______________________________________________
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/intel-gfx

^ permalink raw reply	[flat|nested] 25+ messages in thread

* [Intel-gfx] ✗ Fi.CI.CHECKPATCH: warning for series starting with [1/6] lib/scatterlist: add sg_set_dma_addr() function
  2020-03-11 13:51 [Intel-gfx] P2P for DMA-buf Christian König
                   ` (5 preceding siblings ...)
  2020-03-11 13:51 ` [Intel-gfx] [PATCH 6/6] drm/amdgpu: improve amdgpu_gem_info debugfs file Christian König
@ 2020-03-11 18:09 ` Patchwork
  2020-03-11 18:34 ` [Intel-gfx] ✗ Fi.CI.BAT: failure " Patchwork
  7 siblings, 0 replies; 25+ messages in thread
From: Patchwork @ 2020-03-11 18:09 UTC (permalink / raw)
  To: Christian König; +Cc: intel-gfx

== Series Details ==

Series: series starting with [1/6] lib/scatterlist: add sg_set_dma_addr() function
URL   : https://patchwork.freedesktop.org/series/74590/
State : warning

== Summary ==

$ dim checkpatch origin/drm-tip
cd419e3a5b69 lib/scatterlist: add sg_set_dma_addr() function
-:54: WARNING:NO_AUTHOR_SIGN_OFF: Missing Signed-off-by: line by nominal patch author 'Christian König <ckoenig.leichtzumerken@gmail.com>'

total: 0 errors, 1 warnings, 0 checks, 29 lines checked
df3f51ce43ed dma-buf: add peer2peer flag
-:61: WARNING:NO_AUTHOR_SIGN_OFF: Missing Signed-off-by: line by nominal patch author 'Christian König <ckoenig.leichtzumerken@gmail.com>'

total: 0 errors, 1 warnings, 0 checks, 36 lines checked
cf6060d15d07 drm/amdgpu: note that we can handle peer2peer DMA-buf
-:24: WARNING:NO_AUTHOR_SIGN_OFF: Missing Signed-off-by: line by nominal patch author 'Christian König <ckoenig.leichtzumerken@gmail.com>'

total: 0 errors, 1 warnings, 0 checks, 7 lines checked
e9cf1c28be4f drm/amdgpu: add checks if DMA-buf P2P is supported
-:34: WARNING:NO_AUTHOR_SIGN_OFF: Missing Signed-off-by: line by nominal patch author 'Christian König <ckoenig.leichtzumerken@gmail.com>'

total: 0 errors, 1 warnings, 0 checks, 16 lines checked
d50d55f2e67e drm/amdgpu: add support for exporting VRAM using DMA-buf v2
-:31: WARNING:UNSPECIFIED_INT: Prefer 'unsigned int' to bare use of 'unsigned'
#31: FILE: drivers/gpu/drm/amd/amdgpu/amdgpu_dma_buf.c:286:
+		unsigned domains = AMDGPU_GEM_DOMAIN_GTT;

-:253: WARNING:NO_AUTHOR_SIGN_OFF: Missing Signed-off-by: line by nominal patch author 'Christian König <ckoenig.leichtzumerken@gmail.com>'

total: 0 errors, 2 warnings, 0 checks, 220 lines checked
f6450c80542e drm/amdgpu: improve amdgpu_gem_info debugfs file
-:34: WARNING:NO_AUTHOR_SIGN_OFF: Missing Signed-off-by: line by nominal patch author 'Christian König <ckoenig.leichtzumerken@gmail.com>'

total: 0 errors, 1 warnings, 0 checks, 16 lines checked

_______________________________________________
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/intel-gfx

^ permalink raw reply	[flat|nested] 25+ messages in thread

* [Intel-gfx] ✗ Fi.CI.BAT: failure for series starting with [1/6] lib/scatterlist: add sg_set_dma_addr() function
  2020-03-11 13:51 [Intel-gfx] P2P for DMA-buf Christian König
                   ` (6 preceding siblings ...)
  2020-03-11 18:09 ` [Intel-gfx] ✗ Fi.CI.CHECKPATCH: warning for series starting with [1/6] lib/scatterlist: add sg_set_dma_addr() function Patchwork
@ 2020-03-11 18:34 ` Patchwork
  7 siblings, 0 replies; 25+ messages in thread
From: Patchwork @ 2020-03-11 18:34 UTC (permalink / raw)
  To: Christian König; +Cc: intel-gfx

== Series Details ==

Series: series starting with [1/6] lib/scatterlist: add sg_set_dma_addr() function
URL   : https://patchwork.freedesktop.org/series/74590/
State : failure

== Summary ==

CI Bug Log - changes from CI_DRM_8120 -> Patchwork_16930
====================================================

Summary
-------

  **FAILURE**

  Serious unknown changes coming with Patchwork_16930 absolutely need to be
  verified manually.
  
  If you think the reported changes have nothing to do with the changes
  introduced in Patchwork_16930, please notify your bug team to allow them
  to document this new failure mode, which will reduce false positives in CI.

  External URL: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_16930/index.html

Possible new issues
-------------------

  Here are the unknown changes that may have been introduced in Patchwork_16930:

### IGT changes ###

#### Possible regressions ####

  * igt@runner@aborted:
    - fi-kbl-7500u:       NOTRUN -> [FAIL][1]
   [1]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_16930/fi-kbl-7500u/igt@runner@aborted.html

  
Known issues
------------

  Here are the changes found in Patchwork_16930 that come from known issues:

### IGT changes ###

#### Issues hit ####

  * igt@i915_selftest@live@execlists:
    - fi-kbl-7500u:       [PASS][2] -> [INCOMPLETE][3] ([fdo#112259] / [i915#1430])
   [2]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_8120/fi-kbl-7500u/igt@i915_selftest@live@execlists.html
   [3]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_16930/fi-kbl-7500u/igt@i915_selftest@live@execlists.html

  * igt@kms_chamelium@hdmi-edid-read:
    - fi-icl-u2:          [PASS][4] -> [FAIL][5] ([i915#217])
   [4]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_8120/fi-icl-u2/igt@kms_chamelium@hdmi-edid-read.html
   [5]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_16930/fi-icl-u2/igt@kms_chamelium@hdmi-edid-read.html

  
#### Possible fixes ####

  * igt@gem_exec_suspend@basic-s4-devices:
    - fi-tgl-y:           [FAIL][6] ([CI#94]) -> [PASS][7]
   [6]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_8120/fi-tgl-y/igt@gem_exec_suspend@basic-s4-devices.html
   [7]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_16930/fi-tgl-y/igt@gem_exec_suspend@basic-s4-devices.html

  * igt@i915_selftest@live@execlists:
    - fi-bxt-dsi:         [INCOMPLETE][8] ([fdo#103927]) -> [PASS][9]
   [8]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_8120/fi-bxt-dsi/igt@i915_selftest@live@execlists.html
   [9]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_16930/fi-bxt-dsi/igt@i915_selftest@live@execlists.html

  * igt@i915_selftest@live@gem_contexts:
    - fi-skl-lmem:        [INCOMPLETE][10] ([i915#424]) -> [PASS][11]
   [10]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_8120/fi-skl-lmem/igt@i915_selftest@live@gem_contexts.html
   [11]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_16930/fi-skl-lmem/igt@i915_selftest@live@gem_contexts.html

  * igt@i915_selftest@live@gt_heartbeat:
    - fi-kbl-soraka:      [DMESG-FAIL][12] ([i915#541]) -> [PASS][13]
   [12]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_8120/fi-kbl-soraka/igt@i915_selftest@live@gt_heartbeat.html
   [13]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_16930/fi-kbl-soraka/igt@i915_selftest@live@gt_heartbeat.html

  
  [CI#94]: https://gitlab.freedesktop.org/gfx-ci/i915-infra/issues/94
  [fdo#103927]: https://bugs.freedesktop.org/show_bug.cgi?id=103927
  [fdo#112259]: https://bugs.freedesktop.org/show_bug.cgi?id=112259
  [i915#1430]: https://gitlab.freedesktop.org/drm/intel/issues/1430
  [i915#217]: https://gitlab.freedesktop.org/drm/intel/issues/217
  [i915#424]: https://gitlab.freedesktop.org/drm/intel/issues/424
  [i915#541]: https://gitlab.freedesktop.org/drm/intel/issues/541


Participating hosts (49 -> 41)
------------------------------

  Missing    (8): fi-ilk-m540 fi-hsw-4200u fi-skl-6770hq fi-byt-squawks fi-bsw-cyan fi-bwr-2160 fi-ctg-p8600 fi-byt-clapper 


Build changes
-------------

  * CI: CI-20190529 -> None
  * Linux: CI_DRM_8120 -> Patchwork_16930

  CI-20190529: 20190529
  CI_DRM_8120: ce66c439df71f01b018803664c4a50fc61255788 @ git://anongit.freedesktop.org/gfx-ci/linux
  IGT_5505: 8973d811f3fdfb4ace4aabab2095ce0309881648 @ git://anongit.freedesktop.org/xorg/app/intel-gpu-tools
  Patchwork_16930: f6450c80542e34d54589e8b9a048f933f9f8f23f @ git://anongit.freedesktop.org/gfx-ci/linux


== Linux commits ==

f6450c80542e drm/amdgpu: improve amdgpu_gem_info debugfs file
d50d55f2e67e drm/amdgpu: add support for exporting VRAM using DMA-buf v2
e9cf1c28be4f drm/amdgpu: add checks if DMA-buf P2P is supported
cf6060d15d07 drm/amdgpu: note that we can handle peer2peer DMA-buf
df3f51ce43ed dma-buf: add peer2peer flag
cd419e3a5b69 lib/scatterlist: add sg_set_dma_addr() function

== Logs ==

For more details see: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_16930/index.html
_______________________________________________
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/intel-gfx

^ permalink raw reply	[flat|nested] 25+ messages in thread

* Re: [Intel-gfx] [PATCH 1/6] lib/scatterlist: add sg_set_dma_addr() function
  2020-03-11 15:28   ` Christoph Hellwig
@ 2020-03-12 10:14     ` Christian König
  2020-03-12 10:19       ` Christoph Hellwig
  0 siblings, 1 reply; 25+ messages in thread
From: Christian König @ 2020-03-12 10:14 UTC (permalink / raw)
  To: Christoph Hellwig
  Cc: David1.Zhou, intel-gfx, dri-devel, linaro-mm-sig, jgg, linux-media

Am 11.03.20 um 16:28 schrieb Christoph Hellwig:
> On Wed, Mar 11, 2020 at 02:51:53PM +0100, Christian König wrote:
>> This can be used by drivers to setup P2P DMA between device
>> memory which is not backed by struct pages.
>>
>> The drivers of the involved devices are responsible for
>> setting up and tearing down DMA addresses as necessary
>> using dma_map_resource().
>>
>> The page pointer is set to NULL and only the DMA address,
>> length and offset values are valid.
> NAK.  The only valid way to fill DMA address in scatterlists is
> dma_map_sg / dma_map_sg_attr.

How can we then map PCIe BARs into an scatterlist which are not backed 
by struct pages?

Regards,
Christian.
_______________________________________________
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/intel-gfx

^ permalink raw reply	[flat|nested] 25+ messages in thread

* Re: [Intel-gfx] [PATCH 1/6] lib/scatterlist: add sg_set_dma_addr() function
  2020-03-12 10:14     ` Christian König
@ 2020-03-12 10:19       ` Christoph Hellwig
  2020-03-12 10:31         ` Christian König
  0 siblings, 1 reply; 25+ messages in thread
From: Christoph Hellwig @ 2020-03-12 10:19 UTC (permalink / raw)
  To: christian.koenig
  Cc: David1.Zhou, intel-gfx, dri-devel, Christoph Hellwig, jgg,
	linaro-mm-sig, linux-media

On Thu, Mar 12, 2020 at 11:14:22AM +0100, Christian König wrote:
> > > The page pointer is set to NULL and only the DMA address,
> > > length and offset values are valid.
> > NAK.  The only valid way to fill DMA address in scatterlists is
> > dma_map_sg / dma_map_sg_attr.
> 
> How can we then map PCIe BARs into an scatterlist which are not backed by
> struct pages?

You can't.  scatterlists by definition map memory backed by a struct
page.  If you want to map something else struct scatterlist is the
wrong structure and you need to use something else (which you should
anyway as struct scatterlist is a bad design patter, and the above
is only one of the many issues with it).
_______________________________________________
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/intel-gfx

^ permalink raw reply	[flat|nested] 25+ messages in thread

* Re: [Intel-gfx] [PATCH 1/6] lib/scatterlist: add sg_set_dma_addr() function
  2020-03-12 10:19       ` Christoph Hellwig
@ 2020-03-12 10:31         ` Christian König
  2020-03-12 10:47           ` Christoph Hellwig
  0 siblings, 1 reply; 25+ messages in thread
From: Christian König @ 2020-03-12 10:31 UTC (permalink / raw)
  To: Christoph Hellwig
  Cc: David1.Zhou, intel-gfx, dri-devel, linaro-mm-sig, jgg, linux-media

Am 12.03.20 um 11:19 schrieb Christoph Hellwig:
> On Thu, Mar 12, 2020 at 11:14:22AM +0100, Christian König wrote:
>>>> The page pointer is set to NULL and only the DMA address,
>>>> length and offset values are valid.
>>> NAK.  The only valid way to fill DMA address in scatterlists is
>>> dma_map_sg / dma_map_sg_attr.
>> How can we then map PCIe BARs into an scatterlist which are not backed by
>> struct pages?
> You can't.  scatterlists by definition map memory backed by a struct
> page.  If you want to map something else struct scatterlist is the
> wrong structure and you need to use something else (which you should
> anyway as struct scatterlist is a bad design patter, and the above
> is only one of the many issues with it).

But how should we then deal with all the existing interfaces which 
already take a scatterlist/sg_table ?

The whole DMA-buf design and a lot of drivers are build around 
scatterlist/sg_table and to me that actually makes quite a lot of sense.

For TTM I'm also trying for quite a while to just nuke the manual 
dma_address arrays we have and switch over to scatterlist/sg_table.

I mean we could come up with a new structure for this, but to me that 
just looks like reinventing the wheel. Especially since drivers need to 
be able to handle both I/O to system memory and I/O to PCIe BARs.

Regards,
Christian.


_______________________________________________
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/intel-gfx

^ permalink raw reply	[flat|nested] 25+ messages in thread

* Re: [Intel-gfx] [PATCH 1/6] lib/scatterlist: add sg_set_dma_addr() function
  2020-03-12 10:31         ` Christian König
@ 2020-03-12 10:47           ` Christoph Hellwig
  2020-03-12 11:02             ` Christian König
       [not found]             ` <20200312141928.GK31668@ziepe.ca>
  0 siblings, 2 replies; 25+ messages in thread
From: Christoph Hellwig @ 2020-03-12 10:47 UTC (permalink / raw)
  To: Christian König
  Cc: David1.Zhou, intel-gfx, dri-devel, Christoph Hellwig, jgg,
	linaro-mm-sig, linux-media

On Thu, Mar 12, 2020 at 11:31:35AM +0100, Christian König wrote:
> But how should we then deal with all the existing interfaces which already
> take a scatterlist/sg_table ?
>
> The whole DMA-buf design and a lot of drivers are build around
> scatterlist/sg_table and to me that actually makes quite a lot of sense.
> 

Replace them with a saner interface that doesn't take a scatterlist.
At very least for new functionality like peer to peer DMA, but
especially this code would also benefit from a general move away
from the scatterlist.

> For TTM I'm also trying for quite a while to just nuke the manual
> dma_address arrays we have and switch over to scatterlist/sg_table.

Which is a move in the wrong direction.

> I mean we could come up with a new structure for this, but to me that just
> looks like reinventing the wheel. Especially since drivers need to be able
> to handle both I/O to system memory and I/O to PCIe BARs.

The structure for holding the struct page side of the scatterlist is
called struct bio_vec, so far mostly used by the block and networking
code.  The structure for holding dma addresses doesn't really exist
in a generic form, but would be an array of these structures:

struct dma_sg {
	dma_addr_t	addr;
	u32		len;
};

Keeping them separate is important as most IOMMU drivers will return
less entries than you can feed them.  E.g. if your input boundaries
are 4k aligned you will usually just get a single IOVA entry back.
I will soon also have a dma mapping interface that will take advantage
of that fact.
_______________________________________________
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/intel-gfx

^ permalink raw reply	[flat|nested] 25+ messages in thread

* Re: [Intel-gfx] [PATCH 1/6] lib/scatterlist: add sg_set_dma_addr() function
  2020-03-12 10:47           ` Christoph Hellwig
@ 2020-03-12 11:02             ` Christian König
       [not found]             ` <20200312141928.GK31668@ziepe.ca>
  1 sibling, 0 replies; 25+ messages in thread
From: Christian König @ 2020-03-12 11:02 UTC (permalink / raw)
  To: Christoph Hellwig
  Cc: David1.Zhou, intel-gfx, dri-devel, linaro-mm-sig, jgg, linux-media

Am 12.03.20 um 11:47 schrieb Christoph Hellwig:
> On Thu, Mar 12, 2020 at 11:31:35AM +0100, Christian König wrote:
> [SNIP]
>> I mean we could come up with a new structure for this, but to me that just
>> looks like reinventing the wheel. Especially since drivers need to be able
>> to handle both I/O to system memory and I/O to PCIe BARs.
> The structure for holding the struct page side of the scatterlist is
> called struct bio_vec, so far mostly used by the block and networking
> code.

Yeah, I'm aware of this.

> The structure for holding dma addresses doesn't really exist
> in a generic form, but would be an array of these structures:
>
> struct dma_sg {
> 	dma_addr_t	addr;
> 	u32		len;
> };

So the whole idea is to nuke scatterlist/sg_table in the long term and 
switch over to using bio_vec as input and dma_sg as output for a DMA 
mapping operation.

Is that correct? If yes I could live with that, but it makes my patchset 
much more complicated.

> Keeping them separate is important as most IOMMU drivers will return
> less entries than you can feed them.  E.g. if your input boundaries
> are 4k aligned you will usually just get a single IOVA entry back.
> I will soon also have a dma mapping interface that will take advantage
> of that fact.

Yeah, I noticed as well that this is not really well handled.

Thanks for the feedback,
Christian.
_______________________________________________
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/intel-gfx

^ permalink raw reply	[flat|nested] 25+ messages in thread

* Re: [Intel-gfx] [PATCH 1/6] lib/scatterlist: add sg_set_dma_addr() function
       [not found]             ` <20200312141928.GK31668@ziepe.ca>
@ 2020-03-12 15:39               ` Christian König
  2020-03-12 16:13               ` Logan Gunthorpe
  2020-03-13 11:21               ` Christoph Hellwig
  2 siblings, 0 replies; 25+ messages in thread
From: Christian König @ 2020-03-12 15:39 UTC (permalink / raw)
  To: Jason Gunthorpe, Christoph Hellwig
  Cc: David1.Zhou, intel-gfx, dri-devel, linaro-mm-sig,
	Logan Gunthorpe, linux-media

Am 12.03.20 um 15:19 schrieb Jason Gunthorpe:
> On Thu, Mar 12, 2020 at 03:47:29AM -0700, Christoph Hellwig wrote:
>> On Thu, Mar 12, 2020 at 11:31:35AM +0100, Christian König wrote:
>>> But how should we then deal with all the existing interfaces which already
>>> take a scatterlist/sg_table ?
>>>
>>> The whole DMA-buf design and a lot of drivers are build around
>>> scatterlist/sg_table and to me that actually makes quite a lot of sense.
>>>
>> Replace them with a saner interface that doesn't take a scatterlist.
>> At very least for new functionality like peer to peer DMA, but
>> especially this code would also benefit from a general move away
>> from the scatterlist.
> If dma buf can do P2P I'd like to see support for consuming a dmabuf
> in RDMA.

That would indeed be awesome.

> Looking at how.. there is an existing sgl based path starting
> from get_user_pages through dma map to the drivers. (ib_umem)
>
> I can replace the driver part with something else (dma_sg), but not
> until we get a way to DMA map pages directly into that something
> else..
>
> The non-page scatterlist is also a big concern for RDMA as we have
> drivers that want the page list, so even if we did as this series
> contemplates I'd have still have to split the drivers and create the
> notion of a dma-only SGL.

Yeah that's my concern as well. For GPU drivers I don't think we need 
the struct pages anywhere, but that might not be true for others.

>>> I mean we could come up with a new structure for this, but to me that just
>>> looks like reinventing the wheel. Especially since drivers need to be able
>>> to handle both I/O to system memory and I/O to PCIe BARs.
>> The structure for holding the struct page side of the scatterlist is
>> called struct bio_vec, so far mostly used by the block and networking
>> code.
> I haven't used bio_vecs before, do they support chaining like SGL so
> they can be very big? RDMA dma maps gigabytes of memory
>
>> The structure for holding dma addresses doesn't really exist
>> in a generic form, but would be an array of these structures:
>>
>> struct dma_sg {
>> 	dma_addr_t	addr;
>> 	u32		len;
>> };
> Same question, RDMA needs to represent gigabytes of pages in a DMA
> list, we will need some generic way to handle that. I suspect GPU has
> a similar need? Can it be accomidated in some generic dma_sg?

Yes, we easily have ranges of >1GB. So I would certainly say u64 for the 
len here.

> So I'm guessing the path forward is something like
>
>   - Add some generic dma_sg data structure and helper
>   - Add dma mapping code to go from pages to dma_sg
>   - Rework RDMA to use dma_sg and the new dma mapping code
>   - Rework dmabuf to support dma mapping to a dma_sg
>   - Rework GPU drivers to use dma_sg
>   - Teach p2pdma to generate a dma_sg from a BAR page list
>   - This series
>
> ?

Sounds pretty much like a plan to me, but unfortunately like a rather 
huge one.

Because of this and cause I don't know if all drivers can live with 
dma_sg I'm not sure if we shouldn't have the switch from scatterlist to 
dma_sg separately to this peer2peer work.

Christian.

>
> Jason

_______________________________________________
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/intel-gfx

^ permalink raw reply	[flat|nested] 25+ messages in thread

* Re: [Intel-gfx] [PATCH 1/6] lib/scatterlist: add sg_set_dma_addr() function
       [not found]             ` <20200312141928.GK31668@ziepe.ca>
  2020-03-12 15:39               ` Christian König
@ 2020-03-12 16:13               ` Logan Gunthorpe
  2020-03-13 11:21               ` Christoph Hellwig
  2 siblings, 0 replies; 25+ messages in thread
From: Logan Gunthorpe @ 2020-03-12 16:13 UTC (permalink / raw)
  To: Jason Gunthorpe, Christoph Hellwig
  Cc: David1.Zhou, intel-gfx, dri-devel, linaro-mm-sig,
	Christian König, linux-media



On 2020-03-12 8:19 a.m., Jason Gunthorpe wrote:
> On Thu, Mar 12, 2020 at 03:47:29AM -0700, Christoph Hellwig wrote:
>> On Thu, Mar 12, 2020 at 11:31:35AM +0100, Christian König wrote:
>>> But how should we then deal with all the existing interfaces which already
>>> take a scatterlist/sg_table ?
>>>
>>> The whole DMA-buf design and a lot of drivers are build around
>>> scatterlist/sg_table and to me that actually makes quite a lot of sense.
>>>
>>
>> Replace them with a saner interface that doesn't take a scatterlist.
>> At very least for new functionality like peer to peer DMA, but
>> especially this code would also benefit from a general move away
>> from the scatterlist.
> 
> If dma buf can do P2P I'd like to see support for consuming a dmabuf
> in RDMA. Looking at how.. there is an existing sgl based path starting
> from get_user_pages through dma map to the drivers. (ib_umem)
> 
> I can replace the driver part with something else (dma_sg), but not
> until we get a way to DMA map pages directly into that something
> else..
> 
> The non-page scatterlist is also a big concern for RDMA as we have
> drivers that want the page list, so even if we did as this series
> contemplates I'd have still have to split the drivers and create the
> notion of a dma-only SGL.
> 
>>> I mean we could come up with a new structure for this, but to me that just
>>> looks like reinventing the wheel. Especially since drivers need to be able
>>> to handle both I/O to system memory and I/O to PCIe BARs.
>>
>> The structure for holding the struct page side of the scatterlist is
>> called struct bio_vec, so far mostly used by the block and networking
>> code.
> 
> I haven't used bio_vecs before, do they support chaining like SGL so
> they can be very big? RDMA dma maps gigabytes of memory

bio_vec's themselves don't support chaining... In the block layer they
are used in a struct bio which handles chaining, splitting and other
features. Each bio, though, has a limit of 256 segments to avoid higher
order allocations. Depending on your use case, you could reuse bios or
write your own container to chain bio_vecs.

>> The structure for holding dma addresses doesn't really exist
>> in a generic form, but would be an array of these structures:
>>
>> struct dma_sg {
>> 	dma_addr_t	addr;
>> 	u32		len;
>> };


> Yes, we easily have ranges of >1GB. So I would certainly say u64 for the len here.

I'd probably avoid the u64 here and leave space for some flags or
something. If you have >1GB to map you can always just have mulitple
segments. With 4GB per segment and 256 segments per page, a page of DMA
sgs can easily map 1TB of memory in a single call and with chaining or
larger allocations you can extend that further, if needed.

Logan

_______________________________________________
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/intel-gfx

^ permalink raw reply	[flat|nested] 25+ messages in thread

* Re: [Intel-gfx] [PATCH 1/6] lib/scatterlist: add sg_set_dma_addr() function
       [not found]             ` <20200312141928.GK31668@ziepe.ca>
  2020-03-12 15:39               ` Christian König
  2020-03-12 16:13               ` Logan Gunthorpe
@ 2020-03-13 11:21               ` Christoph Hellwig
  2020-03-13 13:33                 ` Christian König
       [not found]                 ` <20200313121742.GZ31668@ziepe.ca>
  2 siblings, 2 replies; 25+ messages in thread
From: Christoph Hellwig @ 2020-03-13 11:21 UTC (permalink / raw)
  To: Jason Gunthorpe
  Cc: David1.Zhou, Logan Gunthorpe, intel-gfx, dri-devel,
	Christoph Hellwig, linaro-mm-sig, Christian König,
	linux-media

On Thu, Mar 12, 2020 at 11:19:28AM -0300, Jason Gunthorpe wrote:
> The non-page scatterlist is also a big concern for RDMA as we have
> drivers that want the page list, so even if we did as this series
> contemplates I'd have still have to split the drivers and create the
> notion of a dma-only SGL.

The drivers I looked at want a list of IOVA address, aligned to the
device "page size".  What other data do drivers want?  Execept for the
software protocol stack drivers, which of couse need pages for the
stack futher down.

> I haven't used bio_vecs before, do they support chaining like SGL so
> they can be very big? RDMA dma maps gigabytes of memory

bio_vecs itself don't have the chaining, but the bios build around them
do.  But each entry can map a huge pile.  If needed we could use the
same chaining scheme we use for scatterlists for bio_vecs as well, but
lets see if we really end up needing that.

> So I'm guessing the path forward is something like
> 
>  - Add some generic dma_sg data structure and helper
>  - Add dma mapping code to go from pages to dma_sg

That has been on my todo list for a while.  All the DMA consolidatation
is to prepare for that and we're finally getting close.
_______________________________________________
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/intel-gfx

^ permalink raw reply	[flat|nested] 25+ messages in thread

* Re: [Intel-gfx] [PATCH 1/6] lib/scatterlist: add sg_set_dma_addr() function
  2020-03-13 11:21               ` Christoph Hellwig
@ 2020-03-13 13:33                 ` Christian König
       [not found]                 ` <20200313121742.GZ31668@ziepe.ca>
  1 sibling, 0 replies; 25+ messages in thread
From: Christian König @ 2020-03-13 13:33 UTC (permalink / raw)
  To: Christoph Hellwig, Jason Gunthorpe
  Cc: David1.Zhou, intel-gfx, dri-devel, linaro-mm-sig,
	Logan Gunthorpe, linux-media

Am 13.03.20 um 12:21 schrieb Christoph Hellwig:
> On Thu, Mar 12, 2020 at 11:19:28AM -0300, Jason Gunthorpe wrote:
>> The non-page scatterlist is also a big concern for RDMA as we have
>> drivers that want the page list, so even if we did as this series
>> contemplates I'd have still have to split the drivers and create the
>> notion of a dma-only SGL.
> The drivers I looked at want a list of IOVA address, aligned to the
> device "page size".  What other data do drivers want?

Well for GPUs I have the requirement that those IOVA addresses allow 
random access.

That's the reason why we currently convert the sg_table into a linear 
arrays of addresses and pages. To solve that keeping the length in 
separate optional array would be ideal for us.

But this is so a special use case that I'm not sure if we want to 
support this in the common framework or not.

> Execept for the software protocol stack drivers, which of couse need pages for the
> stack futher down.

Yes completely agree.

For the GPUs I will propose a patch to stop copying the page from the 
sg_table over into our linear arrays and see if anybody starts to scream.

I don't think so, but probably better to double check.

Thanks,
Christian.

>
>> I haven't used bio_vecs before, do they support chaining like SGL so
>> they can be very big? RDMA dma maps gigabytes of memory
> bio_vecs itself don't have the chaining, but the bios build around them
> do.  But each entry can map a huge pile.  If needed we could use the
> same chaining scheme we use for scatterlists for bio_vecs as well, but
> lets see if we really end up needing that.
>
>> So I'm guessing the path forward is something like
>>
>>   - Add some generic dma_sg data structure and helper
>>   - Add dma mapping code to go from pages to dma_sg
> That has been on my todo list for a while.  All the DMA consolidatation
> is to prepare for that and we're finally getting close.

_______________________________________________
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/intel-gfx

^ permalink raw reply	[flat|nested] 25+ messages in thread

* Re: [Intel-gfx] [PATCH 1/6] lib/scatterlist: add sg_set_dma_addr() function
       [not found]                 ` <20200313121742.GZ31668@ziepe.ca>
@ 2020-03-16  8:56                   ` Christoph Hellwig
  2020-03-16  9:41                     ` Christian König
  0 siblings, 1 reply; 25+ messages in thread
From: Christoph Hellwig @ 2020-03-16  8:56 UTC (permalink / raw)
  To: Jason Gunthorpe
  Cc: David1.Zhou, Logan Gunthorpe, intel-gfx, dri-devel,
	Christoph Hellwig, linaro-mm-sig, Christian König,
	linux-media

On Fri, Mar 13, 2020 at 09:17:42AM -0300, Jason Gunthorpe wrote:
> On Fri, Mar 13, 2020 at 04:21:39AM -0700, Christoph Hellwig wrote:
> > On Thu, Mar 12, 2020 at 11:19:28AM -0300, Jason Gunthorpe wrote:
> > > The non-page scatterlist is also a big concern for RDMA as we have
> > > drivers that want the page list, so even if we did as this series
> > > contemplates I'd have still have to split the drivers and create the
> > > notion of a dma-only SGL.
> > 
> > The drivers I looked at want a list of IOVA address, aligned to the
> > device "page size".  What other data do drivers want?  Execept for the
> > software protocol stack drivers, which of couse need pages for the
> > stack futher down.
> 
> In principle it is possible to have just an aligned page list -
> however the page size is variable, following certain rules, and today
> the drivers still determine the correct page size largely on their
> own.  
> 
> Some progress was made recently to consolidate this, but more is
> needed.
> 
> If the common code doesn't know the device page size in advance then
> today's approach of sending largest possible dma mapped SGLs into the
> device driver is best. The driver only has to do splitting.

The point was that drivers don't need pages, drivers need IOVAs.  In
what form they are stuffed into the hardware is the drivers problem.
_______________________________________________
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/intel-gfx

^ permalink raw reply	[flat|nested] 25+ messages in thread

* Re: [Intel-gfx] [PATCH 1/6] lib/scatterlist: add sg_set_dma_addr() function
  2020-03-16  8:56                   ` Christoph Hellwig
@ 2020-03-16  9:41                     ` Christian König
  2020-03-16  9:52                       ` Christoph Hellwig
  0 siblings, 1 reply; 25+ messages in thread
From: Christian König @ 2020-03-16  9:41 UTC (permalink / raw)
  To: Christoph Hellwig, Jason Gunthorpe
  Cc: David1.Zhou, intel-gfx, dri-devel, linaro-mm-sig,
	Logan Gunthorpe, linux-media

Am 16.03.20 um 09:56 schrieb Christoph Hellwig:
> On Fri, Mar 13, 2020 at 09:17:42AM -0300, Jason Gunthorpe wrote:
>> On Fri, Mar 13, 2020 at 04:21:39AM -0700, Christoph Hellwig wrote:
>>> On Thu, Mar 12, 2020 at 11:19:28AM -0300, Jason Gunthorpe wrote:
>>>> The non-page scatterlist is also a big concern for RDMA as we have
>>>> drivers that want the page list, so even if we did as this series
>>>> contemplates I'd have still have to split the drivers and create the
>>>> notion of a dma-only SGL.
>>> The drivers I looked at want a list of IOVA address, aligned to the
>>> device "page size".  What other data do drivers want?  Execept for the
>>> software protocol stack drivers, which of couse need pages for the
>>> stack futher down.
>> In principle it is possible to have just an aligned page list -
>> however the page size is variable, following certain rules, and today
>> the drivers still determine the correct page size largely on their
>> own.
>>
>> Some progress was made recently to consolidate this, but more is
>> needed.
>>
>> If the common code doesn't know the device page size in advance then
>> today's approach of sending largest possible dma mapped SGLs into the
>> device driver is best. The driver only has to do splitting.
> The point was that drivers don't need pages, drivers need IOVAs.  In
> what form they are stuffed into the hardware is the drivers problem.

Well I would prefer if the drivers can somehow express their 
requirements and get IOVA structures already in the form they need.

Converting the IOVA data from one form to another is sometimes quite 
costly. Especially when it is only temporarily needed.

Regards,
Christian.
_______________________________________________
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/intel-gfx

^ permalink raw reply	[flat|nested] 25+ messages in thread

* Re: [Intel-gfx] [PATCH 1/6] lib/scatterlist: add sg_set_dma_addr() function
  2020-03-16  9:41                     ` Christian König
@ 2020-03-16  9:52                       ` Christoph Hellwig
  0 siblings, 0 replies; 25+ messages in thread
From: Christoph Hellwig @ 2020-03-16  9:52 UTC (permalink / raw)
  To: Christian König
  Cc: linaro-mm-sig, David1.Zhou, intel-gfx, dri-devel,
	Christoph Hellwig, Jason Gunthorpe, Logan Gunthorpe, linux-media

On Mon, Mar 16, 2020 at 10:41:42AM +0100, Christian König wrote:
> Well I would prefer if the drivers can somehow express their requirements
> and get IOVA structures already in the form they need.
> 
> Converting the IOVA data from one form to another is sometimes quite costly.
> Especially when it is only temporarily needed.

We basically have two ways to generate the IOVA:

  - a linear translation for the direct mapping case or some dumb IOMMU
    drivers - in that case case there is a 1:1 mapping between input
    segments and output segments in DMA mapping
  - a non-trivial IOMMU where all aligned segments are merged into
    a single IOVA range

So I don't really see how the dma layer could help much with any
limitation beyond existing max size and dma boundary ones.
_______________________________________________
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/intel-gfx

^ permalink raw reply	[flat|nested] 25+ messages in thread

end of thread, other threads:[~2020-03-16 14:18 UTC | newest]

Thread overview: 25+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2020-03-11 13:51 [Intel-gfx] P2P for DMA-buf Christian König
2020-03-11 13:51 ` [Intel-gfx] [PATCH 1/6] lib/scatterlist: add sg_set_dma_addr() function Christian König
2020-03-11 15:28   ` Christoph Hellwig
2020-03-12 10:14     ` Christian König
2020-03-12 10:19       ` Christoph Hellwig
2020-03-12 10:31         ` Christian König
2020-03-12 10:47           ` Christoph Hellwig
2020-03-12 11:02             ` Christian König
     [not found]             ` <20200312141928.GK31668@ziepe.ca>
2020-03-12 15:39               ` Christian König
2020-03-12 16:13               ` Logan Gunthorpe
2020-03-13 11:21               ` Christoph Hellwig
2020-03-13 13:33                 ` Christian König
     [not found]                 ` <20200313121742.GZ31668@ziepe.ca>
2020-03-16  8:56                   ` Christoph Hellwig
2020-03-16  9:41                     ` Christian König
2020-03-16  9:52                       ` Christoph Hellwig
2020-03-11 13:51 ` [Intel-gfx] [PATCH 2/6] dma-buf: add peer2peer flag Christian König
2020-03-11 13:51 ` [Intel-gfx] [PATCH 3/6] drm/amdgpu: note that we can handle peer2peer DMA-buf Christian König
2020-03-11 13:51 ` [Intel-gfx] [PATCH 4/6] drm/amdgpu: add checks if DMA-buf P2P is supported Christian König
     [not found]   ` <20200311140415.GB31668@ziepe.ca>
2020-03-11 14:33     ` Christian König
     [not found]       ` <20200311143835.GD31668@ziepe.ca>
2020-03-11 14:43         ` Christian König
2020-03-11 13:51 ` [Intel-gfx] [PATCH 5/6] drm/amdgpu: add support for exporting VRAM using DMA-buf v2 Christian König
2020-03-11 15:08   ` Alex Deucher
2020-03-11 13:51 ` [Intel-gfx] [PATCH 6/6] drm/amdgpu: improve amdgpu_gem_info debugfs file Christian König
2020-03-11 18:09 ` [Intel-gfx] ✗ Fi.CI.CHECKPATCH: warning for series starting with [1/6] lib/scatterlist: add sg_set_dma_addr() function Patchwork
2020-03-11 18:34 ` [Intel-gfx] ✗ Fi.CI.BAT: failure " Patchwork

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).