[PATCH v2 0/4] Fixes related to the DMA max_segment_size parameter

[PATCH v2 0/4] Fixes related to the DMA max_segment_size parameter

[Bart Van Assche]

Hi Jason,

These four patches are what I came up with while analyzing a kernel warning
triggered by the ib_srp kernel driver. Please consider these patches for
inclusion in the Linux kernel.

Thanks,

Bart.

Changes compared to v1:
- Restored the dma_set_max_seg_size() call in setup_dma_device(). Reordered
  this patch series to keep it bisectable.

Bart Van Assche (4):
  RDMA/core: Fix ib_dma_max_seg_size()
  rdma_rxe: Increase DMA max_segment_size parameter
  siw: Increase DMA max_segment_size parameter
  RDMA/core: Set DMA parameters correctly

 drivers/infiniband/core/device.c      | 16 ++++++++++++++--
 drivers/infiniband/sw/rxe/rxe_verbs.c |  3 +++
 drivers/infiniband/sw/rxe/rxe_verbs.h |  1 +
 drivers/infiniband/sw/siw/siw.h       |  1 +
 drivers/infiniband/sw/siw/siw_main.c  |  3 +++
 include/rdma/ib_verbs.h               |  4 +---
 6 files changed, 23 insertions(+), 5 deletions(-)

[PATCH v2 1/4] RDMA/core: Fix ib_dma_max_seg_size()

[Bart Van Assche]

If dev->dma_device->params == NULL then the maximum DMA segment size is
64 KB. See also the dma_get_max_seg_size() implementation. This patch
fixes the following kernel warning:

DMA-API: infiniband rxe0: mapping sg segment longer than device claims to support [len=126976] [max=65536]
WARNING: CPU: 4 PID: 4848 at kernel/dma/debug.c:1220 debug_dma_map_sg+0x3d9/0x450
RIP: 0010:debug_dma_map_sg+0x3d9/0x450
Call Trace:
 srp_queuecommand+0x626/0x18d0 [ib_srp]
 scsi_queue_rq+0xd02/0x13e0 [scsi_mod]
 __blk_mq_try_issue_directly+0x2b3/0x3f0
 blk_mq_request_issue_directly+0xac/0xf0
 blk_insert_cloned_request+0xdf/0x170
 dm_mq_queue_rq+0x43d/0x830 [dm_mod]
 __blk_mq_try_issue_directly+0x2b3/0x3f0
 blk_mq_request_issue_directly+0xac/0xf0
 blk_mq_try_issue_list_directly+0xb8/0x170
 blk_mq_sched_insert_requests+0x23c/0x3b0
 blk_mq_flush_plug_list+0x529/0x730
 blk_flush_plug_list+0x21f/0x260
 blk_mq_make_request+0x56b/0xf20
 generic_make_request+0x196/0x660
 submit_bio+0xae/0x290
 blkdev_direct_IO+0x822/0x900
 generic_file_direct_write+0x110/0x200
 __generic_file_write_iter+0x124/0x2a0
 blkdev_write_iter+0x168/0x270
 aio_write+0x1c4/0x310
 io_submit_one+0x971/0x1390
 __x64_sys_io_submit+0x12a/0x390
 do_syscall_64+0x6f/0x2e0
 entry_SYSCALL_64_after_hwframe+0x49/0xbe

Cc: Christoph Hellwig <hch@lst.de>
Cc: <stable@vger.kernel.org>
Fixes: 0b5cb3300ae5 ("RDMA/srp: Increase max_segment_size")
Signed-off-by: Bart Van Assche <bvanassche@acm.org>
---
 include/rdma/ib_verbs.h | 4 +---
 1 file changed, 1 insertion(+), 3 deletions(-)

diff --git a/include/rdma/ib_verbs.h b/include/rdma/ib_verbs.h
index cca9985b4cbc..0626b62ed107 100644
--- a/include/rdma/ib_verbs.h
+++ b/include/rdma/ib_verbs.h
@@ -4057,9 +4057,7 @@ static inline void ib_dma_unmap_sg_attrs(struct ib_device *dev,
  */
 static inline unsigned int ib_dma_max_seg_size(struct ib_device *dev)
 {
-	struct device_dma_parameters *p = dev->dma_device->dma_parms;
-
-	return p ? p->max_segment_size : UINT_MAX;
+	return dma_get_max_seg_size(dev->dma_device);
 }
 
 /**
-- 
2.24.0.rc0.303.g954a862665-goog

[PATCH v2 2/4] rdma_rxe: Increase DMA max_segment_size parameter

[Bart Van Assche]

Increase the DMA max_segment_size parameter from 64 KB to 2 GB.

Cc: Christoph Hellwig <hch@lst.de>
Signed-off-by: Bart Van Assche <bvanassche@acm.org>
---
 drivers/infiniband/sw/rxe/rxe_verbs.c | 3 +++
 drivers/infiniband/sw/rxe/rxe_verbs.h | 1 +
 2 files changed, 4 insertions(+)

diff --git a/drivers/infiniband/sw/rxe/rxe_verbs.c b/drivers/infiniband/sw/rxe/rxe_verbs.c
index fa47bdcc7f54..9dd4bd7aea92 100644
--- a/drivers/infiniband/sw/rxe/rxe_verbs.c
+++ b/drivers/infiniband/sw/rxe/rxe_verbs.c
@@ -1175,6 +1175,9 @@ int rxe_register_device(struct rxe_dev *rxe, const char *ibdev_name)
 	addrconf_addr_eui48((unsigned char *)&dev->node_guid,
 			    rxe->ndev->dev_addr);
 	dev->dev.dma_ops = &dma_virt_ops;
+	dev->dev.dma_parms = &rxe->dma_parms;
+	rxe->dma_parms = (struct device_dma_parameters)
+		{ .max_segment_size = SZ_2G };
 	dma_coerce_mask_and_coherent(&dev->dev,
 				     dma_get_required_mask(&dev->dev));
 
diff --git a/drivers/infiniband/sw/rxe/rxe_verbs.h b/drivers/infiniband/sw/rxe/rxe_verbs.h
index 5c4b2239129c..95834206c80c 100644
--- a/drivers/infiniband/sw/rxe/rxe_verbs.h
+++ b/drivers/infiniband/sw/rxe/rxe_verbs.h
@@ -384,6 +384,7 @@ struct rxe_port {
 struct rxe_dev {
 	struct ib_device	ib_dev;
 	struct ib_device_attr	attr;
+	struct device_dma_parameters dma_parms;
 	int			max_ucontext;
 	int			max_inline_data;
 	struct mutex	usdev_lock;
-- 
2.24.0.rc0.303.g954a862665-goog

[PATCH v2 3/4] siw: Increase DMA max_segment_size parameter

[Bart Van Assche]

Increase the DMA max_segment_size parameter from 64 KB to 2 GB.

Cc: Christoph Hellwig <hch@lst.de>
Cc: Bernard Metzler <bmt@zurich.ibm.com>
Signed-off-by: Bart Van Assche <bvanassche@acm.org>
---
 drivers/infiniband/sw/siw/siw.h      | 1 +
 drivers/infiniband/sw/siw/siw_main.c | 3 +++
 2 files changed, 4 insertions(+)

diff --git a/drivers/infiniband/sw/siw/siw.h b/drivers/infiniband/sw/siw/siw.h
index dba4535494ab..1ea3ed249e7b 100644
--- a/drivers/infiniband/sw/siw/siw.h
+++ b/drivers/infiniband/sw/siw/siw.h
@@ -70,6 +70,7 @@ struct siw_pd {
 
 struct siw_device {
 	struct ib_device base_dev;
+	struct device_dma_parameters dma_parms;
 	struct net_device *netdev;
 	struct siw_dev_cap attrs;
 
diff --git a/drivers/infiniband/sw/siw/siw_main.c b/drivers/infiniband/sw/siw/siw_main.c
index b8fe43074ad6..48e45a852b51 100644
--- a/drivers/infiniband/sw/siw/siw_main.c
+++ b/drivers/infiniband/sw/siw/siw_main.c
@@ -382,6 +382,9 @@ static struct siw_device *siw_device_create(struct net_device *netdev)
 	base_dev->phys_port_cnt = 1;
 	base_dev->dev.parent = parent;
 	base_dev->dev.dma_ops = &dma_virt_ops;
+	base_dev->dev.dma_parms = &sdev->dma_parms;
+	sdev->dma_parms = (struct device_dma_parameters)
+		{ .max_segment_size = SZ_2G };
 	base_dev->num_comp_vectors = num_possible_cpus();
 
 	ib_set_device_ops(base_dev, &siw_device_ops);
-- 
2.24.0.rc0.303.g954a862665-goog

[PATCH v2 4/4] RDMA/core: Set DMA parameters correctly

[Bart Van Assche]

The dma_set_max_seg_size() call in setup_dma_device() does not have any
effect since device->dev.dma_parms is NULL. Fix this by initializing
device->dev.dma_parms first.

Cc: Michael J. Ruhl <michael.j.ruhl@intel.com>
Cc: Ira Weiny <ira.weiny@intel.com>
Cc: Adit Ranadive <aditr@vmware.com>
Cc: Shiraz Saleem <shiraz.saleem@intel.com>
Cc: Gal Pressman <galpress@amazon.com>
Cc: Selvin Xavier <selvin.xavier@broadcom.com>
Fixes: d10bcf947a3e ("RDMA/umem: Combine contiguous PAGE_SIZE regions in SGEs")
Signed-off-by: Bart Van Assche <bvanassche@acm.org>
---
 drivers/infiniband/core/device.c | 16 ++++++++++++++--
 1 file changed, 14 insertions(+), 2 deletions(-)

diff --git a/drivers/infiniband/core/device.c b/drivers/infiniband/core/device.c
index eb35b663a742..a93c23867fb5 100644
--- a/drivers/infiniband/core/device.c
+++ b/drivers/infiniband/core/device.c
@@ -1196,9 +1196,21 @@ static void setup_dma_device(struct ib_device *device)
 		WARN_ON_ONCE(!parent);
 		device->dma_device = parent;
 	}
-	/* Setup default max segment size for all IB devices */
-	dma_set_max_seg_size(device->dma_device, SZ_2G);
 
+	if (!device->dev.dma_parms) {
+		if (parent) {
+			/*
+			 * The caller did not provide DMA parameters, so
+			 * 'parent' probably represents a PCI device. The PCI
+			 * core sets the maximum segment size to 64
+			 * KB. Increase this parameter to 2 GB.
+			 */
+			device->dev.dma_parms = parent->dma_parms;
+			dma_set_max_seg_size(device->dma_device, SZ_2G);
+		} else {
+			WARN_ON_ONCE(true);
+		}
+	}
 }
 
 /*
-- 
2.24.0.rc0.303.g954a862665-goog

Re: [PATCH v2 0/4] Fixes related to the DMA max_segment_size parameter

[Jason Gunthorpe]

On Fri, Oct 25, 2019 at 03:58:26PM -0700, Bart Van Assche wrote:
> Hi Jason,
> 
> These four patches are what I came up with while analyzing a kernel warning
> triggered by the ib_srp kernel driver. Please consider these patches for
> inclusion in the Linux kernel.
> 
> Thanks,
> 
> Bart.
> 
> Changes compared to v1:
> - Restored the dma_set_max_seg_size() call in setup_dma_device(). Reordered
>   this patch series to keep it bisectable.
> 
> Bart Van Assche (4):
>   RDMA/core: Fix ib_dma_max_seg_size()
>   rdma_rxe: Increase DMA max_segment_size parameter
>   siw: Increase DMA max_segment_size parameter
>   RDMA/core: Set DMA parameters correctly

Applied to for-next, thanks

Jason