* [PATCH v2 1/4] RDMA/core: Fix ib_dma_max_seg_size()
2019-10-25 22:58 [PATCH v2 0/4] Fixes related to the DMA max_segment_size parameter Bart Van Assche
@ 2019-10-25 22:58 ` Bart Van Assche
2019-10-25 22:58 ` [PATCH v2 2/4] rdma_rxe: Increase DMA max_segment_size parameter Bart Van Assche
` (3 subsequent siblings)
4 siblings, 0 replies; 6+ messages in thread
From: Bart Van Assche @ 2019-10-25 22:58 UTC (permalink / raw)
To: Jason Gunthorpe
Cc: Leon Romanovsky, Doug Ledford, linux-rdma, Bart Van Assche,
Christoph Hellwig, stable
If dev->dma_device->params == NULL then the maximum DMA segment size is
64 KB. See also the dma_get_max_seg_size() implementation. This patch
fixes the following kernel warning:
DMA-API: infiniband rxe0: mapping sg segment longer than device claims to support [len=126976] [max=65536]
WARNING: CPU: 4 PID: 4848 at kernel/dma/debug.c:1220 debug_dma_map_sg+0x3d9/0x450
RIP: 0010:debug_dma_map_sg+0x3d9/0x450
Call Trace:
srp_queuecommand+0x626/0x18d0 [ib_srp]
scsi_queue_rq+0xd02/0x13e0 [scsi_mod]
__blk_mq_try_issue_directly+0x2b3/0x3f0
blk_mq_request_issue_directly+0xac/0xf0
blk_insert_cloned_request+0xdf/0x170
dm_mq_queue_rq+0x43d/0x830 [dm_mod]
__blk_mq_try_issue_directly+0x2b3/0x3f0
blk_mq_request_issue_directly+0xac/0xf0
blk_mq_try_issue_list_directly+0xb8/0x170
blk_mq_sched_insert_requests+0x23c/0x3b0
blk_mq_flush_plug_list+0x529/0x730
blk_flush_plug_list+0x21f/0x260
blk_mq_make_request+0x56b/0xf20
generic_make_request+0x196/0x660
submit_bio+0xae/0x290
blkdev_direct_IO+0x822/0x900
generic_file_direct_write+0x110/0x200
__generic_file_write_iter+0x124/0x2a0
blkdev_write_iter+0x168/0x270
aio_write+0x1c4/0x310
io_submit_one+0x971/0x1390
__x64_sys_io_submit+0x12a/0x390
do_syscall_64+0x6f/0x2e0
entry_SYSCALL_64_after_hwframe+0x49/0xbe
Cc: Christoph Hellwig <hch@lst.de>
Cc: <stable@vger.kernel.org>
Fixes: 0b5cb3300ae5 ("RDMA/srp: Increase max_segment_size")
Signed-off-by: Bart Van Assche <bvanassche@acm.org>
---
include/rdma/ib_verbs.h | 4 +---
1 file changed, 1 insertion(+), 3 deletions(-)
diff --git a/include/rdma/ib_verbs.h b/include/rdma/ib_verbs.h
index cca9985b4cbc..0626b62ed107 100644
--- a/include/rdma/ib_verbs.h
+++ b/include/rdma/ib_verbs.h
@@ -4057,9 +4057,7 @@ static inline void ib_dma_unmap_sg_attrs(struct ib_device *dev,
*/
static inline unsigned int ib_dma_max_seg_size(struct ib_device *dev)
{
- struct device_dma_parameters *p = dev->dma_device->dma_parms;
-
- return p ? p->max_segment_size : UINT_MAX;
+ return dma_get_max_seg_size(dev->dma_device);
}
/**
--
2.24.0.rc0.303.g954a862665-goog
^ permalink raw reply related [flat|nested] 6+ messages in thread* [PATCH v2 2/4] rdma_rxe: Increase DMA max_segment_size parameter
2019-10-25 22:58 [PATCH v2 0/4] Fixes related to the DMA max_segment_size parameter Bart Van Assche
2019-10-25 22:58 ` [PATCH v2 1/4] RDMA/core: Fix ib_dma_max_seg_size() Bart Van Assche
@ 2019-10-25 22:58 ` Bart Van Assche
2019-10-25 22:58 ` [PATCH v2 3/4] siw: " Bart Van Assche
` (2 subsequent siblings)
4 siblings, 0 replies; 6+ messages in thread
From: Bart Van Assche @ 2019-10-25 22:58 UTC (permalink / raw)
To: Jason Gunthorpe
Cc: Leon Romanovsky, Doug Ledford, linux-rdma, Bart Van Assche,
Christoph Hellwig
Increase the DMA max_segment_size parameter from 64 KB to 2 GB.
Cc: Christoph Hellwig <hch@lst.de>
Signed-off-by: Bart Van Assche <bvanassche@acm.org>
---
drivers/infiniband/sw/rxe/rxe_verbs.c | 3 +++
drivers/infiniband/sw/rxe/rxe_verbs.h | 1 +
2 files changed, 4 insertions(+)
diff --git a/drivers/infiniband/sw/rxe/rxe_verbs.c b/drivers/infiniband/sw/rxe/rxe_verbs.c
index fa47bdcc7f54..9dd4bd7aea92 100644
--- a/drivers/infiniband/sw/rxe/rxe_verbs.c
+++ b/drivers/infiniband/sw/rxe/rxe_verbs.c
@@ -1175,6 +1175,9 @@ int rxe_register_device(struct rxe_dev *rxe, const char *ibdev_name)
addrconf_addr_eui48((unsigned char *)&dev->node_guid,
rxe->ndev->dev_addr);
dev->dev.dma_ops = &dma_virt_ops;
+ dev->dev.dma_parms = &rxe->dma_parms;
+ rxe->dma_parms = (struct device_dma_parameters)
+ { .max_segment_size = SZ_2G };
dma_coerce_mask_and_coherent(&dev->dev,
dma_get_required_mask(&dev->dev));
diff --git a/drivers/infiniband/sw/rxe/rxe_verbs.h b/drivers/infiniband/sw/rxe/rxe_verbs.h
index 5c4b2239129c..95834206c80c 100644
--- a/drivers/infiniband/sw/rxe/rxe_verbs.h
+++ b/drivers/infiniband/sw/rxe/rxe_verbs.h
@@ -384,6 +384,7 @@ struct rxe_port {
struct rxe_dev {
struct ib_device ib_dev;
struct ib_device_attr attr;
+ struct device_dma_parameters dma_parms;
int max_ucontext;
int max_inline_data;
struct mutex usdev_lock;
--
2.24.0.rc0.303.g954a862665-goog
^ permalink raw reply related [flat|nested] 6+ messages in thread* [PATCH v2 3/4] siw: Increase DMA max_segment_size parameter
2019-10-25 22:58 [PATCH v2 0/4] Fixes related to the DMA max_segment_size parameter Bart Van Assche
2019-10-25 22:58 ` [PATCH v2 1/4] RDMA/core: Fix ib_dma_max_seg_size() Bart Van Assche
2019-10-25 22:58 ` [PATCH v2 2/4] rdma_rxe: Increase DMA max_segment_size parameter Bart Van Assche
@ 2019-10-25 22:58 ` Bart Van Assche
2019-10-25 22:58 ` [PATCH v2 4/4] RDMA/core: Set DMA parameters correctly Bart Van Assche
2019-10-28 17:58 ` [PATCH v2 0/4] Fixes related to the DMA max_segment_size parameter Jason Gunthorpe
4 siblings, 0 replies; 6+ messages in thread
From: Bart Van Assche @ 2019-10-25 22:58 UTC (permalink / raw)
To: Jason Gunthorpe
Cc: Leon Romanovsky, Doug Ledford, linux-rdma, Bart Van Assche,
Christoph Hellwig, Bernard Metzler
Increase the DMA max_segment_size parameter from 64 KB to 2 GB.
Cc: Christoph Hellwig <hch@lst.de>
Cc: Bernard Metzler <bmt@zurich.ibm.com>
Signed-off-by: Bart Van Assche <bvanassche@acm.org>
---
drivers/infiniband/sw/siw/siw.h | 1 +
drivers/infiniband/sw/siw/siw_main.c | 3 +++
2 files changed, 4 insertions(+)
diff --git a/drivers/infiniband/sw/siw/siw.h b/drivers/infiniband/sw/siw/siw.h
index dba4535494ab..1ea3ed249e7b 100644
--- a/drivers/infiniband/sw/siw/siw.h
+++ b/drivers/infiniband/sw/siw/siw.h
@@ -70,6 +70,7 @@ struct siw_pd {
struct siw_device {
struct ib_device base_dev;
+ struct device_dma_parameters dma_parms;
struct net_device *netdev;
struct siw_dev_cap attrs;
diff --git a/drivers/infiniband/sw/siw/siw_main.c b/drivers/infiniband/sw/siw/siw_main.c
index b8fe43074ad6..48e45a852b51 100644
--- a/drivers/infiniband/sw/siw/siw_main.c
+++ b/drivers/infiniband/sw/siw/siw_main.c
@@ -382,6 +382,9 @@ static struct siw_device *siw_device_create(struct net_device *netdev)
base_dev->phys_port_cnt = 1;
base_dev->dev.parent = parent;
base_dev->dev.dma_ops = &dma_virt_ops;
+ base_dev->dev.dma_parms = &sdev->dma_parms;
+ sdev->dma_parms = (struct device_dma_parameters)
+ { .max_segment_size = SZ_2G };
base_dev->num_comp_vectors = num_possible_cpus();
ib_set_device_ops(base_dev, &siw_device_ops);
--
2.24.0.rc0.303.g954a862665-goog
^ permalink raw reply related [flat|nested] 6+ messages in thread* [PATCH v2 4/4] RDMA/core: Set DMA parameters correctly
2019-10-25 22:58 [PATCH v2 0/4] Fixes related to the DMA max_segment_size parameter Bart Van Assche
` (2 preceding siblings ...)
2019-10-25 22:58 ` [PATCH v2 3/4] siw: " Bart Van Assche
@ 2019-10-25 22:58 ` Bart Van Assche
2019-10-28 17:58 ` [PATCH v2 0/4] Fixes related to the DMA max_segment_size parameter Jason Gunthorpe
4 siblings, 0 replies; 6+ messages in thread
From: Bart Van Assche @ 2019-10-25 22:58 UTC (permalink / raw)
To: Jason Gunthorpe
Cc: Leon Romanovsky, Doug Ledford, linux-rdma, Bart Van Assche,
Michael J . Ruhl, Ira Weiny, Adit Ranadive, Shiraz Saleem,
Gal Pressman, Selvin Xavier
The dma_set_max_seg_size() call in setup_dma_device() does not have any
effect since device->dev.dma_parms is NULL. Fix this by initializing
device->dev.dma_parms first.
Cc: Michael J. Ruhl <michael.j.ruhl@intel.com>
Cc: Ira Weiny <ira.weiny@intel.com>
Cc: Adit Ranadive <aditr@vmware.com>
Cc: Shiraz Saleem <shiraz.saleem@intel.com>
Cc: Gal Pressman <galpress@amazon.com>
Cc: Selvin Xavier <selvin.xavier@broadcom.com>
Fixes: d10bcf947a3e ("RDMA/umem: Combine contiguous PAGE_SIZE regions in SGEs")
Signed-off-by: Bart Van Assche <bvanassche@acm.org>
---
drivers/infiniband/core/device.c | 16 ++++++++++++++--
1 file changed, 14 insertions(+), 2 deletions(-)
diff --git a/drivers/infiniband/core/device.c b/drivers/infiniband/core/device.c
index eb35b663a742..a93c23867fb5 100644
--- a/drivers/infiniband/core/device.c
+++ b/drivers/infiniband/core/device.c
@@ -1196,9 +1196,21 @@ static void setup_dma_device(struct ib_device *device)
WARN_ON_ONCE(!parent);
device->dma_device = parent;
}
- /* Setup default max segment size for all IB devices */
- dma_set_max_seg_size(device->dma_device, SZ_2G);
+ if (!device->dev.dma_parms) {
+ if (parent) {
+ /*
+ * The caller did not provide DMA parameters, so
+ * 'parent' probably represents a PCI device. The PCI
+ * core sets the maximum segment size to 64
+ * KB. Increase this parameter to 2 GB.
+ */
+ device->dev.dma_parms = parent->dma_parms;
+ dma_set_max_seg_size(device->dma_device, SZ_2G);
+ } else {
+ WARN_ON_ONCE(true);
+ }
+ }
}
/*
--
2.24.0.rc0.303.g954a862665-goog
^ permalink raw reply related [flat|nested] 6+ messages in thread* Re: [PATCH v2 0/4] Fixes related to the DMA max_segment_size parameter
2019-10-25 22:58 [PATCH v2 0/4] Fixes related to the DMA max_segment_size parameter Bart Van Assche
` (3 preceding siblings ...)
2019-10-25 22:58 ` [PATCH v2 4/4] RDMA/core: Set DMA parameters correctly Bart Van Assche
@ 2019-10-28 17:58 ` Jason Gunthorpe
4 siblings, 0 replies; 6+ messages in thread
From: Jason Gunthorpe @ 2019-10-28 17:58 UTC (permalink / raw)
To: Bart Van Assche; +Cc: Leon Romanovsky, Doug Ledford, linux-rdma
On Fri, Oct 25, 2019 at 03:58:26PM -0700, Bart Van Assche wrote:
> Hi Jason,
>
> These four patches are what I came up with while analyzing a kernel warning
> triggered by the ib_srp kernel driver. Please consider these patches for
> inclusion in the Linux kernel.
>
> Thanks,
>
> Bart.
>
> Changes compared to v1:
> - Restored the dma_set_max_seg_size() call in setup_dma_device(). Reordered
> this patch series to keep it bisectable.
>
> Bart Van Assche (4):
> RDMA/core: Fix ib_dma_max_seg_size()
> rdma_rxe: Increase DMA max_segment_size parameter
> siw: Increase DMA max_segment_size parameter
> RDMA/core: Set DMA parameters correctly
Applied to for-next, thanks
Jason
^ permalink raw reply [flat|nested] 6+ messages in thread