[PATCH v2 0/8] IB/srp bug fixes

[PATCH v2 0/8] IB/srp bug fixes

[Bart Van Assche]

Hello Doug,

The patches in this series are the initiator patches I came up with while
testing the SRP initiator and target drivers. Please consider these patches
for inclusion in the upstream kernel.

The changes compared to the previous version of this series are:
- Left out the "Use the IB_DEVICE_SG_GAPS_REG HCA feature if supported" patch.
- Added a patch to stop using memory regions with gaps since Laurence ran
  into an issue with these memory regions that cannot be explained by
  analzying the SRP initiator code only.
- Addressed Leon's comments about patch "Avoid that duplicate responses
  trigger a kernel bug" and "Add support for draining IB_POLL_DIRECT
  completion queues".

The patches in this series are:

Bart Van Assche (8):
  IB/SRP: Avoid using IB_MR_TYPE_SG_GAPS
  IB/srp: Avoid that duplicate responses trigger a kernel bug
  IB/srp: Fix race conditions related to task management
  IB/srp: Document locking conventions
  IB/srp: Make a diagnostic message more informative
  IB/srp: Improve an error path
  IB/core: Add support for draining IB_POLL_DIRECT completion queues
  IB/srp: Drain the send queue before destroying a QP

 drivers/infiniband/core/cq.c        |  4 +-
 drivers/infiniband/core/verbs.c     | 35 ++++++--------
 drivers/infiniband/ulp/srp/ib_srp.c | 92 ++++++++++++++++++++++++-------------
 drivers/infiniband/ulp/srp/ib_srp.h |  1 +
 4 files changed, 77 insertions(+), 55 deletions(-)

Thanks,

Bart.

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[PATCH v2 1/8] IB/SRP: Avoid using IB_MR_TYPE_SG_GAPS

[Bart Van Assche]

Tests have shown that the following error message is reported when
using SG-GAPS registration with an mlx5 adapter:

scsi host1: ib_srp: failed RECV status WR flushed (5) for CQE ffff880bd4270eb0
00000000 00000000 00000000 00000000
00000000 00000000 00000000 00000000
00000000 00000000 00000000 00000000
00000000 0f007806 2500002a ad9fafd1
scsi host1: ib_srp: reconnect succeeded
mlx5_0:dump_cqe:262:(pid 7369): dump error cqe
00000000 00000000 00000000 00000000
00000000 00000000 00000000 00000000
00000000 00000000 00000000 00000000
00000000 0f007806 25000032 00105dd0
scsi host1: ib_srp: failed FAST REG status memory management operation error (6) for CQE ffff880b92860138

Hence avoid using SG-GAPS memory registrations. Additionally,
always configure the blk_queue_virt_boundary() to avoid to trigger
a mapping failure when using adapters that support SG-GAPS (e.g.
mlx5).

Fixes: commit ad8e66b4a801 ("IB/srp: fix mr allocation when the device supports sg gaps")
Fixes: commit 509c5f33f4f6 ("IB/srp: Prevent mapping failures")
Reported-by: Laurence Oberman <loberman-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>
Signed-off-by: Bart Van Assche <bart.vanassche-XdAiOPVOjttBDgjK7y7TUQ@public.gmane.org>
Cc: Israel Rukshin <israelr-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
Cc: Max Gurtovoy <maxg-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
Cc: Leon Romanovsky <leonro-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
Cc: Mark Bloch <markb-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
Cc: Yuval Shaia <yuval.shaia-QHcLZuEGTsvQT0dZR+AlfA@public.gmane.org>
Cc: <stable-u79uwXL29TY76Z2rM5mHXA@public.gmane.org> # 4.7+
---
 drivers/infiniband/ulp/srp/ib_srp.c | 11 +++--------
 1 file changed, 3 insertions(+), 8 deletions(-)

diff --git a/drivers/infiniband/ulp/srp/ib_srp.c b/drivers/infiniband/ulp/srp/ib_srp.c
index 79bf48477ddb..07e3613e4798 100644
--- a/drivers/infiniband/ulp/srp/ib_srp.c
+++ b/drivers/infiniband/ulp/srp/ib_srp.c
@@ -371,7 +371,6 @@ static struct srp_fr_pool *srp_create_fr_pool(struct ib_device *device,
 	struct srp_fr_desc *d;
 	struct ib_mr *mr;
 	int i, ret = -EINVAL;
-	enum ib_mr_type mr_type;
 
 	if (pool_size <= 0)
 		goto err;
@@ -385,13 +384,9 @@ static struct srp_fr_pool *srp_create_fr_pool(struct ib_device *device,
 	spin_lock_init(&pool->lock);
 	INIT_LIST_HEAD(&pool->free_list);
 
-	if (device->attrs.device_cap_flags & IB_DEVICE_SG_GAPS_REG)
-		mr_type = IB_MR_TYPE_SG_GAPS;
-	else
-		mr_type = IB_MR_TYPE_MEM_REG;
-
 	for (i = 0, d = &pool->desc[0]; i < pool->size; i++, d++) {
-		mr = ib_alloc_mr(pd, mr_type, max_page_list_len);
+		mr = ib_alloc_mr(pd, IB_MR_TYPE_MEM_REG,
+				 max_page_list_len);
 		if (IS_ERR(mr)) {
 			ret = PTR_ERR(mr);
 			if (ret == -ENOMEM)
@@ -2666,7 +2661,7 @@ static int srp_slave_alloc(struct scsi_device *sdev)
 	struct srp_device *srp_dev = target->srp_host->srp_dev;
 	struct ib_device *ibdev = srp_dev->dev;
 
-	if (!(ibdev->attrs.device_cap_flags & IB_DEVICE_SG_GAPS_REG))
+	if (true)
 		blk_queue_virt_boundary(sdev->request_queue,
 					~srp_dev->mr_page_mask);
 
-- 
2.11.0

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[PATCH v2 1/8] IB/SRP: Avoid using IB_MR_TYPE_SG_GAPS

[Bart Van Assche]

Tests have shown that the following error message is reported when
using SG-GAPS registration with an mlx5 adapter:

scsi host1: ib_srp: failed RECV status WR flushed (5) for CQE ffff880bd4270eb0
00000000 00000000 00000000 00000000
00000000 00000000 00000000 00000000
00000000 00000000 00000000 00000000
00000000 0f007806 2500002a ad9fafd1
scsi host1: ib_srp: reconnect succeeded
mlx5_0:dump_cqe:262:(pid 7369): dump error cqe
00000000 00000000 00000000 00000000
00000000 00000000 00000000 00000000
00000000 00000000 00000000 00000000
00000000 0f007806 25000032 00105dd0
scsi host1: ib_srp: failed FAST REG status memory management operation error (6) for CQE ffff880b92860138

Hence avoid using SG-GAPS memory registrations. Additionally,
always configure the blk_queue_virt_boundary() to avoid to trigger
a mapping failure when using adapters that support SG-GAPS (e.g.
mlx5).

Fixes: commit ad8e66b4a801 ("IB/srp: fix mr allocation when the device supports sg gaps")
Fixes: commit 509c5f33f4f6 ("IB/srp: Prevent mapping failures")
Reported-by: Laurence Oberman <loberman@redhat.com>
Signed-off-by: Bart Van Assche <bart.vanassche@sandisk.com>
Cc: Israel Rukshin <israelr@mellanox.com>
Cc: Max Gurtovoy <maxg@mellanox.com>
Cc: Leon Romanovsky <leonro@mellanox.com>
Cc: Mark Bloch <markb@mellanox.com>
Cc: Yuval Shaia <yuval.shaia@oracle.com>
Cc: <stable@vger.kernel.org> # 4.7+
---
 drivers/infiniband/ulp/srp/ib_srp.c | 11 +++--------
 1 file changed, 3 insertions(+), 8 deletions(-)

diff --git a/drivers/infiniband/ulp/srp/ib_srp.c b/drivers/infiniband/ulp/srp/ib_srp.c
index 79bf48477ddb..07e3613e4798 100644
--- a/drivers/infiniband/ulp/srp/ib_srp.c
+++ b/drivers/infiniband/ulp/srp/ib_srp.c
@@ -371,7 +371,6 @@ static struct srp_fr_pool *srp_create_fr_pool(struct ib_device *device,
 	struct srp_fr_desc *d;
 	struct ib_mr *mr;
 	int i, ret = -EINVAL;
-	enum ib_mr_type mr_type;
 
 	if (pool_size <= 0)
 		goto err;
@@ -385,13 +384,9 @@ static struct srp_fr_pool *srp_create_fr_pool(struct ib_device *device,
 	spin_lock_init(&pool->lock);
 	INIT_LIST_HEAD(&pool->free_list);
 
-	if (device->attrs.device_cap_flags & IB_DEVICE_SG_GAPS_REG)
-		mr_type = IB_MR_TYPE_SG_GAPS;
-	else
-		mr_type = IB_MR_TYPE_MEM_REG;
-
 	for (i = 0, d = &pool->desc[0]; i < pool->size; i++, d++) {
-		mr = ib_alloc_mr(pd, mr_type, max_page_list_len);
+		mr = ib_alloc_mr(pd, IB_MR_TYPE_MEM_REG,
+				 max_page_list_len);
 		if (IS_ERR(mr)) {
 			ret = PTR_ERR(mr);
 			if (ret == -ENOMEM)
@@ -2666,7 +2661,7 @@ static int srp_slave_alloc(struct scsi_device *sdev)
 	struct srp_device *srp_dev = target->srp_host->srp_dev;
 	struct ib_device *ibdev = srp_dev->dev;
 
-	if (!(ibdev->attrs.device_cap_flags & IB_DEVICE_SG_GAPS_REG))
+	if (true)
 		blk_queue_virt_boundary(sdev->request_queue,
 					~srp_dev->mr_page_mask);
 
-- 
2.11.0

Re: [PATCH v2 1/8] IB/SRP: Avoid using IB_MR_TYPE_SG_GAPS

[Bart Van Assche]

On Tue, 2017-02-14 at 10:56 -0800, Bart Van Assche wrote:
> @@ -2666,7 +2661,7 @@ static int srp_slave_alloc(struct scsi_device *sdev)
>  	struct srp_device *srp_dev = target->srp_host->srp_dev;
>  	struct ib_device *ibdev = srp_dev->dev;
>  
> -	if (!(ibdev->attrs.device_cap_flags & IB_DEVICE_SG_GAPS_REG))
> +	if (true)
>  		blk_queue_virt_boundary(sdev->request_queue,
>  					~srp_dev->mr_page_mask);

Hello Doug,

This patch introduces a new warning when building with W=1, namely that
variable ibdev is not used. The build bot figured this out. Do you want me
to address this?

Thanks,

Bart.--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH v2 1/8] IB/SRP: Avoid using IB_MR_TYPE_SG_GAPS

[Bart Van Assche]

On Tue, 2017-02-14 at 10:56 -0800, Bart Van Assche wrote:
> @@ -2666,7 +2661,7 @@ static int srp_slave_alloc(struct scsi_device *sdev)
>  	struct srp_device *srp_dev = target->srp_host->srp_dev;
>  	struct ib_device *ibdev = srp_dev->dev;
>  
> -	if (!(ibdev->attrs.device_cap_flags & IB_DEVICE_SG_GAPS_REG))
> +	if (true)
>  		blk_queue_virt_boundary(sdev->request_queue,
>  					~srp_dev->mr_page_mask);

Hello Doug,

This patch introduces a new warning when building with W=1, namely that
variable ibdev is not used. The build bot figured this out. Do you want me
to address this?

Thanks,

Bart.

Re: [PATCH v2 1/8] IB/SRP: Avoid using IB_MR_TYPE_SG_GAPS

[Leon Romanovsky]

[-- Attachment #1: Type: text/plain, Size: 3655 bytes --]

On Tue, Feb 14, 2017 at 10:56:29AM -0800, Bart Van Assche wrote:
> Tests have shown that the following error message is reported when
> using SG-GAPS registration with an mlx5 adapter:
>
> scsi host1: ib_srp: failed RECV status WR flushed (5) for CQE ffff880bd4270eb0
> 00000000 00000000 00000000 00000000
> 00000000 00000000 00000000 00000000
> 00000000 00000000 00000000 00000000
> 00000000 0f007806 2500002a ad9fafd1
> scsi host1: ib_srp: reconnect succeeded
> mlx5_0:dump_cqe:262:(pid 7369): dump error cqe
> 00000000 00000000 00000000 00000000
> 00000000 00000000 00000000 00000000
> 00000000 00000000 00000000 00000000
> 00000000 0f007806 25000032 00105dd0
> scsi host1: ib_srp: failed FAST REG status memory management operation error (6) for CQE ffff880b92860138
>
> Hence avoid using SG-GAPS memory registrations. Additionally,
> always configure the blk_queue_virt_boundary() to avoid to trigger
> a mapping failure when using adapters that support SG-GAPS (e.g.
> mlx5).

According to the error dump, we have an issue with max_page_list_len supplied and/or
internal calculations from that value to the UMR byte count.

I assume that iser works with SG_GAPS.

>
> Fixes: commit ad8e66b4a801 ("IB/srp: fix mr allocation when the device supports sg gaps")
> Fixes: commit 509c5f33f4f6 ("IB/srp: Prevent mapping failures")
> Reported-by: Laurence Oberman <loberman-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>
> Signed-off-by: Bart Van Assche <bart.vanassche-XdAiOPVOjttBDgjK7y7TUQ@public.gmane.org>
> Cc: Israel Rukshin <israelr-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
> Cc: Max Gurtovoy <maxg-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
> Cc: Leon Romanovsky <leonro-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
> Cc: Mark Bloch <markb-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
> Cc: Yuval Shaia <yuval.shaia-QHcLZuEGTsvQT0dZR+AlfA@public.gmane.org>
> Cc: <stable-u79uwXL29TY76Z2rM5mHXA@public.gmane.org> # 4.7+
> ---
>  drivers/infiniband/ulp/srp/ib_srp.c | 11 +++--------
>  1 file changed, 3 insertions(+), 8 deletions(-)
>
> diff --git a/drivers/infiniband/ulp/srp/ib_srp.c b/drivers/infiniband/ulp/srp/ib_srp.c
> index 79bf48477ddb..07e3613e4798 100644
> --- a/drivers/infiniband/ulp/srp/ib_srp.c
> +++ b/drivers/infiniband/ulp/srp/ib_srp.c
> @@ -371,7 +371,6 @@ static struct srp_fr_pool *srp_create_fr_pool(struct ib_device *device,
>  	struct srp_fr_desc *d;
>  	struct ib_mr *mr;
>  	int i, ret = -EINVAL;
> -	enum ib_mr_type mr_type;
>
>  	if (pool_size <= 0)
>  		goto err;
> @@ -385,13 +384,9 @@ static struct srp_fr_pool *srp_create_fr_pool(struct ib_device *device,
>  	spin_lock_init(&pool->lock);
>  	INIT_LIST_HEAD(&pool->free_list);
>
> -	if (device->attrs.device_cap_flags & IB_DEVICE_SG_GAPS_REG)
> -		mr_type = IB_MR_TYPE_SG_GAPS;
> -	else
> -		mr_type = IB_MR_TYPE_MEM_REG;
> -
>  	for (i = 0, d = &pool->desc[0]; i < pool->size; i++, d++) {
> -		mr = ib_alloc_mr(pd, mr_type, max_page_list_len);
> +		mr = ib_alloc_mr(pd, IB_MR_TYPE_MEM_REG,
> +				 max_page_list_len);
>  		if (IS_ERR(mr)) {
>  			ret = PTR_ERR(mr);
>  			if (ret == -ENOMEM)
> @@ -2666,7 +2661,7 @@ static int srp_slave_alloc(struct scsi_device *sdev)
>  	struct srp_device *srp_dev = target->srp_host->srp_dev;
>  	struct ib_device *ibdev = srp_dev->dev;
>
> -	if (!(ibdev->attrs.device_cap_flags & IB_DEVICE_SG_GAPS_REG))
> +	if (true)
>  		blk_queue_virt_boundary(sdev->request_queue,
>  					~srp_dev->mr_page_mask);
>
> --
> 2.11.0
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [PATCH v2 1/8] IB/SRP: Avoid using IB_MR_TYPE_SG_GAPS

[Leon Romanovsky]

[-- Attachment #1: Type: text/plain, Size: 3413 bytes --]

On Tue, Feb 14, 2017 at 10:56:29AM -0800, Bart Van Assche wrote:
> Tests have shown that the following error message is reported when
> using SG-GAPS registration with an mlx5 adapter:
>
> scsi host1: ib_srp: failed RECV status WR flushed (5) for CQE ffff880bd4270eb0
> 00000000 00000000 00000000 00000000
> 00000000 00000000 00000000 00000000
> 00000000 00000000 00000000 00000000
> 00000000 0f007806 2500002a ad9fafd1
> scsi host1: ib_srp: reconnect succeeded
> mlx5_0:dump_cqe:262:(pid 7369): dump error cqe
> 00000000 00000000 00000000 00000000
> 00000000 00000000 00000000 00000000
> 00000000 00000000 00000000 00000000
> 00000000 0f007806 25000032 00105dd0
> scsi host1: ib_srp: failed FAST REG status memory management operation error (6) for CQE ffff880b92860138
>
> Hence avoid using SG-GAPS memory registrations. Additionally,
> always configure the blk_queue_virt_boundary() to avoid to trigger
> a mapping failure when using adapters that support SG-GAPS (e.g.
> mlx5).

According to the error dump, we have an issue with max_page_list_len supplied and/or
internal calculations from that value to the UMR byte count.

I assume that iser works with SG_GAPS.

>
> Fixes: commit ad8e66b4a801 ("IB/srp: fix mr allocation when the device supports sg gaps")
> Fixes: commit 509c5f33f4f6 ("IB/srp: Prevent mapping failures")
> Reported-by: Laurence Oberman <loberman@redhat.com>
> Signed-off-by: Bart Van Assche <bart.vanassche@sandisk.com>
> Cc: Israel Rukshin <israelr@mellanox.com>
> Cc: Max Gurtovoy <maxg@mellanox.com>
> Cc: Leon Romanovsky <leonro@mellanox.com>
> Cc: Mark Bloch <markb@mellanox.com>
> Cc: Yuval Shaia <yuval.shaia@oracle.com>
> Cc: <stable@vger.kernel.org> # 4.7+
> ---
>  drivers/infiniband/ulp/srp/ib_srp.c | 11 +++--------
>  1 file changed, 3 insertions(+), 8 deletions(-)
>
> diff --git a/drivers/infiniband/ulp/srp/ib_srp.c b/drivers/infiniband/ulp/srp/ib_srp.c
> index 79bf48477ddb..07e3613e4798 100644
> --- a/drivers/infiniband/ulp/srp/ib_srp.c
> +++ b/drivers/infiniband/ulp/srp/ib_srp.c
> @@ -371,7 +371,6 @@ static struct srp_fr_pool *srp_create_fr_pool(struct ib_device *device,
>  	struct srp_fr_desc *d;
>  	struct ib_mr *mr;
>  	int i, ret = -EINVAL;
> -	enum ib_mr_type mr_type;
>
>  	if (pool_size <= 0)
>  		goto err;
> @@ -385,13 +384,9 @@ static struct srp_fr_pool *srp_create_fr_pool(struct ib_device *device,
>  	spin_lock_init(&pool->lock);
>  	INIT_LIST_HEAD(&pool->free_list);
>
> -	if (device->attrs.device_cap_flags & IB_DEVICE_SG_GAPS_REG)
> -		mr_type = IB_MR_TYPE_SG_GAPS;
> -	else
> -		mr_type = IB_MR_TYPE_MEM_REG;
> -
>  	for (i = 0, d = &pool->desc[0]; i < pool->size; i++, d++) {
> -		mr = ib_alloc_mr(pd, mr_type, max_page_list_len);
> +		mr = ib_alloc_mr(pd, IB_MR_TYPE_MEM_REG,
> +				 max_page_list_len);
>  		if (IS_ERR(mr)) {
>  			ret = PTR_ERR(mr);
>  			if (ret == -ENOMEM)
> @@ -2666,7 +2661,7 @@ static int srp_slave_alloc(struct scsi_device *sdev)
>  	struct srp_device *srp_dev = target->srp_host->srp_dev;
>  	struct ib_device *ibdev = srp_dev->dev;
>
> -	if (!(ibdev->attrs.device_cap_flags & IB_DEVICE_SG_GAPS_REG))
> +	if (true)
>  		blk_queue_virt_boundary(sdev->request_queue,
>  					~srp_dev->mr_page_mask);
>
> --
> 2.11.0
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [PATCH v2 1/8] IB/SRP: Avoid using IB_MR_TYPE_SG_GAPS

[Leon Romanovsky]

[-- Attachment #1: Type: text/plain, Size: 2163 bytes --]

On Wed, Feb 15, 2017 at 09:14:49AM +0200, Leon Romanovsky wrote:
> On Tue, Feb 14, 2017 at 10:56:29AM -0800, Bart Van Assche wrote:
> > Tests have shown that the following error message is reported when
> > using SG-GAPS registration with an mlx5 adapter:
> >
> > scsi host1: ib_srp: failed RECV status WR flushed (5) for CQE ffff880bd4270eb0
> > 00000000 00000000 00000000 00000000
> > 00000000 00000000 00000000 00000000
> > 00000000 00000000 00000000 00000000
> > 00000000 0f007806 2500002a ad9fafd1
> > scsi host1: ib_srp: reconnect succeeded
> > mlx5_0:dump_cqe:262:(pid 7369): dump error cqe
> > 00000000 00000000 00000000 00000000
> > 00000000 00000000 00000000 00000000
> > 00000000 00000000 00000000 00000000
> > 00000000 0f007806 25000032 00105dd0
> > scsi host1: ib_srp: failed FAST REG status memory management operation error (6) for CQE ffff880b92860138
> >
> > Hence avoid using SG-GAPS memory registrations. Additionally,
> > always configure the blk_queue_virt_boundary() to avoid to trigger
> > a mapping failure when using adapters that support SG-GAPS (e.g.
> > mlx5).
>
> According to the error dump, we have an issue with max_page_list_len supplied and/or
> internal calculations from that value to the UMR byte count.

Hi Bart,

Do you mind to try your test on my branch rdma-next [1] with the following
fixup?

diff --git a/drivers/infiniband/hw/mlx5/mr.c b/drivers/infiniband/hw/mlx5/mr.c
index 3c1f483d003f..3e59dce10d5e 100644
--- a/drivers/infiniband/hw/mlx5/mr.c
+++ b/drivers/infiniband/hw/mlx5/mr.c
@@ -1045,8 +1045,9 @@ int mlx5_ib_update_xlt(struct mlx5_ib_mr *mr, u64 idx, int npages,
 	for (pages_mapped = 0;
 	     pages_mapped < pages_to_map && !err;
 	     pages_mapped += pages_iter, idx += pages_iter) {
+		npages = min_t(int, pages_iter, pages_to_map - pages_mapped);
 		dma_sync_single_for_cpu(ddev, dma, size, DMA_TO_DEVICE);
-		npages = populate_xlt(mr, idx, pages_iter, xlt,
+		npages = populate_xlt(mr, idx, npages, xlt,
 				      page_shift, size, flags);

 		dma_sync_single_for_device(ddev, dma, size, DMA_TO_DEVICE);

[1] https://git.kernel.org/cgit/linux/kernel/git/leon/linux-rdma.git/log/?h=rdma-next

Thanks

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [PATCH v2 1/8] IB/SRP: Avoid using IB_MR_TYPE_SG_GAPS

[Leon Romanovsky]

[-- Attachment #1: Type: text/plain, Size: 2163 bytes --]

On Wed, Feb 15, 2017 at 09:14:49AM +0200, Leon Romanovsky wrote:
> On Tue, Feb 14, 2017 at 10:56:29AM -0800, Bart Van Assche wrote:
> > Tests have shown that the following error message is reported when
> > using SG-GAPS registration with an mlx5 adapter:
> >
> > scsi host1: ib_srp: failed RECV status WR flushed (5) for CQE ffff880bd4270eb0
> > 00000000 00000000 00000000 00000000
> > 00000000 00000000 00000000 00000000
> > 00000000 00000000 00000000 00000000
> > 00000000 0f007806 2500002a ad9fafd1
> > scsi host1: ib_srp: reconnect succeeded
> > mlx5_0:dump_cqe:262:(pid 7369): dump error cqe
> > 00000000 00000000 00000000 00000000
> > 00000000 00000000 00000000 00000000
> > 00000000 00000000 00000000 00000000
> > 00000000 0f007806 25000032 00105dd0
> > scsi host1: ib_srp: failed FAST REG status memory management operation error (6) for CQE ffff880b92860138
> >
> > Hence avoid using SG-GAPS memory registrations. Additionally,
> > always configure the blk_queue_virt_boundary() to avoid to trigger
> > a mapping failure when using adapters that support SG-GAPS (e.g.
> > mlx5).
>
> According to the error dump, we have an issue with max_page_list_len supplied and/or
> internal calculations from that value to the UMR byte count.

Hi Bart,

Do you mind to try your test on my branch rdma-next [1] with the following
fixup?

diff --git a/drivers/infiniband/hw/mlx5/mr.c b/drivers/infiniband/hw/mlx5/mr.c
index 3c1f483d003f..3e59dce10d5e 100644
--- a/drivers/infiniband/hw/mlx5/mr.c
+++ b/drivers/infiniband/hw/mlx5/mr.c
@@ -1045,8 +1045,9 @@ int mlx5_ib_update_xlt(struct mlx5_ib_mr *mr, u64 idx, int npages,
 	for (pages_mapped = 0;
 	     pages_mapped < pages_to_map && !err;
 	     pages_mapped += pages_iter, idx += pages_iter) {
+		npages = min_t(int, pages_iter, pages_to_map - pages_mapped);
 		dma_sync_single_for_cpu(ddev, dma, size, DMA_TO_DEVICE);
-		npages = populate_xlt(mr, idx, pages_iter, xlt,
+		npages = populate_xlt(mr, idx, npages, xlt,
 				      page_shift, size, flags);

 		dma_sync_single_for_device(ddev, dma, size, DMA_TO_DEVICE);

[1] https://git.kernel.org/cgit/linux/kernel/git/leon/linux-rdma.git/log/?h=rdma-next

Thanks

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [PATCH v2 1/8] IB/SRP: Avoid using IB_MR_TYPE_SG_GAPS

[Laurence Oberman]

----- Original Message -----
> From: "Leon Romanovsky" <leonro-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
> To: "Bart Van Assche" <bart.vanassche-XdAiOPVOjttBDgjK7y7TUQ@public.gmane.org>, "Max Gurtovoy" <maxg-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
> Cc: "Doug Ledford" <dledford-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>, linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org, "Israel Rukshin" <israelr-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>, "Mark
> Bloch" <markb-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>, "Yuval Shaia" <yuval.shaia-QHcLZuEGTsvQT0dZR+AlfA@public.gmane.org>, "Artemy Kovalyov" <artemyko-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>, "# 4
> . 7+" <stable-u79uwXL29TY76Z2rM5mHXA@public.gmane.org>
> Sent: Wednesday, February 15, 2017 3:19:45 AM
> Subject: Re: [PATCH v2 1/8] IB/SRP: Avoid using IB_MR_TYPE_SG_GAPS
> 
> On Wed, Feb 15, 2017 at 09:14:49AM +0200, Leon Romanovsky wrote:
> > On Tue, Feb 14, 2017 at 10:56:29AM -0800, Bart Van Assche wrote:
> > > Tests have shown that the following error message is reported when
> > > using SG-GAPS registration with an mlx5 adapter:
> > >
> > > scsi host1: ib_srp: failed RECV status WR flushed (5) for CQE
> > > ffff880bd4270eb0
> > > 00000000 00000000 00000000 00000000
> > > 00000000 00000000 00000000 00000000
> > > 00000000 00000000 00000000 00000000
> > > 00000000 0f007806 2500002a ad9fafd1
> > > scsi host1: ib_srp: reconnect succeeded
> > > mlx5_0:dump_cqe:262:(pid 7369): dump error cqe
> > > 00000000 00000000 00000000 00000000
> > > 00000000 00000000 00000000 00000000
> > > 00000000 00000000 00000000 00000000
> > > 00000000 0f007806 25000032 00105dd0
> > > scsi host1: ib_srp: failed FAST REG status memory management operation
> > > error (6) for CQE ffff880b92860138
> > >
> > > Hence avoid using SG-GAPS memory registrations. Additionally,
> > > always configure the blk_queue_virt_boundary() to avoid to trigger
> > > a mapping failure when using adapters that support SG-GAPS (e.g.
> > > mlx5).
> >
> > According to the error dump, we have an issue with max_page_list_len
> > supplied and/or
> > internal calculations from that value to the UMR byte count.
> 
> Hi Bart,
> 
> Do you mind to try your test on my branch rdma-next [1] with the following
> fixup?
> 
> diff --git a/drivers/infiniband/hw/mlx5/mr.c
> b/drivers/infiniband/hw/mlx5/mr.c
> index 3c1f483d003f..3e59dce10d5e 100644
> --- a/drivers/infiniband/hw/mlx5/mr.c
> +++ b/drivers/infiniband/hw/mlx5/mr.c
> @@ -1045,8 +1045,9 @@ int mlx5_ib_update_xlt(struct mlx5_ib_mr *mr, u64 idx,
> int npages,
>  	for (pages_mapped = 0;
>  	     pages_mapped < pages_to_map && !err;
>  	     pages_mapped += pages_iter, idx += pages_iter) {
> +		npages = min_t(int, pages_iter, pages_to_map - pages_mapped);
>  		dma_sync_single_for_cpu(ddev, dma, size, DMA_TO_DEVICE);
> -		npages = populate_xlt(mr, idx, pages_iter, xlt,
> +		npages = populate_xlt(mr, idx, npages, xlt,
>  				      page_shift, size, flags);
> 
>  		dma_sync_single_for_device(ddev, dma, size, DMA_TO_DEVICE);
> 
> [1]
> https://git.kernel.org/cgit/linux/kernel/git/leon/linux-rdma.git/log/?h=rdma-next
> 
> Thanks
> 

Hello Leon
Replied earlier but I dont know if my reply made it.
I will have to test this.

is this repo https://git.kernel.org/cgit/linux/kernel/git/leon/linux-rdma.git/log/?h=rdma-next already patched with the change you want.
If not can I just take the patch and apply to my earlier tree based just on Linus's tree where I reverted the patch.

Thanks
Laurence
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH v2 1/8] IB/SRP: Avoid using IB_MR_TYPE_SG_GAPS

[Laurence Oberman]

----- Original Message -----
> From: "Leon Romanovsky" <leonro@mellanox.com>
> To: "Bart Van Assche" <bart.vanassche@sandisk.com>, "Max Gurtovoy" <maxg@mellanox.com>
> Cc: "Doug Ledford" <dledford@redhat.com>, linux-rdma@vger.kernel.org, "Israel Rukshin" <israelr@mellanox.com>, "Mark
> Bloch" <markb@mellanox.com>, "Yuval Shaia" <yuval.shaia@oracle.com>, "Artemy Kovalyov" <artemyko@mellanox.com>, "# 4
> . 7+" <stable@vger.kernel.org>
> Sent: Wednesday, February 15, 2017 3:19:45 AM
> Subject: Re: [PATCH v2 1/8] IB/SRP: Avoid using IB_MR_TYPE_SG_GAPS
> 
> On Wed, Feb 15, 2017 at 09:14:49AM +0200, Leon Romanovsky wrote:
> > On Tue, Feb 14, 2017 at 10:56:29AM -0800, Bart Van Assche wrote:
> > > Tests have shown that the following error message is reported when
> > > using SG-GAPS registration with an mlx5 adapter:
> > >
> > > scsi host1: ib_srp: failed RECV status WR flushed (5) for CQE
> > > ffff880bd4270eb0
> > > 00000000 00000000 00000000 00000000
> > > 00000000 00000000 00000000 00000000
> > > 00000000 00000000 00000000 00000000
> > > 00000000 0f007806 2500002a ad9fafd1
> > > scsi host1: ib_srp: reconnect succeeded
> > > mlx5_0:dump_cqe:262:(pid 7369): dump error cqe
> > > 00000000 00000000 00000000 00000000
> > > 00000000 00000000 00000000 00000000
> > > 00000000 00000000 00000000 00000000
> > > 00000000 0f007806 25000032 00105dd0
> > > scsi host1: ib_srp: failed FAST REG status memory management operation
> > > error (6) for CQE ffff880b92860138
> > >
> > > Hence avoid using SG-GAPS memory registrations. Additionally,
> > > always configure the blk_queue_virt_boundary() to avoid to trigger
> > > a mapping failure when using adapters that support SG-GAPS (e.g.
> > > mlx5).
> >
> > According to the error dump, we have an issue with max_page_list_len
> > supplied and/or
> > internal calculations from that value to the UMR byte count.
> 
> Hi Bart,
> 
> Do you mind to try your test on my branch rdma-next [1] with the following
> fixup?
> 
> diff --git a/drivers/infiniband/hw/mlx5/mr.c
> b/drivers/infiniband/hw/mlx5/mr.c
> index 3c1f483d003f..3e59dce10d5e 100644
> --- a/drivers/infiniband/hw/mlx5/mr.c
> +++ b/drivers/infiniband/hw/mlx5/mr.c
> @@ -1045,8 +1045,9 @@ int mlx5_ib_update_xlt(struct mlx5_ib_mr *mr, u64 idx,
> int npages,
>  	for (pages_mapped = 0;
>  	     pages_mapped < pages_to_map && !err;
>  	     pages_mapped += pages_iter, idx += pages_iter) {
> +		npages = min_t(int, pages_iter, pages_to_map - pages_mapped);
>  		dma_sync_single_for_cpu(ddev, dma, size, DMA_TO_DEVICE);
> -		npages = populate_xlt(mr, idx, pages_iter, xlt,
> +		npages = populate_xlt(mr, idx, npages, xlt,
>  				      page_shift, size, flags);
> 
>  		dma_sync_single_for_device(ddev, dma, size, DMA_TO_DEVICE);
> 
> [1]
> https://git.kernel.org/cgit/linux/kernel/git/leon/linux-rdma.git/log/?h=rdma-next
> 
> Thanks
> 

Hello Leon
Replied earlier but I dont know if my reply made it.
I will have to test this.

is this repo https://git.kernel.org/cgit/linux/kernel/git/leon/linux-rdma.git/log/?h=rdma-next already patched with the change you want.
If not can I just take the patch and apply to my earlier tree based just on Linus's tree where I reverted the patch.

Thanks
Laurence

Re: [PATCH v2 1/8] IB/SRP: Avoid using IB_MR_TYPE_SG_GAPS

[Laurence Oberman]

----- Original Message -----
> From: "Laurence Oberman" <loberman@redhat.com>
> To: "Leon Romanovsky" <leonro@mellanox.com>
> Cc: "Bart Van Assche" <bart.vanassche@sandisk.com>, "Max Gurtovoy" <maxg@mellanox.com>, "Doug Ledford"
> <dledford@redhat.com>, linux-rdma@vger.kernel.org, "Israel Rukshin" <israelr@mellanox.com>, "Mark Bloch"
> <markb@mellanox.com>, "Yuval Shaia" <yuval.shaia@oracle.com>, "Artemy Kovalyov" <artemyko@mellanox.com>, "# 4 . 7+"
> <stable@vger.kernel.org>
> Sent: Wednesday, February 15, 2017 8:33:35 AM
> Subject: Re: [PATCH v2 1/8] IB/SRP: Avoid using IB_MR_TYPE_SG_GAPS
> 
> 
> 
> ----- Original Message -----
> > From: "Leon Romanovsky" <leonro@mellanox.com>
> > To: "Bart Van Assche" <bart.vanassche@sandisk.com>, "Max Gurtovoy"
> > <maxg@mellanox.com>
> > Cc: "Doug Ledford" <dledford@redhat.com>, linux-rdma@vger.kernel.org,
> > "Israel Rukshin" <israelr@mellanox.com>, "Mark
> > Bloch" <markb@mellanox.com>, "Yuval Shaia" <yuval.shaia@oracle.com>,
> > "Artemy Kovalyov" <artemyko@mellanox.com>, "# 4
> > . 7+" <stable@vger.kernel.org>
> > Sent: Wednesday, February 15, 2017 3:19:45 AM
> > Subject: Re: [PATCH v2 1/8] IB/SRP: Avoid using IB_MR_TYPE_SG_GAPS
> > 
> > On Wed, Feb 15, 2017 at 09:14:49AM +0200, Leon Romanovsky wrote:
> > > On Tue, Feb 14, 2017 at 10:56:29AM -0800, Bart Van Assche wrote:
> > > > Tests have shown that the following error message is reported when
> > > > using SG-GAPS registration with an mlx5 adapter:
> > > >
> > > > scsi host1: ib_srp: failed RECV status WR flushed (5) for CQE
> > > > ffff880bd4270eb0
> > > > 00000000 00000000 00000000 00000000
> > > > 00000000 00000000 00000000 00000000
> > > > 00000000 00000000 00000000 00000000
> > > > 00000000 0f007806 2500002a ad9fafd1
> > > > scsi host1: ib_srp: reconnect succeeded
> > > > mlx5_0:dump_cqe:262:(pid 7369): dump error cqe
> > > > 00000000 00000000 00000000 00000000
> > > > 00000000 00000000 00000000 00000000
> > > > 00000000 00000000 00000000 00000000
> > > > 00000000 0f007806 25000032 00105dd0
> > > > scsi host1: ib_srp: failed FAST REG status memory management operation
> > > > error (6) for CQE ffff880b92860138
> > > >
> > > > Hence avoid using SG-GAPS memory registrations. Additionally,
> > > > always configure the blk_queue_virt_boundary() to avoid to trigger
> > > > a mapping failure when using adapters that support SG-GAPS (e.g.
> > > > mlx5).
> > >
> > > According to the error dump, we have an issue with max_page_list_len
> > > supplied and/or
> > > internal calculations from that value to the UMR byte count.
> > 
> > Hi Bart,
> > 
> > Do you mind to try your test on my branch rdma-next [1] with the following
> > fixup?
> > 
> > diff --git a/drivers/infiniband/hw/mlx5/mr.c
> > b/drivers/infiniband/hw/mlx5/mr.c
> > index 3c1f483d003f..3e59dce10d5e 100644
> > --- a/drivers/infiniband/hw/mlx5/mr.c
> > +++ b/drivers/infiniband/hw/mlx5/mr.c
> > @@ -1045,8 +1045,9 @@ int mlx5_ib_update_xlt(struct mlx5_ib_mr *mr, u64
> > idx,
> > int npages,
> >  	for (pages_mapped = 0;
> >  	     pages_mapped < pages_to_map && !err;
> >  	     pages_mapped += pages_iter, idx += pages_iter) {
> > +		npages = min_t(int, pages_iter, pages_to_map - pages_mapped);
> >  		dma_sync_single_for_cpu(ddev, dma, size, DMA_TO_DEVICE);
> > -		npages = populate_xlt(mr, idx, pages_iter, xlt,
> > +		npages = populate_xlt(mr, idx, npages, xlt,
> >  				      page_shift, size, flags);
> > 
> >  		dma_sync_single_for_device(ddev, dma, size, DMA_TO_DEVICE);
> > 
> > [1]
> > https://git.kernel.org/cgit/linux/kernel/git/leon/linux-rdma.git/log/?h=rdma-next
> > 
> > Thanks
> > 
> 
> Hello Leon
> Replied earlier but I dont know if my reply made it.
> I will have to test this.
> 
> is this repo
> https://git.kernel.org/cgit/linux/kernel/git/leon/linux-rdma.git/log/?h=rdma-next
> already patched with the change you want.
> If not can I just take the patch and apply to my earlier tree based just on
> Linus's tree where I reverted the patch.
> 
> Thanks
> Laurence
> --
> To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 

I guess not, I looked at applying this patch to Linus's tree but the function 
mlx5_ib_update_xlt() is not in there yet.

I will see if I can get your tree staged and test this for you.

Thanks
Laurence

Re: [PATCH v2 1/8] IB/SRP: Avoid using IB_MR_TYPE_SG_GAPS

[Leon Romanovsky]

[-- Attachment #1: Type: text/plain, Size: 5018 bytes --]

On Wed, Feb 15, 2017 at 08:43:38AM -0500, Laurence Oberman wrote:
>
>
> ----- Original Message -----
> > From: "Laurence Oberman" <loberman@redhat.com>
> > To: "Leon Romanovsky" <leonro@mellanox.com>
> > Cc: "Bart Van Assche" <bart.vanassche@sandisk.com>, "Max Gurtovoy" <maxg@mellanox.com>, "Doug Ledford"
> > <dledford@redhat.com>, linux-rdma@vger.kernel.org, "Israel Rukshin" <israelr@mellanox.com>, "Mark Bloch"
> > <markb@mellanox.com>, "Yuval Shaia" <yuval.shaia@oracle.com>, "Artemy Kovalyov" <artemyko@mellanox.com>, "# 4 . 7+"
> > <stable@vger.kernel.org>
> > Sent: Wednesday, February 15, 2017 8:33:35 AM
> > Subject: Re: [PATCH v2 1/8] IB/SRP: Avoid using IB_MR_TYPE_SG_GAPS
> >
> >
> >
> > ----- Original Message -----
> > > From: "Leon Romanovsky" <leonro@mellanox.com>
> > > To: "Bart Van Assche" <bart.vanassche@sandisk.com>, "Max Gurtovoy"
> > > <maxg@mellanox.com>
> > > Cc: "Doug Ledford" <dledford@redhat.com>, linux-rdma@vger.kernel.org,
> > > "Israel Rukshin" <israelr@mellanox.com>, "Mark
> > > Bloch" <markb@mellanox.com>, "Yuval Shaia" <yuval.shaia@oracle.com>,
> > > "Artemy Kovalyov" <artemyko@mellanox.com>, "# 4
> > > . 7+" <stable@vger.kernel.org>
> > > Sent: Wednesday, February 15, 2017 3:19:45 AM
> > > Subject: Re: [PATCH v2 1/8] IB/SRP: Avoid using IB_MR_TYPE_SG_GAPS
> > >
> > > On Wed, Feb 15, 2017 at 09:14:49AM +0200, Leon Romanovsky wrote:
> > > > On Tue, Feb 14, 2017 at 10:56:29AM -0800, Bart Van Assche wrote:
> > > > > Tests have shown that the following error message is reported when
> > > > > using SG-GAPS registration with an mlx5 adapter:
> > > > >
> > > > > scsi host1: ib_srp: failed RECV status WR flushed (5) for CQE
> > > > > ffff880bd4270eb0
> > > > > 00000000 00000000 00000000 00000000
> > > > > 00000000 00000000 00000000 00000000
> > > > > 00000000 00000000 00000000 00000000
> > > > > 00000000 0f007806 2500002a ad9fafd1
> > > > > scsi host1: ib_srp: reconnect succeeded
> > > > > mlx5_0:dump_cqe:262:(pid 7369): dump error cqe
> > > > > 00000000 00000000 00000000 00000000
> > > > > 00000000 00000000 00000000 00000000
> > > > > 00000000 00000000 00000000 00000000
> > > > > 00000000 0f007806 25000032 00105dd0
> > > > > scsi host1: ib_srp: failed FAST REG status memory management operation
> > > > > error (6) for CQE ffff880b92860138
> > > > >
> > > > > Hence avoid using SG-GAPS memory registrations. Additionally,
> > > > > always configure the blk_queue_virt_boundary() to avoid to trigger
> > > > > a mapping failure when using adapters that support SG-GAPS (e.g.
> > > > > mlx5).
> > > >
> > > > According to the error dump, we have an issue with max_page_list_len
> > > > supplied and/or
> > > > internal calculations from that value to the UMR byte count.
> > >
> > > Hi Bart,
> > >
> > > Do you mind to try your test on my branch rdma-next [1] with the following
> > > fixup?
> > >
> > > diff --git a/drivers/infiniband/hw/mlx5/mr.c
> > > b/drivers/infiniband/hw/mlx5/mr.c
> > > index 3c1f483d003f..3e59dce10d5e 100644
> > > --- a/drivers/infiniband/hw/mlx5/mr.c
> > > +++ b/drivers/infiniband/hw/mlx5/mr.c
> > > @@ -1045,8 +1045,9 @@ int mlx5_ib_update_xlt(struct mlx5_ib_mr *mr, u64
> > > idx,
> > > int npages,
> > >  	for (pages_mapped = 0;
> > >  	     pages_mapped < pages_to_map && !err;
> > >  	     pages_mapped += pages_iter, idx += pages_iter) {
> > > +		npages = min_t(int, pages_iter, pages_to_map - pages_mapped);
> > >  		dma_sync_single_for_cpu(ddev, dma, size, DMA_TO_DEVICE);
> > > -		npages = populate_xlt(mr, idx, pages_iter, xlt,
> > > +		npages = populate_xlt(mr, idx, npages, xlt,
> > >  				      page_shift, size, flags);
> > >
> > >  		dma_sync_single_for_device(ddev, dma, size, DMA_TO_DEVICE);
> > >
> > > [1]
> > > https://git.kernel.org/cgit/linux/kernel/git/leon/linux-rdma.git/log/?h=rdma-next
> > >
> > > Thanks
> > >
> >
> > Hello Leon
> > Replied earlier but I dont know if my reply made it.
> > I will have to test this.
> >
> > is this repo
> > https://git.kernel.org/cgit/linux/kernel/git/leon/linux-rdma.git/log/?h=rdma-next
> > already patched with the change you want.
> > If not can I just take the patch and apply to my earlier tree based just on
> > Linus's tree where I reverted the patch.
> >
> > Thanks
> > Laurence
> > --
> > To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
> > the body of a message to majordomo@vger.kernel.org
> > More majordomo info at  http://vger.kernel.org/majordomo-info.html
> >
>
> I guess not, I looked at applying this patch to Linus's tree but the function
> mlx5_ib_update_xlt() is not in there yet.

This is why I asked to use my tree, there is a chance that wrong
calculation was before we introduced mlx5_ib_update_xlt() function.

>
> I will see if I can get your tree staged and test this for you.

Thanks a lot.

>
> Thanks
> Laurence
> --
> To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [PATCH v2 1/8] IB/SRP: Avoid using IB_MR_TYPE_SG_GAPS

[Leon Romanovsky]

[-- Attachment #1: Type: text/plain, Size: 5018 bytes --]

On Wed, Feb 15, 2017 at 08:43:38AM -0500, Laurence Oberman wrote:
>
>
> ----- Original Message -----
> > From: "Laurence Oberman" <loberman@redhat.com>
> > To: "Leon Romanovsky" <leonro@mellanox.com>
> > Cc: "Bart Van Assche" <bart.vanassche@sandisk.com>, "Max Gurtovoy" <maxg@mellanox.com>, "Doug Ledford"
> > <dledford@redhat.com>, linux-rdma@vger.kernel.org, "Israel Rukshin" <israelr@mellanox.com>, "Mark Bloch"
> > <markb@mellanox.com>, "Yuval Shaia" <yuval.shaia@oracle.com>, "Artemy Kovalyov" <artemyko@mellanox.com>, "# 4 . 7+"
> > <stable@vger.kernel.org>
> > Sent: Wednesday, February 15, 2017 8:33:35 AM
> > Subject: Re: [PATCH v2 1/8] IB/SRP: Avoid using IB_MR_TYPE_SG_GAPS
> >
> >
> >
> > ----- Original Message -----
> > > From: "Leon Romanovsky" <leonro@mellanox.com>
> > > To: "Bart Van Assche" <bart.vanassche@sandisk.com>, "Max Gurtovoy"
> > > <maxg@mellanox.com>
> > > Cc: "Doug Ledford" <dledford@redhat.com>, linux-rdma@vger.kernel.org,
> > > "Israel Rukshin" <israelr@mellanox.com>, "Mark
> > > Bloch" <markb@mellanox.com>, "Yuval Shaia" <yuval.shaia@oracle.com>,
> > > "Artemy Kovalyov" <artemyko@mellanox.com>, "# 4
> > > . 7+" <stable@vger.kernel.org>
> > > Sent: Wednesday, February 15, 2017 3:19:45 AM
> > > Subject: Re: [PATCH v2 1/8] IB/SRP: Avoid using IB_MR_TYPE_SG_GAPS
> > >
> > > On Wed, Feb 15, 2017 at 09:14:49AM +0200, Leon Romanovsky wrote:
> > > > On Tue, Feb 14, 2017 at 10:56:29AM -0800, Bart Van Assche wrote:
> > > > > Tests have shown that the following error message is reported when
> > > > > using SG-GAPS registration with an mlx5 adapter:
> > > > >
> > > > > scsi host1: ib_srp: failed RECV status WR flushed (5) for CQE
> > > > > ffff880bd4270eb0
> > > > > 00000000 00000000 00000000 00000000
> > > > > 00000000 00000000 00000000 00000000
> > > > > 00000000 00000000 00000000 00000000
> > > > > 00000000 0f007806 2500002a ad9fafd1
> > > > > scsi host1: ib_srp: reconnect succeeded
> > > > > mlx5_0:dump_cqe:262:(pid 7369): dump error cqe
> > > > > 00000000 00000000 00000000 00000000
> > > > > 00000000 00000000 00000000 00000000
> > > > > 00000000 00000000 00000000 00000000
> > > > > 00000000 0f007806 25000032 00105dd0
> > > > > scsi host1: ib_srp: failed FAST REG status memory management operation
> > > > > error (6) for CQE ffff880b92860138
> > > > >
> > > > > Hence avoid using SG-GAPS memory registrations. Additionally,
> > > > > always configure the blk_queue_virt_boundary() to avoid to trigger
> > > > > a mapping failure when using adapters that support SG-GAPS (e.g.
> > > > > mlx5).
> > > >
> > > > According to the error dump, we have an issue with max_page_list_len
> > > > supplied and/or
> > > > internal calculations from that value to the UMR byte count.
> > >
> > > Hi Bart,
> > >
> > > Do you mind to try your test on my branch rdma-next [1] with the following
> > > fixup?
> > >
> > > diff --git a/drivers/infiniband/hw/mlx5/mr.c
> > > b/drivers/infiniband/hw/mlx5/mr.c
> > > index 3c1f483d003f..3e59dce10d5e 100644
> > > --- a/drivers/infiniband/hw/mlx5/mr.c
> > > +++ b/drivers/infiniband/hw/mlx5/mr.c
> > > @@ -1045,8 +1045,9 @@ int mlx5_ib_update_xlt(struct mlx5_ib_mr *mr, u64
> > > idx,
> > > int npages,
> > >  	for (pages_mapped = 0;
> > >  	     pages_mapped < pages_to_map && !err;
> > >  	     pages_mapped += pages_iter, idx += pages_iter) {
> > > +		npages = min_t(int, pages_iter, pages_to_map - pages_mapped);
> > >  		dma_sync_single_for_cpu(ddev, dma, size, DMA_TO_DEVICE);
> > > -		npages = populate_xlt(mr, idx, pages_iter, xlt,
> > > +		npages = populate_xlt(mr, idx, npages, xlt,
> > >  				      page_shift, size, flags);
> > >
> > >  		dma_sync_single_for_device(ddev, dma, size, DMA_TO_DEVICE);
> > >
> > > [1]
> > > https://git.kernel.org/cgit/linux/kernel/git/leon/linux-rdma.git/log/?h=rdma-next
> > >
> > > Thanks
> > >
> >
> > Hello Leon
> > Replied earlier but I dont know if my reply made it.
> > I will have to test this.
> >
> > is this repo
> > https://git.kernel.org/cgit/linux/kernel/git/leon/linux-rdma.git/log/?h=rdma-next
> > already patched with the change you want.
> > If not can I just take the patch and apply to my earlier tree based just on
> > Linus's tree where I reverted the patch.
> >
> > Thanks
> > Laurence
> > --
> > To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
> > the body of a message to majordomo@vger.kernel.org
> > More majordomo info at  http://vger.kernel.org/majordomo-info.html
> >
>
> I guess not, I looked at applying this patch to Linus's tree but the function
> mlx5_ib_update_xlt() is not in there yet.

This is why I asked to use my tree, there is a chance that wrong
calculation was before we introduced mlx5_ib_update_xlt() function.

>
> I will see if I can get your tree staged and test this for you.

Thanks a lot.

>
> Thanks
> Laurence
> --
> To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [PATCH v2 1/8] IB/SRP: Avoid using IB_MR_TYPE_SG_GAPS

[Laurence Oberman]

----- Original Message -----
> From: "Leon Romanovsky" <leonro@mellanox.com>
> To: "Laurence Oberman" <loberman@redhat.com>
> Cc: "Bart Van Assche" <bart.vanassche@sandisk.com>, "Max Gurtovoy" <maxg@mellanox.com>, "Doug Ledford"
> <dledford@redhat.com>, linux-rdma@vger.kernel.org, "Israel Rukshin" <israelr@mellanox.com>, "Mark Bloch"
> <markb@mellanox.com>, "Yuval Shaia" <yuval.shaia@oracle.com>, "Artemy Kovalyov" <artemyko@mellanox.com>, "# 4 . 7+"
> <stable@vger.kernel.org>
> Sent: Wednesday, February 15, 2017 8:47:07 AM
> Subject: Re: [PATCH v2 1/8] IB/SRP: Avoid using IB_MR_TYPE_SG_GAPS
> 
> On Wed, Feb 15, 2017 at 08:43:38AM -0500, Laurence Oberman wrote:
> >
> >
> > ----- Original Message -----
> > > From: "Laurence Oberman" <loberman@redhat.com>
> > > To: "Leon Romanovsky" <leonro@mellanox.com>
> > > Cc: "Bart Van Assche" <bart.vanassche@sandisk.com>, "Max Gurtovoy"
> > > <maxg@mellanox.com>, "Doug Ledford"
> > > <dledford@redhat.com>, linux-rdma@vger.kernel.org, "Israel Rukshin"
> > > <israelr@mellanox.com>, "Mark Bloch"
> > > <markb@mellanox.com>, "Yuval Shaia" <yuval.shaia@oracle.com>, "Artemy
> > > Kovalyov" <artemyko@mellanox.com>, "# 4 . 7+"
> > > <stable@vger.kernel.org>
> > > Sent: Wednesday, February 15, 2017 8:33:35 AM
> > > Subject: Re: [PATCH v2 1/8] IB/SRP: Avoid using IB_MR_TYPE_SG_GAPS
> > >
> > >
> > >
> > > ----- Original Message -----
> > > > From: "Leon Romanovsky" <leonro@mellanox.com>
> > > > To: "Bart Van Assche" <bart.vanassche@sandisk.com>, "Max Gurtovoy"
> > > > <maxg@mellanox.com>
> > > > Cc: "Doug Ledford" <dledford@redhat.com>, linux-rdma@vger.kernel.org,
> > > > "Israel Rukshin" <israelr@mellanox.com>, "Mark
> > > > Bloch" <markb@mellanox.com>, "Yuval Shaia" <yuval.shaia@oracle.com>,
> > > > "Artemy Kovalyov" <artemyko@mellanox.com>, "# 4
> > > > . 7+" <stable@vger.kernel.org>
> > > > Sent: Wednesday, February 15, 2017 3:19:45 AM
> > > > Subject: Re: [PATCH v2 1/8] IB/SRP: Avoid using IB_MR_TYPE_SG_GAPS
> > > >
> > > > On Wed, Feb 15, 2017 at 09:14:49AM +0200, Leon Romanovsky wrote:
> > > > > On Tue, Feb 14, 2017 at 10:56:29AM -0800, Bart Van Assche wrote:
> > > > > > Tests have shown that the following error message is reported when
> > > > > > using SG-GAPS registration with an mlx5 adapter:
> > > > > >
> > > > > > scsi host1: ib_srp: failed RECV status WR flushed (5) for CQE
> > > > > > ffff880bd4270eb0
> > > > > > 00000000 00000000 00000000 00000000
> > > > > > 00000000 00000000 00000000 00000000
> > > > > > 00000000 00000000 00000000 00000000
> > > > > > 00000000 0f007806 2500002a ad9fafd1
> > > > > > scsi host1: ib_srp: reconnect succeeded
> > > > > > mlx5_0:dump_cqe:262:(pid 7369): dump error cqe
> > > > > > 00000000 00000000 00000000 00000000
> > > > > > 00000000 00000000 00000000 00000000
> > > > > > 00000000 00000000 00000000 00000000
> > > > > > 00000000 0f007806 25000032 00105dd0
> > > > > > scsi host1: ib_srp: failed FAST REG status memory management
> > > > > > operation
> > > > > > error (6) for CQE ffff880b92860138
> > > > > >
> > > > > > Hence avoid using SG-GAPS memory registrations. Additionally,
> > > > > > always configure the blk_queue_virt_boundary() to avoid to trigger
> > > > > > a mapping failure when using adapters that support SG-GAPS (e.g.
> > > > > > mlx5).
> > > > >
> > > > > According to the error dump, we have an issue with max_page_list_len
> > > > > supplied and/or
> > > > > internal calculations from that value to the UMR byte count.
> > > >
> > > > Hi Bart,
> > > >
> > > > Do you mind to try your test on my branch rdma-next [1] with the
> > > > following
> > > > fixup?
> > > >
> > > > diff --git a/drivers/infiniband/hw/mlx5/mr.c
> > > > b/drivers/infiniband/hw/mlx5/mr.c
> > > > index 3c1f483d003f..3e59dce10d5e 100644
> > > > --- a/drivers/infiniband/hw/mlx5/mr.c
> > > > +++ b/drivers/infiniband/hw/mlx5/mr.c
> > > > @@ -1045,8 +1045,9 @@ int mlx5_ib_update_xlt(struct mlx5_ib_mr *mr, u64
> > > > idx,
> > > > int npages,
> > > >  	for (pages_mapped = 0;
> > > >  	     pages_mapped < pages_to_map && !err;
> > > >  	     pages_mapped += pages_iter, idx += pages_iter) {
> > > > +		npages = min_t(int, pages_iter, pages_to_map - pages_mapped);
> > > >  		dma_sync_single_for_cpu(ddev, dma, size, DMA_TO_DEVICE);
> > > > -		npages = populate_xlt(mr, idx, pages_iter, xlt,
> > > > +		npages = populate_xlt(mr, idx, npages, xlt,
> > > >  				      page_shift, size, flags);
> > > >
> > > >  		dma_sync_single_for_device(ddev, dma, size, DMA_TO_DEVICE);
> > > >
> > > > [1]
> > > > https://git.kernel.org/cgit/linux/kernel/git/leon/linux-rdma.git/log/?h=rdma-next
> > > >
> > > > Thanks
> > > >
> > >
> > > Hello Leon
> > > Replied earlier but I dont know if my reply made it.
> > > I will have to test this.
> > >
> > > is this repo
> > > https://git.kernel.org/cgit/linux/kernel/git/leon/linux-rdma.git/log/?h=rdma-next
> > > already patched with the change you want.
> > > If not can I just take the patch and apply to my earlier tree based just
> > > on
> > > Linus's tree where I reverted the patch.
> > >
> > > Thanks
> > > Laurence
> > > --
> > > To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
> > > the body of a message to majordomo@vger.kernel.org
> > > More majordomo info at  http://vger.kernel.org/majordomo-info.html
> > >
> >
> > I guess not, I looked at applying this patch to Linus's tree but the
> > function
> > mlx5_ib_update_xlt() is not in there yet.
> 
> This is why I asked to use my tree, there is a chance that wrong
> calculation was before we introduced mlx5_ib_update_xlt() function.
> 
> >
> > I will see if I can get your tree staged and test this for you.
> 
> Thanks a lot.
> 
> >
> > Thanks
> > Laurence
> > --
> > To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
> > the body of a message to majordomo@vger.kernel.org
> > More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 

Hello Leon
I pulled your tree, ran git checkout rdma-next
I applied the patch manually, built the kernel and started the tests.

Pretty soon I ran into

ibclient login: [  132.640142] mlx5_1:dump_cqe:262:(pid 11417): dump error cqe
[  132.640185] scsi host2: ib_srp: failed RECV status WR flushed (5) for CQE ffff880bd86a97b0
[  132.718106] 00000000 00000000 00000000 00000000
[  132.743616] 00000000 00000000 00000000 00000000
[  132.767790] 00000000 00000000 00000000 00000000
[  132.793626] 00000000 0f007806 2500002a 8ad015d1
[  136.181512] systemd-readahead[701]: open(/var/tmp/dracut.ZR8CYG/initramfs/usr/bin/loginctl) failed: Too many levels of symbolic links
[  136.250779] systemd-readahead[701]: open(/var/tmp/dracut.ZR8CYG/initramfs/usr/lib/systemd/system/dracut-emergency.service) failed: Too many levels of symbolic links
[  147.791028] scsi host2: ib_srp: reconnect succeeded
[  147.827012] scsi host2: ib_srp: failed RECV status WR flushed (5) for CQE ffff880bf5e71330
[  162.908764] scsi host2: ib_srp: reconnect succeeded
[  162.944244] scsi host2: ib_srp: failed RECV status WR flushed (5) for CQE ffff880bf5e71330
[  166.409523] scsi host1: ib_srp: failed RECV status WR flushed (5) for CQE ffff8817f23ec570
[  177.292496] scsi host2: ib_srp: reconnect succeeded
[  177.334396] mlx5_1:dump_cqe:262:(pid 0): dump error cqe
[  177.363793] 00000000 00000000 00000000 00000000
[  177.388434] 00000000 00000000 00000000 00000000
[  177.413918] 00000000 00000000 00000000 00000000
[  177.438911] 00000000 0f007806 25000042 00102dd0
[  177.465048] scsi host2: ib_srp: failed FAST REG status memory management operation error (6) for CQE ffff880bd7c90138
[  181.386124] scsi host1: ib_srp: reconnect succeeded
[  181.422892] scsi host1: ib_srp: failed RECV status WR flushed (5) for CQE ffff8817f23ec5b0
[  192.827678] fast_io_fail_tmo expired for SRP port-2:1 / host2.
[  193.230036] scsi host2: ib_srp: reconnect succeeded
[  193.329290] mlx5_1:dump_cqe:262:(pid 0): dump error cqe
[  193.356245] 00000000 00000000 00000000 00000000
[  193.379488] 00000000 00000000 00000000 00000000
[  193.404853] 00000000 00000000 00000000 00000000
[  193.429633] 00000000 0f007806 2500004a 006c46d0
[  193.455183] scsi host2: ib_srp: failed FAST REG status memory management operation error (6) for CQE ffff880bd7c906f8
[  196.178111] scsi host1: ib_srp: reconnect succeeded
[  196.215197] scsi host1: ib_srp: failed RECV status WR flushed (5) for CQE ffff8817f23ec5b0
[  208.277396] scsi host2: ib_srp: reconnect succeeded
[  208.318454] mlx5_1:dump_cqe:262:(pid 0): dump error cqe
[  208.349012] 00000000 00000000 00000000 00000000
[  208.374873] 00000000 00000000 00000000 00000000
[  208.401070] 00000000 00000000 00000000 00000000
[  208.426873] 00000000 0f007806 25000052 00103dd0
[  208.452788] scsi host2: ib_srp: failed FAST REG status memory management operation error (6) for CQE ffff880bd7c90138
[  211.066945] scsi host1: ib_srp: reconnect succeeded
[  211.101645] scsi host1: ib_srp: failed RECV status WR flushed (5) for CQE ffff8817f23ec5b0
[  223.296126] scsi host2: ib_srp: reconnect succeeded
[  223.396340] mlx5_1:dump_cqe:262:(pid 0): dump error cqe
[  223.423310] 00000000 00000000 00000000 00000000
[  223.448590] 00000000 00000000 00000000 00000000
[  223.473520] 00000000 00000000 00000000 00000000
[  223.499008] 00000000 0f007806 2500005a 006c56d0
[  223.524787] scsi host2: ib_srp: failed FAST REG status memory management operation error (6) for CQE ffff880bd7c906f8
[  225.463312] scsi host1: ib_srp: reconnect succeeded
[  225.499915] scsi host1: ib_srp: failed RECV status WR flushed (5) for CQE ffff8817f23ec5b0
[  238.912126] fast_io_fail_tmo expired for SRP port-2:1 / host2.
[  239.254388] scsi host2: ib_srp: reconnect succeeded
[  239.291843] scsi host2: ib_srp: failed RECV status WR flushed (5) for CQE ffff880bf5e712f0
[  240.619914] scsi host1: ib_srp: reconnect succeeded
[  240.654445] scsi host1: ib_srp: failed RECV status WR flushed (5) for CQE ffff8817f23ec5b0
[  254.620679] scsi host2: ib_srp: reconnect succeeded
[  254.658021] scsi host2: ib_srp: failed RECV status WR flushed (5) for CQE ffff880bf5e712f0
[  255.798616] scsi host1: ib_srp: reconnect succeeded
[  255.832897] scsi host1: ib_srp: failed RECV status WR flushed (5) for CQE ffff8817f23ec5b0
[  269.228783] scsi host2: ib_srp: reconnect succeeded
[  269.272055] mlx5_1:dump_cqe:262:(pid 0): dump error cqe
[  269.300655] 00000000 00000000 00000000 00000000
[  269.325885] 00000000 00000000 00000000 00000000
[  269.350457] 00000000 00000000 00000000 00000000
[  269.375757] 00000000 0f007806 25000072 005847d0
[  269.402013] scsi host2: ib_srp: failed FAST REG status memory management operation error (6) for CQE ffff880bd7c905b8
[  269.856464] scsi host1: ib_srp: reconnect succeeded
[  269.893203] scsi host1: ib_srp: failed RECV status WR flushed (5) for CQE ffff8817f23ec5b0

Thanks
Laurence

Re: [PATCH v2 1/8] IB/SRP: Avoid using IB_MR_TYPE_SG_GAPS

[Leon Romanovsky]

[-- Attachment #1: Type: text/plain, Size: 12548 bytes --]

On Wed, Feb 15, 2017 at 09:53:56AM -0500, Laurence Oberman wrote:
>
>
> ----- Original Message -----
> > From: "Leon Romanovsky" <leonro-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
> > To: "Laurence Oberman" <loberman-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>
> > Cc: "Bart Van Assche" <bart.vanassche-XdAiOPVOjttBDgjK7y7TUQ@public.gmane.org>, "Max Gurtovoy" <maxg-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>, "Doug Ledford"
> > <dledford-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>, linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org, "Israel Rukshin" <israelr-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>, "Mark Bloch"
> > <markb-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>, "Yuval Shaia" <yuval.shaia-QHcLZuEGTsvQT0dZR+AlfA@public.gmane.org>, "Artemy Kovalyov" <artemyko-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>, "# 4 . 7+"
> > <stable-u79uwXL29TY76Z2rM5mHXA@public.gmane.org>
> > Sent: Wednesday, February 15, 2017 8:47:07 AM
> > Subject: Re: [PATCH v2 1/8] IB/SRP: Avoid using IB_MR_TYPE_SG_GAPS
> >
> > On Wed, Feb 15, 2017 at 08:43:38AM -0500, Laurence Oberman wrote:
> > >
> > >
> > > ----- Original Message -----
> > > > From: "Laurence Oberman" <loberman-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>
> > > > To: "Leon Romanovsky" <leonro-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
> > > > Cc: "Bart Van Assche" <bart.vanassche-XdAiOPVOjttBDgjK7y7TUQ@public.gmane.org>, "Max Gurtovoy"
> > > > <maxg-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>, "Doug Ledford"
> > > > <dledford-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>, linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org, "Israel Rukshin"
> > > > <israelr-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>, "Mark Bloch"
> > > > <markb-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>, "Yuval Shaia" <yuval.shaia-QHcLZuEGTsvQT0dZR+AlfA@public.gmane.org>, "Artemy
> > > > Kovalyov" <artemyko-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>, "# 4 . 7+"
> > > > <stable-u79uwXL29TY76Z2rM5mHXA@public.gmane.org>
> > > > Sent: Wednesday, February 15, 2017 8:33:35 AM
> > > > Subject: Re: [PATCH v2 1/8] IB/SRP: Avoid using IB_MR_TYPE_SG_GAPS
> > > >
> > > >
> > > >
> > > > ----- Original Message -----
> > > > > From: "Leon Romanovsky" <leonro-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
> > > > > To: "Bart Van Assche" <bart.vanassche-XdAiOPVOjttBDgjK7y7TUQ@public.gmane.org>, "Max Gurtovoy"
> > > > > <maxg-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
> > > > > Cc: "Doug Ledford" <dledford-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>, linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org,
> > > > > "Israel Rukshin" <israelr-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>, "Mark
> > > > > Bloch" <markb-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>, "Yuval Shaia" <yuval.shaia-QHcLZuEGTsvQT0dZR+AlfA@public.gmane.org>,
> > > > > "Artemy Kovalyov" <artemyko-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>, "# 4
> > > > > . 7+" <stable-u79uwXL29TY76Z2rM5mHXA@public.gmane.org>
> > > > > Sent: Wednesday, February 15, 2017 3:19:45 AM
> > > > > Subject: Re: [PATCH v2 1/8] IB/SRP: Avoid using IB_MR_TYPE_SG_GAPS
> > > > >
> > > > > On Wed, Feb 15, 2017 at 09:14:49AM +0200, Leon Romanovsky wrote:
> > > > > > On Tue, Feb 14, 2017 at 10:56:29AM -0800, Bart Van Assche wrote:
> > > > > > > Tests have shown that the following error message is reported when
> > > > > > > using SG-GAPS registration with an mlx5 adapter:
> > > > > > >
> > > > > > > scsi host1: ib_srp: failed RECV status WR flushed (5) for CQE
> > > > > > > ffff880bd4270eb0
> > > > > > > 00000000 00000000 00000000 00000000
> > > > > > > 00000000 00000000 00000000 00000000
> > > > > > > 00000000 00000000 00000000 00000000
> > > > > > > 00000000 0f007806 2500002a ad9fafd1
> > > > > > > scsi host1: ib_srp: reconnect succeeded
> > > > > > > mlx5_0:dump_cqe:262:(pid 7369): dump error cqe
> > > > > > > 00000000 00000000 00000000 00000000
> > > > > > > 00000000 00000000 00000000 00000000
> > > > > > > 00000000 00000000 00000000 00000000
> > > > > > > 00000000 0f007806 25000032 00105dd0
> > > > > > > scsi host1: ib_srp: failed FAST REG status memory management
> > > > > > > operation
> > > > > > > error (6) for CQE ffff880b92860138
> > > > > > >
> > > > > > > Hence avoid using SG-GAPS memory registrations. Additionally,
> > > > > > > always configure the blk_queue_virt_boundary() to avoid to trigger
> > > > > > > a mapping failure when using adapters that support SG-GAPS (e.g.
> > > > > > > mlx5).
> > > > > >
> > > > > > According to the error dump, we have an issue with max_page_list_len
> > > > > > supplied and/or
> > > > > > internal calculations from that value to the UMR byte count.
> > > > >
> > > > > Hi Bart,
> > > > >
> > > > > Do you mind to try your test on my branch rdma-next [1] with the
> > > > > following
> > > > > fixup?
> > > > >
> > > > > diff --git a/drivers/infiniband/hw/mlx5/mr.c
> > > > > b/drivers/infiniband/hw/mlx5/mr.c
> > > > > index 3c1f483d003f..3e59dce10d5e 100644
> > > > > --- a/drivers/infiniband/hw/mlx5/mr.c
> > > > > +++ b/drivers/infiniband/hw/mlx5/mr.c
> > > > > @@ -1045,8 +1045,9 @@ int mlx5_ib_update_xlt(struct mlx5_ib_mr *mr, u64
> > > > > idx,
> > > > > int npages,
> > > > >  	for (pages_mapped = 0;
> > > > >  	     pages_mapped < pages_to_map && !err;
> > > > >  	     pages_mapped += pages_iter, idx += pages_iter) {
> > > > > +		npages = min_t(int, pages_iter, pages_to_map - pages_mapped);
> > > > >  		dma_sync_single_for_cpu(ddev, dma, size, DMA_TO_DEVICE);
> > > > > -		npages = populate_xlt(mr, idx, pages_iter, xlt,
> > > > > +		npages = populate_xlt(mr, idx, npages, xlt,
> > > > >  				      page_shift, size, flags);
> > > > >
> > > > >  		dma_sync_single_for_device(ddev, dma, size, DMA_TO_DEVICE);
> > > > >
> > > > > [1]
> > > > > https://git.kernel.org/cgit/linux/kernel/git/leon/linux-rdma.git/log/?h=rdma-next
> > > > >
> > > > > Thanks
> > > > >
> > > >
> > > > Hello Leon
> > > > Replied earlier but I dont know if my reply made it.
> > > > I will have to test this.
> > > >
> > > > is this repo
> > > > https://git.kernel.org/cgit/linux/kernel/git/leon/linux-rdma.git/log/?h=rdma-next
> > > > already patched with the change you want.
> > > > If not can I just take the patch and apply to my earlier tree based just
> > > > on
> > > > Linus's tree where I reverted the patch.
> > > >
> > > > Thanks
> > > > Laurence
> > > > --
> > > > To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
> > > > the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> > > > More majordomo info at  http://vger.kernel.org/majordomo-info.html
> > > >
> > >
> > > I guess not, I looked at applying this patch to Linus's tree but the
> > > function
> > > mlx5_ib_update_xlt() is not in there yet.
> >
> > This is why I asked to use my tree, there is a chance that wrong
> > calculation was before we introduced mlx5_ib_update_xlt() function.
> >
> > >
> > > I will see if I can get your tree staged and test this for you.
> >
> > Thanks a lot.
> >
> > >
> > > Thanks
> > > Laurence
> > > --
> > > To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
> > > the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> > > More majordomo info at  http://vger.kernel.org/majordomo-info.html
> >
>
> Hello Leon
> I pulled your tree, ran git checkout rdma-next
> I applied the patch manually, built the kernel and started the tests.
>
> Pretty soon I ran into
>
> ibclient login: [  132.640142] mlx5_1:dump_cqe:262:(pid 11417): dump error cqe
> [  132.640185] scsi host2: ib_srp: failed RECV status WR flushed (5) for CQE ffff880bd86a97b0
> [  132.718106] 00000000 00000000 00000000 00000000
> [  132.743616] 00000000 00000000 00000000 00000000
> [  132.767790] 00000000 00000000 00000000 00000000
> [  132.793626] 00000000 0f007806 2500002a 8ad015d1
> [  136.181512] systemd-readahead[701]: open(/var/tmp/dracut.ZR8CYG/initramfs/usr/bin/loginctl) failed: Too many levels of symbolic links
> [  136.250779] systemd-readahead[701]: open(/var/tmp/dracut.ZR8CYG/initramfs/usr/lib/systemd/system/dracut-emergency.service) failed: Too many levels of symbolic links
> [  147.791028] scsi host2: ib_srp: reconnect succeeded
> [  147.827012] scsi host2: ib_srp: failed RECV status WR flushed (5) for CQE ffff880bf5e71330
> [  162.908764] scsi host2: ib_srp: reconnect succeeded
> [  162.944244] scsi host2: ib_srp: failed RECV status WR flushed (5) for CQE ffff880bf5e71330
> [  166.409523] scsi host1: ib_srp: failed RECV status WR flushed (5) for CQE ffff8817f23ec570
> [  177.292496] scsi host2: ib_srp: reconnect succeeded
> [  177.334396] mlx5_1:dump_cqe:262:(pid 0): dump error cqe
> [  177.363793] 00000000 00000000 00000000 00000000
> [  177.388434] 00000000 00000000 00000000 00000000
> [  177.413918] 00000000 00000000 00000000 00000000
> [  177.438911] 00000000 0f007806 25000042 00102dd0
> [  177.465048] scsi host2: ib_srp: failed FAST REG status memory management operation error (6) for CQE ffff880bd7c90138
> [  181.386124] scsi host1: ib_srp: reconnect succeeded
> [  181.422892] scsi host1: ib_srp: failed RECV status WR flushed (5) for CQE ffff8817f23ec5b0
> [  192.827678] fast_io_fail_tmo expired for SRP port-2:1 / host2.
> [  193.230036] scsi host2: ib_srp: reconnect succeeded
> [  193.329290] mlx5_1:dump_cqe:262:(pid 0): dump error cqe
> [  193.356245] 00000000 00000000 00000000 00000000
> [  193.379488] 00000000 00000000 00000000 00000000
> [  193.404853] 00000000 00000000 00000000 00000000
> [  193.429633] 00000000 0f007806 2500004a 006c46d0
> [  193.455183] scsi host2: ib_srp: failed FAST REG status memory management operation error (6) for CQE ffff880bd7c906f8
> [  196.178111] scsi host1: ib_srp: reconnect succeeded
> [  196.215197] scsi host1: ib_srp: failed RECV status WR flushed (5) for CQE ffff8817f23ec5b0
> [  208.277396] scsi host2: ib_srp: reconnect succeeded
> [  208.318454] mlx5_1:dump_cqe:262:(pid 0): dump error cqe
> [  208.349012] 00000000 00000000 00000000 00000000
> [  208.374873] 00000000 00000000 00000000 00000000
> [  208.401070] 00000000 00000000 00000000 00000000
> [  208.426873] 00000000 0f007806 25000052 00103dd0
> [  208.452788] scsi host2: ib_srp: failed FAST REG status memory management operation error (6) for CQE ffff880bd7c90138
> [  211.066945] scsi host1: ib_srp: reconnect succeeded
> [  211.101645] scsi host1: ib_srp: failed RECV status WR flushed (5) for CQE ffff8817f23ec5b0
> [  223.296126] scsi host2: ib_srp: reconnect succeeded
> [  223.396340] mlx5_1:dump_cqe:262:(pid 0): dump error cqe
> [  223.423310] 00000000 00000000 00000000 00000000
> [  223.448590] 00000000 00000000 00000000 00000000
> [  223.473520] 00000000 00000000 00000000 00000000
> [  223.499008] 00000000 0f007806 2500005a 006c56d0
> [  223.524787] scsi host2: ib_srp: failed FAST REG status memory management operation error (6) for CQE ffff880bd7c906f8
> [  225.463312] scsi host1: ib_srp: reconnect succeeded
> [  225.499915] scsi host1: ib_srp: failed RECV status WR flushed (5) for CQE ffff8817f23ec5b0
> [  238.912126] fast_io_fail_tmo expired for SRP port-2:1 / host2.
> [  239.254388] scsi host2: ib_srp: reconnect succeeded
> [  239.291843] scsi host2: ib_srp: failed RECV status WR flushed (5) for CQE ffff880bf5e712f0
> [  240.619914] scsi host1: ib_srp: reconnect succeeded
> [  240.654445] scsi host1: ib_srp: failed RECV status WR flushed (5) for CQE ffff8817f23ec5b0
> [  254.620679] scsi host2: ib_srp: reconnect succeeded
> [  254.658021] scsi host2: ib_srp: failed RECV status WR flushed (5) for CQE ffff880bf5e712f0
> [  255.798616] scsi host1: ib_srp: reconnect succeeded
> [  255.832897] scsi host1: ib_srp: failed RECV status WR flushed (5) for CQE ffff8817f23ec5b0
> [  269.228783] scsi host2: ib_srp: reconnect succeeded
> [  269.272055] mlx5_1:dump_cqe:262:(pid 0): dump error cqe
> [  269.300655] 00000000 00000000 00000000 00000000
> [  269.325885] 00000000 00000000 00000000 00000000
> [  269.350457] 00000000 00000000 00000000 00000000
> [  269.375757] 00000000 0f007806 25000072 005847d0
> [  269.402013] scsi host2: ib_srp: failed FAST REG status memory management operation error (6) for CQE ffff880bd7c905b8
> [  269.856464] scsi host1: ib_srp: reconnect succeeded
> [  269.893203] scsi host1: ib_srp: failed RECV status WR flushed (5) for CQE ffff8817f23ec5b0
>

Thanks a lot for your effort, we tried fast path to debug, but it didn't
work :(.

> Thanks
> Laurence
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [PATCH v2 1/8] IB/SRP: Avoid using IB_MR_TYPE_SG_GAPS

[Leon Romanovsky]

[-- Attachment #1: Type: text/plain, Size: 11611 bytes --]

On Wed, Feb 15, 2017 at 09:53:56AM -0500, Laurence Oberman wrote:
>
>
> ----- Original Message -----
> > From: "Leon Romanovsky" <leonro@mellanox.com>
> > To: "Laurence Oberman" <loberman@redhat.com>
> > Cc: "Bart Van Assche" <bart.vanassche@sandisk.com>, "Max Gurtovoy" <maxg@mellanox.com>, "Doug Ledford"
> > <dledford@redhat.com>, linux-rdma@vger.kernel.org, "Israel Rukshin" <israelr@mellanox.com>, "Mark Bloch"
> > <markb@mellanox.com>, "Yuval Shaia" <yuval.shaia@oracle.com>, "Artemy Kovalyov" <artemyko@mellanox.com>, "# 4 . 7+"
> > <stable@vger.kernel.org>
> > Sent: Wednesday, February 15, 2017 8:47:07 AM
> > Subject: Re: [PATCH v2 1/8] IB/SRP: Avoid using IB_MR_TYPE_SG_GAPS
> >
> > On Wed, Feb 15, 2017 at 08:43:38AM -0500, Laurence Oberman wrote:
> > >
> > >
> > > ----- Original Message -----
> > > > From: "Laurence Oberman" <loberman@redhat.com>
> > > > To: "Leon Romanovsky" <leonro@mellanox.com>
> > > > Cc: "Bart Van Assche" <bart.vanassche@sandisk.com>, "Max Gurtovoy"
> > > > <maxg@mellanox.com>, "Doug Ledford"
> > > > <dledford@redhat.com>, linux-rdma@vger.kernel.org, "Israel Rukshin"
> > > > <israelr@mellanox.com>, "Mark Bloch"
> > > > <markb@mellanox.com>, "Yuval Shaia" <yuval.shaia@oracle.com>, "Artemy
> > > > Kovalyov" <artemyko@mellanox.com>, "# 4 . 7+"
> > > > <stable@vger.kernel.org>
> > > > Sent: Wednesday, February 15, 2017 8:33:35 AM
> > > > Subject: Re: [PATCH v2 1/8] IB/SRP: Avoid using IB_MR_TYPE_SG_GAPS
> > > >
> > > >
> > > >
> > > > ----- Original Message -----
> > > > > From: "Leon Romanovsky" <leonro@mellanox.com>
> > > > > To: "Bart Van Assche" <bart.vanassche@sandisk.com>, "Max Gurtovoy"
> > > > > <maxg@mellanox.com>
> > > > > Cc: "Doug Ledford" <dledford@redhat.com>, linux-rdma@vger.kernel.org,
> > > > > "Israel Rukshin" <israelr@mellanox.com>, "Mark
> > > > > Bloch" <markb@mellanox.com>, "Yuval Shaia" <yuval.shaia@oracle.com>,
> > > > > "Artemy Kovalyov" <artemyko@mellanox.com>, "# 4
> > > > > . 7+" <stable@vger.kernel.org>
> > > > > Sent: Wednesday, February 15, 2017 3:19:45 AM
> > > > > Subject: Re: [PATCH v2 1/8] IB/SRP: Avoid using IB_MR_TYPE_SG_GAPS
> > > > >
> > > > > On Wed, Feb 15, 2017 at 09:14:49AM +0200, Leon Romanovsky wrote:
> > > > > > On Tue, Feb 14, 2017 at 10:56:29AM -0800, Bart Van Assche wrote:
> > > > > > > Tests have shown that the following error message is reported when
> > > > > > > using SG-GAPS registration with an mlx5 adapter:
> > > > > > >
> > > > > > > scsi host1: ib_srp: failed RECV status WR flushed (5) for CQE
> > > > > > > ffff880bd4270eb0
> > > > > > > 00000000 00000000 00000000 00000000
> > > > > > > 00000000 00000000 00000000 00000000
> > > > > > > 00000000 00000000 00000000 00000000
> > > > > > > 00000000 0f007806 2500002a ad9fafd1
> > > > > > > scsi host1: ib_srp: reconnect succeeded
> > > > > > > mlx5_0:dump_cqe:262:(pid 7369): dump error cqe
> > > > > > > 00000000 00000000 00000000 00000000
> > > > > > > 00000000 00000000 00000000 00000000
> > > > > > > 00000000 00000000 00000000 00000000
> > > > > > > 00000000 0f007806 25000032 00105dd0
> > > > > > > scsi host1: ib_srp: failed FAST REG status memory management
> > > > > > > operation
> > > > > > > error (6) for CQE ffff880b92860138
> > > > > > >
> > > > > > > Hence avoid using SG-GAPS memory registrations. Additionally,
> > > > > > > always configure the blk_queue_virt_boundary() to avoid to trigger
> > > > > > > a mapping failure when using adapters that support SG-GAPS (e.g.
> > > > > > > mlx5).
> > > > > >
> > > > > > According to the error dump, we have an issue with max_page_list_len
> > > > > > supplied and/or
> > > > > > internal calculations from that value to the UMR byte count.
> > > > >
> > > > > Hi Bart,
> > > > >
> > > > > Do you mind to try your test on my branch rdma-next [1] with the
> > > > > following
> > > > > fixup?
> > > > >
> > > > > diff --git a/drivers/infiniband/hw/mlx5/mr.c
> > > > > b/drivers/infiniband/hw/mlx5/mr.c
> > > > > index 3c1f483d003f..3e59dce10d5e 100644
> > > > > --- a/drivers/infiniband/hw/mlx5/mr.c
> > > > > +++ b/drivers/infiniband/hw/mlx5/mr.c
> > > > > @@ -1045,8 +1045,9 @@ int mlx5_ib_update_xlt(struct mlx5_ib_mr *mr, u64
> > > > > idx,
> > > > > int npages,
> > > > >  	for (pages_mapped = 0;
> > > > >  	     pages_mapped < pages_to_map && !err;
> > > > >  	     pages_mapped += pages_iter, idx += pages_iter) {
> > > > > +		npages = min_t(int, pages_iter, pages_to_map - pages_mapped);
> > > > >  		dma_sync_single_for_cpu(ddev, dma, size, DMA_TO_DEVICE);
> > > > > -		npages = populate_xlt(mr, idx, pages_iter, xlt,
> > > > > +		npages = populate_xlt(mr, idx, npages, xlt,
> > > > >  				      page_shift, size, flags);
> > > > >
> > > > >  		dma_sync_single_for_device(ddev, dma, size, DMA_TO_DEVICE);
> > > > >
> > > > > [1]
> > > > > https://git.kernel.org/cgit/linux/kernel/git/leon/linux-rdma.git/log/?h=rdma-next
> > > > >
> > > > > Thanks
> > > > >
> > > >
> > > > Hello Leon
> > > > Replied earlier but I dont know if my reply made it.
> > > > I will have to test this.
> > > >
> > > > is this repo
> > > > https://git.kernel.org/cgit/linux/kernel/git/leon/linux-rdma.git/log/?h=rdma-next
> > > > already patched with the change you want.
> > > > If not can I just take the patch and apply to my earlier tree based just
> > > > on
> > > > Linus's tree where I reverted the patch.
> > > >
> > > > Thanks
> > > > Laurence
> > > > --
> > > > To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
> > > > the body of a message to majordomo@vger.kernel.org
> > > > More majordomo info at  http://vger.kernel.org/majordomo-info.html
> > > >
> > >
> > > I guess not, I looked at applying this patch to Linus's tree but the
> > > function
> > > mlx5_ib_update_xlt() is not in there yet.
> >
> > This is why I asked to use my tree, there is a chance that wrong
> > calculation was before we introduced mlx5_ib_update_xlt() function.
> >
> > >
> > > I will see if I can get your tree staged and test this for you.
> >
> > Thanks a lot.
> >
> > >
> > > Thanks
> > > Laurence
> > > --
> > > To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
> > > the body of a message to majordomo@vger.kernel.org
> > > More majordomo info at  http://vger.kernel.org/majordomo-info.html
> >
>
> Hello Leon
> I pulled your tree, ran git checkout rdma-next
> I applied the patch manually, built the kernel and started the tests.
>
> Pretty soon I ran into
>
> ibclient login: [  132.640142] mlx5_1:dump_cqe:262:(pid 11417): dump error cqe
> [  132.640185] scsi host2: ib_srp: failed RECV status WR flushed (5) for CQE ffff880bd86a97b0
> [  132.718106] 00000000 00000000 00000000 00000000
> [  132.743616] 00000000 00000000 00000000 00000000
> [  132.767790] 00000000 00000000 00000000 00000000
> [  132.793626] 00000000 0f007806 2500002a 8ad015d1
> [  136.181512] systemd-readahead[701]: open(/var/tmp/dracut.ZR8CYG/initramfs/usr/bin/loginctl) failed: Too many levels of symbolic links
> [  136.250779] systemd-readahead[701]: open(/var/tmp/dracut.ZR8CYG/initramfs/usr/lib/systemd/system/dracut-emergency.service) failed: Too many levels of symbolic links
> [  147.791028] scsi host2: ib_srp: reconnect succeeded
> [  147.827012] scsi host2: ib_srp: failed RECV status WR flushed (5) for CQE ffff880bf5e71330
> [  162.908764] scsi host2: ib_srp: reconnect succeeded
> [  162.944244] scsi host2: ib_srp: failed RECV status WR flushed (5) for CQE ffff880bf5e71330
> [  166.409523] scsi host1: ib_srp: failed RECV status WR flushed (5) for CQE ffff8817f23ec570
> [  177.292496] scsi host2: ib_srp: reconnect succeeded
> [  177.334396] mlx5_1:dump_cqe:262:(pid 0): dump error cqe
> [  177.363793] 00000000 00000000 00000000 00000000
> [  177.388434] 00000000 00000000 00000000 00000000
> [  177.413918] 00000000 00000000 00000000 00000000
> [  177.438911] 00000000 0f007806 25000042 00102dd0
> [  177.465048] scsi host2: ib_srp: failed FAST REG status memory management operation error (6) for CQE ffff880bd7c90138
> [  181.386124] scsi host1: ib_srp: reconnect succeeded
> [  181.422892] scsi host1: ib_srp: failed RECV status WR flushed (5) for CQE ffff8817f23ec5b0
> [  192.827678] fast_io_fail_tmo expired for SRP port-2:1 / host2.
> [  193.230036] scsi host2: ib_srp: reconnect succeeded
> [  193.329290] mlx5_1:dump_cqe:262:(pid 0): dump error cqe
> [  193.356245] 00000000 00000000 00000000 00000000
> [  193.379488] 00000000 00000000 00000000 00000000
> [  193.404853] 00000000 00000000 00000000 00000000
> [  193.429633] 00000000 0f007806 2500004a 006c46d0
> [  193.455183] scsi host2: ib_srp: failed FAST REG status memory management operation error (6) for CQE ffff880bd7c906f8
> [  196.178111] scsi host1: ib_srp: reconnect succeeded
> [  196.215197] scsi host1: ib_srp: failed RECV status WR flushed (5) for CQE ffff8817f23ec5b0
> [  208.277396] scsi host2: ib_srp: reconnect succeeded
> [  208.318454] mlx5_1:dump_cqe:262:(pid 0): dump error cqe
> [  208.349012] 00000000 00000000 00000000 00000000
> [  208.374873] 00000000 00000000 00000000 00000000
> [  208.401070] 00000000 00000000 00000000 00000000
> [  208.426873] 00000000 0f007806 25000052 00103dd0
> [  208.452788] scsi host2: ib_srp: failed FAST REG status memory management operation error (6) for CQE ffff880bd7c90138
> [  211.066945] scsi host1: ib_srp: reconnect succeeded
> [  211.101645] scsi host1: ib_srp: failed RECV status WR flushed (5) for CQE ffff8817f23ec5b0
> [  223.296126] scsi host2: ib_srp: reconnect succeeded
> [  223.396340] mlx5_1:dump_cqe:262:(pid 0): dump error cqe
> [  223.423310] 00000000 00000000 00000000 00000000
> [  223.448590] 00000000 00000000 00000000 00000000
> [  223.473520] 00000000 00000000 00000000 00000000
> [  223.499008] 00000000 0f007806 2500005a 006c56d0
> [  223.524787] scsi host2: ib_srp: failed FAST REG status memory management operation error (6) for CQE ffff880bd7c906f8
> [  225.463312] scsi host1: ib_srp: reconnect succeeded
> [  225.499915] scsi host1: ib_srp: failed RECV status WR flushed (5) for CQE ffff8817f23ec5b0
> [  238.912126] fast_io_fail_tmo expired for SRP port-2:1 / host2.
> [  239.254388] scsi host2: ib_srp: reconnect succeeded
> [  239.291843] scsi host2: ib_srp: failed RECV status WR flushed (5) for CQE ffff880bf5e712f0
> [  240.619914] scsi host1: ib_srp: reconnect succeeded
> [  240.654445] scsi host1: ib_srp: failed RECV status WR flushed (5) for CQE ffff8817f23ec5b0
> [  254.620679] scsi host2: ib_srp: reconnect succeeded
> [  254.658021] scsi host2: ib_srp: failed RECV status WR flushed (5) for CQE ffff880bf5e712f0
> [  255.798616] scsi host1: ib_srp: reconnect succeeded
> [  255.832897] scsi host1: ib_srp: failed RECV status WR flushed (5) for CQE ffff8817f23ec5b0
> [  269.228783] scsi host2: ib_srp: reconnect succeeded
> [  269.272055] mlx5_1:dump_cqe:262:(pid 0): dump error cqe
> [  269.300655] 00000000 00000000 00000000 00000000
> [  269.325885] 00000000 00000000 00000000 00000000
> [  269.350457] 00000000 00000000 00000000 00000000
> [  269.375757] 00000000 0f007806 25000072 005847d0
> [  269.402013] scsi host2: ib_srp: failed FAST REG status memory management operation error (6) for CQE ffff880bd7c905b8
> [  269.856464] scsi host1: ib_srp: reconnect succeeded
> [  269.893203] scsi host1: ib_srp: failed RECV status WR flushed (5) for CQE ffff8817f23ec5b0
>

Thanks a lot for your effort, we tried fast path to debug, but it didn't
work :(.

> Thanks
> Laurence
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [PATCH v2 1/8] IB/SRP: Avoid using IB_MR_TYPE_SG_GAPS

[Sagi Grimberg]

> Hello Leon
> I pulled your tree, ran git checkout rdma-next
> I applied the patch manually, built the kernel and started the tests.
>
> Pretty soon I ran into
>
> ibclient login: [  132.640142] mlx5_1:dump_cqe:262:(pid 11417): dump error cqe
> [  132.640185] scsi host2: ib_srp: failed RECV status WR flushed (5) for CQE ffff880bd86a97b0
> [  132.718106] 00000000 00000000 00000000 00000000
> [  132.743616] 00000000 00000000 00000000 00000000
> [  132.767790] 00000000 00000000 00000000 00000000
> [  132.793626] 00000000 0f007806 2500002a 8ad015d1

Not surprising given that the ib_map_mr_sg code never visits there...

Re: [PATCH v2 1/8] IB/SRP: Avoid using IB_MR_TYPE_SG_GAPS

[Sagi Grimberg]

> Tests have shown that the following error message is reported when
> using SG-GAPS registration with an mlx5 adapter:
>
> scsi host1: ib_srp: failed RECV status WR flushed (5) for CQE ffff880bd4270eb0
> 00000000 00000000 00000000 00000000
> 00000000 00000000 00000000 00000000
> 00000000 00000000 00000000 00000000
> 00000000 0f007806 2500002a ad9fafd1
> scsi host1: ib_srp: reconnect succeeded
> mlx5_0:dump_cqe:262:(pid 7369): dump error cqe
> 00000000 00000000 00000000 00000000
> 00000000 00000000 00000000 00000000
> 00000000 00000000 00000000 00000000
> 00000000 0f007806 25000032 00105dd0
> scsi host1: ib_srp: failed FAST REG status memory management operation error (6) for CQE ffff880b92860138
>
> Hence avoid using SG-GAPS memory registrations. Additionally,
> always configure the blk_queue_virt_boundary() to avoid to trigger
> a mapping failure when using adapters that support SG-GAPS (e.g.
> mlx5).

Hi Guys,

Sorry for addressing this late, but has this failure been investigated?

Max, Israel, what does this error syndrome map to?

Looking at mlx5_ib_sg_to_klms, I think the mr->length is incorrectly
incremented. Does the following change fix the problem?
--
diff --git a/drivers/infiniband/hw/mlx5/mr.c 
b/drivers/infiniband/hw/mlx5/mr.c
index 8f608debe141..c21c9eee37f6 100644
--- a/drivers/infiniband/hw/mlx5/mr.c
+++ b/drivers/infiniband/hw/mlx5/mr.c
@@ -1832,7 +1832,7 @@ mlx5_ib_sg_to_klms(struct mlx5_ib_mr *mr,
                 klms[i].va = cpu_to_be64(sg_dma_address(sg) + sg_offset);
                 klms[i].bcount = cpu_to_be32(sg_dma_len(sg) - sg_offset);
                 klms[i].key = cpu_to_be32(lkey);
-               mr->ibmr.length += sg_dma_len(sg);
+               mr->ibmr.length += sg_dma_len(sg) - sg_offset;

                 sg_offset = 0;
         }

Re: [PATCH v2 1/8] IB/SRP: Avoid using IB_MR_TYPE_SG_GAPS

[Sagi Grimberg]

> Tests have shown that the following error message is reported when
> using SG-GAPS registration with an mlx5 adapter:
>
> scsi host1: ib_srp: failed RECV status WR flushed (5) for CQE ffff880bd4270eb0
> 00000000 00000000 00000000 00000000
> 00000000 00000000 00000000 00000000
> 00000000 00000000 00000000 00000000
> 00000000 0f007806 2500002a ad9fafd1
> scsi host1: ib_srp: reconnect succeeded
> mlx5_0:dump_cqe:262:(pid 7369): dump error cqe
> 00000000 00000000 00000000 00000000
> 00000000 00000000 00000000 00000000
> 00000000 00000000 00000000 00000000
> 00000000 0f007806 25000032 00105dd0
> scsi host1: ib_srp: failed FAST REG status memory management operation error (6) for CQE ffff880b92860138
>
> Hence avoid using SG-GAPS memory registrations. Additionally,
> always configure the blk_queue_virt_boundary() to avoid to trigger
> a mapping failure when using adapters that support SG-GAPS (e.g.
> mlx5).

Hi Guys,

Sorry for addressing this late, but has this failure been investigated?

Max, Israel, what does this error syndrome map to?

Looking at mlx5_ib_sg_to_klms, I think the mr->length is incorrectly
incremented. Does the following change fix the problem?
--
diff --git a/drivers/infiniband/hw/mlx5/mr.c 
b/drivers/infiniband/hw/mlx5/mr.c
index 8f608debe141..c21c9eee37f6 100644
--- a/drivers/infiniband/hw/mlx5/mr.c
+++ b/drivers/infiniband/hw/mlx5/mr.c
@@ -1832,7 +1832,7 @@ mlx5_ib_sg_to_klms(struct mlx5_ib_mr *mr,
                 klms[i].va = cpu_to_be64(sg_dma_address(sg) + sg_offset);
                 klms[i].bcount = cpu_to_be32(sg_dma_len(sg) - sg_offset);
                 klms[i].key = cpu_to_be32(lkey);
-               mr->ibmr.length += sg_dma_len(sg);
+               mr->ibmr.length += sg_dma_len(sg) - sg_offset;

                 sg_offset = 0;
         }
--

Re: [PATCH v2 1/8] IB/SRP: Avoid using IB_MR_TYPE_SG_GAPS

[Laurence Oberman]

----- Original Message -----
> From: "Sagi Grimberg" <sagi-NQWnxTmZq1alnMjI0IkVqw@public.gmane.org>
> To: "Bart Van Assche" <bart.vanassche-XdAiOPVOjttBDgjK7y7TUQ@public.gmane.org>, "Doug Ledford" <dledford-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>
> Cc: linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org, "Israel Rukshin" <israelr-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>, "Max Gurtovoy" <maxg-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>, "Leon
> Romanovsky" <leonro-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>, "Mark Bloch" <markb-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>, "Yuval Shaia" <yuval.shaia-QHcLZuEGTsvQT0dZR+AlfA@public.gmane.org>, "# 4 .
> 7+" <stable-u79uwXL29TY76Z2rM5mHXA@public.gmane.org>
> Sent: Wednesday, February 15, 2017 10:38:06 AM
> Subject: Re: [PATCH v2 1/8] IB/SRP: Avoid using IB_MR_TYPE_SG_GAPS
> 
> 
> > Tests have shown that the following error message is reported when
> > using SG-GAPS registration with an mlx5 adapter:
> >
> > scsi host1: ib_srp: failed RECV status WR flushed (5) for CQE
> > ffff880bd4270eb0
> > 00000000 00000000 00000000 00000000
> > 00000000 00000000 00000000 00000000
> > 00000000 00000000 00000000 00000000
> > 00000000 0f007806 2500002a ad9fafd1
> > scsi host1: ib_srp: reconnect succeeded
> > mlx5_0:dump_cqe:262:(pid 7369): dump error cqe
> > 00000000 00000000 00000000 00000000
> > 00000000 00000000 00000000 00000000
> > 00000000 00000000 00000000 00000000
> > 00000000 0f007806 25000032 00105dd0
> > scsi host1: ib_srp: failed FAST REG status memory management operation
> > error (6) for CQE ffff880b92860138
> >
> > Hence avoid using SG-GAPS memory registrations. Additionally,
> > always configure the blk_queue_virt_boundary() to avoid to trigger
> > a mapping failure when using adapters that support SG-GAPS (e.g.
> > mlx5).
> 
> Hi Guys,
> 
> Sorry for addressing this late, but has this failure been investigated?
> 
> Max, Israel, what does this error syndrome map to?
> 
> Looking at mlx5_ib_sg_to_klms, I think the mr->length is incorrectly
> incremented. Does the following change fix the problem?
> --
> diff --git a/drivers/infiniband/hw/mlx5/mr.c
> b/drivers/infiniband/hw/mlx5/mr.c
> index 8f608debe141..c21c9eee37f6 100644
> --- a/drivers/infiniband/hw/mlx5/mr.c
> +++ b/drivers/infiniband/hw/mlx5/mr.c
> @@ -1832,7 +1832,7 @@ mlx5_ib_sg_to_klms(struct mlx5_ib_mr *mr,
>                  klms[i].va = cpu_to_be64(sg_dma_address(sg) + sg_offset);
>                  klms[i].bcount = cpu_to_be32(sg_dma_len(sg) - sg_offset);
>                  klms[i].key = cpu_to_be32(lkey);
> -               mr->ibmr.length += sg_dma_len(sg);
> +               mr->ibmr.length += sg_dma_len(sg) - sg_offset;
> 
>                  sg_offset = 0;
>          }
> --
> --
> To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 

Hello Sagi,

I will get this tested.
Thanks
Laurence
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH v2 1/8] IB/SRP: Avoid using IB_MR_TYPE_SG_GAPS

[Laurence Oberman]

----- Original Message -----
> From: "Sagi Grimberg" <sagi@grimberg.me>
> To: "Bart Van Assche" <bart.vanassche@sandisk.com>, "Doug Ledford" <dledford@redhat.com>
> Cc: linux-rdma@vger.kernel.org, "Israel Rukshin" <israelr@mellanox.com>, "Max Gurtovoy" <maxg@mellanox.com>, "Leon
> Romanovsky" <leonro@mellanox.com>, "Mark Bloch" <markb@mellanox.com>, "Yuval Shaia" <yuval.shaia@oracle.com>, "# 4 .
> 7+" <stable@vger.kernel.org>
> Sent: Wednesday, February 15, 2017 10:38:06 AM
> Subject: Re: [PATCH v2 1/8] IB/SRP: Avoid using IB_MR_TYPE_SG_GAPS
> 
> 
> > Tests have shown that the following error message is reported when
> > using SG-GAPS registration with an mlx5 adapter:
> >
> > scsi host1: ib_srp: failed RECV status WR flushed (5) for CQE
> > ffff880bd4270eb0
> > 00000000 00000000 00000000 00000000
> > 00000000 00000000 00000000 00000000
> > 00000000 00000000 00000000 00000000
> > 00000000 0f007806 2500002a ad9fafd1
> > scsi host1: ib_srp: reconnect succeeded
> > mlx5_0:dump_cqe:262:(pid 7369): dump error cqe
> > 00000000 00000000 00000000 00000000
> > 00000000 00000000 00000000 00000000
> > 00000000 00000000 00000000 00000000
> > 00000000 0f007806 25000032 00105dd0
> > scsi host1: ib_srp: failed FAST REG status memory management operation
> > error (6) for CQE ffff880b92860138
> >
> > Hence avoid using SG-GAPS memory registrations. Additionally,
> > always configure the blk_queue_virt_boundary() to avoid to trigger
> > a mapping failure when using adapters that support SG-GAPS (e.g.
> > mlx5).
> 
> Hi Guys,
> 
> Sorry for addressing this late, but has this failure been investigated?
> 
> Max, Israel, what does this error syndrome map to?
> 
> Looking at mlx5_ib_sg_to_klms, I think the mr->length is incorrectly
> incremented. Does the following change fix the problem?
> --
> diff --git a/drivers/infiniband/hw/mlx5/mr.c
> b/drivers/infiniband/hw/mlx5/mr.c
> index 8f608debe141..c21c9eee37f6 100644
> --- a/drivers/infiniband/hw/mlx5/mr.c
> +++ b/drivers/infiniband/hw/mlx5/mr.c
> @@ -1832,7 +1832,7 @@ mlx5_ib_sg_to_klms(struct mlx5_ib_mr *mr,
>                  klms[i].va = cpu_to_be64(sg_dma_address(sg) + sg_offset);
>                  klms[i].bcount = cpu_to_be32(sg_dma_len(sg) - sg_offset);
>                  klms[i].key = cpu_to_be32(lkey);
> -               mr->ibmr.length += sg_dma_len(sg);
> +               mr->ibmr.length += sg_dma_len(sg) - sg_offset;
> 
>                  sg_offset = 0;
>          }
> --
> --
> To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 

Hello Sagi,

I will get this tested.
Thanks
Laurence

Re: [PATCH v2 1/8] IB/SRP: Avoid using IB_MR_TYPE_SG_GAPS

[Max Gurtovoy]

On 2/15/2017 5:38 PM, Sagi Grimberg wrote:
>
>> Tests have shown that the following error message is reported when
>> using SG-GAPS registration with an mlx5 adapter:
>>
>> scsi host1: ib_srp: failed RECV status WR flushed (5) for CQE
>> ffff880bd4270eb0
>> 00000000 00000000 00000000 00000000
>> 00000000 00000000 00000000 00000000
>> 00000000 00000000 00000000 00000000
>> 00000000 0f007806 2500002a ad9fafd1
>> scsi host1: ib_srp: reconnect succeeded
>> mlx5_0:dump_cqe:262:(pid 7369): dump error cqe
>> 00000000 00000000 00000000 00000000
>> 00000000 00000000 00000000 00000000
>> 00000000 00000000 00000000 00000000
>> 00000000 0f007806 25000032 00105dd0
>> scsi host1: ib_srp: failed FAST REG status memory management operation
>> error (6) for CQE ffff880b92860138
>>
>> Hence avoid using SG-GAPS memory registrations. Additionally,
>> always configure the blk_queue_virt_boundary() to avoid to trigger
>> a mapping failure when using adapters that support SG-GAPS (e.g.
>> mlx5).
>
> Hi Guys,
>
> Sorry for addressing this late, but has this failure been investigated?
>
> Max, Israel, what does this error syndrome map to?

Sagi,
this syndrome says that number of klms to write is bigger than number of 
mtts.

Artemy started investigating it and proposed solution that were tested 
by Laurence.
Let's see if your fix will help.

>
> Looking at mlx5_ib_sg_to_klms, I think the mr->length is incorrectly
> incremented. Does the following change fix the problem?
> --
> diff --git a/drivers/infiniband/hw/mlx5/mr.c
> b/drivers/infiniband/hw/mlx5/mr.c
> index 8f608debe141..c21c9eee37f6 100644
> --- a/drivers/infiniband/hw/mlx5/mr.c
> +++ b/drivers/infiniband/hw/mlx5/mr.c
> @@ -1832,7 +1832,7 @@ mlx5_ib_sg_to_klms(struct mlx5_ib_mr *mr,
>                 klms[i].va = cpu_to_be64(sg_dma_address(sg) + sg_offset);
>                 klms[i].bcount = cpu_to_be32(sg_dma_len(sg) - sg_offset);
>                 klms[i].key = cpu_to_be32(lkey);
> -               mr->ibmr.length += sg_dma_len(sg);
> +               mr->ibmr.length += sg_dma_len(sg) - sg_offset;
>
>                 sg_offset = 0;
>         }
> --
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH v2 1/8] IB/SRP: Avoid using IB_MR_TYPE_SG_GAPS

[Max Gurtovoy]

On 2/15/2017 5:38 PM, Sagi Grimberg wrote:
>
>> Tests have shown that the following error message is reported when
>> using SG-GAPS registration with an mlx5 adapter:
>>
>> scsi host1: ib_srp: failed RECV status WR flushed (5) for CQE
>> ffff880bd4270eb0
>> 00000000 00000000 00000000 00000000
>> 00000000 00000000 00000000 00000000
>> 00000000 00000000 00000000 00000000
>> 00000000 0f007806 2500002a ad9fafd1
>> scsi host1: ib_srp: reconnect succeeded
>> mlx5_0:dump_cqe:262:(pid 7369): dump error cqe
>> 00000000 00000000 00000000 00000000
>> 00000000 00000000 00000000 00000000
>> 00000000 00000000 00000000 00000000
>> 00000000 0f007806 25000032 00105dd0
>> scsi host1: ib_srp: failed FAST REG status memory management operation
>> error (6) for CQE ffff880b92860138
>>
>> Hence avoid using SG-GAPS memory registrations. Additionally,
>> always configure the blk_queue_virt_boundary() to avoid to trigger
>> a mapping failure when using adapters that support SG-GAPS (e.g.
>> mlx5).
>
> Hi Guys,
>
> Sorry for addressing this late, but has this failure been investigated?
>
> Max, Israel, what does this error syndrome map to?

Sagi,
this syndrome says that number of klms to write is bigger than number of 
mtts.

Artemy started investigating it and proposed solution that were tested 
by Laurence.
Let's see if your fix will help.

>
> Looking at mlx5_ib_sg_to_klms, I think the mr->length is incorrectly
> incremented. Does the following change fix the problem?
> --
> diff --git a/drivers/infiniband/hw/mlx5/mr.c
> b/drivers/infiniband/hw/mlx5/mr.c
> index 8f608debe141..c21c9eee37f6 100644
> --- a/drivers/infiniband/hw/mlx5/mr.c
> +++ b/drivers/infiniband/hw/mlx5/mr.c
> @@ -1832,7 +1832,7 @@ mlx5_ib_sg_to_klms(struct mlx5_ib_mr *mr,
>                 klms[i].va = cpu_to_be64(sg_dma_address(sg) + sg_offset);
>                 klms[i].bcount = cpu_to_be32(sg_dma_len(sg) - sg_offset);
>                 klms[i].key = cpu_to_be32(lkey);
> -               mr->ibmr.length += sg_dma_len(sg);
> +               mr->ibmr.length += sg_dma_len(sg) - sg_offset;
>
>                 sg_offset = 0;
>         }
> --

Re: [PATCH v2 1/8] IB/SRP: Avoid using IB_MR_TYPE_SG_GAPS

[Sagi Grimberg]

>> Hi Guys,
>>
>> Sorry for addressing this late, but has this failure been investigated?
>>
>> Max, Israel, what does this error syndrome map to?
>
> Sagi,
> this syndrome says that number of klms to write is bigger than number of
> mtts.

That is strange, given that we check for it explicitly...

if this is indeed the case, then mlx5 reports wrong
max_fast_reg_page_list_len.

> Artemy started investigating it and proposed solution that were tested
> by Laurence.
> Let's see if your fix will help.

Not sure it will help if the syndrome is what you say it is...

Re: [PATCH v2 1/8] IB/SRP: Avoid using IB_MR_TYPE_SG_GAPS

[Leon Romanovsky]

[-- Attachment #1: Type: text/plain, Size: 2582 bytes --]

On Wed, Feb 15, 2017 at 06:18:02PM +0200, Max Gurtovoy wrote:
>
>
> On 2/15/2017 5:38 PM, Sagi Grimberg wrote:
> >
> > > Tests have shown that the following error message is reported when
> > > using SG-GAPS registration with an mlx5 adapter:
> > >
> > > scsi host1: ib_srp: failed RECV status WR flushed (5) for CQE
> > > ffff880bd4270eb0
> > > 00000000 00000000 00000000 00000000
> > > 00000000 00000000 00000000 00000000
> > > 00000000 00000000 00000000 00000000
> > > 00000000 0f007806 2500002a ad9fafd1
> > > scsi host1: ib_srp: reconnect succeeded
> > > mlx5_0:dump_cqe:262:(pid 7369): dump error cqe
> > > 00000000 00000000 00000000 00000000
> > > 00000000 00000000 00000000 00000000
> > > 00000000 00000000 00000000 00000000
> > > 00000000 0f007806 25000032 00105dd0
> > > scsi host1: ib_srp: failed FAST REG status memory management operation
> > > error (6) for CQE ffff880b92860138
> > >
> > > Hence avoid using SG-GAPS memory registrations. Additionally,
> > > always configure the blk_queue_virt_boundary() to avoid to trigger
> > > a mapping failure when using adapters that support SG-GAPS (e.g.
> > > mlx5).
> >
> > Hi Guys,
> >
> > Sorry for addressing this late, but has this failure been investigated?
> >
> > Max, Israel, what does this error syndrome map to?
>
> Sagi,
> this syndrome says that number of klms to write is bigger than number of
> mtts.
>
> Artemy started investigating it and proposed solution that were tested by
> Laurence.
> Let's see if your fix will help.

No, Artemy's change doesn't fix it.

>
> >
> > Looking at mlx5_ib_sg_to_klms, I think the mr->length is incorrectly
> > incremented. Does the following change fix the problem?
> > --
> > diff --git a/drivers/infiniband/hw/mlx5/mr.c
> > b/drivers/infiniband/hw/mlx5/mr.c
> > index 8f608debe141..c21c9eee37f6 100644
> > --- a/drivers/infiniband/hw/mlx5/mr.c
> > +++ b/drivers/infiniband/hw/mlx5/mr.c
> > @@ -1832,7 +1832,7 @@ mlx5_ib_sg_to_klms(struct mlx5_ib_mr *mr,
> >                 klms[i].va = cpu_to_be64(sg_dma_address(sg) + sg_offset);
> >                 klms[i].bcount = cpu_to_be32(sg_dma_len(sg) - sg_offset);
> >                 klms[i].key = cpu_to_be32(lkey);
> > -               mr->ibmr.length += sg_dma_len(sg);
> > +               mr->ibmr.length += sg_dma_len(sg) - sg_offset;
> >
> >                 sg_offset = 0;
> >         }
> > --
> --
> To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [PATCH v2 1/8] IB/SRP: Avoid using IB_MR_TYPE_SG_GAPS

[Leon Romanovsky]

[-- Attachment #1: Type: text/plain, Size: 2558 bytes --]

On Wed, Feb 15, 2017 at 06:18:02PM +0200, Max Gurtovoy wrote:
>
>
> On 2/15/2017 5:38 PM, Sagi Grimberg wrote:
> >
> > > Tests have shown that the following error message is reported when
> > > using SG-GAPS registration with an mlx5 adapter:
> > >
> > > scsi host1: ib_srp: failed RECV status WR flushed (5) for CQE
> > > ffff880bd4270eb0
> > > 00000000 00000000 00000000 00000000
> > > 00000000 00000000 00000000 00000000
> > > 00000000 00000000 00000000 00000000
> > > 00000000 0f007806 2500002a ad9fafd1
> > > scsi host1: ib_srp: reconnect succeeded
> > > mlx5_0:dump_cqe:262:(pid 7369): dump error cqe
> > > 00000000 00000000 00000000 00000000
> > > 00000000 00000000 00000000 00000000
> > > 00000000 00000000 00000000 00000000
> > > 00000000 0f007806 25000032 00105dd0
> > > scsi host1: ib_srp: failed FAST REG status memory management operation
> > > error (6) for CQE ffff880b92860138
> > >
> > > Hence avoid using SG-GAPS memory registrations. Additionally,
> > > always configure the blk_queue_virt_boundary() to avoid to trigger
> > > a mapping failure when using adapters that support SG-GAPS (e.g.
> > > mlx5).
> >
> > Hi Guys,
> >
> > Sorry for addressing this late, but has this failure been investigated?
> >
> > Max, Israel, what does this error syndrome map to?
>
> Sagi,
> this syndrome says that number of klms to write is bigger than number of
> mtts.
>
> Artemy started investigating it and proposed solution that were tested by
> Laurence.
> Let's see if your fix will help.

No, Artemy's change doesn't fix it.

>
> >
> > Looking at mlx5_ib_sg_to_klms, I think the mr->length is incorrectly
> > incremented. Does the following change fix the problem?
> > --
> > diff --git a/drivers/infiniband/hw/mlx5/mr.c
> > b/drivers/infiniband/hw/mlx5/mr.c
> > index 8f608debe141..c21c9eee37f6 100644
> > --- a/drivers/infiniband/hw/mlx5/mr.c
> > +++ b/drivers/infiniband/hw/mlx5/mr.c
> > @@ -1832,7 +1832,7 @@ mlx5_ib_sg_to_klms(struct mlx5_ib_mr *mr,
> >                 klms[i].va = cpu_to_be64(sg_dma_address(sg) + sg_offset);
> >                 klms[i].bcount = cpu_to_be32(sg_dma_len(sg) - sg_offset);
> >                 klms[i].key = cpu_to_be32(lkey);
> > -               mr->ibmr.length += sg_dma_len(sg);
> > +               mr->ibmr.length += sg_dma_len(sg) - sg_offset;
> >
> >                 sg_offset = 0;
> >         }
> > --
> --
> To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [PATCH v2 1/8] IB/SRP: Avoid using IB_MR_TYPE_SG_GAPS

[Laurence Oberman]

----- Original Message -----
> From: "Sagi Grimberg" <sagi-NQWnxTmZq1alnMjI0IkVqw@public.gmane.org>
> To: "Bart Van Assche" <bart.vanassche-XdAiOPVOjttBDgjK7y7TUQ@public.gmane.org>, "Doug Ledford" <dledford-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>
> Cc: linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org, "Israel Rukshin" <israelr-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>, "Max Gurtovoy" <maxg-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>, "Leon
> Romanovsky" <leonro-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>, "Mark Bloch" <markb-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>, "Yuval Shaia" <yuval.shaia-QHcLZuEGTsvQT0dZR+AlfA@public.gmane.org>, "# 4 .
> 7+" <stable-u79uwXL29TY76Z2rM5mHXA@public.gmane.org>
> Sent: Wednesday, February 15, 2017 10:38:06 AM
> Subject: Re: [PATCH v2 1/8] IB/SRP: Avoid using IB_MR_TYPE_SG_GAPS
> 
> 
> > Tests have shown that the following error message is reported when
> > using SG-GAPS registration with an mlx5 adapter:
> >
> > scsi host1: ib_srp: failed RECV status WR flushed (5) for CQE
> > ffff880bd4270eb0
> > 00000000 00000000 00000000 00000000
> > 00000000 00000000 00000000 00000000
> > 00000000 00000000 00000000 00000000
> > 00000000 0f007806 2500002a ad9fafd1
> > scsi host1: ib_srp: reconnect succeeded
> > mlx5_0:dump_cqe:262:(pid 7369): dump error cqe
> > 00000000 00000000 00000000 00000000
> > 00000000 00000000 00000000 00000000
> > 00000000 00000000 00000000 00000000
> > 00000000 0f007806 25000032 00105dd0
> > scsi host1: ib_srp: failed FAST REG status memory management operation
> > error (6) for CQE ffff880b92860138
> >
> > Hence avoid using SG-GAPS memory registrations. Additionally,
> > always configure the blk_queue_virt_boundary() to avoid to trigger
> > a mapping failure when using adapters that support SG-GAPS (e.g.
> > mlx5).
> 
> Hi Guys,
> 
> Sorry for addressing this late, but has this failure been investigated?
> 
> Max, Israel, what does this error syndrome map to?
> 
> Looking at mlx5_ib_sg_to_klms, I think the mr->length is incorrectly
> incremented. Does the following change fix the problem?
> --
> diff --git a/drivers/infiniband/hw/mlx5/mr.c
> b/drivers/infiniband/hw/mlx5/mr.c
> index 8f608debe141..c21c9eee37f6 100644
> --- a/drivers/infiniband/hw/mlx5/mr.c
> +++ b/drivers/infiniband/hw/mlx5/mr.c
> @@ -1832,7 +1832,7 @@ mlx5_ib_sg_to_klms(struct mlx5_ib_mr *mr,
>                  klms[i].va = cpu_to_be64(sg_dma_address(sg) + sg_offset);
>                  klms[i].bcount = cpu_to_be32(sg_dma_len(sg) - sg_offset);
>                  klms[i].key = cpu_to_be32(lkey);
> -               mr->ibmr.length += sg_dma_len(sg);
> +               mr->ibmr.length += sg_dma_len(sg) - sg_offset;
> 
>                  sg_offset = 0;
>          }
> --
> --
> To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 

Started with Linus's tree, applied the change requested by Sagi, built the kernel, rebooted and started the tests.

Linux ibclient 4.10.0-rc8.sagi+ #1 SMP Wed Feb 15 11:09:44 EST 2017 x86_64 x86_64 x86_64 GNU/Linux

Very quickly get to this

[  180.990285] mlx5_0:dump_cqe:262:(pid 0): dump error cqe
[  181.016899] 00000000 00000000 00000000 00000000
[  181.040949] 00000000 00000000 00000000 00000000
[  181.066960] 00000000 00000000 00000000 00000000
[  181.092030] 00000000 0f007806 2500002a bf1913d0
[  181.117254] scsi host2: ib_srp: failed FAST REG status memory management operation error (6) for CQE ffff880bdbe88778
[  196.288933] fast_io_fail_tmo expired for SRP port-2:1 / host2.
[  197.090886] scsi host2: ib_srp: reconnect succeeded
[  197.127628] scsi host2: ib_srp: failed RECV status WR flushed (5) for CQE ffff8817f09b6f30

So does not help.
I think my and Barts suggestion to revert for now is the best way forward.
I have already tested this in-depth from Bart's tree and its been sent to Doug as V2 of Bart'recent 8 patch series.

Thanks
Laurence
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH v2 1/8] IB/SRP: Avoid using IB_MR_TYPE_SG_GAPS

[Laurence Oberman]

----- Original Message -----
> From: "Sagi Grimberg" <sagi@grimberg.me>
> To: "Bart Van Assche" <bart.vanassche@sandisk.com>, "Doug Ledford" <dledford@redhat.com>
> Cc: linux-rdma@vger.kernel.org, "Israel Rukshin" <israelr@mellanox.com>, "Max Gurtovoy" <maxg@mellanox.com>, "Leon
> Romanovsky" <leonro@mellanox.com>, "Mark Bloch" <markb@mellanox.com>, "Yuval Shaia" <yuval.shaia@oracle.com>, "# 4 .
> 7+" <stable@vger.kernel.org>
> Sent: Wednesday, February 15, 2017 10:38:06 AM
> Subject: Re: [PATCH v2 1/8] IB/SRP: Avoid using IB_MR_TYPE_SG_GAPS
> 
> 
> > Tests have shown that the following error message is reported when
> > using SG-GAPS registration with an mlx5 adapter:
> >
> > scsi host1: ib_srp: failed RECV status WR flushed (5) for CQE
> > ffff880bd4270eb0
> > 00000000 00000000 00000000 00000000
> > 00000000 00000000 00000000 00000000
> > 00000000 00000000 00000000 00000000
> > 00000000 0f007806 2500002a ad9fafd1
> > scsi host1: ib_srp: reconnect succeeded
> > mlx5_0:dump_cqe:262:(pid 7369): dump error cqe
> > 00000000 00000000 00000000 00000000
> > 00000000 00000000 00000000 00000000
> > 00000000 00000000 00000000 00000000
> > 00000000 0f007806 25000032 00105dd0
> > scsi host1: ib_srp: failed FAST REG status memory management operation
> > error (6) for CQE ffff880b92860138
> >
> > Hence avoid using SG-GAPS memory registrations. Additionally,
> > always configure the blk_queue_virt_boundary() to avoid to trigger
> > a mapping failure when using adapters that support SG-GAPS (e.g.
> > mlx5).
> 
> Hi Guys,
> 
> Sorry for addressing this late, but has this failure been investigated?
> 
> Max, Israel, what does this error syndrome map to?
> 
> Looking at mlx5_ib_sg_to_klms, I think the mr->length is incorrectly
> incremented. Does the following change fix the problem?
> --
> diff --git a/drivers/infiniband/hw/mlx5/mr.c
> b/drivers/infiniband/hw/mlx5/mr.c
> index 8f608debe141..c21c9eee37f6 100644
> --- a/drivers/infiniband/hw/mlx5/mr.c
> +++ b/drivers/infiniband/hw/mlx5/mr.c
> @@ -1832,7 +1832,7 @@ mlx5_ib_sg_to_klms(struct mlx5_ib_mr *mr,
>                  klms[i].va = cpu_to_be64(sg_dma_address(sg) + sg_offset);
>                  klms[i].bcount = cpu_to_be32(sg_dma_len(sg) - sg_offset);
>                  klms[i].key = cpu_to_be32(lkey);
> -               mr->ibmr.length += sg_dma_len(sg);
> +               mr->ibmr.length += sg_dma_len(sg) - sg_offset;
> 
>                  sg_offset = 0;
>          }
> --
> --
> To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 

Started with Linus's tree, applied the change requested by Sagi, built the kernel, rebooted and started the tests.

Linux ibclient 4.10.0-rc8.sagi+ #1 SMP Wed Feb 15 11:09:44 EST 2017 x86_64 x86_64 x86_64 GNU/Linux

Very quickly get to this

[  180.990285] mlx5_0:dump_cqe:262:(pid 0): dump error cqe
[  181.016899] 00000000 00000000 00000000 00000000
[  181.040949] 00000000 00000000 00000000 00000000
[  181.066960] 00000000 00000000 00000000 00000000
[  181.092030] 00000000 0f007806 2500002a bf1913d0
[  181.117254] scsi host2: ib_srp: failed FAST REG status memory management operation error (6) for CQE ffff880bdbe88778
[  196.288933] fast_io_fail_tmo expired for SRP port-2:1 / host2.
[  197.090886] scsi host2: ib_srp: reconnect succeeded
[  197.127628] scsi host2: ib_srp: failed RECV status WR flushed (5) for CQE ffff8817f09b6f30

So does not help.
I think my and Barts suggestion to revert for now is the best way forward.
I have already tested this in-depth from Bart's tree and its been sent to Doug as V2 of Bart'recent 8 patch series.

Thanks
Laurence

Re: [PATCH v2 1/8] IB/SRP: Avoid using IB_MR_TYPE_SG_GAPS

[Sagi Grimberg]

> Started with Linus's tree, applied the change requested by Sagi, built the kernel, rebooted and started the tests.
>
> Linux ibclient 4.10.0-rc8.sagi+ #1 SMP Wed Feb 15 11:09:44 EST 2017 x86_64 x86_64 x86_64 GNU/Linux
>
> Very quickly get to this
>
> [  180.990285] mlx5_0:dump_cqe:262:(pid 0): dump error cqe
> [  181.016899] 00000000 00000000 00000000 00000000
> [  181.040949] 00000000 00000000 00000000 00000000
> [  181.066960] 00000000 00000000 00000000 00000000
> [  181.092030] 00000000 0f007806 2500002a bf1913d0
> [  181.117254] scsi host2: ib_srp: failed FAST REG status memory management operation error (6) for CQE ffff880bdbe88778
> [  196.288933] fast_io_fail_tmo expired for SRP port-2:1 / host2.
> [  197.090886] scsi host2: ib_srp: reconnect succeeded
> [  197.127628] scsi host2: ib_srp: failed RECV status WR flushed (5) for CQE ffff8817f09b6f30
>
> So does not help.
> I think my and Barts suggestion to revert for now is the best way forward.
> I have already tested this in-depth from Bart's tree and its been sent to Doug as V2 of Bart'recent 8 patch series.

Yea, probably this is the best way forward.

Bart, I think the change I suggested is still needed regardless,
do you agree?

Max, Leon, is it possible that the max number of klms pr mr is
less than what reported in device capabilities for page_list_len?

If so, this means that either:
1. mlx5 needs to expose the minimum between pages and sg elems (sucks)
2. we need yet another capability for SG_GAPS (sucks^2 because the
whole point was to make it transparent to the user)
3. mlx5 does not support SG_GAPS (sucks^3 because we now have something
thats not supported by any device).

Re: [PATCH v2 1/8] IB/SRP: Avoid using IB_MR_TYPE_SG_GAPS

[Bart Van Assche]

On Wed, 2017-02-15 at 18:55 +0200, Sagi Grimberg wrote:
> Bart, I think the change I suggested is still needed regardless,
> do you agree?

I'm not sure. I have not yet had a close look at that part of the mlx5 driver.

Bart.

Re: [PATCH v2 1/8] IB/SRP: Avoid using IB_MR_TYPE_SG_GAPS

[Leon Romanovsky]

[-- Attachment #1: Type: text/plain, Size: 2157 bytes --]

On Wed, Feb 15, 2017 at 06:55:52PM +0200, Sagi Grimberg wrote:
>
> > Started with Linus's tree, applied the change requested by Sagi, built the kernel, rebooted and started the tests.
> >
> > Linux ibclient 4.10.0-rc8.sagi+ #1 SMP Wed Feb 15 11:09:44 EST 2017 x86_64 x86_64 x86_64 GNU/Linux
> >
> > Very quickly get to this
> >
> > [  180.990285] mlx5_0:dump_cqe:262:(pid 0): dump error cqe
> > [  181.016899] 00000000 00000000 00000000 00000000
> > [  181.040949] 00000000 00000000 00000000 00000000
> > [  181.066960] 00000000 00000000 00000000 00000000
> > [  181.092030] 00000000 0f007806 2500002a bf1913d0
> > [  181.117254] scsi host2: ib_srp: failed FAST REG status memory management operation error (6) for CQE ffff880bdbe88778
> > [  196.288933] fast_io_fail_tmo expired for SRP port-2:1 / host2.
> > [  197.090886] scsi host2: ib_srp: reconnect succeeded
> > [  197.127628] scsi host2: ib_srp: failed RECV status WR flushed (5) for CQE ffff8817f09b6f30
> >
> > So does not help.
> > I think my and Barts suggestion to revert for now is the best way forward.
> > I have already tested this in-depth from Bart's tree and its been sent to Doug as V2 of Bart'recent 8 patch series.
>
> Yea, probably this is the best way forward.
>
> Bart, I think the change I suggested is still needed regardless,
> do you agree?
>
> Max, Leon, is it possible that the max number of klms pr mr is
> less than what reported in device capabilities for page_list_len?

I hope no and we will check.
I already asked it, but didn't get any response, and I'll repeat it again.
ISER has similar code with SG_GAPS, does it work?

>
> If so, this means that either:
> 1. mlx5 needs to expose the minimum between pages and sg elems (sucks)
> 2. we need yet another capability for SG_GAPS (sucks^2 because the
> whole point was to make it transparent to the user)
> 3. mlx5 does not support SG_GAPS (sucks^3 because we now have something
> thats not supported by any device).
> --
> To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [PATCH v2 1/8] IB/SRP: Avoid using IB_MR_TYPE_SG_GAPS

[Leon Romanovsky]

[-- Attachment #1: Type: text/plain, Size: 2157 bytes --]

On Wed, Feb 15, 2017 at 06:55:52PM +0200, Sagi Grimberg wrote:
>
> > Started with Linus's tree, applied the change requested by Sagi, built the kernel, rebooted and started the tests.
> >
> > Linux ibclient 4.10.0-rc8.sagi+ #1 SMP Wed Feb 15 11:09:44 EST 2017 x86_64 x86_64 x86_64 GNU/Linux
> >
> > Very quickly get to this
> >
> > [  180.990285] mlx5_0:dump_cqe:262:(pid 0): dump error cqe
> > [  181.016899] 00000000 00000000 00000000 00000000
> > [  181.040949] 00000000 00000000 00000000 00000000
> > [  181.066960] 00000000 00000000 00000000 00000000
> > [  181.092030] 00000000 0f007806 2500002a bf1913d0
> > [  181.117254] scsi host2: ib_srp: failed FAST REG status memory management operation error (6) for CQE ffff880bdbe88778
> > [  196.288933] fast_io_fail_tmo expired for SRP port-2:1 / host2.
> > [  197.090886] scsi host2: ib_srp: reconnect succeeded
> > [  197.127628] scsi host2: ib_srp: failed RECV status WR flushed (5) for CQE ffff8817f09b6f30
> >
> > So does not help.
> > I think my and Barts suggestion to revert for now is the best way forward.
> > I have already tested this in-depth from Bart's tree and its been sent to Doug as V2 of Bart'recent 8 patch series.
>
> Yea, probably this is the best way forward.
>
> Bart, I think the change I suggested is still needed regardless,
> do you agree?
>
> Max, Leon, is it possible that the max number of klms pr mr is
> less than what reported in device capabilities for page_list_len?

I hope no and we will check.
I already asked it, but didn't get any response, and I'll repeat it again.
ISER has similar code with SG_GAPS, does it work?

>
> If so, this means that either:
> 1. mlx5 needs to expose the minimum between pages and sg elems (sucks)
> 2. we need yet another capability for SG_GAPS (sucks^2 because the
> whole point was to make it transparent to the user)
> 3. mlx5 does not support SG_GAPS (sucks^3 because we now have something
> thats not supported by any device).
> --
> To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [PATCH v2 1/8] IB/SRP: Avoid using IB_MR_TYPE_SG_GAPS

[Max Gurtovoy]

On 2/16/2017 8:14 AM, Leon Romanovsky wrote:
> On Wed, Feb 15, 2017 at 06:55:52PM +0200, Sagi Grimberg wrote:
>>
>>> Started with Linus's tree, applied the change requested by Sagi, built the kernel, rebooted and started the tests.
>>>
>>> Linux ibclient 4.10.0-rc8.sagi+ #1 SMP Wed Feb 15 11:09:44 EST 2017 x86_64 x86_64 x86_64 GNU/Linux
>>>
>>> Very quickly get to this
>>>
>>> [  180.990285] mlx5_0:dump_cqe:262:(pid 0): dump error cqe
>>> [  181.016899] 00000000 00000000 00000000 00000000
>>> [  181.040949] 00000000 00000000 00000000 00000000
>>> [  181.066960] 00000000 00000000 00000000 00000000
>>> [  181.092030] 00000000 0f007806 2500002a bf1913d0
>>> [  181.117254] scsi host2: ib_srp: failed FAST REG status memory management operation error (6) for CQE ffff880bdbe88778
>>> [  196.288933] fast_io_fail_tmo expired for SRP port-2:1 / host2.
>>> [  197.090886] scsi host2: ib_srp: reconnect succeeded
>>> [  197.127628] scsi host2: ib_srp: failed RECV status WR flushed (5) for CQE ffff8817f09b6f30
>>>
>>> So does not help.
>>> I think my and Barts suggestion to revert for now is the best way forward.
>>> I have already tested this in-depth from Bart's tree and its been sent to Doug as V2 of Bart'recent 8 patch series.
>>
>> Yea, probably this is the best way forward.
>>
>> Bart, I think the change I suggested is still needed regardless,
>> do you agree?
>>
>> Max, Leon, is it possible that the max number of klms pr mr is
>> less than what reported in device capabilities for page_list_len?
>
> I hope no and we will check.
> I already asked it, but didn't get any response, and I'll repeat it again.
> ISER has similar code with SG_GAPS, does it work?

Yes, I haven't seen issues with that in iSER.
We need to continue with the debug.

>
>>
>> If so, this means that either:
>> 1. mlx5 needs to expose the minimum between pages and sg elems (sucks)
>> 2. we need yet another capability for SG_GAPS (sucks^2 because the
>> whole point was to make it transparent to the user)
>> 3. mlx5 does not support SG_GAPS (sucks^3 because we now have something
>> thats not supported by any device).
>> --
>> To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH v2 1/8] IB/SRP: Avoid using IB_MR_TYPE_SG_GAPS

[Max Gurtovoy]

On 2/16/2017 8:14 AM, Leon Romanovsky wrote:
> On Wed, Feb 15, 2017 at 06:55:52PM +0200, Sagi Grimberg wrote:
>>
>>> Started with Linus's tree, applied the change requested by Sagi, built the kernel, rebooted and started the tests.
>>>
>>> Linux ibclient 4.10.0-rc8.sagi+ #1 SMP Wed Feb 15 11:09:44 EST 2017 x86_64 x86_64 x86_64 GNU/Linux
>>>
>>> Very quickly get to this
>>>
>>> [  180.990285] mlx5_0:dump_cqe:262:(pid 0): dump error cqe
>>> [  181.016899] 00000000 00000000 00000000 00000000
>>> [  181.040949] 00000000 00000000 00000000 00000000
>>> [  181.066960] 00000000 00000000 00000000 00000000
>>> [  181.092030] 00000000 0f007806 2500002a bf1913d0
>>> [  181.117254] scsi host2: ib_srp: failed FAST REG status memory management operation error (6) for CQE ffff880bdbe88778
>>> [  196.288933] fast_io_fail_tmo expired for SRP port-2:1 / host2.
>>> [  197.090886] scsi host2: ib_srp: reconnect succeeded
>>> [  197.127628] scsi host2: ib_srp: failed RECV status WR flushed (5) for CQE ffff8817f09b6f30
>>>
>>> So does not help.
>>> I think my and Barts suggestion to revert for now is the best way forward.
>>> I have already tested this in-depth from Bart's tree and its been sent to Doug as V2 of Bart'recent 8 patch series.
>>
>> Yea, probably this is the best way forward.
>>
>> Bart, I think the change I suggested is still needed regardless,
>> do you agree?
>>
>> Max, Leon, is it possible that the max number of klms pr mr is
>> less than what reported in device capabilities for page_list_len?
>
> I hope no and we will check.
> I already asked it, but didn't get any response, and I'll repeat it again.
> ISER has similar code with SG_GAPS, does it work?

Yes, I haven't seen issues with that in iSER.
We need to continue with the debug.

>
>>
>> If so, this means that either:
>> 1. mlx5 needs to expose the minimum between pages and sg elems (sucks)
>> 2. we need yet another capability for SG_GAPS (sucks^2 because the
>> whole point was to make it transparent to the user)
>> 3. mlx5 does not support SG_GAPS (sucks^3 because we now have something
>> thats not supported by any device).
>> --
>> To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html

[PATCH v2 2/8] IB/srp: Avoid that duplicate responses trigger a kernel bug

[Bart Van Assche]

After srp_process_rsp() returns there is a short time during which
the scsi_host_find_tag() call will return a pointer to the SCSI
command that is being completed. If during that time a duplicate
response is received, avoid that the following call stack appears:

BUG: unable to handle kernel NULL pointer dereference at           (null)
IP: srp_recv_done+0x450/0x6b0 [ib_srp]
Oops: 0000 [#1] SMP
CPU: 10 PID: 0 Comm: swapper/10 Not tainted 4.10.0-rc7-dbg+ #1
Call Trace:
 <IRQ>
 __ib_process_cq+0x4b/0xd0 [ib_core]
 ib_poll_handler+0x1d/0x70 [ib_core]
 irq_poll_softirq+0xba/0x120
 __do_softirq+0xba/0x4c0
 irq_exit+0xbe/0xd0
 smp_apic_timer_interrupt+0x38/0x50
 apic_timer_interrupt+0x90/0xa0
 </IRQ>
RIP: srp_recv_done+0x450/0x6b0 [ib_srp] RSP: ffff88046f483e20

Signed-off-by: Bart Van Assche <bart.vanassche-XdAiOPVOjttBDgjK7y7TUQ@public.gmane.org>
Cc: Israel Rukshin <israelr-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
Cc: Max Gurtovoy <maxg-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
Cc: Laurence Oberman <loberman-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>
Cc: Steve Feeley <Steve.Feeley-XdAiOPVOjttBDgjK7y7TUQ@public.gmane.org>
Cc: <stable-u79uwXL29TY76Z2rM5mHXA@public.gmane.org>
---
 drivers/infiniband/ulp/srp/ib_srp.c | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/drivers/infiniband/ulp/srp/ib_srp.c b/drivers/infiniband/ulp/srp/ib_srp.c
index 07e3613e4798..12027e3a4dec 100644
--- a/drivers/infiniband/ulp/srp/ib_srp.c
+++ b/drivers/infiniband/ulp/srp/ib_srp.c
@@ -1892,9 +1892,11 @@ static void srp_process_rsp(struct srp_rdma_ch *ch, struct srp_rsp *rsp)
 		complete(&ch->tsk_mgmt_done);
 	} else {
 		scmnd = scsi_host_find_tag(target->scsi_host, rsp->tag);
-		if (scmnd) {
+		if (scmnd && scmnd->host_scribble) {
 			req = (void *)scmnd->host_scribble;
 			scmnd = srp_claim_req(ch, req, NULL, scmnd);
+		} else {
+			scmnd = NULL;
 		}
 		if (!scmnd) {
 			shost_printk(KERN_ERR, target->scsi_host,
-- 
2.11.0

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[PATCH v2 2/8] IB/srp: Avoid that duplicate responses trigger a kernel bug

[Bart Van Assche]

After srp_process_rsp() returns there is a short time during which
the scsi_host_find_tag() call will return a pointer to the SCSI
command that is being completed. If during that time a duplicate
response is received, avoid that the following call stack appears:

BUG: unable to handle kernel NULL pointer dereference at           (null)
IP: srp_recv_done+0x450/0x6b0 [ib_srp]
Oops: 0000 [#1] SMP
CPU: 10 PID: 0 Comm: swapper/10 Not tainted 4.10.0-rc7-dbg+ #1
Call Trace:
 <IRQ>
 __ib_process_cq+0x4b/0xd0 [ib_core]
 ib_poll_handler+0x1d/0x70 [ib_core]
 irq_poll_softirq+0xba/0x120
 __do_softirq+0xba/0x4c0
 irq_exit+0xbe/0xd0
 smp_apic_timer_interrupt+0x38/0x50
 apic_timer_interrupt+0x90/0xa0
 </IRQ>
RIP: srp_recv_done+0x450/0x6b0 [ib_srp] RSP: ffff88046f483e20

Signed-off-by: Bart Van Assche <bart.vanassche@sandisk.com>
Cc: Israel Rukshin <israelr@mellanox.com>
Cc: Max Gurtovoy <maxg@mellanox.com>
Cc: Laurence Oberman <loberman@redhat.com>
Cc: Steve Feeley <Steve.Feeley@sandisk.com>
Cc: <stable@vger.kernel.org>
---
 drivers/infiniband/ulp/srp/ib_srp.c | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/drivers/infiniband/ulp/srp/ib_srp.c b/drivers/infiniband/ulp/srp/ib_srp.c
index 07e3613e4798..12027e3a4dec 100644
--- a/drivers/infiniband/ulp/srp/ib_srp.c
+++ b/drivers/infiniband/ulp/srp/ib_srp.c
@@ -1892,9 +1892,11 @@ static void srp_process_rsp(struct srp_rdma_ch *ch, struct srp_rsp *rsp)
 		complete(&ch->tsk_mgmt_done);
 	} else {
 		scmnd = scsi_host_find_tag(target->scsi_host, rsp->tag);
-		if (scmnd) {
+		if (scmnd && scmnd->host_scribble) {
 			req = (void *)scmnd->host_scribble;
 			scmnd = srp_claim_req(ch, req, NULL, scmnd);
+		} else {
+			scmnd = NULL;
 		}
 		if (!scmnd) {
 			shost_printk(KERN_ERR, target->scsi_host,
-- 
2.11.0

Re: [PATCH v2 2/8] IB/srp: Avoid that duplicate responses trigger a kernel bug

[Leon Romanovsky]

[-- Attachment #1: Type: text/plain, Size: 1304 bytes --]

On Tue, Feb 14, 2017 at 10:56:30AM -0800, Bart Van Assche wrote:
> After srp_process_rsp() returns there is a short time during which
> the scsi_host_find_tag() call will return a pointer to the SCSI
> command that is being completed. If during that time a duplicate
> response is received, avoid that the following call stack appears:
>
> BUG: unable to handle kernel NULL pointer dereference at           (null)
> IP: srp_recv_done+0x450/0x6b0 [ib_srp]
> Oops: 0000 [#1] SMP
> CPU: 10 PID: 0 Comm: swapper/10 Not tainted 4.10.0-rc7-dbg+ #1
> Call Trace:
>  <IRQ>
>  __ib_process_cq+0x4b/0xd0 [ib_core]
>  ib_poll_handler+0x1d/0x70 [ib_core]
>  irq_poll_softirq+0xba/0x120
>  __do_softirq+0xba/0x4c0
>  irq_exit+0xbe/0xd0
>  smp_apic_timer_interrupt+0x38/0x50
>  apic_timer_interrupt+0x90/0xa0
>  </IRQ>
> RIP: srp_recv_done+0x450/0x6b0 [ib_srp] RSP: ffff88046f483e20
>
> Signed-off-by: Bart Van Assche <bart.vanassche@sandisk.com>
> Cc: Israel Rukshin <israelr@mellanox.com>
> Cc: Max Gurtovoy <maxg@mellanox.com>
> Cc: Laurence Oberman <loberman@redhat.com>
> Cc: Steve Feeley <Steve.Feeley@sandisk.com>
> Cc: <stable@vger.kernel.org>
> ---
>  drivers/infiniband/ulp/srp/ib_srp.c | 4 +++-
>  1 file changed, 3 insertions(+), 1 deletion(-)
>

Thanks,
Reviewed-by: Leon Romanovsky <leonro@mellanox.com>

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

[PATCH v2 3/8] IB/srp: Fix race conditions related to task management

[Bart Van Assche]

Avoid that srp_process_rsp() overwrites the status information
in ch if the SRP target response timed out and processing of
another task management function has already started. Avoid that
issuing multiple task management functions concurrently triggers
list corruption. This patch prevents that the following stack
trace appears in the system log:

WARNING: CPU: 8 PID: 9269 at lib/list_debug.c:52 __list_del_entry_valid+0xbc/0xc0
list_del corruption. prev->next should be ffffc90004bb7b00, but was ffff8804052ecc68
CPU: 8 PID: 9269 Comm: sg_reset Tainted: G        W       4.10.0-rc7-dbg+ #3
Call Trace:
 dump_stack+0x68/0x93
 __warn+0xc6/0xe0
 warn_slowpath_fmt+0x4a/0x50
 __list_del_entry_valid+0xbc/0xc0
 wait_for_completion_timeout+0x12e/0x170
 srp_send_tsk_mgmt+0x1ef/0x2d0 [ib_srp]
 srp_reset_device+0x5b/0x110 [ib_srp]
 scsi_ioctl_reset+0x1c7/0x290
 scsi_ioctl+0x12a/0x420
 sd_ioctl+0x9d/0x100
 blkdev_ioctl+0x51e/0x9f0
 block_ioctl+0x38/0x40
 do_vfs_ioctl+0x8f/0x700
 SyS_ioctl+0x3c/0x70
 entry_SYSCALL_64_fastpath+0x18/0xad

Signed-off-by: Bart Van Assche <bart.vanassche-XdAiOPVOjttBDgjK7y7TUQ@public.gmane.org>
Cc: Israel Rukshin <israelr-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
Cc: Max Gurtovoy <maxg-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
Cc: Laurence Oberman <loberman-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>
Cc: Steve Feeley <Steve.Feeley-XdAiOPVOjttBDgjK7y7TUQ@public.gmane.org>
Cc: <stable-u79uwXL29TY76Z2rM5mHXA@public.gmane.org>
---
 drivers/infiniband/ulp/srp/ib_srp.c | 45 ++++++++++++++++++++++++-------------
 drivers/infiniband/ulp/srp/ib_srp.h |  1 +
 2 files changed, 30 insertions(+), 16 deletions(-)

diff --git a/drivers/infiniband/ulp/srp/ib_srp.c b/drivers/infiniband/ulp/srp/ib_srp.c
index 12027e3a4dec..8d6ce03da1e8 100644
--- a/drivers/infiniband/ulp/srp/ib_srp.c
+++ b/drivers/infiniband/ulp/srp/ib_srp.c
@@ -1884,12 +1884,17 @@ static void srp_process_rsp(struct srp_rdma_ch *ch, struct srp_rsp *rsp)
 	if (unlikely(rsp->tag & SRP_TAG_TSK_MGMT)) {
 		spin_lock_irqsave(&ch->lock, flags);
 		ch->req_lim += be32_to_cpu(rsp->req_lim_delta);
+		if (rsp->tag == ch->tsk_mgmt_tag) {
+			ch->tsk_mgmt_status = -1;
+			if (be32_to_cpu(rsp->resp_data_len) >= 4)
+				ch->tsk_mgmt_status = rsp->data[3];
+			complete(&ch->tsk_mgmt_done);
+		} else {
+			shost_printk(KERN_ERR, target->scsi_host,
+				     "Received tsk mgmt response too late for tag %#llx\n",
+				     rsp->tag);
+		}
 		spin_unlock_irqrestore(&ch->lock, flags);
-
-		ch->tsk_mgmt_status = -1;
-		if (be32_to_cpu(rsp->resp_data_len) >= 4)
-			ch->tsk_mgmt_status = rsp->data[3];
-		complete(&ch->tsk_mgmt_done);
 	} else {
 		scmnd = scsi_host_find_tag(target->scsi_host, rsp->tag);
 		if (scmnd && scmnd->host_scribble) {
@@ -2528,19 +2533,18 @@ srp_change_queue_depth(struct scsi_device *sdev, int qdepth)
 }
 
 static int srp_send_tsk_mgmt(struct srp_rdma_ch *ch, u64 req_tag, u64 lun,
-			     u8 func)
+			     u8 func, u8 *status)
 {
 	struct srp_target_port *target = ch->target;
 	struct srp_rport *rport = target->rport;
 	struct ib_device *dev = target->srp_host->srp_dev->dev;
 	struct srp_iu *iu;
 	struct srp_tsk_mgmt *tsk_mgmt;
+	int res;
 
 	if (!ch->connected || target->qp_in_error)
 		return -1;
 
-	init_completion(&ch->tsk_mgmt_done);
-
 	/*
 	 * Lock the rport mutex to avoid that srp_create_ch_ib() is
 	 * invoked while a task management function is being sent.
@@ -2563,10 +2567,16 @@ static int srp_send_tsk_mgmt(struct srp_rdma_ch *ch, u64 req_tag, u64 lun,
 
 	tsk_mgmt->opcode 	= SRP_TSK_MGMT;
 	int_to_scsilun(lun, &tsk_mgmt->lun);
-	tsk_mgmt->tag		= req_tag | SRP_TAG_TSK_MGMT;
 	tsk_mgmt->tsk_mgmt_func = func;
 	tsk_mgmt->task_tag	= req_tag;
 
+	spin_lock_irq(&ch->lock);
+	ch->tsk_mgmt_tag = (ch->tsk_mgmt_tag + 1) | SRP_TAG_TSK_MGMT;
+	tsk_mgmt->tag = ch->tsk_mgmt_tag;
+	spin_unlock_irq(&ch->lock);
+
+	init_completion(&ch->tsk_mgmt_done);
+
 	ib_dma_sync_single_for_device(dev, iu->dma, sizeof *tsk_mgmt,
 				      DMA_TO_DEVICE);
 	if (srp_post_send(ch, iu, sizeof(*tsk_mgmt))) {
@@ -2575,13 +2585,15 @@ static int srp_send_tsk_mgmt(struct srp_rdma_ch *ch, u64 req_tag, u64 lun,
 
 		return -1;
 	}
+	res = wait_for_completion_timeout(&ch->tsk_mgmt_done,
+					msecs_to_jiffies(SRP_ABORT_TIMEOUT_MS));
+	if (res > 0 && status)
+		*status = ch->tsk_mgmt_status;
 	mutex_unlock(&rport->mutex);
 
-	if (!wait_for_completion_timeout(&ch->tsk_mgmt_done,
-					 msecs_to_jiffies(SRP_ABORT_TIMEOUT_MS)))
-		return -1;
+	WARN_ON_ONCE(res < 0);
 
-	return 0;
+	return res > 0 ? 0 : -1;
 }
 
 static int srp_abort(struct scsi_cmnd *scmnd)
@@ -2607,7 +2619,7 @@ static int srp_abort(struct scsi_cmnd *scmnd)
 	shost_printk(KERN_ERR, target->scsi_host,
 		     "Sending SRP abort for tag %#x\n", tag);
 	if (srp_send_tsk_mgmt(ch, tag, scmnd->device->lun,
-			      SRP_TSK_ABORT_TASK) == 0)
+			      SRP_TSK_ABORT_TASK, NULL) == 0)
 		ret = SUCCESS;
 	else if (target->rport->state == SRP_RPORT_LOST)
 		ret = FAST_IO_FAIL;
@@ -2625,14 +2637,15 @@ static int srp_reset_device(struct scsi_cmnd *scmnd)
 	struct srp_target_port *target = host_to_target(scmnd->device->host);
 	struct srp_rdma_ch *ch;
 	int i;
+	u8 status;
 
 	shost_printk(KERN_ERR, target->scsi_host, "SRP reset_device called\n");
 
 	ch = &target->ch[0];
 	if (srp_send_tsk_mgmt(ch, SRP_TAG_NO_REQ, scmnd->device->lun,
-			      SRP_TSK_LUN_RESET))
+			      SRP_TSK_LUN_RESET, &status))
 		return FAILED;
-	if (ch->tsk_mgmt_status)
+	if (status)
 		return FAILED;
 
 	for (i = 0; i < target->ch_count; i++) {
diff --git a/drivers/infiniband/ulp/srp/ib_srp.h b/drivers/infiniband/ulp/srp/ib_srp.h
index 21c69695f9d4..32ed40db3ca2 100644
--- a/drivers/infiniband/ulp/srp/ib_srp.h
+++ b/drivers/infiniband/ulp/srp/ib_srp.h
@@ -163,6 +163,7 @@ struct srp_rdma_ch {
 	int			max_ti_iu_len;
 	int			comp_vector;
 
+	u64			tsk_mgmt_tag;
 	struct completion	tsk_mgmt_done;
 	u8			tsk_mgmt_status;
 	bool			connected;
-- 
2.11.0

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[PATCH v2 3/8] IB/srp: Fix race conditions related to task management

[Bart Van Assche]

Avoid that srp_process_rsp() overwrites the status information
in ch if the SRP target response timed out and processing of
another task management function has already started. Avoid that
issuing multiple task management functions concurrently triggers
list corruption. This patch prevents that the following stack
trace appears in the system log:

WARNING: CPU: 8 PID: 9269 at lib/list_debug.c:52 __list_del_entry_valid+0xbc/0xc0
list_del corruption. prev->next should be ffffc90004bb7b00, but was ffff8804052ecc68
CPU: 8 PID: 9269 Comm: sg_reset Tainted: G        W       4.10.0-rc7-dbg+ #3
Call Trace:
 dump_stack+0x68/0x93
 __warn+0xc6/0xe0
 warn_slowpath_fmt+0x4a/0x50
 __list_del_entry_valid+0xbc/0xc0
 wait_for_completion_timeout+0x12e/0x170
 srp_send_tsk_mgmt+0x1ef/0x2d0 [ib_srp]
 srp_reset_device+0x5b/0x110 [ib_srp]
 scsi_ioctl_reset+0x1c7/0x290
 scsi_ioctl+0x12a/0x420
 sd_ioctl+0x9d/0x100
 blkdev_ioctl+0x51e/0x9f0
 block_ioctl+0x38/0x40
 do_vfs_ioctl+0x8f/0x700
 SyS_ioctl+0x3c/0x70
 entry_SYSCALL_64_fastpath+0x18/0xad

Signed-off-by: Bart Van Assche <bart.vanassche@sandisk.com>
Cc: Israel Rukshin <israelr@mellanox.com>
Cc: Max Gurtovoy <maxg@mellanox.com>
Cc: Laurence Oberman <loberman@redhat.com>
Cc: Steve Feeley <Steve.Feeley@sandisk.com>
Cc: <stable@vger.kernel.org>
---
 drivers/infiniband/ulp/srp/ib_srp.c | 45 ++++++++++++++++++++++++-------------
 drivers/infiniband/ulp/srp/ib_srp.h |  1 +
 2 files changed, 30 insertions(+), 16 deletions(-)

diff --git a/drivers/infiniband/ulp/srp/ib_srp.c b/drivers/infiniband/ulp/srp/ib_srp.c
index 12027e3a4dec..8d6ce03da1e8 100644
--- a/drivers/infiniband/ulp/srp/ib_srp.c
+++ b/drivers/infiniband/ulp/srp/ib_srp.c
@@ -1884,12 +1884,17 @@ static void srp_process_rsp(struct srp_rdma_ch *ch, struct srp_rsp *rsp)
 	if (unlikely(rsp->tag & SRP_TAG_TSK_MGMT)) {
 		spin_lock_irqsave(&ch->lock, flags);
 		ch->req_lim += be32_to_cpu(rsp->req_lim_delta);
+		if (rsp->tag == ch->tsk_mgmt_tag) {
+			ch->tsk_mgmt_status = -1;
+			if (be32_to_cpu(rsp->resp_data_len) >= 4)
+				ch->tsk_mgmt_status = rsp->data[3];
+			complete(&ch->tsk_mgmt_done);
+		} else {
+			shost_printk(KERN_ERR, target->scsi_host,
+				     "Received tsk mgmt response too late for tag %#llx\n",
+				     rsp->tag);
+		}
 		spin_unlock_irqrestore(&ch->lock, flags);
-
-		ch->tsk_mgmt_status = -1;
-		if (be32_to_cpu(rsp->resp_data_len) >= 4)
-			ch->tsk_mgmt_status = rsp->data[3];
-		complete(&ch->tsk_mgmt_done);
 	} else {
 		scmnd = scsi_host_find_tag(target->scsi_host, rsp->tag);
 		if (scmnd && scmnd->host_scribble) {
@@ -2528,19 +2533,18 @@ srp_change_queue_depth(struct scsi_device *sdev, int qdepth)
 }
 
 static int srp_send_tsk_mgmt(struct srp_rdma_ch *ch, u64 req_tag, u64 lun,
-			     u8 func)
+			     u8 func, u8 *status)
 {
 	struct srp_target_port *target = ch->target;
 	struct srp_rport *rport = target->rport;
 	struct ib_device *dev = target->srp_host->srp_dev->dev;
 	struct srp_iu *iu;
 	struct srp_tsk_mgmt *tsk_mgmt;
+	int res;
 
 	if (!ch->connected || target->qp_in_error)
 		return -1;
 
-	init_completion(&ch->tsk_mgmt_done);
-
 	/*
 	 * Lock the rport mutex to avoid that srp_create_ch_ib() is
 	 * invoked while a task management function is being sent.
@@ -2563,10 +2567,16 @@ static int srp_send_tsk_mgmt(struct srp_rdma_ch *ch, u64 req_tag, u64 lun,
 
 	tsk_mgmt->opcode 	= SRP_TSK_MGMT;
 	int_to_scsilun(lun, &tsk_mgmt->lun);
-	tsk_mgmt->tag		= req_tag | SRP_TAG_TSK_MGMT;
 	tsk_mgmt->tsk_mgmt_func = func;
 	tsk_mgmt->task_tag	= req_tag;
 
+	spin_lock_irq(&ch->lock);
+	ch->tsk_mgmt_tag = (ch->tsk_mgmt_tag + 1) | SRP_TAG_TSK_MGMT;
+	tsk_mgmt->tag = ch->tsk_mgmt_tag;
+	spin_unlock_irq(&ch->lock);
+
+	init_completion(&ch->tsk_mgmt_done);
+
 	ib_dma_sync_single_for_device(dev, iu->dma, sizeof *tsk_mgmt,
 				      DMA_TO_DEVICE);
 	if (srp_post_send(ch, iu, sizeof(*tsk_mgmt))) {
@@ -2575,13 +2585,15 @@ static int srp_send_tsk_mgmt(struct srp_rdma_ch *ch, u64 req_tag, u64 lun,
 
 		return -1;
 	}
+	res = wait_for_completion_timeout(&ch->tsk_mgmt_done,
+					msecs_to_jiffies(SRP_ABORT_TIMEOUT_MS));
+	if (res > 0 && status)
+		*status = ch->tsk_mgmt_status;
 	mutex_unlock(&rport->mutex);
 
-	if (!wait_for_completion_timeout(&ch->tsk_mgmt_done,
-					 msecs_to_jiffies(SRP_ABORT_TIMEOUT_MS)))
-		return -1;
+	WARN_ON_ONCE(res < 0);
 
-	return 0;
+	return res > 0 ? 0 : -1;
 }
 
 static int srp_abort(struct scsi_cmnd *scmnd)
@@ -2607,7 +2619,7 @@ static int srp_abort(struct scsi_cmnd *scmnd)
 	shost_printk(KERN_ERR, target->scsi_host,
 		     "Sending SRP abort for tag %#x\n", tag);
 	if (srp_send_tsk_mgmt(ch, tag, scmnd->device->lun,
-			      SRP_TSK_ABORT_TASK) == 0)
+			      SRP_TSK_ABORT_TASK, NULL) == 0)
 		ret = SUCCESS;
 	else if (target->rport->state == SRP_RPORT_LOST)
 		ret = FAST_IO_FAIL;
@@ -2625,14 +2637,15 @@ static int srp_reset_device(struct scsi_cmnd *scmnd)
 	struct srp_target_port *target = host_to_target(scmnd->device->host);
 	struct srp_rdma_ch *ch;
 	int i;
+	u8 status;
 
 	shost_printk(KERN_ERR, target->scsi_host, "SRP reset_device called\n");
 
 	ch = &target->ch[0];
 	if (srp_send_tsk_mgmt(ch, SRP_TAG_NO_REQ, scmnd->device->lun,
-			      SRP_TSK_LUN_RESET))
+			      SRP_TSK_LUN_RESET, &status))
 		return FAILED;
-	if (ch->tsk_mgmt_status)
+	if (status)
 		return FAILED;
 
 	for (i = 0; i < target->ch_count; i++) {
diff --git a/drivers/infiniband/ulp/srp/ib_srp.h b/drivers/infiniband/ulp/srp/ib_srp.h
index 21c69695f9d4..32ed40db3ca2 100644
--- a/drivers/infiniband/ulp/srp/ib_srp.h
+++ b/drivers/infiniband/ulp/srp/ib_srp.h
@@ -163,6 +163,7 @@ struct srp_rdma_ch {
 	int			max_ti_iu_len;
 	int			comp_vector;
 
+	u64			tsk_mgmt_tag;
 	struct completion	tsk_mgmt_done;
 	u8			tsk_mgmt_status;
 	bool			connected;
-- 
2.11.0

[PATCH v2 4/8] IB/srp: Document locking conventions

[Bart Van Assche]

Use lockdep_assert_held() statements to verify at run-time
whether the proper locks are held.

Signed-off-by: Bart Van Assche <bart.vanassche-XdAiOPVOjttBDgjK7y7TUQ@public.gmane.org>
Cc: Israel Rukshin <israelr-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
Cc: Max Gurtovoy <maxg-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
Cc: Laurence Oberman <loberman-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>
---
 drivers/infiniband/ulp/srp/ib_srp.c | 5 +++++
 1 file changed, 5 insertions(+)

diff --git a/drivers/infiniband/ulp/srp/ib_srp.c b/drivers/infiniband/ulp/srp/ib_srp.c
index 8d6ce03da1e8..01da4400f5bb 100644
--- a/drivers/infiniband/ulp/srp/ib_srp.c
+++ b/drivers/infiniband/ulp/srp/ib_srp.c
@@ -40,6 +40,7 @@
 #include <linux/parser.h>
 #include <linux/random.h>
 #include <linux/jiffies.h>
+#include <linux/lockdep.h>
 #include <rdma/ib_cache.h>
 
 #include <linux/atomic.h>
@@ -1799,6 +1800,8 @@ static struct srp_iu *__srp_get_tx_iu(struct srp_rdma_ch *ch,
 	s32 rsv = (iu_type == SRP_IU_TSK_MGMT) ? 0 : SRP_TSK_MGMT_SQ_SIZE;
 	struct srp_iu *iu;
 
+	lockdep_assert_held(&ch->lock);
+
 	ib_process_cq_direct(ch->send_cq, -1);
 
 	if (list_empty(&ch->free_tx))
@@ -1829,6 +1832,8 @@ static void srp_send_done(struct ib_cq *cq, struct ib_wc *wc)
 		return;
 	}
 
+	lockdep_assert_held(&ch->lock);
+
 	list_add(&iu->list, &ch->free_tx);
 }
 
-- 
2.11.0

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[PATCH v2 5/8] IB/srp: Make a diagnostic message more informative

[Bart Van Assche]

Report the destination port GID if connecting fails.

Signed-off-by: Bart Van Assche <bart.vanassche-XdAiOPVOjttBDgjK7y7TUQ@public.gmane.org>
Cc: Israel Rukshin <israelr-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
Cc: Max Gurtovoy <maxg-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
Cc: Laurence Oberman <loberman-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>
---
 drivers/infiniband/ulp/srp/ib_srp.c | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)

diff --git a/drivers/infiniband/ulp/srp/ib_srp.c b/drivers/infiniband/ulp/srp/ib_srp.c
index 01da4400f5bb..9115d2902ad4 100644
--- a/drivers/infiniband/ulp/srp/ib_srp.c
+++ b/drivers/infiniband/ulp/srp/ib_srp.c
@@ -3436,9 +3436,10 @@ static ssize_t srp_create_target(struct device *dev,
 			ret = srp_connect_ch(ch, multich);
 			if (ret) {
 				shost_printk(KERN_ERR, target->scsi_host,
-					     PFX "Connection %d/%d failed\n",
+					     PFX "Connection %d/%d to %pI6 failed\n",
 					     ch_start + cpu_idx,
-					     target->ch_count);
+					     target->ch_count,
+					     ch->target->orig_dgid.raw);
 				if (node_idx == 0 && cpu_idx == 0) {
 					goto err_disconnect;
 				} else {
-- 
2.11.0

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[PATCH v2 6/8] IB/srp: Improve an error path

[Bart Van Assche]

Avoid that the following message is printed if login fails:

scsi host0: ib_srp: Sending CM DREQ failed

Signed-off-by: Bart Van Assche <bart.vanassche-XdAiOPVOjttBDgjK7y7TUQ@public.gmane.org>
Cc: Israel Rukshin <israelr-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
Cc: Max Gurtovoy <maxg-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
Cc: Laurence Oberman <loberman-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>
---
 drivers/infiniband/ulp/srp/ib_srp.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/drivers/infiniband/ulp/srp/ib_srp.c b/drivers/infiniband/ulp/srp/ib_srp.c
index 9115d2902ad4..b303b4331e98 100644
--- a/drivers/infiniband/ulp/srp/ib_srp.c
+++ b/drivers/infiniband/ulp/srp/ib_srp.c
@@ -3441,7 +3441,7 @@ static ssize_t srp_create_target(struct device *dev,
 					     target->ch_count,
 					     ch->target->orig_dgid.raw);
 				if (node_idx == 0 && cpu_idx == 0) {
-					goto err_disconnect;
+					goto free_ch;
 				} else {
 					srp_free_ch_ib(target, ch);
 					srp_free_req_data(target, ch);
@@ -3488,6 +3488,7 @@ static ssize_t srp_create_target(struct device *dev,
 err_disconnect:
 	srp_disconnect_target(target);
 
+free_ch:
 	for (i = 0; i < target->ch_count; i++) {
 		ch = &target->ch[i];
 		srp_free_ch_ib(target, ch);
-- 
2.11.0

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[PATCH v2 7/8] IB/core: Add support for draining IB_POLL_DIRECT completion queues

[Bart Van Assche]

Signed-off-by: Bart Van Assche <bart.vanassche-XdAiOPVOjttBDgjK7y7TUQ@public.gmane.org>
Cc: Steve Wise <swise-7bPotxP6k4+P2YhJcF5u+vpXobYPEAuW@public.gmane.org>
Cc: Chuck Lever <chuck.lever-QHcLZuEGTsvQT0dZR+AlfA@public.gmane.org>
Cc: Christoph Hellwig <hch-jcswGhMUV9g@public.gmane.org>
Cc: Max Gurtovoy <maxg-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
---
 drivers/infiniband/core/cq.c    |  4 ++--
 drivers/infiniband/core/verbs.c | 35 +++++++++++++++--------------------
 2 files changed, 17 insertions(+), 22 deletions(-)

diff --git a/drivers/infiniband/core/cq.c b/drivers/infiniband/core/cq.c
index a754fc727de5..48242785b323 100644
--- a/drivers/infiniband/core/cq.c
+++ b/drivers/infiniband/core/cq.c
@@ -58,8 +58,8 @@ static int __ib_process_cq(struct ib_cq *cq, int budget)
  * %IB_POLL_DIRECT CQ.  It does not offload CQ processing to a different
  * context and does not ask for completion interrupts from the HCA.
  *
- * Note: for compatibility reasons -1 can be passed in %budget for unlimited
- * polling.  Do not use this feature in new code, it will be removed soon.
+ * Note: do not pass -1 as %budget unless it is guaranteed that the number
+ * of completions that will be processed is small.
  */
 int ib_process_cq_direct(struct ib_cq *cq, int budget)
 {
diff --git a/drivers/infiniband/core/verbs.c b/drivers/infiniband/core/verbs.c
index 71580cc28c9e..42f8927b542c 100644
--- a/drivers/infiniband/core/verbs.c
+++ b/drivers/infiniband/core/verbs.c
@@ -1949,17 +1949,12 @@ static void ib_drain_qp_done(struct ib_cq *cq, struct ib_wc *wc)
  */
 static void __ib_drain_sq(struct ib_qp *qp)
 {
+	struct ib_cq *cq = qp->send_cq;
 	struct ib_qp_attr attr = { .qp_state = IB_QPS_ERR };
 	struct ib_drain_cqe sdrain;
 	struct ib_send_wr swr = {}, *bad_swr;
 	int ret;
 
-	if (qp->send_cq->poll_ctx == IB_POLL_DIRECT) {
-		WARN_ONCE(qp->send_cq->poll_ctx == IB_POLL_DIRECT,
-			  "IB_POLL_DIRECT poll_ctx not supported for drain\n");
-		return;
-	}
-
 	swr.wr_cqe = &sdrain.cqe;
 	sdrain.cqe.done = ib_drain_qp_done;
 	init_completion(&sdrain.done);
@@ -1976,7 +1971,11 @@ static void __ib_drain_sq(struct ib_qp *qp)
 		return;
 	}
 
-	wait_for_completion(&sdrain.done);
+	if (cq->poll_ctx == IB_POLL_DIRECT)
+		while (wait_for_completion_timeout(&sdrain.done, HZ / 10) <= 0)
+			ib_process_cq_direct(cq, -1);
+	else
+		wait_for_completion(&sdrain.done);
 }
 
 /*
@@ -1984,17 +1983,12 @@ static void __ib_drain_sq(struct ib_qp *qp)
  */
 static void __ib_drain_rq(struct ib_qp *qp)
 {
+	struct ib_cq *cq = qp->recv_cq;
 	struct ib_qp_attr attr = { .qp_state = IB_QPS_ERR };
 	struct ib_drain_cqe rdrain;
 	struct ib_recv_wr rwr = {}, *bad_rwr;
 	int ret;
 
-	if (qp->recv_cq->poll_ctx == IB_POLL_DIRECT) {
-		WARN_ONCE(qp->recv_cq->poll_ctx == IB_POLL_DIRECT,
-			  "IB_POLL_DIRECT poll_ctx not supported for drain\n");
-		return;
-	}
-
 	rwr.wr_cqe = &rdrain.cqe;
 	rdrain.cqe.done = ib_drain_qp_done;
 	init_completion(&rdrain.done);
@@ -2011,7 +2005,11 @@ static void __ib_drain_rq(struct ib_qp *qp)
 		return;
 	}
 
-	wait_for_completion(&rdrain.done);
+	if (cq->poll_ctx == IB_POLL_DIRECT)
+		while (wait_for_completion_timeout(&rdrain.done, HZ / 10) <= 0)
+			ib_process_cq_direct(cq, -1);
+	else
+		wait_for_completion(&rdrain.done);
 }
 
 /**
@@ -2028,8 +2026,7 @@ static void __ib_drain_rq(struct ib_qp *qp)
  * ensure there is room in the CQ and SQ for the drain work request and
  * completion.
  *
- * allocate the CQ using ib_alloc_cq() and the CQ poll context cannot be
- * IB_POLL_DIRECT.
+ * allocate the CQ using ib_alloc_cq().
  *
  * ensure that there are no other contexts that are posting WRs concurrently.
  * Otherwise the drain is not guaranteed.
@@ -2057,8 +2054,7 @@ EXPORT_SYMBOL(ib_drain_sq);
  * ensure there is room in the CQ and RQ for the drain work request and
  * completion.
  *
- * allocate the CQ using ib_alloc_cq() and the CQ poll context cannot be
- * IB_POLL_DIRECT.
+ * allocate the CQ using ib_alloc_cq().
  *
  * ensure that there are no other contexts that are posting WRs concurrently.
  * Otherwise the drain is not guaranteed.
@@ -2082,8 +2078,7 @@ EXPORT_SYMBOL(ib_drain_rq);
  * ensure there is room in the CQ(s), SQ, and RQ for drain work requests
  * and completions.
  *
- * allocate the CQs using ib_alloc_cq() and the CQ poll context cannot be
- * IB_POLL_DIRECT.
+ * allocate the CQs using ib_alloc_cq().
  *
  * ensure that there are no other contexts that are posting WRs concurrently.
  * Otherwise the drain is not guaranteed.
-- 
2.11.0

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: [PATCH v2 7/8] IB/core: Add support for draining IB_POLL_DIRECT completion queues

[Steve Wise]

> Subject: [PATCH v2 7/8] IB/core: Add support for draining IB_POLL_DIRECT
> completion queues
> 
> Signed-off-by: Bart Van Assche <bart.vanassche-XdAiOPVOjttBDgjK7y7TUQ@public.gmane.org>
> Cc: Steve Wise <swise-7bPotxP6k4+P2YhJcF5u+vpXobYPEAuW@public.gmane.org>
> Cc: Chuck Lever <chuck.lever-QHcLZuEGTsvQT0dZR+AlfA@public.gmane.org>
> Cc: Christoph Hellwig <hch-jcswGhMUV9g@public.gmane.org>
> Cc: Max Gurtovoy <maxg-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
> ---
>  drivers/infiniband/core/cq.c    |  4 ++--
>  drivers/infiniband/core/verbs.c | 35 +++++++++++++++--------------------
>  2 files changed, 17 insertions(+), 22 deletions(-)
> 

Looks ok to me.

Reviewed-by: Steve Wise <swise-7bPotxP6k4+P2YhJcF5u+vpXobYPEAuW@public.gmane.org>


--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH v2 7/8] IB/core: Add support for draining IB_POLL_DIRECT completion queues

[Leon Romanovsky]

[-- Attachment #1: Type: text/plain, Size: 1438 bytes --]

On Tue, Feb 14, 2017 at 10:56:35AM -0800, Bart Van Assche wrote:
> Signed-off-by: Bart Van Assche <bart.vanassche-XdAiOPVOjttBDgjK7y7TUQ@public.gmane.org>
> Cc: Steve Wise <swise-7bPotxP6k4+P2YhJcF5u+vpXobYPEAuW@public.gmane.org>
> Cc: Chuck Lever <chuck.lever-QHcLZuEGTsvQT0dZR+AlfA@public.gmane.org>
> Cc: Christoph Hellwig <hch-jcswGhMUV9g@public.gmane.org>
> Cc: Max Gurtovoy <maxg-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
> ---
>  drivers/infiniband/core/cq.c    |  4 ++--
>  drivers/infiniband/core/verbs.c | 35 +++++++++++++++--------------------
>  2 files changed, 17 insertions(+), 22 deletions(-)
>
> diff --git a/drivers/infiniband/core/cq.c b/drivers/infiniband/core/cq.c
> index a754fc727de5..48242785b323 100644
> --- a/drivers/infiniband/core/cq.c
> +++ b/drivers/infiniband/core/cq.c
> @@ -58,8 +58,8 @@ static int __ib_process_cq(struct ib_cq *cq, int budget)
>   * %IB_POLL_DIRECT CQ.  It does not offload CQ processing to a different
>   * context and does not ask for completion interrupts from the HCA.
>   *
> - * Note: for compatibility reasons -1 can be passed in %budget for unlimited
> - * polling.  Do not use this feature in new code, it will be removed soon.
> + * Note: do not pass -1 as %budget unless it is guaranteed that the number
> + * of completions that will be processed is small.
>   */

Interesting solution.

Thanks,
Reviewed-by: Leon Romanovsky <leonro-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

[PATCH v2 8/8] IB/srp: Drain the send queue before destroying a QP

[Bart Van Assche]

A quote from the IB spec:

However, if the Consumer does not wait for the Affiliated Asynchronous
Last WQE Reached Event, then WQE and Data Segment leakage may occur.
Therefore, it is good programming practice to tear down a QP that is
associated with an SRQ by using the following process:
* Put the QP in the Error State;
* wait for the Affiliated Asynchronous Last WQE Reached Event;
* either:
  * drain the CQ by invoking the Poll CQ verb and either wait for CQ
    to be empty or the number of Poll CQ operations has exceeded CQ
    capacity size; or
  * post another WR that completes on the same CQ and wait for this WR to return as a WC;
* and then invoke a Destroy QP or Reset QP.

Signed-off-by: Bart Van Assche <bart.vanassche-XdAiOPVOjttBDgjK7y7TUQ@public.gmane.org>
Cc: Christoph Hellwig <hch-jcswGhMUV9g@public.gmane.org>
Cc: Israel Rukshin <israelr-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
Cc: Max Gurtovoy <maxg-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
Cc: Laurence Oberman <loberman-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>
---
 drivers/infiniband/ulp/srp/ib_srp.c | 19 ++++++++++++++-----
 1 file changed, 14 insertions(+), 5 deletions(-)

diff --git a/drivers/infiniband/ulp/srp/ib_srp.c b/drivers/infiniband/ulp/srp/ib_srp.c
index b303b4331e98..39bc0f4e1303 100644
--- a/drivers/infiniband/ulp/srp/ib_srp.c
+++ b/drivers/infiniband/ulp/srp/ib_srp.c
@@ -466,9 +466,13 @@ static struct srp_fr_pool *srp_alloc_fr_pool(struct srp_target_port *target)
  * completion handler can access the queue pair while it is
  * being destroyed.
  */
-static void srp_destroy_qp(struct ib_qp *qp)
+static void srp_destroy_qp(struct srp_rdma_ch *ch, struct ib_qp *qp)
 {
-	ib_drain_rq(qp);
+	spin_lock_irq(&ch->lock);
+	ib_process_cq_direct(ch->send_cq, -1);
+	spin_unlock_irq(&ch->lock);
+
+	ib_drain_qp(qp);
 	ib_destroy_qp(qp);
 }
 
@@ -542,7 +546,7 @@ static int srp_create_ch_ib(struct srp_rdma_ch *ch)
 	}
 
 	if (ch->qp)
-		srp_destroy_qp(ch->qp);
+		srp_destroy_qp(ch, ch->qp);
 	if (ch->recv_cq)
 		ib_free_cq(ch->recv_cq);
 	if (ch->send_cq)
@@ -566,7 +570,7 @@ static int srp_create_ch_ib(struct srp_rdma_ch *ch)
 	return 0;
 
 err_qp:
-	srp_destroy_qp(qp);
+	srp_destroy_qp(ch, qp);
 
 err_send_cq:
 	ib_free_cq(send_cq);
@@ -609,7 +613,7 @@ static void srp_free_ch_ib(struct srp_target_port *target,
 			ib_destroy_fmr_pool(ch->fmr_pool);
 	}
 
-	srp_destroy_qp(ch->qp);
+	srp_destroy_qp(ch, ch->qp);
 	ib_free_cq(ch->send_cq);
 	ib_free_cq(ch->recv_cq);
 
@@ -1822,6 +1826,11 @@ static struct srp_iu *__srp_get_tx_iu(struct srp_rdma_ch *ch,
 	return iu;
 }
 
+/*
+ * Note: if this function is called from inside ib_drain_sq() then it will
+ * be called without ch->lock being held. If ib_drain_sq() dequeues a WQE
+ * with status IB_WC_SUCCESS then that's a bug.
+ */
 static void srp_send_done(struct ib_cq *cq, struct ib_wc *wc)
 {
 	struct srp_iu *iu = container_of(wc->wr_cqe, struct srp_iu, cqe);
-- 
2.11.0

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH v2 0/8] IB/srp bug fixes

[Doug Ledford]

[-- Attachment #1: Type: text/plain, Size: 1888 bytes --]

On Tue, 2017-02-14 at 10:56 -0800, Bart Van Assche wrote:
> Hello Doug,
> 
> The patches in this series are the initiator patches I came up with
> while
> testing the SRP initiator and target drivers. Please consider these
> patches
> for inclusion in the upstream kernel.
> 
> The changes compared to the previous version of this series are:
> - Left out the "Use the IB_DEVICE_SG_GAPS_REG HCA feature if
> supported" patch.
> - Added a patch to stop using memory regions with gaps since Laurence
> ran
>   into an issue with these memory regions that cannot be explained by
>   analzying the SRP initiator code only.
> - Addressed Leon's comments about patch "Avoid that duplicate
> responses
>   trigger a kernel bug" and "Add support for draining IB_POLL_DIRECT
>   completion queues".
> 
> The patches in this series are:
> 
> Bart Van Assche (8):
>   IB/SRP: Avoid using IB_MR_TYPE_SG_GAPS
>   IB/srp: Avoid that duplicate responses trigger a kernel bug
>   IB/srp: Fix race conditions related to task management
>   IB/srp: Document locking conventions
>   IB/srp: Make a diagnostic message more informative
>   IB/srp: Improve an error path
>   IB/core: Add support for draining IB_POLL_DIRECT completion queues
>   IB/srp: Drain the send queue before destroying a QP
> 
>  drivers/infiniband/core/cq.c        |  4 +-
>  drivers/infiniband/core/verbs.c     | 35 ++++++--------
>  drivers/infiniband/ulp/srp/ib_srp.c | 92 ++++++++++++++++++++++++---
> ----------
>  drivers/infiniband/ulp/srp/ib_srp.h |  1 +
>  4 files changed, 77 insertions(+), 55 deletions(-)
> 
> Thanks,
> 
> Bart.

Thanks Bart, series applied.

-- 
Doug Ledford <dledford-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>
    GPG KeyID: B826A3330E572FDD
   
Key fingerprint = AE6B 1BDA 122B 23B4 265B  1274 B826 A333 0E57 2FDD

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: [PATCH v2 1/8] IB/SRP: Avoid using IB_MR_TYPE_SG_GAPS

[Bart Van Assche]

On Wed, 2017-02-15 at 05:50 -0500, Laurence Oberman wrote:
> Bart only has mlx4.

I'll try to find an mlx5 setup.

Bart.

Re: [PATCH v2 1/8] IB/SRP: Avoid using IB_MR_TYPE_SG_GAPS

[Bart Van Assche]

On Wed, 2017-02-15 at 05:50 -0500, Laurence Oberman wrote:
> Bart only has mlx4.

I'll try to find an mlx5 setup.

Bart.

Re: [PATCH v2 1/8] IB/SRP: Avoid using IB_MR_TYPE_SG_GAPS

[Laurence Oberman]

----- Original Message -----
> From: "Bart Van Assche" <Bart.VanAssche-XdAiOPVOjttBDgjK7y7TUQ@public.gmane.org>
> To: leonro-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org, loberman-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org
> Cc: artemyko-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org, linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org, israelr-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org, "yuval shaia" <yuval.shaia-QHcLZuEGTsvQT0dZR+AlfA@public.gmane.org>,
> stable-u79uwXL29TY76Z2rM5mHXA@public.gmane.org, markb-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org, maxg-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org, dledford-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org
> Sent: Wednesday, February 15, 2017 10:31:20 AM
> Subject: Re: [PATCH v2 1/8] IB/SRP: Avoid using IB_MR_TYPE_SG_GAPS
> 
> On Wed, 2017-02-15 at 05:50 -0500, Laurence Oberman wrote:
> > Bart only has mlx4.
> 
> I'll try to find an mlx5 setup.
> 
> Bart.

Hello Bart

Its fine, I tested it already. See later email from me.
I am good to do the mlx5 tests, I have this test bed in place permanently.

Thanks
Laurence
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH v2 1/8] IB/SRP: Avoid using IB_MR_TYPE_SG_GAPS

[Laurence Oberman]

----- Original Message -----
> From: "Bart Van Assche" <Bart.VanAssche@sandisk.com>
> To: leonro@mellanox.com, loberman@redhat.com
> Cc: artemyko@mellanox.com, linux-rdma@vger.kernel.org, israelr@mellanox.com, "yuval shaia" <yuval.shaia@oracle.com>,
> stable@vger.kernel.org, markb@mellanox.com, maxg@mellanox.com, dledford@redhat.com
> Sent: Wednesday, February 15, 2017 10:31:20 AM
> Subject: Re: [PATCH v2 1/8] IB/SRP: Avoid using IB_MR_TYPE_SG_GAPS
> 
> On Wed, 2017-02-15 at 05:50 -0500, Laurence Oberman wrote:
> > Bart only has mlx4.
> 
> I'll try to find an mlx5 setup.
> 
> Bart.

Hello Bart

Its fine, I tested it already. See later email from me.
I am good to do the mlx5 tests, I have this test bed in place permanently.

Thanks
Laurence