[PATCH, untested] mlx5: Avoid that mlx5_ib_sg_to_klms() overflows the klms[] array

[PATCH, untested] mlx5: Avoid that mlx5_ib_sg_to_klms() overflows the klms[] array

[Bart Van Assche]

ib_map_mr_sg() can pass an SG-list to .map_mr_sg() that is larger
than what fits into a single MR. .map_mr_sg() must not attempt to
map more SG-list elements than what fits into a single MR.
Hence make sure that mlx5_ib_sg_to_klms() does not write outside
the MR klms[] array.

Fixes: b005d3164713 ("mlx5: Add arbitrary sg list support")
Signed-off-by: Bart Van Assche <bart.vanassche-XdAiOPVOjttBDgjK7y7TUQ@public.gmane.org>
Reviewed-by: Max Gurtovoy <maxg-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
Cc: Sagi Grimberg <sagi-NQWnxTmZq1alnMjI0IkVqw@public.gmane.org>
Cc: Leon Romanovsky <leonro-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
Cc: Israel Rukshin <israelr-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
Cc: <stable-u79uwXL29TY76Z2rM5mHXA@public.gmane.org>
---
 drivers/infiniband/hw/mlx5/mr.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/drivers/infiniband/hw/mlx5/mr.c b/drivers/infiniband/hw/mlx5/mr.c
index d9c6c0ea750b..99beacfc4716 100644
--- a/drivers/infiniband/hw/mlx5/mr.c
+++ b/drivers/infiniband/hw/mlx5/mr.c
@@ -1777,7 +1777,7 @@ mlx5_ib_sg_to_klms(struct mlx5_ib_mr *mr,
 	mr->ndescs = sg_nents;
 
 	for_each_sg(sgl, sg, sg_nents, i) {
-		if (unlikely(i > mr->max_descs))
+		if (unlikely(i >= mr->max_descs))
 			break;
 		klms[i].va = cpu_to_be64(sg_dma_address(sg) + sg_offset);
 		klms[i].bcount = cpu_to_be32(sg_dma_len(sg) - sg_offset);
-- 
2.12.2

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH, untested] mlx5: Avoid that mlx5_ib_sg_to_klms() overflows the klms[] array

[Laurence Oberman]

----- Original Message -----
> From: "Bart Van Assche" <bart.vanassche-XdAiOPVOjttBDgjK7y7TUQ@public.gmane.org>
> To: "Doug Ledford" <dledford-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>
> Cc: "Max Gurtovoy" <maxg-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>, "Sagi Grimberg" <sagi-NQWnxTmZq1alnMjI0IkVqw@public.gmane.org>, "Leon Romanovsky" <leonro-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>,
> "Israel Rukshin" <israelr-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>, "Laurence Oberman" <loberman-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>, linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> Sent: Monday, April 24, 2017 6:15:28 PM
> Subject: [PATCH, untested] mlx5: Avoid that mlx5_ib_sg_to_klms() overflows the klms[] array
> 
> ib_map_mr_sg() can pass an SG-list to .map_mr_sg() that is larger
> than what fits into a single MR. .map_mr_sg() must not attempt to
> map more SG-list elements than what fits into a single MR.
> Hence make sure that mlx5_ib_sg_to_klms() does not write outside
> the MR klms[] array.
> 
> Fixes: b005d3164713 ("mlx5: Add arbitrary sg list support")
> Signed-off-by: Bart Van Assche <bart.vanassche-XdAiOPVOjttBDgjK7y7TUQ@public.gmane.org>
> Reviewed-by: Max Gurtovoy <maxg-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
> Cc: Sagi Grimberg <sagi-NQWnxTmZq1alnMjI0IkVqw@public.gmane.org>
> Cc: Leon Romanovsky <leonro-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
> Cc: Israel Rukshin <israelr-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
> Cc: <stable-u79uwXL29TY76Z2rM5mHXA@public.gmane.org>
> ---
>  drivers/infiniband/hw/mlx5/mr.c | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/drivers/infiniband/hw/mlx5/mr.c
> b/drivers/infiniband/hw/mlx5/mr.c
> index d9c6c0ea750b..99beacfc4716 100644
> --- a/drivers/infiniband/hw/mlx5/mr.c
> +++ b/drivers/infiniband/hw/mlx5/mr.c
> @@ -1777,7 +1777,7 @@ mlx5_ib_sg_to_klms(struct mlx5_ib_mr *mr,
>  	mr->ndescs = sg_nents;
>  
>  	for_each_sg(sgl, sg, sg_nents, i) {
> -		if (unlikely(i > mr->max_descs))
> +		if (unlikely(i >= mr->max_descs))
>  			break;
>  		klms[i].va = cpu_to_be64(sg_dma_address(sg) + sg_offset);
>  		klms[i].bcount = cpu_to_be32(sg_dma_len(sg) - sg_offset);
> --
> 2.12.2
> 
> 

Thanks Bart as always.
Will get this tested this week,

Regards
Laurence
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH, untested] mlx5: Avoid that mlx5_ib_sg_to_klms() overflows the klms[] array

[Bart Van Assche]

On Mon, 2017-04-24 at 18:39 -0400, Laurence Oberman wrote:
> Will get this tested this week,

Thanks Laurence. BTW, if you want to test this patch with the SRP protocol
you will also have to revert commit d6c58dc40fec ("IB/SRP: Avoid using
IB_MR_TYPE_SG_GAPS"). The code path touched by this patch is namely only
relevant for IB_MR_TYPE_SG_GAPS memory regions. Currently the SRP initiator
driver does not use that MR type. Reverting the aforementioned commit will
make the SRP initiator driver use that MR type.

Please also apply Sagi's "mlx5: Fix mlx5_ib_map_mr_sg mr length" patch
before starting any tests.

Thanks,

Bart.--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH, untested] mlx5: Avoid that mlx5_ib_sg_to_klms() overflows the klms[] array

[Laurence Oberman]

----- Original Message -----
> From: "Bart Van Assche" <Bart.VanAssche-XdAiOPVOjttBDgjK7y7TUQ@public.gmane.org>
> To: loberman-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org
> Cc: maxg-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org, israelr-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org, leonro-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org, linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org, dledford-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org,
> sagi-NQWnxTmZq1alnMjI0IkVqw@public.gmane.org
> Sent: Monday, April 24, 2017 6:46:30 PM
> Subject: Re: [PATCH, untested] mlx5: Avoid that mlx5_ib_sg_to_klms() overflows the klms[] array
> 
> On Mon, 2017-04-24 at 18:39 -0400, Laurence Oberman wrote:
> > Will get this tested this week,
> 
> Thanks Laurence. BTW, if you want to test this patch with the SRP protocol
> you will also have to revert commit d6c58dc40fec ("IB/SRP: Avoid using
> IB_MR_TYPE_SG_GAPS"). The code path touched by this patch is namely only
> relevant for IB_MR_TYPE_SG_GAPS memory regions. Currently the SRP initiator
> driver does not use that MR type. Reverting the aforementioned commit will
> make the SRP initiator driver use that MR type.
> 
> Please also apply Sagi's "mlx5: Fix mlx5_ib_map_mr_sg mr length" patch
> before starting any tests.
> 
> Thanks,
> 
> Bart.--
> To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 

Understood
Regards
Laurence
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH, untested] mlx5: Avoid that mlx5_ib_sg_to_klms() overflows the klms[] array

[Leon Romanovsky]

[-- Attachment #1: Type: text/plain, Size: 1072 bytes --]

On Mon, Apr 24, 2017 at 03:15:28PM -0700, Bart Van Assche wrote:
> ib_map_mr_sg() can pass an SG-list to .map_mr_sg() that is larger
> than what fits into a single MR. .map_mr_sg() must not attempt to
> map more SG-list elements than what fits into a single MR.
> Hence make sure that mlx5_ib_sg_to_klms() does not write outside
> the MR klms[] array.
>
> Fixes: b005d3164713 ("mlx5: Add arbitrary sg list support")
> Signed-off-by: Bart Van Assche <bart.vanassche-XdAiOPVOjttBDgjK7y7TUQ@public.gmane.org>
> Reviewed-by: Max Gurtovoy <maxg-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
> Cc: Sagi Grimberg <sagi-NQWnxTmZq1alnMjI0IkVqw@public.gmane.org>
> Cc: Leon Romanovsky <leonro-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
> Cc: Israel Rukshin <israelr-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
> Cc: <stable-u79uwXL29TY76Z2rM5mHXA@public.gmane.org>
> ---
>  drivers/infiniband/hw/mlx5/mr.c | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
>

Bart,

Thanks a lot, it indeed looks right.
Acked-by: Leon Romanovsky <leonro-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>

Thanks

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [PATCH, untested] mlx5: Avoid that mlx5_ib_sg_to_klms() overflows the klms[] array

[Laurence Oberman]

----- Original Message -----
> From: "Leon Romanovsky" <leonro-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
> To: "Bart Van Assche" <bart.vanassche-XdAiOPVOjttBDgjK7y7TUQ@public.gmane.org>
> Cc: "Doug Ledford" <dledford-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>, "Max Gurtovoy" <maxg-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>, "Sagi Grimberg" <sagi-NQWnxTmZq1alnMjI0IkVqw@public.gmane.org>,
> "Israel Rukshin" <israelr-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>, "Laurence Oberman" <loberman-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>, linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> Sent: Tuesday, April 25, 2017 1:58:49 PM
> Subject: Re: [PATCH, untested] mlx5: Avoid that mlx5_ib_sg_to_klms() overflows the klms[] array
> 
> On Mon, Apr 24, 2017 at 03:15:28PM -0700, Bart Van Assche wrote:
> > ib_map_mr_sg() can pass an SG-list to .map_mr_sg() that is larger
> > than what fits into a single MR. .map_mr_sg() must not attempt to
> > map more SG-list elements than what fits into a single MR.
> > Hence make sure that mlx5_ib_sg_to_klms() does not write outside
> > the MR klms[] array.
> >
> > Fixes: b005d3164713 ("mlx5: Add arbitrary sg list support")
> > Signed-off-by: Bart Van Assche <bart.vanassche-XdAiOPVOjttBDgjK7y7TUQ@public.gmane.org>
> > Reviewed-by: Max Gurtovoy <maxg-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
> > Cc: Sagi Grimberg <sagi-NQWnxTmZq1alnMjI0IkVqw@public.gmane.org>
> > Cc: Leon Romanovsky <leonro-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
> > Cc: Israel Rukshin <israelr-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
> > Cc: <stable-u79uwXL29TY76Z2rM5mHXA@public.gmane.org>
> > ---
> >  drivers/infiniband/hw/mlx5/mr.c | 2 +-
> >  1 file changed, 1 insertion(+), 1 deletion(-)
> >
> 
> Bart,
> 
> Thanks a lot, it indeed looks right.
> Acked-by: Leon Romanovsky <leonro-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
> 
> Thanks
> 


Hello Bart, Leon, Max and Israel.

I cloned off Barts tree.

git clone https://github.com/bvanassche/linux
cd linux
git checkout block-scsi-for-next

I checked all patches were in for this test.

a83e404 IB/srp: Reenable IB_MR_TYPE_SG_GAPS
dfa5a2b mlx5: Avoid that mlx5_ib_sg_to_klms() overflows the klms[] array
f759c80 mlx5: Fix mlx5_ib_map_mr_sg mr lengt

Built and tested the kernel.

However this issue is not resolved :(

[ 2707.931909] scsi host1: ib_srp: failed RECV status WR flushed (5) for CQE ffff8817edca86b0
[ 2708.089806] mlx5_0:dump_cqe:262:(pid 20129): dump error cqe
[ 2708.121342] 00000000 00000000 00000000 00000000
[ 2708.147104] 00000000 00000000 00000000 00000000
[ 2708.172633] 00000000 00000000 00000000 00000000
[ 2708.198702] 00000000 0f007806 2500002a 14a527d0
[ 2732.434127] scsi host1: ib_srp: reconnect succeeded
[ 2733.048023] scsi host1: ib_srp: failed RECV status WR flushed (5) for CQE ffff8817ed0a9c30

[root@localhost ~]# [ 2746.413277] mlx5_0:dump_cqe:262:(pid 15877): dump error cqe
[ 2746.443240] 00000000 00000000 00000000 00000000
[ 2746.469323] 00000000 00000000 00000000 00000000
[ 2746.495310] 00000000 00000000 00000000 00000000
[ 2746.521407] 00000000 0f007806 25000032 003c7ad0
[ 2752.445899] scsi host1: ib_srp: reconnect succeeded
[ 2752.481835] scsi host1: ib_srp: failed RECV status WR flushed (5) for CQE ffff8817ed0a9cf0
[ 2763.267386] mlx5_0:dump_cqe:262:(pid 15877): dump error cqe
[ 2763.297826] 00000000 00000000 00000000 00000000
[ 2763.323352] 00000000 00000000 00000000 00000000
[ 2763.348722] 00000000 00000000 00000000 00000000
[ 2763.374681] 00000000 0f007806 2500003a 00084bd0

[root@localhost ~]# [ 2769.385203] fast_io_fail_tmo expired for SRP port-1:1 / host1.
[ 2769.415956] scsi host1: ib_srp: reconnect succeeded
[ 2769.450258] scsi host1: ib_srp: failed RECV status WR flushed (5) for CQE ffff8817ed0a9cf0
[ 2780.064627] mlx5_0:dump_cqe:262:(pid 18771): dump error cqe
[ 2780.093520] 00000000 00000000 00000000 00000000
[ 2780.120067] 00000000 00000000 00000000 00000000
[ 2780.145575] 00000000 00000000 00000000 00000000
[ 2780.171153] 00000000 0f007806 25000042 000833d0
[ 2785.923399] scsi host1: ib_srp: reconnect succeeded
[ 2785.957504] scsi host1: ib_srp: failed RECV status WR flushed (5) for CQE ffff8817ed0a9cf0
[ 2796.463426] mlx5_0:dump_cqe:262:(pid 18771): dump error cqe
[ 2796.495257] 00000000 00000000 00000000 00000000
[ 2796.521506] 00000000 00000000 00000000 00000000
[ 2796.547640] 00000000 00000000 00000000 00000000
[ 2796.573120] 00000000 0f007806 2500004a 00083bd0
[ 2802.562578] scsi host1: ib_srp: reconnect succeeded
[ 2802.596880] scsi host1: ib_srp: failed RECV status WR flushed (5) for CQE ffff8817ed0a9cf0

Regards
Laurence
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH, untested] mlx5: Avoid that mlx5_ib_sg_to_klms() overflows the klms[] array

[Bart Van Assche]

On Tue, 2017-04-25 at 16:37 -0400, Laurence Oberman wrote:
> Hello Bart, Leon, Max and Israel.
> 
> I cloned off Barts tree.
> 
> git clone https://github.com/bvanassche/linux
> cd linux
> git checkout block-scsi-for-next
> 
> I checked all patches were in for this test.
> 
> a83e404 IB/srp: Reenable IB_MR_TYPE_SG_GAPS
> dfa5a2b mlx5: Avoid that mlx5_ib_sg_to_klms() overflows the klms[] array
> f759c80 mlx5: Fix mlx5_ib_map_mr_sg mr lengt
> 
> Built and tested the kernel.
> 
> However this issue is not resolved :(
> 
> [ 2707.931909] scsi host1: ib_srp: failed RECV status WR flushed (5) for CQE ffff8817edca86b0
> [ 2708.089806] mlx5_0:dump_cqe:262:(pid 20129): dump error cqe
> [ 2708.121342] 00000000 00000000 00000000 00000000
> [ 2708.147104] 00000000 00000000 00000000 00000000
> [ 2708.172633] 00000000 00000000 00000000 00000000
> [ 2708.198702] 00000000 0f007806 2500002a 14a527d0
> [ 2732.434127] scsi host1: ib_srp: reconnect succeeded
> [ 2733.048023] scsi host1: ib_srp: failed RECV status WR flushed (5) for CQE ffff8817ed0a9c30

Hello Laurence,

Thank you for having run this test. But are you aware that if a flush error
is reported at the initiator side that does not necessarily mean that there
is a bug at the initiator side? If e.g. the target system would initiate a
disconnect that would also trigger this kind of flush errors. What kind of
SRP target system was used in this test? Were the clocks of initiator and
target system synchronized? Are the logs of the target system available? If
so, can you have a look whether anything interesting can be found in the
target log around the time the initiator reported the flush error?

Thanks,

Bart.

Re: [PATCH, untested] mlx5: Avoid that mlx5_ib_sg_to_klms() overflows the klms[] array

[Laurence Oberman]

----- Original Message -----
> From: "Bart Van Assche" <Bart.VanAssche-XdAiOPVOjttBDgjK7y7TUQ@public.gmane.org>
> To: leonro-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org, loberman-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org
> Cc: maxg-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org, israelr-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org, linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org, dledford-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org, sagi-NQWnxTmZq1alnMjI0IkVqw@public.gmane.org
> Sent: Tuesday, April 25, 2017 11:39:12 PM
> Subject: Re: [PATCH, untested] mlx5: Avoid that mlx5_ib_sg_to_klms() overflows the klms[] array
> 
> On Tue, 2017-04-25 at 16:37 -0400, Laurence Oberman wrote:
> > Hello Bart, Leon, Max and Israel.
> > 
> > I cloned off Barts tree.
> > 
> > git clone https://github.com/bvanassche/linux
> > cd linux
> > git checkout block-scsi-for-next
> > 
> > I checked all patches were in for this test.
> > 
> > a83e404 IB/srp: Reenable IB_MR_TYPE_SG_GAPS
> > dfa5a2b mlx5: Avoid that mlx5_ib_sg_to_klms() overflows the klms[] array
> > f759c80 mlx5: Fix mlx5_ib_map_mr_sg mr lengt
> > 
> > Built and tested the kernel.
> > 
> > However this issue is not resolved :(
> > 
> > [ 2707.931909] scsi host1: ib_srp: failed RECV status WR flushed (5) for
> > CQE ffff8817edca86b0
> > [ 2708.089806] mlx5_0:dump_cqe:262:(pid 20129): dump error cqe
> > [ 2708.121342] 00000000 00000000 00000000 00000000
> > [ 2708.147104] 00000000 00000000 00000000 00000000
> > [ 2708.172633] 00000000 00000000 00000000 00000000
> > [ 2708.198702] 00000000 0f007806 2500002a 14a527d0
> > [ 2732.434127] scsi host1: ib_srp: reconnect succeeded
> > [ 2733.048023] scsi host1: ib_srp: failed RECV status WR flushed (5) for
> > CQE ffff8817ed0a9c30
> 
> Hello Laurence,
> 
> Thank you for having run this test. But are you aware that if a flush error
> is reported at the initiator side that does not necessarily mean that there
> is a bug at the initiator side? If e.g. the target system would initiate a
> disconnect that would also trigger this kind of flush errors. What kind of
> SRP target system was used in this test? Were the clocks of initiator and
> target system synchronized? Are the logs of the target system available? If
> so, can you have a look whether anything interesting can be found in the
> target log around the time the initiator reported the flush error?
> 
> Thanks,
> 
> Bart.

Hi Bart

Its the same target that is stable for all other tests.
This is the same issue I originally reported when we then reverted the SG+GAPS.
Remember when I reverted that we were stable again.

This happens on the initiator first

[root@localhost ~]# [  512.375904] mlx5_0:dump_cqe:262:(pid 4653): dump error cqe
[  512.376648] scsi host2: ib_srp: failed RECV status WR flushed (5) for CQE ffff8817c596f770
[  512.454276] 00000000 00000000 00000000 00000000
[  512.478734] 00000000 00000000 00000000 00000000
[  512.504170] 00000000 00000000 00000000 00000000
[  512.529457] 00000000 0f007806 2500002a 0548e2d0
[  532.128455] scsi host2: ib_srp: reconnect succeeded
[  532.232126] scsi host2: ib_srp: failed RECV status WR flushed (5) for CQE ffff880bf2bb3bf0
[  532.780107] mlx5_0:dump_cqe:262:(pid 511): dump error cqe
[  532.811863] 00000000 00000000 00000000 00000000
[  532.837984] 00000000 00000000 00000000 00000000
[  532.863955] 00000000 00000000 00000000 00000000
[  532.889885] 00000000 0f007806 25000032 00683bd0

Only afterwards do I see the target complain

[root@fedstorage ~]# [  537.105985] ib_srpt Received CM TimeWait exit for ch 0x4e6e72000390fe7c7cfe900300726ed2-48.
[  537.152767] ib_srpt Received CM TimeWait exit for ch 0x4e6e72000390fe7c7cfe900300726ed2-47.
[  537.200585] ib_srpt Received CM TimeWait exit for ch 0x4e6e72000390fe7c7cfe900300726ed2-46.
[  537.247864] ib_srpt Received CM TimeWait exit for ch 0x4e6e72000390fe7c7cfe900300726ed2-45.
[  537.296822] ib_srpt Received CM TimeWait exit for ch 0x4e6e72000390fe7c7cfe900300726ed2-44.
[  537.345001] ib_srpt Received CM TimeWait exit for ch 0x4e6e72000390fe7c7cfe900300726ed2-43.
[  537.394146] ib_srpt Received CM TimeWait exit for ch 0x4e6e72000390fe7c7cfe900300726ed2-42.
[  537.442148] ib_srpt Received CM TimeWait exit for ch 0x4e6e72000390fe7c7cfe900300726ed2-41.
[  537.490011] ib_srpt sending response for ioctx 0xffff8800951ed800 failed with status 5
[  539.774018] ib_srpt Received SRP_LOGIN_REQ with i_port_id 0x4e6e72000390fe7c:0x7cfe900300726ed2, t_port_id 0x7cfe900300726e4e:0x7cfe900300726e4e and it_iu_len 4148 on port 1 (guid=0xfe80000000000000:0x7cfe900300726e4e)
[  539.887987] ib_srpt Received SRP_LOGIN_REQ with i_port_id 0x4e6e72000390fe7c:0x7cfe900300726ed2, t_port_id 0x7cfe900300726e4e:0x7cfe900300726e4e and it_iu_len 4148 on port 1 (guid=0xfe80000000000000:0x7cfe900300726e4e)
[  540.001241] ib_srpt Received SRP_LOGIN_REQ with i_port_id 0x4e6e72000390fe7c:0x7cfe900300726ed2, t_port_id 0x7cfe900300726e4e:0x7cfe900300726e4e and it_iu_len 4148 on port 1 (guid=0xfe80000000000000:0x7cfe900300726e4e)
[  540.111455] ib_srpt Received SRP_LOGIN_REQ with i_port_id 0x4e6e72000390fe7c:0x7cfe900300726ed2, t_port_id 0x7cfe900300726e4e:0x7cfe900300726e4e and it_iu_len 4148 on port 1 (guid=0xfe80000000000000:0x7cfe900300726e4e)
[  540.224780] ib_srpt Received SRP_LOGIN_REQ with i_port_id 0x4e6e72000390fe7c:0x7cfe900300726ed2, t_port_id 0x7cfe900300726e4e:0x7cfe900300726e4e and it_iu_len 4148 on port 1 (guid=0xfe80000000000000:0x7cfe900300726e4e)
[  540.340522] ib_srpt Received SRP_LOGIN_REQ with i_port_id 0x4e6e72000390fe7c:0x7cfe900300726ed2, t_port_id 0x7cfe900300726e4e:0x7cfe900300726e4e and it_iu_len 4148 on port 1 (guid=0xfe80000000000000:0x7cfe900300726e4e)
[  540.453736] ib_srpt Received SRP_LOGIN_REQ with i_port_id 0x4e6e72000390fe7c:0x7cfe900300726ed2, t_port_id 0x7cfe900300726e4e:0x7cfe900300726e4e and it_iu_len 4148 on port 1 (guid=0xfe80000000000000:0x7cfe900300726e4e)
[  540.567043] ib_srpt Received SRP_LOGIN_REQ with i_port_id 0x4e6e72000390fe7c:0x7cfe900300726ed2, t_port_id 0x7cfe900300726e4e:0x7cfe900300726e4e and it_iu_len 4148 on port 1 (guid=0xfe80000000000000:0x7cfe900300726e4e)

Thanks
Laurence

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH, untested] mlx5: Avoid that mlx5_ib_sg_to_klms() overflows the klms[] array

[Bart Van Assche]

On Wed, 2017-04-26 at 07:46 -0400, Laurence Oberman wrote:
> Its the same target that is stable for all other tests.
> This is the same issue I originally reported when we then reverted the SG+GAPS.
> Remember when I reverted that we were stable again.
> 
> This happens on the initiator first
> 
> [...]
> 
> Only afterwards do I see the target complain
> 
> [...]

Thanks Laurence. I think this confirms that we have to continue analyzing the
initiator side further.

Bart.--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH, untested] mlx5: Avoid that mlx5_ib_sg_to_klms() overflows the klms[] array

[Leon Romanovsky]

[-- Attachment #1: Type: text/plain, Size: 3330 bytes --]

On Tue, Apr 25, 2017 at 04:37:35PM -0400, Laurence Oberman wrote:
>
>
> ----- Original Message -----
> > From: "Leon Romanovsky" <leonro-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
> > To: "Bart Van Assche" <bart.vanassche-XdAiOPVOjttBDgjK7y7TUQ@public.gmane.org>
> > Cc: "Doug Ledford" <dledford-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>, "Max Gurtovoy" <maxg-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>, "Sagi Grimberg" <sagi-NQWnxTmZq1alnMjI0IkVqw@public.gmane.org>,
> > "Israel Rukshin" <israelr-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>, "Laurence Oberman" <loberman-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>, linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> > Sent: Tuesday, April 25, 2017 1:58:49 PM
> > Subject: Re: [PATCH, untested] mlx5: Avoid that mlx5_ib_sg_to_klms() overflows the klms[] array
> >
> > On Mon, Apr 24, 2017 at 03:15:28PM -0700, Bart Van Assche wrote:
> > > ib_map_mr_sg() can pass an SG-list to .map_mr_sg() that is larger
> > > than what fits into a single MR. .map_mr_sg() must not attempt to
> > > map more SG-list elements than what fits into a single MR.
> > > Hence make sure that mlx5_ib_sg_to_klms() does not write outside
> > > the MR klms[] array.
> > >
> > > Fixes: b005d3164713 ("mlx5: Add arbitrary sg list support")
> > > Signed-off-by: Bart Van Assche <bart.vanassche-XdAiOPVOjttBDgjK7y7TUQ@public.gmane.org>
> > > Reviewed-by: Max Gurtovoy <maxg-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
> > > Cc: Sagi Grimberg <sagi-NQWnxTmZq1alnMjI0IkVqw@public.gmane.org>
> > > Cc: Leon Romanovsky <leonro-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
> > > Cc: Israel Rukshin <israelr-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
> > > Cc: <stable-u79uwXL29TY76Z2rM5mHXA@public.gmane.org>
> > > ---
> > >  drivers/infiniband/hw/mlx5/mr.c | 2 +-
> > >  1 file changed, 1 insertion(+), 1 deletion(-)
> > >
> >
> > Bart,
> >
> > Thanks a lot, it indeed looks right.
> > Acked-by: Leon Romanovsky <leonro-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
> >
> > Thanks
> >
>
>
> Hello Bart, Leon, Max and Israel.
>
> I cloned off Barts tree.
>
> git clone https://github.com/bvanassche/linux
> cd linux
> git checkout block-scsi-for-next
>
> I checked all patches were in for this test.
>
> a83e404 IB/srp: Reenable IB_MR_TYPE_SG_GAPS
> dfa5a2b mlx5: Avoid that mlx5_ib_sg_to_klms() overflows the klms[] array
> f759c80 mlx5: Fix mlx5_ib_map_mr_sg mr lengt
>
> Built and tested the kernel.
>
> However this issue is not resolved :(
>
> [ 2707.931909] scsi host1: ib_srp: failed RECV status WR flushed (5) for CQE ffff8817edca86b0
> [ 2708.089806] mlx5_0:dump_cqe:262:(pid 20129): dump error cqe
> [ 2708.121342] 00000000 00000000 00000000 00000000
> [ 2708.147104] 00000000 00000000 00000000 00000000
> [ 2708.172633] 00000000 00000000 00000000 00000000
> [ 2708.198702] 00000000 0f007806 2500002a 14a527d0

Parsed version:
	hw_error_syndrome                : 0xf
	hw_syndrome_type                 : 0x0
	vendor_error_syndrome            : 0x78
	syndrome                         : MEMORY_WINDOW_BIND_ERROR (0x6)
	s_wqe_opcode                     : UMR (0x25)
	opcode                           : REQUESTOR_ERROR (0xd)
	cqe_format                       : NO_INLINE_DATA (0x0)
	owner                            : 0x0

Description:
	umr.klm_octoword_count > mkey.mtt_octoword_count

Sagi, Max,
Any idea where can it be?

Thanks

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [PATCH, untested] mlx5: Avoid that mlx5_ib_sg_to_klms() overflows the klms[] array

[Max Gurtovoy]

On 4/26/2017 9:16 AM, Leon Romanovsky wrote:
> On Tue, Apr 25, 2017 at 04:37:35PM -0400, Laurence Oberman wrote:
>>
>>
>> ----- Original Message -----
>>> From: "Leon Romanovsky" <leonro-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
>>> To: "Bart Van Assche" <bart.vanassche-XdAiOPVOjttBDgjK7y7TUQ@public.gmane.org>
>>> Cc: "Doug Ledford" <dledford-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>, "Max Gurtovoy" <maxg-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>, "Sagi Grimberg" <sagi-NQWnxTmZq1alnMjI0IkVqw@public.gmane.org>,
>>> "Israel Rukshin" <israelr-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>, "Laurence Oberman" <loberman-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>, linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
>>> Sent: Tuesday, April 25, 2017 1:58:49 PM
>>> Subject: Re: [PATCH, untested] mlx5: Avoid that mlx5_ib_sg_to_klms() overflows the klms[] array
>>>
>>> On Mon, Apr 24, 2017 at 03:15:28PM -0700, Bart Van Assche wrote:
>>>> ib_map_mr_sg() can pass an SG-list to .map_mr_sg() that is larger
>>>> than what fits into a single MR. .map_mr_sg() must not attempt to
>>>> map more SG-list elements than what fits into a single MR.
>>>> Hence make sure that mlx5_ib_sg_to_klms() does not write outside
>>>> the MR klms[] array.
>>>>
>>>> Fixes: b005d3164713 ("mlx5: Add arbitrary sg list support")
>>>> Signed-off-by: Bart Van Assche <bart.vanassche-XdAiOPVOjttBDgjK7y7TUQ@public.gmane.org>
>>>> Reviewed-by: Max Gurtovoy <maxg-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
>>>> Cc: Sagi Grimberg <sagi-NQWnxTmZq1alnMjI0IkVqw@public.gmane.org>
>>>> Cc: Leon Romanovsky <leonro-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
>>>> Cc: Israel Rukshin <israelr-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
>>>> Cc: <stable-u79uwXL29TY76Z2rM5mHXA@public.gmane.org>
>>>> ---
>>>>  drivers/infiniband/hw/mlx5/mr.c | 2 +-
>>>>  1 file changed, 1 insertion(+), 1 deletion(-)
>>>>
>>>
>>> Bart,
>>>
>>> Thanks a lot, it indeed looks right.
>>> Acked-by: Leon Romanovsky <leonro-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
>>>
>>> Thanks
>>>
>>
>>
>> Hello Bart, Leon, Max and Israel.
>>
>> I cloned off Barts tree.
>>
>> git clone https://github.com/bvanassche/linux
>> cd linux
>> git checkout block-scsi-for-next
>>
>> I checked all patches were in for this test.
>>
>> a83e404 IB/srp: Reenable IB_MR_TYPE_SG_GAPS
>> dfa5a2b mlx5: Avoid that mlx5_ib_sg_to_klms() overflows the klms[] array
>> f759c80 mlx5: Fix mlx5_ib_map_mr_sg mr lengt
>>
>> Built and tested the kernel.
>>
>> However this issue is not resolved :(
>>
>> [ 2707.931909] scsi host1: ib_srp: failed RECV status WR flushed (5) for CQE ffff8817edca86b0
>> [ 2708.089806] mlx5_0:dump_cqe:262:(pid 20129): dump error cqe
>> [ 2708.121342] 00000000 00000000 00000000 00000000
>> [ 2708.147104] 00000000 00000000 00000000 00000000
>> [ 2708.172633] 00000000 00000000 00000000 00000000
>> [ 2708.198702] 00000000 0f007806 2500002a 14a527d0
>
> Parsed version:
> 	hw_error_syndrome                : 0xf
> 	hw_syndrome_type                 : 0x0
> 	vendor_error_syndrome            : 0x78
> 	syndrome                         : MEMORY_WINDOW_BIND_ERROR (0x6)
> 	s_wqe_opcode                     : UMR (0x25)
> 	opcode                           : REQUESTOR_ERROR (0xd)
> 	cqe_format                       : NO_INLINE_DATA (0x0)
> 	owner                            : 0x0
>
> Description:
> 	umr.klm_octoword_count > mkey.mtt_octoword_count
>
> Sagi, Max,
> Any idea where can it be?

Sagi,
I see this code in drivers/infiniband/hw/mlx5/mr.c:

"
...
else if (mr_type == IB_MR_TYPE_SG_GAPS) {
                 mr->access_mode = MLX5_MKC_ACCESS_MODE_KLMS;

                 err = mlx5_alloc_priv_descs(pd->device, mr,
                                             ndescs, sizeof(struct 
mlx5_klm));
                 if (err)
                         goto err_free_in;
                 mr->desc_size = sizeof(struct mlx5_klm);
                 mr->max_descs = ndescs;
"

while in the past it was:
"
} else if (mr_type == IB_MR_INDIRECT_REG) {
                 MLX5_SET(mkc, mkc, translations_octword_size,
                          ALIGN(max_num_sg + 1, 4));
                 mr->access_mode = MLX5_MKC_ACCESS_MODE_KLMS | 
MLX5_PERM_UMR_EN;
                 mr->max_descs = ndescs;

"

in INDIRECT_REG it was + 1...

maybe this is the issue ?

Max.

>
> Thanks
>
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH, untested] mlx5: Avoid that mlx5_ib_sg_to_klms() overflows the klms[] array

[Sagi Grimberg]

>> Hello Bart, Leon, Max and Israel.
>>
>> I cloned off Barts tree.
>>
>> git clone https://github.com/bvanassche/linux
>> cd linux
>> git checkout block-scsi-for-next
>>
>> I checked all patches were in for this test.
>>
>> a83e404 IB/srp: Reenable IB_MR_TYPE_SG_GAPS
>> dfa5a2b mlx5: Avoid that mlx5_ib_sg_to_klms() overflows the klms[] array
>> f759c80 mlx5: Fix mlx5_ib_map_mr_sg mr lengt
>>
>> Built and tested the kernel.
>>
>> However this issue is not resolved :(
>>
>> [ 2707.931909] scsi host1: ib_srp: failed RECV status WR flushed (5) for CQE ffff8817edca86b0
>> [ 2708.089806] mlx5_0:dump_cqe:262:(pid 20129): dump error cqe
>> [ 2708.121342] 00000000 00000000 00000000 00000000
>> [ 2708.147104] 00000000 00000000 00000000 00000000
>> [ 2708.172633] 00000000 00000000 00000000 00000000
>> [ 2708.198702] 00000000 0f007806 2500002a 14a527d0
>
> Parsed version:
> 	hw_error_syndrome                : 0xf
> 	hw_syndrome_type                 : 0x0
> 	vendor_error_syndrome            : 0x78
> 	syndrome                         : MEMORY_WINDOW_BIND_ERROR (0x6)
> 	s_wqe_opcode                     : UMR (0x25)
> 	opcode                           : REQUESTOR_ERROR (0xd)
> 	cqe_format                       : NO_INLINE_DATA (0x0)
> 	owner                            : 0x0
>
> Description:
> 	umr.klm_octoword_count > mkey.mtt_octoword_count
>
> Sagi, Max,
> Any idea where can it be?

Laurence, Max,

We need to make sure that we never overflow number of mapping
elements.

Looking at the code, it seems that some of it was reworked by
Artemy for ODP.

Laurence, can you try and retest the below patch:
--
diff --git a/drivers/infiniband/hw/mlx5/qp.c 
b/drivers/infiniband/hw/mlx5/qp.c
index ad8a2638e339..76f3857ecd53 100644
--- a/drivers/infiniband/hw/mlx5/qp.c
+++ b/drivers/infiniband/hw/mlx5/qp.c
@@ -3224,22 +3224,19 @@ static void set_reg_mkey_seg(struct 
mlx5_mkey_seg *seg,
                              struct mlx5_ib_mr *mr,
                              u32 key, int access)
  {
-       int ndescs = ALIGN(mr->ndescs, 8) >> 1;
+       int size = mr->ndescs * mr->desc_size;

         memset(seg, 0, sizeof(*seg));

         if (mr->access_mode == MLX5_MKC_ACCESS_MODE_MTT)
                 seg->log2_page_size = ilog2(mr->ibmr.page_size);
-       else if (mr->access_mode == MLX5_MKC_ACCESS_MODE_KLMS)
-               /* KLMs take twice the size of MTTs */
-               ndescs *= 2;

         seg->flags = get_umr_flags(access) | mr->access_mode;
         seg->qpn_mkey7_0 = cpu_to_be32((key & 0xff) | 0xffffff00);
         seg->flags_pd = cpu_to_be32(MLX5_MKEY_REMOTE_INVAL);
         seg->start_addr = cpu_to_be64(mr->ibmr.iova);
         seg->len = cpu_to_be64(mr->ibmr.length);
-       seg->xlt_oct_size = cpu_to_be32(ndescs);
+       seg->xlt_oct_size = cpu_to_be32(get_xlt_octo(size));
  }

  static void set_linv_mkey_seg(struct mlx5_mkey_seg *seg)
--
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH, untested] mlx5: Avoid that mlx5_ib_sg_to_klms() overflows the klms[] array

[Laurence Oberman]

----- Original Message -----
> From: "Sagi Grimberg" <sagi-NQWnxTmZq1alnMjI0IkVqw@public.gmane.org>
> To: "Leon Romanovsky" <leonro-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>, "Laurence Oberman" <loberman-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>
> Cc: "Bart Van Assche" <bart.vanassche-XdAiOPVOjttBDgjK7y7TUQ@public.gmane.org>, "Doug Ledford" <dledford-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>, "Max Gurtovoy"
> <maxg-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>, "Israel Rukshin" <israelr-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>, linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> Sent: Wednesday, May 3, 2017 4:18:38 AM
> Subject: Re: [PATCH, untested] mlx5: Avoid that mlx5_ib_sg_to_klms() overflows the klms[] array
> 
> 
> >> Hello Bart, Leon, Max and Israel.
> >>
> >> I cloned off Barts tree.
> >>
> >> git clone https://github.com/bvanassche/linux
> >> cd linux
> >> git checkout block-scsi-for-next
> >>
> >> I checked all patches were in for this test.
> >>
> >> a83e404 IB/srp: Reenable IB_MR_TYPE_SG_GAPS
> >> dfa5a2b mlx5: Avoid that mlx5_ib_sg_to_klms() overflows the klms[] array
> >> f759c80 mlx5: Fix mlx5_ib_map_mr_sg mr lengt
> >>
> >> Built and tested the kernel.
> >>
> >> However this issue is not resolved :(
> >>
> >> [ 2707.931909] scsi host1: ib_srp: failed RECV status WR flushed (5) for
> >> CQE ffff8817edca86b0
> >> [ 2708.089806] mlx5_0:dump_cqe:262:(pid 20129): dump error cqe
> >> [ 2708.121342] 00000000 00000000 00000000 00000000
> >> [ 2708.147104] 00000000 00000000 00000000 00000000
> >> [ 2708.172633] 00000000 00000000 00000000 00000000
> >> [ 2708.198702] 00000000 0f007806 2500002a 14a527d0
> >
> > Parsed version:
> > 	hw_error_syndrome                : 0xf
> > 	hw_syndrome_type                 : 0x0
> > 	vendor_error_syndrome            : 0x78
> > 	syndrome                         : MEMORY_WINDOW_BIND_ERROR (0x6)
> > 	s_wqe_opcode                     : UMR (0x25)
> > 	opcode                           : REQUESTOR_ERROR (0xd)
> > 	cqe_format                       : NO_INLINE_DATA (0x0)
> > 	owner                            : 0x0
> >
> > Description:
> > 	umr.klm_octoword_count > mkey.mtt_octoword_count
> >
> > Sagi, Max,
> > Any idea where can it be?
> 
> Laurence, Max,
> 
> We need to make sure that we never overflow number of mapping
> elements.
> 
> Looking at the code, it seems that some of it was reworked by
> Artemy for ODP.
> 
> Laurence, can you try and retest the below patch:
> --
> diff --git a/drivers/infiniband/hw/mlx5/qp.c
> b/drivers/infiniband/hw/mlx5/qp.c
> index ad8a2638e339..76f3857ecd53 100644
> --- a/drivers/infiniband/hw/mlx5/qp.c
> +++ b/drivers/infiniband/hw/mlx5/qp.c
> @@ -3224,22 +3224,19 @@ static void set_reg_mkey_seg(struct
> mlx5_mkey_seg *seg,
>                               struct mlx5_ib_mr *mr,
>                               u32 key, int access)
>   {
> -       int ndescs = ALIGN(mr->ndescs, 8) >> 1;
> +       int size = mr->ndescs * mr->desc_size;
> 
>          memset(seg, 0, sizeof(*seg));
> 
>          if (mr->access_mode == MLX5_MKC_ACCESS_MODE_MTT)
>                  seg->log2_page_size = ilog2(mr->ibmr.page_size);
> -       else if (mr->access_mode == MLX5_MKC_ACCESS_MODE_KLMS)
> -               /* KLMs take twice the size of MTTs */
> -               ndescs *= 2;
> 
>          seg->flags = get_umr_flags(access) | mr->access_mode;
>          seg->qpn_mkey7_0 = cpu_to_be32((key & 0xff) | 0xffffff00);
>          seg->flags_pd = cpu_to_be32(MLX5_MKEY_REMOTE_INVAL);
>          seg->start_addr = cpu_to_be64(mr->ibmr.iova);
>          seg->len = cpu_to_be64(mr->ibmr.length);
> -       seg->xlt_oct_size = cpu_to_be32(ndescs);
> +       seg->xlt_oct_size = cpu_to_be32(get_xlt_octo(size));
>   }
> 
>   static void set_linv_mkey_seg(struct mlx5_mkey_seg *seg)
> --
> --
> To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 

Hello Sagi
Against Bart's tree again

a83e404 IB/srp: Reenable IB_MR_TYPE_SG_GAPS
dfa5a2b mlx5: Avoid that mlx5_ib_sg_to_klms() overflows the klms[] array
f759c80 mlx5: Fix mlx5_ib_map_mr_sg mr lengt

Above are all in
Added your most recent patch above

Same behavior.
[  579.368733] scsi host1: ib_srp: failed RECV status WR flushed (5) for CQE ffff8817de9c57b0
[  579.369875] mlx5_1:dump_cqe:262:(pid 15140): dump error cqe
[  579.369877] 00000000 00000000 00000000 00000000
[  579.369877] 00000000 00000000 00000000 00000000
[  579.369878] 00000000 00000000 00000000 00000000
[  579.369878] 00000000 0f007806 2500002b 1c528dd0
[  579.369883] scsi host1: ib_srp: failed FAST REG status memory management operation error (6) for CQE ffff88179a460af8
[  594.814222] scsi host1: ib_srp: reconnect succeeded
[  594.916876] scsi host1: ib_srp: failed RECV status WR flushed (5) for CQE ffff8817e1d4a6b0
[  595.494532] mlx5_1:dump_cqe:262:(pid 15205): dump error cqe
[  595.525995] 00000000 00000000 00000000 00000000
[  595.552125] 00000000 00000000 00000000 00000000
[  595.578204] 00000000 00000000 00000000 00000000
[  595.603670] 00000000 0f007806 25000033 002d77d0
^C[  610.821911] scsi host1: ib_srp: reconnect succeeded
[  610.933298] scsi host1: ib_srp: failed RECV status WR flushed (5) for CQE ffff8817e1d4a170
[  611.514234] mlx5_1:dump_cqe:262:(pid 15242): dump error cqe
[  611.543083] 00000000 00000000 00000000 00000000
[  611.568670] 00000000 00000000 00000000 00000000
[  611.594064] 00000000 00000000 00000000 00000000
[  611.620142] 00000000 0f007806 2500003b 003161d0

I will capture the function traces with your patch applied and the additional logging asked for by Max.
Thanks
Laurence
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH, untested] mlx5: Avoid that mlx5_ib_sg_to_klms() overflows the klms[] array

[Sagi Grimberg]

> Hello Sagi
> Against Bart's tree again
>
> a83e404 IB/srp: Reenable IB_MR_TYPE_SG_GAPS
> dfa5a2b mlx5: Avoid that mlx5_ib_sg_to_klms() overflows the klms[] array
> f759c80 mlx5: Fix mlx5_ib_map_mr_sg mr lengt
>
> Above are all in
> Added your most recent patch above
>
> Same behavior.
> [  579.368733] scsi host1: ib_srp: failed RECV status WR flushed (5) for CQE ffff8817de9c57b0
> [  579.369875] mlx5_1:dump_cqe:262:(pid 15140): dump error cqe
> [  579.369877] 00000000 00000000 00000000 00000000
> [  579.369877] 00000000 00000000 00000000 00000000
> [  579.369878] 00000000 00000000 00000000 00000000
> [  579.369878] 00000000 0f007806 2500002b 1c528dd0
> [  579.369883] scsi host1: ib_srp: failed FAST REG status memory management operation error (6) for CQE ffff88179a460af8
> [  594.814222] scsi host1: ib_srp: reconnect succeeded
> [  594.916876] scsi host1: ib_srp: failed RECV status WR flushed (5) for CQE ffff8817e1d4a6b0
> [  595.494532] mlx5_1:dump_cqe:262:(pid 15205): dump error cqe
> [  595.525995] 00000000 00000000 00000000 00000000
> [  595.552125] 00000000 00000000 00000000 00000000
> [  595.578204] 00000000 00000000 00000000 00000000
> [  595.603670] 00000000 0f007806 25000033 002d77d0
> ^C[  610.821911] scsi host1: ib_srp: reconnect succeeded
> [  610.933298] scsi host1: ib_srp: failed RECV status WR flushed (5) for CQE ffff8817e1d4a170
> [  611.514234] mlx5_1:dump_cqe:262:(pid 15242): dump error cqe
> [  611.543083] 00000000 00000000 00000000 00000000
> [  611.568670] 00000000 00000000 00000000 00000000
> [  611.594064] 00000000 00000000 00000000 00000000
> [  611.620142] 00000000 0f007806 2500003b 003161d0
>
> I will capture the function traces with your patch applied and the additional logging asked for by Max.

Thanks, that would be helpful,

Can you try the following patch, just to see if there is an off by 1 case:

--
diff --git a/drivers/infiniband/hw/mlx5/mr.c 
b/drivers/infiniband/hw/mlx5/mr.c
index b8f9382a8b7d..3d6ef7bce7d9 100644
--- a/drivers/infiniband/hw/mlx5/mr.c
+++ b/drivers/infiniband/hw/mlx5/mr.c
@@ -1525,7 +1525,7 @@ struct ib_mr *mlx5_ib_alloc_mr(struct ib_pd *pd,
  {
         struct mlx5_ib_dev *dev = to_mdev(pd->device);
         int inlen = MLX5_ST_SZ_BYTES(create_mkey_in);
-       int ndescs = ALIGN(max_num_sg, 4);
+       int ndescs = ALIGN(max_num_sg + 1, 4);
         struct mlx5_ib_mr *mr;
         void *mkc;
         u32 *in;
--

It's not a fix, but if it works it can give us a clue...
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH, untested] mlx5: Avoid that mlx5_ib_sg_to_klms() overflows the klms[] array

[Laurence Oberman]

----- Original Message -----
> From: "Sagi Grimberg" <sagi-NQWnxTmZq1alnMjI0IkVqw@public.gmane.org>
> To: "Laurence Oberman" <loberman-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>
> Cc: "Leon Romanovsky" <leonro-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>, "Bart Van Assche" <bart.vanassche-XdAiOPVOjttBDgjK7y7TUQ@public.gmane.org>, "Doug Ledford"
> <dledford-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>, "Max Gurtovoy" <maxg-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>, "Israel Rukshin" <israelr-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>,
> linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> Sent: Wednesday, May 3, 2017 10:58:43 AM
> Subject: Re: [PATCH, untested] mlx5: Avoid that mlx5_ib_sg_to_klms() overflows the klms[] array
> 
> 
> > Hello Sagi
> > Against Bart's tree again
> >
> > a83e404 IB/srp: Reenable IB_MR_TYPE_SG_GAPS
> > dfa5a2b mlx5: Avoid that mlx5_ib_sg_to_klms() overflows the klms[] array
> > f759c80 mlx5: Fix mlx5_ib_map_mr_sg mr lengt
> >
> > Above are all in
> > Added your most recent patch above
> >
> > Same behavior.
> > [  579.368733] scsi host1: ib_srp: failed RECV status WR flushed (5) for
> > CQE ffff8817de9c57b0
> > [  579.369875] mlx5_1:dump_cqe:262:(pid 15140): dump error cqe
> > [  579.369877] 00000000 00000000 00000000 00000000
> > [  579.369877] 00000000 00000000 00000000 00000000
> > [  579.369878] 00000000 00000000 00000000 00000000
> > [  579.369878] 00000000 0f007806 2500002b 1c528dd0
> > [  579.369883] scsi host1: ib_srp: failed FAST REG status memory management
> > operation error (6) for CQE ffff88179a460af8
> > [  594.814222] scsi host1: ib_srp: reconnect succeeded
> > [  594.916876] scsi host1: ib_srp: failed RECV status WR flushed (5) for
> > CQE ffff8817e1d4a6b0
> > [  595.494532] mlx5_1:dump_cqe:262:(pid 15205): dump error cqe
> > [  595.525995] 00000000 00000000 00000000 00000000
> > [  595.552125] 00000000 00000000 00000000 00000000
> > [  595.578204] 00000000 00000000 00000000 00000000
> > [  595.603670] 00000000 0f007806 25000033 002d77d0
> > ^C[  610.821911] scsi host1: ib_srp: reconnect succeeded
> > [  610.933298] scsi host1: ib_srp: failed RECV status WR flushed (5) for
> > CQE ffff8817e1d4a170
> > [  611.514234] mlx5_1:dump_cqe:262:(pid 15242): dump error cqe
> > [  611.543083] 00000000 00000000 00000000 00000000
> > [  611.568670] 00000000 00000000 00000000 00000000
> > [  611.594064] 00000000 00000000 00000000 00000000
> > [  611.620142] 00000000 0f007806 2500003b 003161d0
> >
> > I will capture the function traces with your patch applied and the
> > additional logging asked for by Max.
> 
> Thanks, that would be helpful,
> 
> Can you try the following patch, just to see if there is an off by 1 case:
> 
> --
> diff --git a/drivers/infiniband/hw/mlx5/mr.c
> b/drivers/infiniband/hw/mlx5/mr.c
> index b8f9382a8b7d..3d6ef7bce7d9 100644
> --- a/drivers/infiniband/hw/mlx5/mr.c
> +++ b/drivers/infiniband/hw/mlx5/mr.c
> @@ -1525,7 +1525,7 @@ struct ib_mr *mlx5_ib_alloc_mr(struct ib_pd *pd,
>   {
>          struct mlx5_ib_dev *dev = to_mdev(pd->device);
>          int inlen = MLX5_ST_SZ_BYTES(create_mkey_in);
> -       int ndescs = ALIGN(max_num_sg, 4);
> +       int ndescs = ALIGN(max_num_sg + 1, 4);
>          struct mlx5_ib_mr *mr;
>          void *mkc;
>          u32 *in;
> --
> 
> It's not a fix, but if it works it can give us a clue...
> 

Sorry, been delayed this week, will get this done this weekend.
Thanks
Laurence
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH, untested] mlx5: Avoid that mlx5_ib_sg_to_klms() overflows the klms[] array

[Laurence Oberman]

----- Original Message -----
> From: "Laurence Oberman" <loberman-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>
> To: "Sagi Grimberg" <sagi-NQWnxTmZq1alnMjI0IkVqw@public.gmane.org>
> Cc: "Leon Romanovsky" <leonro-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>, "Bart Van Assche" <bart.vanassche-XdAiOPVOjttBDgjK7y7TUQ@public.gmane.org>, "Doug Ledford"
> <dledford-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>, "Max Gurtovoy" <maxg-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>, "Israel Rukshin" <israelr-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>,
> linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> Sent: Friday, May 5, 2017 12:31:06 PM
> Subject: Re: [PATCH, untested] mlx5: Avoid that mlx5_ib_sg_to_klms() overflows the klms[] array
> 
> 
> 
> ----- Original Message -----
> > From: "Sagi Grimberg" <sagi-NQWnxTmZq1alnMjI0IkVqw@public.gmane.org>
> > To: "Laurence Oberman" <loberman-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>
> > Cc: "Leon Romanovsky" <leonro-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>, "Bart Van Assche"
> > <bart.vanassche-XdAiOPVOjttBDgjK7y7TUQ@public.gmane.org>, "Doug Ledford"
> > <dledford-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>, "Max Gurtovoy" <maxg-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>, "Israel Rukshin"
> > <israelr-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>,
> > linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> > Sent: Wednesday, May 3, 2017 10:58:43 AM
> > Subject: Re: [PATCH, untested] mlx5: Avoid that mlx5_ib_sg_to_klms()
> > overflows the klms[] array
> > 
> > 
> > > Hello Sagi
> > > Against Bart's tree again
> > >
> > > a83e404 IB/srp: Reenable IB_MR_TYPE_SG_GAPS
> > > dfa5a2b mlx5: Avoid that mlx5_ib_sg_to_klms() overflows the klms[] array
> > > f759c80 mlx5: Fix mlx5_ib_map_mr_sg mr lengt
> > >
> > > Above are all in
> > > Added your most recent patch above
> > >
> > > Same behavior.
> > > [  579.368733] scsi host1: ib_srp: failed RECV status WR flushed (5) for
> > > CQE ffff8817de9c57b0
> > > [  579.369875] mlx5_1:dump_cqe:262:(pid 15140): dump error cqe
> > > [  579.369877] 00000000 00000000 00000000 00000000
> > > [  579.369877] 00000000 00000000 00000000 00000000
> > > [  579.369878] 00000000 00000000 00000000 00000000
> > > [  579.369878] 00000000 0f007806 2500002b 1c528dd0
> > > [  579.369883] scsi host1: ib_srp: failed FAST REG status memory
> > > management
> > > operation error (6) for CQE ffff88179a460af8
> > > [  594.814222] scsi host1: ib_srp: reconnect succeeded
> > > [  594.916876] scsi host1: ib_srp: failed RECV status WR flushed (5) for
> > > CQE ffff8817e1d4a6b0
> > > [  595.494532] mlx5_1:dump_cqe:262:(pid 15205): dump error cqe
> > > [  595.525995] 00000000 00000000 00000000 00000000
> > > [  595.552125] 00000000 00000000 00000000 00000000
> > > [  595.578204] 00000000 00000000 00000000 00000000
> > > [  595.603670] 00000000 0f007806 25000033 002d77d0
> > > ^C[  610.821911] scsi host1: ib_srp: reconnect succeeded
> > > [  610.933298] scsi host1: ib_srp: failed RECV status WR flushed (5) for
> > > CQE ffff8817e1d4a170
> > > [  611.514234] mlx5_1:dump_cqe:262:(pid 15242): dump error cqe
> > > [  611.543083] 00000000 00000000 00000000 00000000
> > > [  611.568670] 00000000 00000000 00000000 00000000
> > > [  611.594064] 00000000 00000000 00000000 00000000
> > > [  611.620142] 00000000 0f007806 2500003b 003161d0
> > >
> > > I will capture the function traces with your patch applied and the
> > > additional logging asked for by Max.
> > 
> > Thanks, that would be helpful,
> > 
> > Can you try the following patch, just to see if there is an off by 1 case:
> > 
> > --
> > diff --git a/drivers/infiniband/hw/mlx5/mr.c
> > b/drivers/infiniband/hw/mlx5/mr.c
> > index b8f9382a8b7d..3d6ef7bce7d9 100644
> > --- a/drivers/infiniband/hw/mlx5/mr.c
> > +++ b/drivers/infiniband/hw/mlx5/mr.c
> > @@ -1525,7 +1525,7 @@ struct ib_mr *mlx5_ib_alloc_mr(struct ib_pd *pd,
> >   {
> >          struct mlx5_ib_dev *dev = to_mdev(pd->device);
> >          int inlen = MLX5_ST_SZ_BYTES(create_mkey_in);
> > -       int ndescs = ALIGN(max_num_sg, 4);
> > +       int ndescs = ALIGN(max_num_sg + 1, 4);
> >          struct mlx5_ib_mr *mr;
> >          void *mkc;
> >          u32 *in;
> > --
> > 
> > It's not a fix, but if it works it can give us a clue...
> > 
> 
> Sorry, been delayed this week, will get this done this weekend.
> Thanks
> Laurence
> --
> To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 

Sagi, Max 

With the patch below against Barts tree we still see the cqe_dump issue.

Is what is in the everything you wanted applied.
Please check I did not miss anything before I start the tracing.

May  9 17:16:00 localhost kernel: scsi host2: ib_srp: failed RECV status WR flushed (5) for CQE ffff8817ed75c830
May  9 17:16:00 localhost kernel: mlx5_1:dump_cqe:262:(pid 14567): dump error cqe
May  9 17:16:00 localhost kernel: 00000000 00000000 00000000 00000000
May  9 17:16:00 localhost kernel: 00000000 00000000 00000000 00000000
May  9 17:16:00 localhost kernel: 00000000 00000000 00000000 00000000
May  9 17:16:00 localhost kernel: 00000000 0f007806 2500002a 0b670bd0
May  9 17:16:00 localhost kernel: scsi host2: ib_srp: failed FAST REG status memory management operation error (6) for CQE ffff8817972ac278
May  9 17:16:16 localhost kernel: scsi host2: ib_srp: reconnect succeeded
May  9 17:16:16 localhost kernel: scsi host2: ib_srp: failed RECV status WR flushed (5) for CQE ffff8817d819b130


diff --git a/drivers/infiniband/hw/mlx5/mr.c b/drivers/infiniband/hw/mlx5/mr.c
index 99beacf..cf899b4 100644
--- a/drivers/infiniband/hw/mlx5/mr.c
+++ b/drivers/infiniband/hw/mlx5/mr.c
@@ -1525,7 +1525,8 @@ struct ib_mr *mlx5_ib_alloc_mr(struct ib_pd *pd,
 {
        struct mlx5_ib_dev *dev = to_mdev(pd->device);
        int inlen = MLX5_ST_SZ_BYTES(create_mkey_in);
-       int ndescs = ALIGN(max_num_sg, 4);
+       //int ndescs = ALIGN(max_num_sg, 4);
+       int ndescs = ALIGN(max_num_sg + 1, 4);
        struct mlx5_ib_mr *mr;
        void *mkc;
        u32 *in;
diff --git a/drivers/infiniband/hw/mlx5/qp.c b/drivers/infiniband/hw/mlx5/qp.c
index ad8a263..cb726a5 100644
--- a/drivers/infiniband/hw/mlx5/qp.c
+++ b/drivers/infiniband/hw/mlx5/qp.c
@@ -3224,22 +3224,19 @@ static void set_reg_mkey_seg(struct mlx5_mkey_seg *seg,
                             struct mlx5_ib_mr *mr,
                             u32 key, int access)
 {
-       int ndescs = ALIGN(mr->ndescs, 8) >> 1;
+        int size = mr->ndescs * mr->desc_size;
 
        memset(seg, 0, sizeof(*seg));
 
        if (mr->access_mode == MLX5_MKC_ACCESS_MODE_MTT)
                seg->log2_page_size = ilog2(mr->ibmr.page_size);
-       else if (mr->access_mode == MLX5_MKC_ACCESS_MODE_KLMS)
-               /* KLMs take twice the size of MTTs */
-               ndescs *= 2;
 
        seg->flags = get_umr_flags(access) | mr->access_mode;
        seg->qpn_mkey7_0 = cpu_to_be32((key & 0xff) | 0xffffff00);
        seg->flags_pd = cpu_to_be32(MLX5_MKEY_REMOTE_INVAL);
        seg->start_addr = cpu_to_be64(mr->ibmr.iova);
        seg->len = cpu_to_be64(mr->ibmr.length);
-       seg->xlt_oct_size = cpu_to_be32(ndescs);
+        seg->xlt_oct_size = cpu_to_be32(get_xlt_octo(size));
 }

I will see about capturing traces, but I am writing to a RAM disk on the target so likely will have a flood of trace data.

Thanks
Laurence
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH, untested] mlx5: Avoid that mlx5_ib_sg_to_klms() overflows the klms[] array

[Max Gurtovoy]

On 4/25/2017 11:37 PM, Laurence Oberman wrote:
>
>
> ----- Original Message -----
>> From: "Leon Romanovsky" <leonro-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
>> To: "Bart Van Assche" <bart.vanassche-XdAiOPVOjttBDgjK7y7TUQ@public.gmane.org>
>> Cc: "Doug Ledford" <dledford-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>, "Max Gurtovoy" <maxg-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>, "Sagi Grimberg" <sagi-NQWnxTmZq1alnMjI0IkVqw@public.gmane.org>,
>> "Israel Rukshin" <israelr-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>, "Laurence Oberman" <loberman-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>, linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
>> Sent: Tuesday, April 25, 2017 1:58:49 PM
>> Subject: Re: [PATCH, untested] mlx5: Avoid that mlx5_ib_sg_to_klms() overflows the klms[] array
>>
>> On Mon, Apr 24, 2017 at 03:15:28PM -0700, Bart Van Assche wrote:
>>> ib_map_mr_sg() can pass an SG-list to .map_mr_sg() that is larger
>>> than what fits into a single MR. .map_mr_sg() must not attempt to
>>> map more SG-list elements than what fits into a single MR.
>>> Hence make sure that mlx5_ib_sg_to_klms() does not write outside
>>> the MR klms[] array.
>>>
>>> Fixes: b005d3164713 ("mlx5: Add arbitrary sg list support")
>>> Signed-off-by: Bart Van Assche <bart.vanassche-XdAiOPVOjttBDgjK7y7TUQ@public.gmane.org>
>>> Reviewed-by: Max Gurtovoy <maxg-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
>>> Cc: Sagi Grimberg <sagi-NQWnxTmZq1alnMjI0IkVqw@public.gmane.org>
>>> Cc: Leon Romanovsky <leonro-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
>>> Cc: Israel Rukshin <israelr-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
>>> Cc: <stable-u79uwXL29TY76Z2rM5mHXA@public.gmane.org>
>>> ---
>>>  drivers/infiniband/hw/mlx5/mr.c | 2 +-
>>>  1 file changed, 1 insertion(+), 1 deletion(-)
>>>
>>
>> Bart,
>>
>> Thanks a lot, it indeed looks right.
>> Acked-by: Leon Romanovsky <leonro-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
>>
>> Thanks
>>
>
>
> Hello Bart, Leon, Max and Israel.
>
> I cloned off Barts tree.
>
> git clone https://github.com/bvanassche/linux
> cd linux
> git checkout block-scsi-for-next
>
> I checked all patches were in for this test.
>
> a83e404 IB/srp: Reenable IB_MR_TYPE_SG_GAPS
> dfa5a2b mlx5: Avoid that mlx5_ib_sg_to_klms() overflows the klms[] array
> f759c80 mlx5: Fix mlx5_ib_map_mr_sg mr lengt

Hi,
copying Sagi's request from different thread:

"
Can you please enable srp_add_one debug:

echo "func srp_add_one +p" > /sys/kernel/debug/dynamic_debug/control

In addition apply the following:
-- 
diff --git a/drivers/infiniband/hw/mlx5/mr.c 
b/drivers/infiniband/hw/mlx5/mr.c
index d9c6c0ea750b..040fbc387e4f 100644
--- a/drivers/infiniband/hw/mlx5/mr.c
+++ b/drivers/infiniband/hw/mlx5/mr.c
@@ -1403,6 +1403,8 @@ mlx5_alloc_priv_descs(struct ib_device *device,
         int add_size;
         int ret;

+       WARN_ON_ONCE(ndescs > device->attr.max_fast_reg_page_list_len);
+
         add_size = max_t(int, MLX5_UMR_ALIGN - ARCH_KMALLOC_MINALIGN, 0);

         mr->descs_alloc = kzalloc(size + add_size, GFP_KERNEL);

"

Max.

>
> Built and tested the kernel.
>
> However this issue is not resolved :(
>
> [ 2707.931909] scsi host1: ib_srp: failed RECV status WR flushed (5) for CQE ffff8817edca86b0
> [ 2708.089806] mlx5_0:dump_cqe:262:(pid 20129): dump error cqe
> [ 2708.121342] 00000000 00000000 00000000 00000000
> [ 2708.147104] 00000000 00000000 00000000 00000000
> [ 2708.172633] 00000000 00000000 00000000 00000000
> [ 2708.198702] 00000000 0f007806 2500002a 14a527d0
> [ 2732.434127] scsi host1: ib_srp: reconnect succeeded
> [ 2733.048023] scsi host1: ib_srp: failed RECV status WR flushed (5) for CQE ffff8817ed0a9c30
>
> [root@localhost ~]# [ 2746.413277] mlx5_0:dump_cqe:262:(pid 15877): dump error cqe
> [ 2746.443240] 00000000 00000000 00000000 00000000
> [ 2746.469323] 00000000 00000000 00000000 00000000
> [ 2746.495310] 00000000 00000000 00000000 00000000
> [ 2746.521407] 00000000 0f007806 25000032 003c7ad0
> [ 2752.445899] scsi host1: ib_srp: reconnect succeeded
> [ 2752.481835] scsi host1: ib_srp: failed RECV status WR flushed (5) for CQE ffff8817ed0a9cf0
> [ 2763.267386] mlx5_0:dump_cqe:262:(pid 15877): dump error cqe
> [ 2763.297826] 00000000 00000000 00000000 00000000
> [ 2763.323352] 00000000 00000000 00000000 00000000
> [ 2763.348722] 00000000 00000000 00000000 00000000
> [ 2763.374681] 00000000 0f007806 2500003a 00084bd0
>
> [root@localhost ~]# [ 2769.385203] fast_io_fail_tmo expired for SRP port-1:1 / host1.
> [ 2769.415956] scsi host1: ib_srp: reconnect succeeded
> [ 2769.450258] scsi host1: ib_srp: failed RECV status WR flushed (5) for CQE ffff8817ed0a9cf0
> [ 2780.064627] mlx5_0:dump_cqe:262:(pid 18771): dump error cqe
> [ 2780.093520] 00000000 00000000 00000000 00000000
> [ 2780.120067] 00000000 00000000 00000000 00000000
> [ 2780.145575] 00000000 00000000 00000000 00000000
> [ 2780.171153] 00000000 0f007806 25000042 000833d0
> [ 2785.923399] scsi host1: ib_srp: reconnect succeeded
> [ 2785.957504] scsi host1: ib_srp: failed RECV status WR flushed (5) for CQE ffff8817ed0a9cf0
> [ 2796.463426] mlx5_0:dump_cqe:262:(pid 18771): dump error cqe
> [ 2796.495257] 00000000 00000000 00000000 00000000
> [ 2796.521506] 00000000 00000000 00000000 00000000
> [ 2796.547640] 00000000 00000000 00000000 00000000
> [ 2796.573120] 00000000 0f007806 2500004a 00083bd0
> [ 2802.562578] scsi host1: ib_srp: reconnect succeeded
> [ 2802.596880] scsi host1: ib_srp: failed RECV status WR flushed (5) for CQE ffff8817ed0a9cf0
>
> Regards
> Laurence
>
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH, untested] mlx5: Avoid that mlx5_ib_sg_to_klms() overflows the klms[] array

[Laurence Oberman]

----- Original Message -----
> From: "Max Gurtovoy" <maxg-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
> To: "Laurence Oberman" <loberman-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>, "Leon Romanovsky" <leonro-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
> Cc: "Bart Van Assche" <bart.vanassche-XdAiOPVOjttBDgjK7y7TUQ@public.gmane.org>, "Doug Ledford" <dledford-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>, "Sagi Grimberg"
> <sagi-NQWnxTmZq1alnMjI0IkVqw@public.gmane.org>, "Israel Rukshin" <israelr-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>, linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> Sent: Wednesday, April 26, 2017 4:31:57 AM
> Subject: Re: [PATCH, untested] mlx5: Avoid that mlx5_ib_sg_to_klms() overflows the klms[] array
> 
> 
> 
> On 4/25/2017 11:37 PM, Laurence Oberman wrote:
> >
> >
> > ----- Original Message -----
> >> From: "Leon Romanovsky" <leonro-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
> >> To: "Bart Van Assche" <bart.vanassche-XdAiOPVOjttBDgjK7y7TUQ@public.gmane.org>
> >> Cc: "Doug Ledford" <dledford-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>, "Max Gurtovoy"
> >> <maxg-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>, "Sagi Grimberg" <sagi-NQWnxTmZq1alnMjI0IkVqw@public.gmane.org>,
> >> "Israel Rukshin" <israelr-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>, "Laurence Oberman"
> >> <loberman-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>, linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> >> Sent: Tuesday, April 25, 2017 1:58:49 PM
> >> Subject: Re: [PATCH, untested] mlx5: Avoid that mlx5_ib_sg_to_klms()
> >> overflows the klms[] array
> >>
> >> On Mon, Apr 24, 2017 at 03:15:28PM -0700, Bart Van Assche wrote:
> >>> ib_map_mr_sg() can pass an SG-list to .map_mr_sg() that is larger
> >>> than what fits into a single MR. .map_mr_sg() must not attempt to
> >>> map more SG-list elements than what fits into a single MR.
> >>> Hence make sure that mlx5_ib_sg_to_klms() does not write outside
> >>> the MR klms[] array.
> >>>
> >>> Fixes: b005d3164713 ("mlx5: Add arbitrary sg list support")
> >>> Signed-off-by: Bart Van Assche <bart.vanassche-XdAiOPVOjttBDgjK7y7TUQ@public.gmane.org>
> >>> Reviewed-by: Max Gurtovoy <maxg-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
> >>> Cc: Sagi Grimberg <sagi-NQWnxTmZq1alnMjI0IkVqw@public.gmane.org>
> >>> Cc: Leon Romanovsky <leonro-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
> >>> Cc: Israel Rukshin <israelr-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
> >>> Cc: <stable-u79uwXL29TY76Z2rM5mHXA@public.gmane.org>
> >>> ---
> >>>  drivers/infiniband/hw/mlx5/mr.c | 2 +-
> >>>  1 file changed, 1 insertion(+), 1 deletion(-)
> >>>
> >>
> >> Bart,
> >>
> >> Thanks a lot, it indeed looks right.
> >> Acked-by: Leon Romanovsky <leonro-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
> >>
> >> Thanks
> >>
> >
> >
> > Hello Bart, Leon, Max and Israel.
> >
> > I cloned off Barts tree.
> >
> > git clone https://github.com/bvanassche/linux
> > cd linux
> > git checkout block-scsi-for-next
> >
> > I checked all patches were in for this test.
> >
> > a83e404 IB/srp: Reenable IB_MR_TYPE_SG_GAPS
> > dfa5a2b mlx5: Avoid that mlx5_ib_sg_to_klms() overflows the klms[] array
> > f759c80 mlx5: Fix mlx5_ib_map_mr_sg mr lengt
> 
> Hi,
> copying Sagi's request from different thread:
> 
> "
> Can you please enable srp_add_one debug:
> 
> echo "func srp_add_one +p" > /sys/kernel/debug/dynamic_debug/control
> 
> In addition apply the following:
> --
> diff --git a/drivers/infiniband/hw/mlx5/mr.c
> b/drivers/infiniband/hw/mlx5/mr.c
> index d9c6c0ea750b..040fbc387e4f 100644
> --- a/drivers/infiniband/hw/mlx5/mr.c
> +++ b/drivers/infiniband/hw/mlx5/mr.c
> @@ -1403,6 +1403,8 @@ mlx5_alloc_priv_descs(struct ib_device *device,
>          int add_size;
>          int ret;
> 
> +       WARN_ON_ONCE(ndescs > device->attr.max_fast_reg_page_list_len);
> +
>          add_size = max_t(int, MLX5_UMR_ALIGN - ARCH_KMALLOC_MINALIGN, 0);
> 
>          mr->descs_alloc = kzalloc(size + add_size, GFP_KERNEL);
> 
> "
> 
> Max.
> 
> >
> > Built and tested the kernel.
> >
> > However this issue is not resolved :(
> >
> > [ 2707.931909] scsi host1: ib_srp: failed RECV status WR flushed (5) for
> > CQE ffff8817edca86b0
> > [ 2708.089806] mlx5_0:dump_cqe:262:(pid 20129): dump error cqe
> > [ 2708.121342] 00000000 00000000 00000000 00000000
> > [ 2708.147104] 00000000 00000000 00000000 00000000
> > [ 2708.172633] 00000000 00000000 00000000 00000000
> > [ 2708.198702] 00000000 0f007806 2500002a 14a527d0
> > [ 2732.434127] scsi host1: ib_srp: reconnect succeeded
> > [ 2733.048023] scsi host1: ib_srp: failed RECV status WR flushed (5) for
> > CQE ffff8817ed0a9c30
> >
> > [root@localhost ~]# [ 2746.413277] mlx5_0:dump_cqe:262:(pid 15877): dump
> > error cqe
> > [ 2746.443240] 00000000 00000000 00000000 00000000
> > [ 2746.469323] 00000000 00000000 00000000 00000000
> > [ 2746.495310] 00000000 00000000 00000000 00000000
> > [ 2746.521407] 00000000 0f007806 25000032 003c7ad0
> > [ 2752.445899] scsi host1: ib_srp: reconnect succeeded
> > [ 2752.481835] scsi host1: ib_srp: failed RECV status WR flushed (5) for
> > CQE ffff8817ed0a9cf0
> > [ 2763.267386] mlx5_0:dump_cqe:262:(pid 15877): dump error cqe
> > [ 2763.297826] 00000000 00000000 00000000 00000000
> > [ 2763.323352] 00000000 00000000 00000000 00000000
> > [ 2763.348722] 00000000 00000000 00000000 00000000
> > [ 2763.374681] 00000000 0f007806 2500003a 00084bd0
> >
> > [root@localhost ~]# [ 2769.385203] fast_io_fail_tmo expired for SRP
> > port-1:1 / host1.
> > [ 2769.415956] scsi host1: ib_srp: reconnect succeeded
> > [ 2769.450258] scsi host1: ib_srp: failed RECV status WR flushed (5) for
> > CQE ffff8817ed0a9cf0
> > [ 2780.064627] mlx5_0:dump_cqe:262:(pid 18771): dump error cqe
> > [ 2780.093520] 00000000 00000000 00000000 00000000
> > [ 2780.120067] 00000000 00000000 00000000 00000000
> > [ 2780.145575] 00000000 00000000 00000000 00000000
> > [ 2780.171153] 00000000 0f007806 25000042 000833d0
> > [ 2785.923399] scsi host1: ib_srp: reconnect succeeded
> > [ 2785.957504] scsi host1: ib_srp: failed RECV status WR flushed (5) for
> > CQE ffff8817ed0a9cf0
> > [ 2796.463426] mlx5_0:dump_cqe:262:(pid 18771): dump error cqe
> > [ 2796.495257] 00000000 00000000 00000000 00000000
> > [ 2796.521506] 00000000 00000000 00000000 00000000
> > [ 2796.547640] 00000000 00000000 00000000 00000000
> > [ 2796.573120] 00000000 0f007806 2500004a 00083bd0
> > [ 2802.562578] scsi host1: ib_srp: reconnect succeeded
> > [ 2802.596880] scsi host1: ib_srp: failed RECV status WR flushed (5) for
> > CQE ffff8817ed0a9cf0
> >
> > Regards
> > Laurence
> >
> 
Doing this now
Thanks
Laurence
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH, untested] mlx5: Avoid that mlx5_ib_sg_to_klms() overflows the klms[] array

[Laurence Oberman]

----- Original Message -----
> From: "Laurence Oberman" <loberman-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>
> To: "Max Gurtovoy" <maxg-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
> Cc: "Leon Romanovsky" <leonro-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>, "Bart Van Assche" <bart.vanassche-XdAiOPVOjttBDgjK7y7TUQ@public.gmane.org>, "Doug Ledford"
> <dledford-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>, "Sagi Grimberg" <sagi-NQWnxTmZq1alnMjI0IkVqw@public.gmane.org>, "Israel Rukshin" <israelr-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>,
> linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> Sent: Wednesday, April 26, 2017 7:47:37 AM
> Subject: Re: [PATCH, untested] mlx5: Avoid that mlx5_ib_sg_to_klms() overflows the klms[] array
> 
> 
> 
> ----- Original Message -----
> > From: "Max Gurtovoy" <maxg-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
> > To: "Laurence Oberman" <loberman-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>, "Leon Romanovsky"
> > <leonro-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
> > Cc: "Bart Van Assche" <bart.vanassche-XdAiOPVOjttBDgjK7y7TUQ@public.gmane.org>, "Doug Ledford"
> > <dledford-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>, "Sagi Grimberg"
> > <sagi-NQWnxTmZq1alnMjI0IkVqw@public.gmane.org>, "Israel Rukshin" <israelr-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>,
> > linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> > Sent: Wednesday, April 26, 2017 4:31:57 AM
> > Subject: Re: [PATCH, untested] mlx5: Avoid that mlx5_ib_sg_to_klms()
> > overflows the klms[] array
> > 
> > 
> > 
> > On 4/25/2017 11:37 PM, Laurence Oberman wrote:
> > >
> > >
> > > ----- Original Message -----
> > >> From: "Leon Romanovsky" <leonro-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
> > >> To: "Bart Van Assche" <bart.vanassche-XdAiOPVOjttBDgjK7y7TUQ@public.gmane.org>
> > >> Cc: "Doug Ledford" <dledford-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>, "Max Gurtovoy"
> > >> <maxg-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>, "Sagi Grimberg" <sagi-NQWnxTmZq1alnMjI0IkVqw@public.gmane.org>,
> > >> "Israel Rukshin" <israelr-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>, "Laurence Oberman"
> > >> <loberman-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>, linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> > >> Sent: Tuesday, April 25, 2017 1:58:49 PM
> > >> Subject: Re: [PATCH, untested] mlx5: Avoid that mlx5_ib_sg_to_klms()
> > >> overflows the klms[] array
> > >>
> > >> On Mon, Apr 24, 2017 at 03:15:28PM -0700, Bart Van Assche wrote:
> > >>> ib_map_mr_sg() can pass an SG-list to .map_mr_sg() that is larger
> > >>> than what fits into a single MR. .map_mr_sg() must not attempt to
> > >>> map more SG-list elements than what fits into a single MR.
> > >>> Hence make sure that mlx5_ib_sg_to_klms() does not write outside
> > >>> the MR klms[] array.
> > >>>
> > >>> Fixes: b005d3164713 ("mlx5: Add arbitrary sg list support")
> > >>> Signed-off-by: Bart Van Assche <bart.vanassche-XdAiOPVOjttBDgjK7y7TUQ@public.gmane.org>
> > >>> Reviewed-by: Max Gurtovoy <maxg-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
> > >>> Cc: Sagi Grimberg <sagi-NQWnxTmZq1alnMjI0IkVqw@public.gmane.org>
> > >>> Cc: Leon Romanovsky <leonro-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
> > >>> Cc: Israel Rukshin <israelr-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
> > >>> Cc: <stable-u79uwXL29TY76Z2rM5mHXA@public.gmane.org>
> > >>> ---
> > >>>  drivers/infiniband/hw/mlx5/mr.c | 2 +-
> > >>>  1 file changed, 1 insertion(+), 1 deletion(-)
> > >>>
> > >>
> > >> Bart,
> > >>
> > >> Thanks a lot, it indeed looks right.
> > >> Acked-by: Leon Romanovsky <leonro-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
> > >>
> > >> Thanks
> > >>
> > >
> > >
> > > Hello Bart, Leon, Max and Israel.
> > >
> > > I cloned off Barts tree.
> > >
> > > git clone https://github.com/bvanassche/linux
> > > cd linux
> > > git checkout block-scsi-for-next
> > >
> > > I checked all patches were in for this test.
> > >
> > > a83e404 IB/srp: Reenable IB_MR_TYPE_SG_GAPS
> > > dfa5a2b mlx5: Avoid that mlx5_ib_sg_to_klms() overflows the klms[] array
> > > f759c80 mlx5: Fix mlx5_ib_map_mr_sg mr lengt
> > 
> > Hi,
> > copying Sagi's request from different thread:
> > 
> > "
> > Can you please enable srp_add_one debug:
> > 
> > echo "func srp_add_one +p" > /sys/kernel/debug/dynamic_debug/control
> > 
> > In addition apply the following:
> > --
> > diff --git a/drivers/infiniband/hw/mlx5/mr.c
> > b/drivers/infiniband/hw/mlx5/mr.c
> > index d9c6c0ea750b..040fbc387e4f 100644
> > --- a/drivers/infiniband/hw/mlx5/mr.c
> > +++ b/drivers/infiniband/hw/mlx5/mr.c
> > @@ -1403,6 +1403,8 @@ mlx5_alloc_priv_descs(struct ib_device *device,
> >          int add_size;
> >          int ret;
> > 
> > +       WARN_ON_ONCE(ndescs > device->attr.max_fast_reg_page_list_len);
> > +
> >          add_size = max_t(int, MLX5_UMR_ALIGN - ARCH_KMALLOC_MINALIGN, 0);
> > 
> >          mr->descs_alloc = kzalloc(size + add_size, GFP_KERNEL);
> > 
> > "
> > 
> > Max.
> > 
> > >
> > > Built and tested the kernel.
> > >
> > > However this issue is not resolved :(
> > >
> > > [ 2707.931909] scsi host1: ib_srp: failed RECV status WR flushed (5) for
> > > CQE ffff8817edca86b0
> > > [ 2708.089806] mlx5_0:dump_cqe:262:(pid 20129): dump error cqe
> > > [ 2708.121342] 00000000 00000000 00000000 00000000
> > > [ 2708.147104] 00000000 00000000 00000000 00000000
> > > [ 2708.172633] 00000000 00000000 00000000 00000000
> > > [ 2708.198702] 00000000 0f007806 2500002a 14a527d0
> > > [ 2732.434127] scsi host1: ib_srp: reconnect succeeded
> > > [ 2733.048023] scsi host1: ib_srp: failed RECV status WR flushed (5) for
> > > CQE ffff8817ed0a9c30
> > >
> > > [root@localhost ~]# [ 2746.413277] mlx5_0:dump_cqe:262:(pid 15877): dump
> > > error cqe
> > > [ 2746.443240] 00000000 00000000 00000000 00000000
> > > [ 2746.469323] 00000000 00000000 00000000 00000000
> > > [ 2746.495310] 00000000 00000000 00000000 00000000
> > > [ 2746.521407] 00000000 0f007806 25000032 003c7ad0
> > > [ 2752.445899] scsi host1: ib_srp: reconnect succeeded
> > > [ 2752.481835] scsi host1: ib_srp: failed RECV status WR flushed (5) for
> > > CQE ffff8817ed0a9cf0
> > > [ 2763.267386] mlx5_0:dump_cqe:262:(pid 15877): dump error cqe
> > > [ 2763.297826] 00000000 00000000 00000000 00000000
> > > [ 2763.323352] 00000000 00000000 00000000 00000000
> > > [ 2763.348722] 00000000 00000000 00000000 00000000
> > > [ 2763.374681] 00000000 0f007806 2500003a 00084bd0
> > >
> > > [root@localhost ~]# [ 2769.385203] fast_io_fail_tmo expired for SRP
> > > port-1:1 / host1.
> > > [ 2769.415956] scsi host1: ib_srp: reconnect succeeded
> > > [ 2769.450258] scsi host1: ib_srp: failed RECV status WR flushed (5) for
> > > CQE ffff8817ed0a9cf0
> > > [ 2780.064627] mlx5_0:dump_cqe:262:(pid 18771): dump error cqe
> > > [ 2780.093520] 00000000 00000000 00000000 00000000
> > > [ 2780.120067] 00000000 00000000 00000000 00000000
> > > [ 2780.145575] 00000000 00000000 00000000 00000000
> > > [ 2780.171153] 00000000 0f007806 25000042 000833d0
> > > [ 2785.923399] scsi host1: ib_srp: reconnect succeeded
> > > [ 2785.957504] scsi host1: ib_srp: failed RECV status WR flushed (5) for
> > > CQE ffff8817ed0a9cf0
> > > [ 2796.463426] mlx5_0:dump_cqe:262:(pid 18771): dump error cqe
> > > [ 2796.495257] 00000000 00000000 00000000 00000000
> > > [ 2796.521506] 00000000 00000000 00000000 00000000
> > > [ 2796.547640] 00000000 00000000 00000000 00000000
> > > [ 2796.573120] 00000000 0f007806 2500004a 00083bd0
> > > [ 2802.562578] scsi host1: ib_srp: reconnect succeeded
> > > [ 2802.596880] scsi host1: ib_srp: failed RECV status WR flushed (5) for
> > > CQE ffff8817ed0a9cf0
> > >
> > > Regards
> > > Laurence
> > >
> > 
> Doing this now
> Thanks
> Laurence

Max

The Patch is not correct.

drivers/infiniband/hw/mlx5/mr.c: In function 'mlx5_alloc_priv_descs':
drivers/infiniband/hw/mlx5/mr.c:1406:30: error: 'struct ib_device' has no member named 'attr'
  WARN_ON_ONCE(ndescs > device->attr.max_fast_reg_page_list_len);
                              ^
./include/asm-generic/bug.h:117:27: note: in definition of macro 'WARN_ON_ONCE'
  int __ret_warn_once = !!(condition);   \

I think you meant to give me

WARN_ON_ONCE(ndescs > ib_device_attr->attr.max_fast_reg_page_list_len);

Can you confirm

Thanks
Laurence
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH, untested] mlx5: Avoid that mlx5_ib_sg_to_klms() overflows the klms[] array

[Laurence Oberman]

----- Original Message -----
> From: "Laurence Oberman" <loberman-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>
> To: "Max Gurtovoy" <maxg-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
> Cc: "Leon Romanovsky" <leonro-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>, "Bart Van Assche" <bart.vanassche-XdAiOPVOjttBDgjK7y7TUQ@public.gmane.org>, "Doug Ledford"
> <dledford-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>, "Sagi Grimberg" <sagi-NQWnxTmZq1alnMjI0IkVqw@public.gmane.org>, "Israel Rukshin" <israelr-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>,
> linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> Sent: Wednesday, April 26, 2017 8:18:13 AM
> Subject: Re: [PATCH, untested] mlx5: Avoid that mlx5_ib_sg_to_klms() overflows the klms[] array
> 
> 
> 
> ----- Original Message -----
> > From: "Laurence Oberman" <loberman-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>
> > To: "Max Gurtovoy" <maxg-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
> > Cc: "Leon Romanovsky" <leonro-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>, "Bart Van Assche"
> > <bart.vanassche-XdAiOPVOjttBDgjK7y7TUQ@public.gmane.org>, "Doug Ledford"
> > <dledford-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>, "Sagi Grimberg" <sagi-NQWnxTmZq1alnMjI0IkVqw@public.gmane.org>, "Israel Rukshin"
> > <israelr-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>,
> > linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> > Sent: Wednesday, April 26, 2017 7:47:37 AM
> > Subject: Re: [PATCH, untested] mlx5: Avoid that mlx5_ib_sg_to_klms()
> > overflows the klms[] array
> > 
> > 
> > 
> > ----- Original Message -----
> > > From: "Max Gurtovoy" <maxg-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
> > > To: "Laurence Oberman" <loberman-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>, "Leon Romanovsky"
> > > <leonro-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
> > > Cc: "Bart Van Assche" <bart.vanassche-XdAiOPVOjttBDgjK7y7TUQ@public.gmane.org>, "Doug Ledford"
> > > <dledford-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>, "Sagi Grimberg"
> > > <sagi-NQWnxTmZq1alnMjI0IkVqw@public.gmane.org>, "Israel Rukshin" <israelr-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>,
> > > linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> > > Sent: Wednesday, April 26, 2017 4:31:57 AM
> > > Subject: Re: [PATCH, untested] mlx5: Avoid that mlx5_ib_sg_to_klms()
> > > overflows the klms[] array
> > > 
> > > 
> > > 
> > > On 4/25/2017 11:37 PM, Laurence Oberman wrote:
> > > >
> > > >
> > > > ----- Original Message -----
> > > >> From: "Leon Romanovsky" <leonro-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
> > > >> To: "Bart Van Assche" <bart.vanassche-XdAiOPVOjttBDgjK7y7TUQ@public.gmane.org>
> > > >> Cc: "Doug Ledford" <dledford-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>, "Max Gurtovoy"
> > > >> <maxg-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>, "Sagi Grimberg" <sagi-NQWnxTmZq1alnMjI0IkVqw@public.gmane.org>,
> > > >> "Israel Rukshin" <israelr-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>, "Laurence Oberman"
> > > >> <loberman-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>, linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> > > >> Sent: Tuesday, April 25, 2017 1:58:49 PM
> > > >> Subject: Re: [PATCH, untested] mlx5: Avoid that mlx5_ib_sg_to_klms()
> > > >> overflows the klms[] array
> > > >>
> > > >> On Mon, Apr 24, 2017 at 03:15:28PM -0700, Bart Van Assche wrote:
> > > >>> ib_map_mr_sg() can pass an SG-list to .map_mr_sg() that is larger
> > > >>> than what fits into a single MR. .map_mr_sg() must not attempt to
> > > >>> map more SG-list elements than what fits into a single MR.
> > > >>> Hence make sure that mlx5_ib_sg_to_klms() does not write outside
> > > >>> the MR klms[] array.
> > > >>>
> > > >>> Fixes: b005d3164713 ("mlx5: Add arbitrary sg list support")
> > > >>> Signed-off-by: Bart Van Assche <bart.vanassche-XdAiOPVOjttBDgjK7y7TUQ@public.gmane.org>
> > > >>> Reviewed-by: Max Gurtovoy <maxg-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
> > > >>> Cc: Sagi Grimberg <sagi-NQWnxTmZq1alnMjI0IkVqw@public.gmane.org>
> > > >>> Cc: Leon Romanovsky <leonro-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
> > > >>> Cc: Israel Rukshin <israelr-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
> > > >>> Cc: <stable-u79uwXL29TY76Z2rM5mHXA@public.gmane.org>
> > > >>> ---
> > > >>>  drivers/infiniband/hw/mlx5/mr.c | 2 +-
> > > >>>  1 file changed, 1 insertion(+), 1 deletion(-)
> > > >>>
> > > >>
> > > >> Bart,
> > > >>
> > > >> Thanks a lot, it indeed looks right.
> > > >> Acked-by: Leon Romanovsky <leonro-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
> > > >>
> > > >> Thanks
> > > >>
> > > >
> > > >
> > > > Hello Bart, Leon, Max and Israel.
> > > >
> > > > I cloned off Barts tree.
> > > >
> > > > git clone https://github.com/bvanassche/linux
> > > > cd linux
> > > > git checkout block-scsi-for-next
> > > >
> > > > I checked all patches were in for this test.
> > > >
> > > > a83e404 IB/srp: Reenable IB_MR_TYPE_SG_GAPS
> > > > dfa5a2b mlx5: Avoid that mlx5_ib_sg_to_klms() overflows the klms[]
> > > > array
> > > > f759c80 mlx5: Fix mlx5_ib_map_mr_sg mr lengt
> > > 
> > > Hi,
> > > copying Sagi's request from different thread:
> > > 
> > > "
> > > Can you please enable srp_add_one debug:
> > > 
> > > echo "func srp_add_one +p" > /sys/kernel/debug/dynamic_debug/control
> > > 
> > > In addition apply the following:
> > > --
> > > diff --git a/drivers/infiniband/hw/mlx5/mr.c
> > > b/drivers/infiniband/hw/mlx5/mr.c
> > > index d9c6c0ea750b..040fbc387e4f 100644
> > > --- a/drivers/infiniband/hw/mlx5/mr.c
> > > +++ b/drivers/infiniband/hw/mlx5/mr.c
> > > @@ -1403,6 +1403,8 @@ mlx5_alloc_priv_descs(struct ib_device *device,
> > >          int add_size;
> > >          int ret;
> > > 
> > > +       WARN_ON_ONCE(ndescs > device->attr.max_fast_reg_page_list_len);
> > > +
> > >          add_size = max_t(int, MLX5_UMR_ALIGN - ARCH_KMALLOC_MINALIGN,
> > >          0);
> > > 
> > >          mr->descs_alloc = kzalloc(size + add_size, GFP_KERNEL);
> > > 
> > > "
> > > 
> > > Max.
> > > 
> > > >
> > > > Built and tested the kernel.
> > > >
> > > > However this issue is not resolved :(
> > > >
> > > > [ 2707.931909] scsi host1: ib_srp: failed RECV status WR flushed (5)
> > > > for
> > > > CQE ffff8817edca86b0
> > > > [ 2708.089806] mlx5_0:dump_cqe:262:(pid 20129): dump error cqe
> > > > [ 2708.121342] 00000000 00000000 00000000 00000000
> > > > [ 2708.147104] 00000000 00000000 00000000 00000000
> > > > [ 2708.172633] 00000000 00000000 00000000 00000000
> > > > [ 2708.198702] 00000000 0f007806 2500002a 14a527d0
> > > > [ 2732.434127] scsi host1: ib_srp: reconnect succeeded
> > > > [ 2733.048023] scsi host1: ib_srp: failed RECV status WR flushed (5)
> > > > for
> > > > CQE ffff8817ed0a9c30
> > > >
> > > > [root@localhost ~]# [ 2746.413277] mlx5_0:dump_cqe:262:(pid 15877):
> > > > dump
> > > > error cqe
> > > > [ 2746.443240] 00000000 00000000 00000000 00000000
> > > > [ 2746.469323] 00000000 00000000 00000000 00000000
> > > > [ 2746.495310] 00000000 00000000 00000000 00000000
> > > > [ 2746.521407] 00000000 0f007806 25000032 003c7ad0
> > > > [ 2752.445899] scsi host1: ib_srp: reconnect succeeded
> > > > [ 2752.481835] scsi host1: ib_srp: failed RECV status WR flushed (5)
> > > > for
> > > > CQE ffff8817ed0a9cf0
> > > > [ 2763.267386] mlx5_0:dump_cqe:262:(pid 15877): dump error cqe
> > > > [ 2763.297826] 00000000 00000000 00000000 00000000
> > > > [ 2763.323352] 00000000 00000000 00000000 00000000
> > > > [ 2763.348722] 00000000 00000000 00000000 00000000
> > > > [ 2763.374681] 00000000 0f007806 2500003a 00084bd0
> > > >
> > > > [root@localhost ~]# [ 2769.385203] fast_io_fail_tmo expired for SRP
> > > > port-1:1 / host1.
> > > > [ 2769.415956] scsi host1: ib_srp: reconnect succeeded
> > > > [ 2769.450258] scsi host1: ib_srp: failed RECV status WR flushed (5)
> > > > for
> > > > CQE ffff8817ed0a9cf0
> > > > [ 2780.064627] mlx5_0:dump_cqe:262:(pid 18771): dump error cqe
> > > > [ 2780.093520] 00000000 00000000 00000000 00000000
> > > > [ 2780.120067] 00000000 00000000 00000000 00000000
> > > > [ 2780.145575] 00000000 00000000 00000000 00000000
> > > > [ 2780.171153] 00000000 0f007806 25000042 000833d0
> > > > [ 2785.923399] scsi host1: ib_srp: reconnect succeeded
> > > > [ 2785.957504] scsi host1: ib_srp: failed RECV status WR flushed (5)
> > > > for
> > > > CQE ffff8817ed0a9cf0
> > > > [ 2796.463426] mlx5_0:dump_cqe:262:(pid 18771): dump error cqe
> > > > [ 2796.495257] 00000000 00000000 00000000 00000000
> > > > [ 2796.521506] 00000000 00000000 00000000 00000000
> > > > [ 2796.547640] 00000000 00000000 00000000 00000000
> > > > [ 2796.573120] 00000000 0f007806 2500004a 00083bd0
> > > > [ 2802.562578] scsi host1: ib_srp: reconnect succeeded
> > > > [ 2802.596880] scsi host1: ib_srp: failed RECV status WR flushed (5)
> > > > for
> > > > CQE ffff8817ed0a9cf0
> > > >
> > > > Regards
> > > > Laurence
> > > >
> > > 
> > Doing this now
> > Thanks
> > Laurence
> 
> Max
> 
> The Patch is not correct.
> 
> drivers/infiniband/hw/mlx5/mr.c: In function 'mlx5_alloc_priv_descs':
> drivers/infiniband/hw/mlx5/mr.c:1406:30: error: 'struct ib_device' has no
> member named 'attr'
>   WARN_ON_ONCE(ndescs > device->attr.max_fast_reg_page_list_len);
>                               ^
> ./include/asm-generic/bug.h:117:27: note: in definition of macro
> 'WARN_ON_ONCE'
>   int __ret_warn_once = !!(condition);   \
> 
> I think you meant to give me
> 
> WARN_ON_ONCE(ndescs > ib_device_attr->attr.max_fast_reg_page_list_len);
> 
> Can you confirm
> 
> Thanks
> Laurence


Oops rather this

WARN_ON_ONCE(ndescs > device->ib_device_attr.max_fast_reg_page_list_len);
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH, untested] mlx5: Avoid that mlx5_ib_sg_to_klms() overflows the klms[] array

[Max Gurtovoy]

On 4/26/2017 3:18 PM, Laurence Oberman wrote:
>
>
> ----- Original Message -----
>> From: "Laurence Oberman" <loberman-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>
>> To: "Max Gurtovoy" <maxg-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
>> Cc: "Leon Romanovsky" <leonro-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>, "Bart Van Assche" <bart.vanassche-XdAiOPVOjttBDgjK7y7TUQ@public.gmane.org>, "Doug Ledford"
>> <dledford-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>, "Sagi Grimberg" <sagi-NQWnxTmZq1alnMjI0IkVqw@public.gmane.org>, "Israel Rukshin" <israelr-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>,
>> linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
>> Sent: Wednesday, April 26, 2017 7:47:37 AM
>> Subject: Re: [PATCH, untested] mlx5: Avoid that mlx5_ib_sg_to_klms() overflows the klms[] array
>>
>>
>>
>> ----- Original Message -----
>>> From: "Max Gurtovoy" <maxg-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
>>> To: "Laurence Oberman" <loberman-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>, "Leon Romanovsky"
>>> <leonro-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
>>> Cc: "Bart Van Assche" <bart.vanassche-XdAiOPVOjttBDgjK7y7TUQ@public.gmane.org>, "Doug Ledford"
>>> <dledford-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>, "Sagi Grimberg"
>>> <sagi-NQWnxTmZq1alnMjI0IkVqw@public.gmane.org>, "Israel Rukshin" <israelr-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>,
>>> linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
>>> Sent: Wednesday, April 26, 2017 4:31:57 AM
>>> Subject: Re: [PATCH, untested] mlx5: Avoid that mlx5_ib_sg_to_klms()
>>> overflows the klms[] array
>>>
>>>
>>>
>>> On 4/25/2017 11:37 PM, Laurence Oberman wrote:
>>>>
>>>>
>>>> ----- Original Message -----
>>>>> From: "Leon Romanovsky" <leonro-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
>>>>> To: "Bart Van Assche" <bart.vanassche-XdAiOPVOjttBDgjK7y7TUQ@public.gmane.org>
>>>>> Cc: "Doug Ledford" <dledford-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>, "Max Gurtovoy"
>>>>> <maxg-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>, "Sagi Grimberg" <sagi-NQWnxTmZq1alnMjI0IkVqw@public.gmane.org>,
>>>>> "Israel Rukshin" <israelr-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>, "Laurence Oberman"
>>>>> <loberman-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>, linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
>>>>> Sent: Tuesday, April 25, 2017 1:58:49 PM
>>>>> Subject: Re: [PATCH, untested] mlx5: Avoid that mlx5_ib_sg_to_klms()
>>>>> overflows the klms[] array
>>>>>
>>>>> On Mon, Apr 24, 2017 at 03:15:28PM -0700, Bart Van Assche wrote:
>>>>>> ib_map_mr_sg() can pass an SG-list to .map_mr_sg() that is larger
>>>>>> than what fits into a single MR. .map_mr_sg() must not attempt to
>>>>>> map more SG-list elements than what fits into a single MR.
>>>>>> Hence make sure that mlx5_ib_sg_to_klms() does not write outside
>>>>>> the MR klms[] array.
>>>>>>
>>>>>> Fixes: b005d3164713 ("mlx5: Add arbitrary sg list support")
>>>>>> Signed-off-by: Bart Van Assche <bart.vanassche-XdAiOPVOjttBDgjK7y7TUQ@public.gmane.org>
>>>>>> Reviewed-by: Max Gurtovoy <maxg-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
>>>>>> Cc: Sagi Grimberg <sagi-NQWnxTmZq1alnMjI0IkVqw@public.gmane.org>
>>>>>> Cc: Leon Romanovsky <leonro-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
>>>>>> Cc: Israel Rukshin <israelr-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
>>>>>> Cc: <stable-u79uwXL29TY76Z2rM5mHXA@public.gmane.org>
>>>>>> ---
>>>>>>  drivers/infiniband/hw/mlx5/mr.c | 2 +-
>>>>>>  1 file changed, 1 insertion(+), 1 deletion(-)
>>>>>>
>>>>>
>>>>> Bart,
>>>>>
>>>>> Thanks a lot, it indeed looks right.
>>>>> Acked-by: Leon Romanovsky <leonro-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
>>>>>
>>>>> Thanks
>>>>>
>>>>
>>>>
>>>> Hello Bart, Leon, Max and Israel.
>>>>
>>>> I cloned off Barts tree.
>>>>
>>>> git clone https://github.com/bvanassche/linux
>>>> cd linux
>>>> git checkout block-scsi-for-next
>>>>
>>>> I checked all patches were in for this test.
>>>>
>>>> a83e404 IB/srp: Reenable IB_MR_TYPE_SG_GAPS
>>>> dfa5a2b mlx5: Avoid that mlx5_ib_sg_to_klms() overflows the klms[] array
>>>> f759c80 mlx5: Fix mlx5_ib_map_mr_sg mr lengt
>>>
>>> Hi,
>>> copying Sagi's request from different thread:
>>>
>>> "
>>> Can you please enable srp_add_one debug:
>>>
>>> echo "func srp_add_one +p" > /sys/kernel/debug/dynamic_debug/control
>>>
>>> In addition apply the following:
>>> --
>>> diff --git a/drivers/infiniband/hw/mlx5/mr.c
>>> b/drivers/infiniband/hw/mlx5/mr.c
>>> index d9c6c0ea750b..040fbc387e4f 100644
>>> --- a/drivers/infiniband/hw/mlx5/mr.c
>>> +++ b/drivers/infiniband/hw/mlx5/mr.c
>>> @@ -1403,6 +1403,8 @@ mlx5_alloc_priv_descs(struct ib_device *device,
>>>          int add_size;
>>>          int ret;
>>>
>>> +       WARN_ON_ONCE(ndescs > device->attr.max_fast_reg_page_list_len);
>>> +
>>>          add_size = max_t(int, MLX5_UMR_ALIGN - ARCH_KMALLOC_MINALIGN, 0);
>>>
>>>          mr->descs_alloc = kzalloc(size + add_size, GFP_KERNEL);
>>>
>>> "
>>>
>>> Max.
>>>
>>>>
>>>> Built and tested the kernel.
>>>>
>>>> However this issue is not resolved :(
>>>>
>>>> [ 2707.931909] scsi host1: ib_srp: failed RECV status WR flushed (5) for
>>>> CQE ffff8817edca86b0
>>>> [ 2708.089806] mlx5_0:dump_cqe:262:(pid 20129): dump error cqe
>>>> [ 2708.121342] 00000000 00000000 00000000 00000000
>>>> [ 2708.147104] 00000000 00000000 00000000 00000000
>>>> [ 2708.172633] 00000000 00000000 00000000 00000000
>>>> [ 2708.198702] 00000000 0f007806 2500002a 14a527d0
>>>> [ 2732.434127] scsi host1: ib_srp: reconnect succeeded
>>>> [ 2733.048023] scsi host1: ib_srp: failed RECV status WR flushed (5) for
>>>> CQE ffff8817ed0a9c30
>>>>
>>>> [root@localhost ~]# [ 2746.413277] mlx5_0:dump_cqe:262:(pid 15877): dump
>>>> error cqe
>>>> [ 2746.443240] 00000000 00000000 00000000 00000000
>>>> [ 2746.469323] 00000000 00000000 00000000 00000000
>>>> [ 2746.495310] 00000000 00000000 00000000 00000000
>>>> [ 2746.521407] 00000000 0f007806 25000032 003c7ad0
>>>> [ 2752.445899] scsi host1: ib_srp: reconnect succeeded
>>>> [ 2752.481835] scsi host1: ib_srp: failed RECV status WR flushed (5) for
>>>> CQE ffff8817ed0a9cf0
>>>> [ 2763.267386] mlx5_0:dump_cqe:262:(pid 15877): dump error cqe
>>>> [ 2763.297826] 00000000 00000000 00000000 00000000
>>>> [ 2763.323352] 00000000 00000000 00000000 00000000
>>>> [ 2763.348722] 00000000 00000000 00000000 00000000
>>>> [ 2763.374681] 00000000 0f007806 2500003a 00084bd0
>>>>
>>>> [root@localhost ~]# [ 2769.385203] fast_io_fail_tmo expired for SRP
>>>> port-1:1 / host1.
>>>> [ 2769.415956] scsi host1: ib_srp: reconnect succeeded
>>>> [ 2769.450258] scsi host1: ib_srp: failed RECV status WR flushed (5) for
>>>> CQE ffff8817ed0a9cf0
>>>> [ 2780.064627] mlx5_0:dump_cqe:262:(pid 18771): dump error cqe
>>>> [ 2780.093520] 00000000 00000000 00000000 00000000
>>>> [ 2780.120067] 00000000 00000000 00000000 00000000
>>>> [ 2780.145575] 00000000 00000000 00000000 00000000
>>>> [ 2780.171153] 00000000 0f007806 25000042 000833d0
>>>> [ 2785.923399] scsi host1: ib_srp: reconnect succeeded
>>>> [ 2785.957504] scsi host1: ib_srp: failed RECV status WR flushed (5) for
>>>> CQE ffff8817ed0a9cf0
>>>> [ 2796.463426] mlx5_0:dump_cqe:262:(pid 18771): dump error cqe
>>>> [ 2796.495257] 00000000 00000000 00000000 00000000
>>>> [ 2796.521506] 00000000 00000000 00000000 00000000
>>>> [ 2796.547640] 00000000 00000000 00000000 00000000
>>>> [ 2796.573120] 00000000 0f007806 2500004a 00083bd0
>>>> [ 2802.562578] scsi host1: ib_srp: reconnect succeeded
>>>> [ 2802.596880] scsi host1: ib_srp: failed RECV status WR flushed (5) for
>>>> CQE ffff8817ed0a9cf0
>>>>
>>>> Regards
>>>> Laurence
>>>>
>>>
>> Doing this now
>> Thanks
>> Laurence
>
> Max
>
> The Patch is not correct.
>
> drivers/infiniband/hw/mlx5/mr.c: In function 'mlx5_alloc_priv_descs':
> drivers/infiniband/hw/mlx5/mr.c:1406:30: error: 'struct ib_device' has no member named 'attr'
>   WARN_ON_ONCE(ndescs > device->attr.max_fast_reg_page_list_len);
>                               ^
> ./include/asm-generic/bug.h:117:27: note: in definition of macro 'WARN_ON_ONCE'
>   int __ret_warn_once = !!(condition);   \
>
> I think you meant to give me
>
> WARN_ON_ONCE(ndescs > ib_device_attr->attr.max_fast_reg_page_list_len);
>
> Can you confirm

Hi Laurence,
should be device->attrs.max_fast_reg_page_list_len.

please check this one that might solve the issue (on top of everything):


diff --git a/drivers/infiniband/hw/mlx5/mr.c 
b/drivers/infiniband/hw/mlx5/mr.c
index b8f9382..063d116 100644
--- a/drivers/infiniband/hw/mlx5/mr.c
+++ b/drivers/infiniband/hw/mlx5/mr.c
@@ -1559,7 +1559,7 @@ struct ib_mr *mlx5_ib_alloc_mr(struct ib_pd *pd,
                 mr->max_descs = ndescs;
         } else if (mr_type == IB_MR_TYPE_SG_GAPS) {
                 mr->access_mode = MLX5_MKC_ACCESS_MODE_KLMS;
-
+               MLX5_SET(mkc, mkc, translations_octword_size, 
ALIGN(max_num_sg + 1, 4));
                 err = mlx5_alloc_priv_descs(pd->device, mr,
                                             ndescs, sizeof(struct 
mlx5_klm));
                 if (err)

thanks,
Max.

>
> Thanks
> Laurence
>
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH, untested] mlx5: Avoid that mlx5_ib_sg_to_klms() overflows the klms[] array

[Laurence Oberman]

----- Original Message -----
> From: "Max Gurtovoy" <maxg-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
> To: "Laurence Oberman" <loberman-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>
> Cc: "Leon Romanovsky" <leonro-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>, "Bart Van Assche" <bart.vanassche-XdAiOPVOjttBDgjK7y7TUQ@public.gmane.org>, "Doug Ledford"
> <dledford-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>, "Sagi Grimberg" <sagi-NQWnxTmZq1alnMjI0IkVqw@public.gmane.org>, "Israel Rukshin" <israelr-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>,
> linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> Sent: Wednesday, April 26, 2017 8:25:30 AM
> Subject: Re: [PATCH, untested] mlx5: Avoid that mlx5_ib_sg_to_klms() overflows the klms[] array
> 
> 
> 
> On 4/26/2017 3:18 PM, Laurence Oberman wrote:
> >
> >
> > ----- Original Message -----
> >> From: "Laurence Oberman" <loberman-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>
> >> To: "Max Gurtovoy" <maxg-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
> >> Cc: "Leon Romanovsky" <leonro-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>, "Bart Van Assche"
> >> <bart.vanassche-XdAiOPVOjttBDgjK7y7TUQ@public.gmane.org>, "Doug Ledford"
> >> <dledford-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>, "Sagi Grimberg" <sagi-NQWnxTmZq1alnMjI0IkVqw@public.gmane.org>, "Israel
> >> Rukshin" <israelr-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>,
> >> linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> >> Sent: Wednesday, April 26, 2017 7:47:37 AM
> >> Subject: Re: [PATCH, untested] mlx5: Avoid that mlx5_ib_sg_to_klms()
> >> overflows the klms[] array
> >>
> >>
> >>
> >> ----- Original Message -----
> >>> From: "Max Gurtovoy" <maxg-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
> >>> To: "Laurence Oberman" <loberman-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>, "Leon Romanovsky"
> >>> <leonro-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
> >>> Cc: "Bart Van Assche" <bart.vanassche-XdAiOPVOjttBDgjK7y7TUQ@public.gmane.org>, "Doug Ledford"
> >>> <dledford-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>, "Sagi Grimberg"
> >>> <sagi-NQWnxTmZq1alnMjI0IkVqw@public.gmane.org>, "Israel Rukshin" <israelr-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>,
> >>> linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> >>> Sent: Wednesday, April 26, 2017 4:31:57 AM
> >>> Subject: Re: [PATCH, untested] mlx5: Avoid that mlx5_ib_sg_to_klms()
> >>> overflows the klms[] array
> >>>
> >>>
> >>>
> >>> On 4/25/2017 11:37 PM, Laurence Oberman wrote:
> >>>>
> >>>>
> >>>> ----- Original Message -----
> >>>>> From: "Leon Romanovsky" <leonro-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
> >>>>> To: "Bart Van Assche" <bart.vanassche-XdAiOPVOjttBDgjK7y7TUQ@public.gmane.org>
> >>>>> Cc: "Doug Ledford" <dledford-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>, "Max Gurtovoy"
> >>>>> <maxg-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>, "Sagi Grimberg" <sagi-NQWnxTmZq1alnMjI0IkVqw@public.gmane.org>,
> >>>>> "Israel Rukshin" <israelr-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>, "Laurence Oberman"
> >>>>> <loberman-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>, linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> >>>>> Sent: Tuesday, April 25, 2017 1:58:49 PM
> >>>>> Subject: Re: [PATCH, untested] mlx5: Avoid that mlx5_ib_sg_to_klms()
> >>>>> overflows the klms[] array
> >>>>>
> >>>>> On Mon, Apr 24, 2017 at 03:15:28PM -0700, Bart Van Assche wrote:
> >>>>>> ib_map_mr_sg() can pass an SG-list to .map_mr_sg() that is larger
> >>>>>> than what fits into a single MR. .map_mr_sg() must not attempt to
> >>>>>> map more SG-list elements than what fits into a single MR.
> >>>>>> Hence make sure that mlx5_ib_sg_to_klms() does not write outside
> >>>>>> the MR klms[] array.
> >>>>>>
> >>>>>> Fixes: b005d3164713 ("mlx5: Add arbitrary sg list support")
> >>>>>> Signed-off-by: Bart Van Assche <bart.vanassche-XdAiOPVOjttBDgjK7y7TUQ@public.gmane.org>
> >>>>>> Reviewed-by: Max Gurtovoy <maxg-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
> >>>>>> Cc: Sagi Grimberg <sagi-NQWnxTmZq1alnMjI0IkVqw@public.gmane.org>
> >>>>>> Cc: Leon Romanovsky <leonro-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
> >>>>>> Cc: Israel Rukshin <israelr-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
> >>>>>> Cc: <stable-u79uwXL29TY76Z2rM5mHXA@public.gmane.org>
> >>>>>> ---
> >>>>>>  drivers/infiniband/hw/mlx5/mr.c | 2 +-
> >>>>>>  1 file changed, 1 insertion(+), 1 deletion(-)
> >>>>>>
> >>>>>
> >>>>> Bart,
> >>>>>
> >>>>> Thanks a lot, it indeed looks right.
> >>>>> Acked-by: Leon Romanovsky <leonro-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
> >>>>>
> >>>>> Thanks
> >>>>>
> >>>>
> >>>>
> >>>> Hello Bart, Leon, Max and Israel.
> >>>>
> >>>> I cloned off Barts tree.
> >>>>
> >>>> git clone https://github.com/bvanassche/linux
> >>>> cd linux
> >>>> git checkout block-scsi-for-next
> >>>>
> >>>> I checked all patches were in for this test.
> >>>>
> >>>> a83e404 IB/srp: Reenable IB_MR_TYPE_SG_GAPS
> >>>> dfa5a2b mlx5: Avoid that mlx5_ib_sg_to_klms() overflows the klms[] array
> >>>> f759c80 mlx5: Fix mlx5_ib_map_mr_sg mr lengt
> >>>
> >>> Hi,
> >>> copying Sagi's request from different thread:
> >>>
> >>> "
> >>> Can you please enable srp_add_one debug:
> >>>
> >>> echo "func srp_add_one +p" > /sys/kernel/debug/dynamic_debug/control
> >>>
> >>> In addition apply the following:
> >>> --
> >>> diff --git a/drivers/infiniband/hw/mlx5/mr.c
> >>> b/drivers/infiniband/hw/mlx5/mr.c
> >>> index d9c6c0ea750b..040fbc387e4f 100644
> >>> --- a/drivers/infiniband/hw/mlx5/mr.c
> >>> +++ b/drivers/infiniband/hw/mlx5/mr.c
> >>> @@ -1403,6 +1403,8 @@ mlx5_alloc_priv_descs(struct ib_device *device,
> >>>          int add_size;
> >>>          int ret;
> >>>
> >>> +       WARN_ON_ONCE(ndescs > device->attr.max_fast_reg_page_list_len);
> >>> +
> >>>          add_size = max_t(int, MLX5_UMR_ALIGN - ARCH_KMALLOC_MINALIGN,
> >>>          0);
> >>>
> >>>          mr->descs_alloc = kzalloc(size + add_size, GFP_KERNEL);
> >>>
> >>> "
> >>>
> >>> Max.
> >>>
> >>>>
> >>>> Built and tested the kernel.
> >>>>
> >>>> However this issue is not resolved :(
> >>>>
> >>>> [ 2707.931909] scsi host1: ib_srp: failed RECV status WR flushed (5) for
> >>>> CQE ffff8817edca86b0
> >>>> [ 2708.089806] mlx5_0:dump_cqe:262:(pid 20129): dump error cqe
> >>>> [ 2708.121342] 00000000 00000000 00000000 00000000
> >>>> [ 2708.147104] 00000000 00000000 00000000 00000000
> >>>> [ 2708.172633] 00000000 00000000 00000000 00000000
> >>>> [ 2708.198702] 00000000 0f007806 2500002a 14a527d0
> >>>> [ 2732.434127] scsi host1: ib_srp: reconnect succeeded
> >>>> [ 2733.048023] scsi host1: ib_srp: failed RECV status WR flushed (5) for
> >>>> CQE ffff8817ed0a9c30
> >>>>
> >>>> [root@localhost ~]# [ 2746.413277] mlx5_0:dump_cqe:262:(pid 15877): dump
> >>>> error cqe
> >>>> [ 2746.443240] 00000000 00000000 00000000 00000000
> >>>> [ 2746.469323] 00000000 00000000 00000000 00000000
> >>>> [ 2746.495310] 00000000 00000000 00000000 00000000
> >>>> [ 2746.521407] 00000000 0f007806 25000032 003c7ad0
> >>>> [ 2752.445899] scsi host1: ib_srp: reconnect succeeded
> >>>> [ 2752.481835] scsi host1: ib_srp: failed RECV status WR flushed (5) for
> >>>> CQE ffff8817ed0a9cf0
> >>>> [ 2763.267386] mlx5_0:dump_cqe:262:(pid 15877): dump error cqe
> >>>> [ 2763.297826] 00000000 00000000 00000000 00000000
> >>>> [ 2763.323352] 00000000 00000000 00000000 00000000
> >>>> [ 2763.348722] 00000000 00000000 00000000 00000000
> >>>> [ 2763.374681] 00000000 0f007806 2500003a 00084bd0
> >>>>
> >>>> [root@localhost ~]# [ 2769.385203] fast_io_fail_tmo expired for SRP
> >>>> port-1:1 / host1.
> >>>> [ 2769.415956] scsi host1: ib_srp: reconnect succeeded
> >>>> [ 2769.450258] scsi host1: ib_srp: failed RECV status WR flushed (5) for
> >>>> CQE ffff8817ed0a9cf0
> >>>> [ 2780.064627] mlx5_0:dump_cqe:262:(pid 18771): dump error cqe
> >>>> [ 2780.093520] 00000000 00000000 00000000 00000000
> >>>> [ 2780.120067] 00000000 00000000 00000000 00000000
> >>>> [ 2780.145575] 00000000 00000000 00000000 00000000
> >>>> [ 2780.171153] 00000000 0f007806 25000042 000833d0
> >>>> [ 2785.923399] scsi host1: ib_srp: reconnect succeeded
> >>>> [ 2785.957504] scsi host1: ib_srp: failed RECV status WR flushed (5) for
> >>>> CQE ffff8817ed0a9cf0
> >>>> [ 2796.463426] mlx5_0:dump_cqe:262:(pid 18771): dump error cqe
> >>>> [ 2796.495257] 00000000 00000000 00000000 00000000
> >>>> [ 2796.521506] 00000000 00000000 00000000 00000000
> >>>> [ 2796.547640] 00000000 00000000 00000000 00000000
> >>>> [ 2796.573120] 00000000 0f007806 2500004a 00083bd0
> >>>> [ 2802.562578] scsi host1: ib_srp: reconnect succeeded
> >>>> [ 2802.596880] scsi host1: ib_srp: failed RECV status WR flushed (5) for
> >>>> CQE ffff8817ed0a9cf0
> >>>>
> >>>> Regards
> >>>> Laurence
> >>>>
> >>>
> >> Doing this now
> >> Thanks
> >> Laurence
> >
> > Max
> >
> > The Patch is not correct.
> >
> > drivers/infiniband/hw/mlx5/mr.c: In function 'mlx5_alloc_priv_descs':
> > drivers/infiniband/hw/mlx5/mr.c:1406:30: error: 'struct ib_device' has no
> > member named 'attr'
> >   WARN_ON_ONCE(ndescs > device->attr.max_fast_reg_page_list_len);
> >                               ^
> > ./include/asm-generic/bug.h:117:27: note: in definition of macro
> > 'WARN_ON_ONCE'
> >   int __ret_warn_once = !!(condition);   \
> >
> > I think you meant to give me
> >
> > WARN_ON_ONCE(ndescs > ib_device_attr->attr.max_fast_reg_page_list_len);
> >
> > Can you confirm
> 
> Hi Laurence,
> should be device->attrs.max_fast_reg_page_list_len.
> 
> please check this one that might solve the issue (on top of everything):
> 
> 
> diff --git a/drivers/infiniband/hw/mlx5/mr.c
> b/drivers/infiniband/hw/mlx5/mr.c
> index b8f9382..063d116 100644
> --- a/drivers/infiniband/hw/mlx5/mr.c
> +++ b/drivers/infiniband/hw/mlx5/mr.c
> @@ -1559,7 +1559,7 @@ struct ib_mr *mlx5_ib_alloc_mr(struct ib_pd *pd,
>                  mr->max_descs = ndescs;
>          } else if (mr_type == IB_MR_TYPE_SG_GAPS) {
>                  mr->access_mode = MLX5_MKC_ACCESS_MODE_KLMS;
> -
> +               MLX5_SET(mkc, mkc, translations_octword_size,
> ALIGN(max_num_sg + 1, 4));
>                  err = mlx5_alloc_priv_descs(pd->device, mr,
>                                              ndescs, sizeof(struct
> mlx5_klm));
>                  if (err)
> 
> thanks,
> Max.
> 
> >
> > Thanks
> > Laurence
> >
> --
> To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 

Hello Max

I have the corrected WARN_ON_ONCE patch and the above patch as well as the rest as it was from Barts tree.

Still fails.

For a baseline I can revert 
a83e404 IB/srp: Reenable IB_MR_TYPE_SG_GAPS

Then test again to make sure we are starting from a good place.

Initiator log

[  280.481951] scsi host1: ib_srp: failed FAST REG status memory management operation error (6) for CQE ffff8817d9a881b8
[  301.149106] scsi host1: ib_srp: reconnect succeeded
[  301.280635] scsi host1: ib_srp: failed RECV status WR flushed (5) for CQE ffff8817ed32f2f0
[  334.596420] scsi host2: ib_srp: failed RECV status WR flushed (5) for CQE ffff8817c592c970
[  334.599689] mlx5_1:dump_cqe:262:(pid 20): dump error cqe
[  334.599691] 00000000 00000000 00000000 00000000
[  334.599692] 00000000 00000000 00000000 00000000
[  334.599692] 00000000 00000000 00000000 00000000
[  334.599693] 00000000 0f007806 2500002d 067b48d0
[  334.599697] scsi host2: ib_srp: failed FAST REG status memory management operation error (6) for CQE ffff8817c6e30078
[  336.117248] mlx5_0:dump_cqe:262:(pid 130): dump error cqe
[  336.145840] 00000000 00000000 00000000 00000000
[  336.171830] 00000000 00000000 00000000 00000000
[  336.197688] 00000000 00000000 00000000 00000000
[  336.223720] 00000000 0f007806 25000032 005408d0
[  339.712706] fast_io_fail_tmo expired for SRP port-1:1 / host1.
[  341.453634] scsi host1: ib_srp: reconnect succeeded
[  341.481600] mlx5_0:dump_cqe:262:(pid 130): dump error cqe
[  341.482145] scsi host1: ib_srp: failed RECV status WR flushed (5) for CQE ffff8817ecaf6970
[  341.559359] 00000000 00000000 00000000 00000000
[  341.585397] 00000000 00000000 00000000 00000000
[  341.610948] 00000000 00000000 00000000 00000000
[  341.637515] 00000000 0f007806 2500003d 000046d0
[  342.297598] sd 1:0:0:9: rejecting I/O to offline device
[  342.297936] sd 1:0:0:9: [sdg] tag#28 FAILED Result: hostbyte=DID_TRANSPORT_FAILFAST driverbyte=DRIVER_OK
[  342.297941] sd 1:0:0:9: [sdg] tag#28 CDB: Write(10) 2a 00 00 00 40 00 00 40 00 00
[  342.297943] blk_update_request: recoverable transport error, dev sdg, sector 16384
[  342.297951] sd 1:0:0:20: [sdar] tag#5 FAILED Result: hostbyte=DID_TRANSPORT_FAILFAST driverbyte=DRIVER_OK
[  342.297952] sd 1:0:0:20: [sdar] tag#15 FAILED Result: hostbyte=DID_TRANSPORT_FAILFAST driverbyte=DRIVER_OK
[  342.297956] sd 1:0:0:20: [sdar] tag#5 CDB: Write(10) 2a 00 00 03 c0 00 00 40 00 00
[  342.297956] sd 1:0:0:20: [sdar] tag#15 CDB: Write(10) 2a 00 00 2c c0 00 00 40 00 00
[  342.297958] blk_update_request: recoverable transport error, dev sdar, sector 245760
[  342.297959] blk_update_request: recoverable transport error, dev sdar, sector 2932736
[  342.298119] device-mapper: multipath: Failing path 8:96.
[  342.298266] sd 1:0:0:9: [sdg] tag#29 FAILED Result: hostbyte=DID_TRANSPORT_FAILFAST driverbyte=DRIVER_OK
[  342.298268] sd 1:0:0:9: [sdg] tag#29 CDB: Write(10) 2a 00 00 00 c0 00 00 40 00 00
[  342.298269] blk_update_request: recoverable transport error, dev sdg, sector 49152
[  342.298300] device-mapper: multipath: Failing path 66:176.
[  342.298486] sd 1:0:0:20: [sdar] tag#16 FAILED Result: hostbyte=DID_TRANSPORT_FAILFAST driverbyte=DRIVER_OK
[  342.298488] sd 1:0:0:20: [sdar] tag#6 FAILED Result: hostbyte=DID_TRANSPORT_FAILFAST driverbyte=DRIVER_OK
[  342.298489] sd 1:0:0:20: [sdar] tag#16 CDB: Write(10) 2a 00 00 2d 40 00 00 40 00 00
[  342.298490] sd 1:0:0:20: [sdar] tag#6 CDB: Write(10) 2a 00 00 04 40 00 00 40 00 00
[  342.298491] blk_update_request: recoverable transport error, dev sdar, sector 2965504
[  342.298492] blk_update_request: recoverable transport error, dev sdar, sector 278528
[  342.298582] sd 1:0:0:9: [sdg] tag#30 FAILED Result: hostbyte=DID_TRANSPORT_FAILFAST driverbyte=DRIVER_OK
[  342.298584] sd 1:0:0:9: [sdg] tag#30 CDB: Write(10) 2a 00 00 01 40 00 00 40 00 00
[  342.298585] blk_update_request: recoverable transport error, dev sdg, sector 81920
[  342.298889] sd 1:0:0:9: [sdg] tag#31 FAILED Result: hostbyte=DID_TRANSPORT_FAILFAST driverbyte=DRIVER_OK
[  342.298890] sd 1:0:0:9: [sdg] tag#31 CDB: Write(10) 2a 00 00 01 c0 00 00 40 00 00
[  342.298891] blk_update_request: recoverable transport error, dev sdg, sector 114688
[  342.298981] sd 1:0:0:20: [sdar] tag#7 FAILED Result: hostbyte=DID_TRANSPORT_FAILFAST driverbyte=DRIVER_OK
[  342.298983] sd 1:0:0:20: [sdar] tag#7 CDB: Write(10) 2a 00 00 04 c0 00 00 40 00 00
[  342.298985] blk_update_request: recoverable transport error, dev sdar, sector 311296
[  342.299004] sd 1:0:0:20: [sdar] tag#17 FAILED Result: hostbyte=DID_TRANSPORT_FAILFAST driverbyte=DRIVER_OK
[  342.299007] sd 1:0:0:20: [sdar] tag#17 CDB: Write(10) 2a 00 00 34 c0 00 00 40 00 00
[  342.299009] blk_update_request: recoverable transport error, dev sdar, sector 3457024
[  342.356353] device-mapper: multipath: Failing path 8:64.
[  342.356489] device-mapper: multipath: Failing path 8:128.
[  342.356628] device-mapper: multipath: Failing path 8:160.
[  342.356699] device-mapper: multipath: Failing path 8:176.
[  342.356767] device-mapper: multipath: Failing path 8:240.
[  342.356834] device-mapper: multipath: Failing path 8:208.
[  342.356900] device-mapper: multipath: Failing path 65:16.
[  342.356967] device-mapper: multipath: Failing path 65:64.
[  342.357035] device-mapper: multipath: Failing path 65:96.
[  342.357103] device-mapper: multipath: Failing path 65:128.
[  342.357169] device-mapper: multipath: Failing path 65:176.
[  342.357237] device-mapper: multipath: Failing path 65:208.
[  342.357303] device-mapper: multipath: Failing path 65:224.
[  342.357371] device-mapper: multipath: Failing path 66:0.
[  342.357454] device-mapper: multipath: Failing path 66:32.
[  342.357521] device-mapper: multipath: Failing path 66:48.
[  342.357647] device-mapper: multipath: Failing path 66:80.
[  342.357714] device-mapper: multipath: Failing path 66:112.
[  342.357781] device-mapper: multipath: Failing path 66:144.
[  342.357936] device-mapper: multipath: Failing path 66:208.
[  342.358019] device-mapper: multipath: Failing path 66:240.
[  342.358115] device-mapper: multipath: Failing path 67:16.
[  342.358183] device-mapper: multipath: Failing path 67:48.
[  342.358264] device-mapper: multipath: Failing path 67:80.
[  342.358359] device-mapper: multipath: Failing path 67:128.
[  342.358442] device-mapper: multipath: Failing path 67:160.
[  342.358594] device-mapper: multipath: Failing path 67:224.
[  342.358671] device-mapper: multipath: Failing path 67:208.
[  350.157728] scsi host2: ib_srp: reconnect succeeded
[  350.189605] mlx5_1:dump_cqe:262:(pid 4756): dump error cqe
[  350.193180] mlx5_1:dump_cqe:262:(pid 1275): dump error cqe
[  350.193182] 00000000 00000000 00000000 00000000
[  350.193182] 00000000 00000000 00000000 00000000
[  350.193183] 00000000 00000000 00000000 00000000
[  350.193183] 00000000 0f007806 25000035 04f569d0
[  350.193187] scsi host2: ib_srp: failed FAST REG status memory management operation error (6) for CQE ffff8817c6e30078
[  350.412637] 00000000 00000000 00000000 00000000
[  350.436431] 00000000 00000000 00000000 00000000
[  350.461871] 00000000 00000000 00000000 00000000
[  350.487549] 00000000 0f007806 25000032 000843d0

Target Log

Thee events happened after the first failures on the initiator

[ 1111.029847] ib_srpt Received CM TimeWait exit for ch 0x4f6e72000390fe7c7cfe900300726ed3-49.
[ 1111.078815] ib_srpt Received CM TimeWait exit for ch 0x4f6e72000390fe7c7cfe900300726ed3-48.
[ 1111.127420] ib_srpt Received CM TimeWait exit for ch 0x4f6e72000390fe7c7cfe900300726ed3-47.
[ 1111.175801] ib_srpt Received CM TimeWait exit for ch 0x4f6e72000390fe7c7cfe900300726ed3-46.
[ 1111.223725] ib_srpt Received CM TimeWait exit for ch 0x4f6e72000390fe7c7cfe900300726ed3-45.
[ 1111.271957] ib_srpt Received CM TimeWait exit for ch 0x4f6e72000390fe7c7cfe900300726ed3-44.
[ 1111.319494] ib_srpt Received CM TimeWait exit for ch 0x4f6e72000390fe7c7cfe900300726ed3-43.
[ 1111.365795] ib_srpt Received CM TimeWait exit for ch 0x4f6e72000390fe7c7cfe900300726ed3-42.

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH, untested] mlx5: Avoid that mlx5_ib_sg_to_klms() overflows the klms[] array

[Laurence Oberman]

----- Original Message -----
> From: "Laurence Oberman" <loberman-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>
> To: "Max Gurtovoy" <maxg-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
> Cc: "Leon Romanovsky" <leonro-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>, "Bart Van Assche" <bart.vanassche-XdAiOPVOjttBDgjK7y7TUQ@public.gmane.org>, "Doug Ledford"
> <dledford-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>, "Sagi Grimberg" <sagi-NQWnxTmZq1alnMjI0IkVqw@public.gmane.org>, "Israel Rukshin" <israelr-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>,
> linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> Sent: Wednesday, April 26, 2017 9:28:37 AM
> Subject: Re: [PATCH, untested] mlx5: Avoid that mlx5_ib_sg_to_klms() overflows the klms[] array
> 
> 
> 
> ----- Original Message -----
> > From: "Max Gurtovoy" <maxg-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
> > To: "Laurence Oberman" <loberman-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>
> > Cc: "Leon Romanovsky" <leonro-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>, "Bart Van Assche"
> > <bart.vanassche-XdAiOPVOjttBDgjK7y7TUQ@public.gmane.org>, "Doug Ledford"
> > <dledford-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>, "Sagi Grimberg" <sagi-NQWnxTmZq1alnMjI0IkVqw@public.gmane.org>, "Israel Rukshin"
> > <israelr-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>,
> > linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> > Sent: Wednesday, April 26, 2017 8:25:30 AM
> > Subject: Re: [PATCH, untested] mlx5: Avoid that mlx5_ib_sg_to_klms()
> > overflows the klms[] array
> > 
> > 
> > 
> > On 4/26/2017 3:18 PM, Laurence Oberman wrote:
> > >
> > >
> > > ----- Original Message -----
> > >> From: "Laurence Oberman" <loberman-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>
> > >> To: "Max Gurtovoy" <maxg-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
> > >> Cc: "Leon Romanovsky" <leonro-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>, "Bart Van Assche"
> > >> <bart.vanassche-XdAiOPVOjttBDgjK7y7TUQ@public.gmane.org>, "Doug Ledford"
> > >> <dledford-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>, "Sagi Grimberg" <sagi-NQWnxTmZq1alnMjI0IkVqw@public.gmane.org>, "Israel
> > >> Rukshin" <israelr-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>,
> > >> linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> > >> Sent: Wednesday, April 26, 2017 7:47:37 AM
> > >> Subject: Re: [PATCH, untested] mlx5: Avoid that mlx5_ib_sg_to_klms()
> > >> overflows the klms[] array
> > >>
> > >>
> > >>
> > >> ----- Original Message -----
> > >>> From: "Max Gurtovoy" <maxg-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
> > >>> To: "Laurence Oberman" <loberman-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>, "Leon Romanovsky"
> > >>> <leonro-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
> > >>> Cc: "Bart Van Assche" <bart.vanassche-XdAiOPVOjttBDgjK7y7TUQ@public.gmane.org>, "Doug Ledford"
> > >>> <dledford-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>, "Sagi Grimberg"
> > >>> <sagi-NQWnxTmZq1alnMjI0IkVqw@public.gmane.org>, "Israel Rukshin" <israelr-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>,
> > >>> linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> > >>> Sent: Wednesday, April 26, 2017 4:31:57 AM
> > >>> Subject: Re: [PATCH, untested] mlx5: Avoid that mlx5_ib_sg_to_klms()
> > >>> overflows the klms[] array
> > >>>
> > >>>
> > >>>
> > >>> On 4/25/2017 11:37 PM, Laurence Oberman wrote:
> > >>>>
> > >>>>
> > >>>> ----- Original Message -----
> > >>>>> From: "Leon Romanovsky" <leonro-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
> > >>>>> To: "Bart Van Assche" <bart.vanassche-XdAiOPVOjttBDgjK7y7TUQ@public.gmane.org>
> > >>>>> Cc: "Doug Ledford" <dledford-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>, "Max Gurtovoy"
> > >>>>> <maxg-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>, "Sagi Grimberg" <sagi-NQWnxTmZq1alnMjI0IkVqw@public.gmane.org>,
> > >>>>> "Israel Rukshin" <israelr-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>, "Laurence Oberman"
> > >>>>> <loberman-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>, linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> > >>>>> Sent: Tuesday, April 25, 2017 1:58:49 PM
> > >>>>> Subject: Re: [PATCH, untested] mlx5: Avoid that mlx5_ib_sg_to_klms()
> > >>>>> overflows the klms[] array
> > >>>>>
> > >>>>> On Mon, Apr 24, 2017 at 03:15:28PM -0700, Bart Van Assche wrote:
> > >>>>>> ib_map_mr_sg() can pass an SG-list to .map_mr_sg() that is larger
> > >>>>>> than what fits into a single MR. .map_mr_sg() must not attempt to
> > >>>>>> map more SG-list elements than what fits into a single MR.
> > >>>>>> Hence make sure that mlx5_ib_sg_to_klms() does not write outside
> > >>>>>> the MR klms[] array.
> > >>>>>>
> > >>>>>> Fixes: b005d3164713 ("mlx5: Add arbitrary sg list support")
> > >>>>>> Signed-off-by: Bart Van Assche <bart.vanassche-XdAiOPVOjttBDgjK7y7TUQ@public.gmane.org>
> > >>>>>> Reviewed-by: Max Gurtovoy <maxg-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
> > >>>>>> Cc: Sagi Grimberg <sagi-NQWnxTmZq1alnMjI0IkVqw@public.gmane.org>
> > >>>>>> Cc: Leon Romanovsky <leonro-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
> > >>>>>> Cc: Israel Rukshin <israelr-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
> > >>>>>> Cc: <stable-u79uwXL29TY76Z2rM5mHXA@public.gmane.org>
> > >>>>>> ---
> > >>>>>>  drivers/infiniband/hw/mlx5/mr.c | 2 +-
> > >>>>>>  1 file changed, 1 insertion(+), 1 deletion(-)
> > >>>>>>
> > >>>>>
> > >>>>> Bart,
> > >>>>>
> > >>>>> Thanks a lot, it indeed looks right.
> > >>>>> Acked-by: Leon Romanovsky <leonro-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
> > >>>>>
> > >>>>> Thanks
> > >>>>>
> > >>>>
> > >>>>
> > >>>> Hello Bart, Leon, Max and Israel.
> > >>>>
> > >>>> I cloned off Barts tree.
> > >>>>
> > >>>> git clone https://github.com/bvanassche/linux
> > >>>> cd linux
> > >>>> git checkout block-scsi-for-next
> > >>>>
> > >>>> I checked all patches were in for this test.
> > >>>>
> > >>>> a83e404 IB/srp: Reenable IB_MR_TYPE_SG_GAPS
> > >>>> dfa5a2b mlx5: Avoid that mlx5_ib_sg_to_klms() overflows the klms[]
> > >>>> array
> > >>>> f759c80 mlx5: Fix mlx5_ib_map_mr_sg mr lengt
> > >>>
> > >>> Hi,
> > >>> copying Sagi's request from different thread:
> > >>>
> > >>> "
> > >>> Can you please enable srp_add_one debug:
> > >>>
> > >>> echo "func srp_add_one +p" > /sys/kernel/debug/dynamic_debug/control
> > >>>
> > >>> In addition apply the following:
> > >>> --
> > >>> diff --git a/drivers/infiniband/hw/mlx5/mr.c
> > >>> b/drivers/infiniband/hw/mlx5/mr.c
> > >>> index d9c6c0ea750b..040fbc387e4f 100644
> > >>> --- a/drivers/infiniband/hw/mlx5/mr.c
> > >>> +++ b/drivers/infiniband/hw/mlx5/mr.c
> > >>> @@ -1403,6 +1403,8 @@ mlx5_alloc_priv_descs(struct ib_device *device,
> > >>>          int add_size;
> > >>>          int ret;
> > >>>
> > >>> +       WARN_ON_ONCE(ndescs > device->attr.max_fast_reg_page_list_len);
> > >>> +
> > >>>          add_size = max_t(int, MLX5_UMR_ALIGN - ARCH_KMALLOC_MINALIGN,
> > >>>          0);
> > >>>
> > >>>          mr->descs_alloc = kzalloc(size + add_size, GFP_KERNEL);
> > >>>
> > >>> "
> > >>>
> > >>> Max.
> > >>>
> > >>>>
> > >>>> Built and tested the kernel.
> > >>>>
> > >>>> However this issue is not resolved :(
> > >>>>
> > >>>> [ 2707.931909] scsi host1: ib_srp: failed RECV status WR flushed (5)
> > >>>> for
> > >>>> CQE ffff8817edca86b0
> > >>>> [ 2708.089806] mlx5_0:dump_cqe:262:(pid 20129): dump error cqe
> > >>>> [ 2708.121342] 00000000 00000000 00000000 00000000
> > >>>> [ 2708.147104] 00000000 00000000 00000000 00000000
> > >>>> [ 2708.172633] 00000000 00000000 00000000 00000000
> > >>>> [ 2708.198702] 00000000 0f007806 2500002a 14a527d0
> > >>>> [ 2732.434127] scsi host1: ib_srp: reconnect succeeded
> > >>>> [ 2733.048023] scsi host1: ib_srp: failed RECV status WR flushed (5)
> > >>>> for
> > >>>> CQE ffff8817ed0a9c30
> > >>>>
> > >>>> [root@localhost ~]# [ 2746.413277] mlx5_0:dump_cqe:262:(pid 15877):
> > >>>> dump
> > >>>> error cqe
> > >>>> [ 2746.443240] 00000000 00000000 00000000 00000000
> > >>>> [ 2746.469323] 00000000 00000000 00000000 00000000
> > >>>> [ 2746.495310] 00000000 00000000 00000000 00000000
> > >>>> [ 2746.521407] 00000000 0f007806 25000032 003c7ad0
> > >>>> [ 2752.445899] scsi host1: ib_srp: reconnect succeeded
> > >>>> [ 2752.481835] scsi host1: ib_srp: failed RECV status WR flushed (5)
> > >>>> for
> > >>>> CQE ffff8817ed0a9cf0
> > >>>> [ 2763.267386] mlx5_0:dump_cqe:262:(pid 15877): dump error cqe
> > >>>> [ 2763.297826] 00000000 00000000 00000000 00000000
> > >>>> [ 2763.323352] 00000000 00000000 00000000 00000000
> > >>>> [ 2763.348722] 00000000 00000000 00000000 00000000
> > >>>> [ 2763.374681] 00000000 0f007806 2500003a 00084bd0
> > >>>>
> > >>>> [root@localhost ~]# [ 2769.385203] fast_io_fail_tmo expired for SRP
> > >>>> port-1:1 / host1.
> > >>>> [ 2769.415956] scsi host1: ib_srp: reconnect succeeded
> > >>>> [ 2769.450258] scsi host1: ib_srp: failed RECV status WR flushed (5)
> > >>>> for
> > >>>> CQE ffff8817ed0a9cf0
> > >>>> [ 2780.064627] mlx5_0:dump_cqe:262:(pid 18771): dump error cqe
> > >>>> [ 2780.093520] 00000000 00000000 00000000 00000000
> > >>>> [ 2780.120067] 00000000 00000000 00000000 00000000
> > >>>> [ 2780.145575] 00000000 00000000 00000000 00000000
> > >>>> [ 2780.171153] 00000000 0f007806 25000042 000833d0
> > >>>> [ 2785.923399] scsi host1: ib_srp: reconnect succeeded
> > >>>> [ 2785.957504] scsi host1: ib_srp: failed RECV status WR flushed (5)
> > >>>> for
> > >>>> CQE ffff8817ed0a9cf0
> > >>>> [ 2796.463426] mlx5_0:dump_cqe:262:(pid 18771): dump error cqe
> > >>>> [ 2796.495257] 00000000 00000000 00000000 00000000
> > >>>> [ 2796.521506] 00000000 00000000 00000000 00000000
> > >>>> [ 2796.547640] 00000000 00000000 00000000 00000000
> > >>>> [ 2796.573120] 00000000 0f007806 2500004a 00083bd0
> > >>>> [ 2802.562578] scsi host1: ib_srp: reconnect succeeded
> > >>>> [ 2802.596880] scsi host1: ib_srp: failed RECV status WR flushed (5)
> > >>>> for
> > >>>> CQE ffff8817ed0a9cf0
> > >>>>
> > >>>> Regards
> > >>>> Laurence
> > >>>>
> > >>>
> > >> Doing this now
> > >> Thanks
> > >> Laurence
> > >
> > > Max
> > >
> > > The Patch is not correct.
> > >
> > > drivers/infiniband/hw/mlx5/mr.c: In function 'mlx5_alloc_priv_descs':
> > > drivers/infiniband/hw/mlx5/mr.c:1406:30: error: 'struct ib_device' has no
> > > member named 'attr'
> > >   WARN_ON_ONCE(ndescs > device->attr.max_fast_reg_page_list_len);
> > >                               ^
> > > ./include/asm-generic/bug.h:117:27: note: in definition of macro
> > > 'WARN_ON_ONCE'
> > >   int __ret_warn_once = !!(condition);   \
> > >
> > > I think you meant to give me
> > >
> > > WARN_ON_ONCE(ndescs > ib_device_attr->attr.max_fast_reg_page_list_len);
> > >
> > > Can you confirm
> > 
> > Hi Laurence,
> > should be device->attrs.max_fast_reg_page_list_len.
> > 
> > please check this one that might solve the issue (on top of everything):
> > 
> > 
> > diff --git a/drivers/infiniband/hw/mlx5/mr.c
> > b/drivers/infiniband/hw/mlx5/mr.c
> > index b8f9382..063d116 100644
> > --- a/drivers/infiniband/hw/mlx5/mr.c
> > +++ b/drivers/infiniband/hw/mlx5/mr.c
> > @@ -1559,7 +1559,7 @@ struct ib_mr *mlx5_ib_alloc_mr(struct ib_pd *pd,
> >                  mr->max_descs = ndescs;
> >          } else if (mr_type == IB_MR_TYPE_SG_GAPS) {
> >                  mr->access_mode = MLX5_MKC_ACCESS_MODE_KLMS;
> > -
> > +               MLX5_SET(mkc, mkc, translations_octword_size,
> > ALIGN(max_num_sg + 1, 4));
> >                  err = mlx5_alloc_priv_descs(pd->device, mr,
> >                                              ndescs, sizeof(struct
> > mlx5_klm));
> >                  if (err)
> > 
> > thanks,
> > Max.
> > 
> > >
> > > Thanks
> > > Laurence
> > >
> > --
> > To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
> > the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> > More majordomo info at  http://vger.kernel.org/majordomo-info.html
> > 
> 
> Hello Max
> 
> I have the corrected WARN_ON_ONCE patch and the above patch as well as the
> rest as it was from Barts tree.
> 
> Still fails.
> 
> For a baseline I can revert
> a83e404 IB/srp: Reenable IB_MR_TYPE_SG_GAPS
> 
> Then test again to make sure we are starting from a good place.
> 
> Initiator log
> 
> [  280.481951] scsi host1: ib_srp: failed FAST REG status memory management
> operation error (6) for CQE ffff8817d9a881b8
> [  301.149106] scsi host1: ib_srp: reconnect succeeded
> [  301.280635] scsi host1: ib_srp: failed RECV status WR flushed (5) for CQE
> ffff8817ed32f2f0
> [  334.596420] scsi host2: ib_srp: failed RECV status WR flushed (5) for CQE
> ffff8817c592c970
> [  334.599689] mlx5_1:dump_cqe:262:(pid 20): dump error cqe
> [  334.599691] 00000000 00000000 00000000 00000000
> [  334.599692] 00000000 00000000 00000000 00000000
> [  334.599692] 00000000 00000000 00000000 00000000
> [  334.599693] 00000000 0f007806 2500002d 067b48d0
> [  334.599697] scsi host2: ib_srp: failed FAST REG status memory management
> operation error (6) for CQE ffff8817c6e30078
> [  336.117248] mlx5_0:dump_cqe:262:(pid 130): dump error cqe
> [  336.145840] 00000000 00000000 00000000 00000000
> [  336.171830] 00000000 00000000 00000000 00000000
> [  336.197688] 00000000 00000000 00000000 00000000
> [  336.223720] 00000000 0f007806 25000032 005408d0
> [  339.712706] fast_io_fail_tmo expired for SRP port-1:1 / host1.
> [  341.453634] scsi host1: ib_srp: reconnect succeeded
> [  341.481600] mlx5_0:dump_cqe:262:(pid 130): dump error cqe
> [  341.482145] scsi host1: ib_srp: failed RECV status WR flushed (5) for CQE
> ffff8817ecaf6970
> [  341.559359] 00000000 00000000 00000000 00000000
> [  341.585397] 00000000 00000000 00000000 00000000
> [  341.610948] 00000000 00000000 00000000 00000000
> [  341.637515] 00000000 0f007806 2500003d 000046d0
> [  342.297598] sd 1:0:0:9: rejecting I/O to offline device
> [  342.297936] sd 1:0:0:9: [sdg] tag#28 FAILED Result:
> hostbyte=DID_TRANSPORT_FAILFAST driverbyte=DRIVER_OK
> [  342.297941] sd 1:0:0:9: [sdg] tag#28 CDB: Write(10) 2a 00 00 00 40 00 00
> 40 00 00
> [  342.297943] blk_update_request: recoverable transport error, dev sdg,
> sector 16384
> [  342.297951] sd 1:0:0:20: [sdar] tag#5 FAILED Result:
> hostbyte=DID_TRANSPORT_FAILFAST driverbyte=DRIVER_OK
> [  342.297952] sd 1:0:0:20: [sdar] tag#15 FAILED Result:
> hostbyte=DID_TRANSPORT_FAILFAST driverbyte=DRIVER_OK
> [  342.297956] sd 1:0:0:20: [sdar] tag#5 CDB: Write(10) 2a 00 00 03 c0 00 00
> 40 00 00
> [  342.297956] sd 1:0:0:20: [sdar] tag#15 CDB: Write(10) 2a 00 00 2c c0 00 00
> 40 00 00
> [  342.297958] blk_update_request: recoverable transport error, dev sdar,
> sector 245760
> [  342.297959] blk_update_request: recoverable transport error, dev sdar,
> sector 2932736
> [  342.298119] device-mapper: multipath: Failing path 8:96.
> [  342.298266] sd 1:0:0:9: [sdg] tag#29 FAILED Result:
> hostbyte=DID_TRANSPORT_FAILFAST driverbyte=DRIVER_OK
> [  342.298268] sd 1:0:0:9: [sdg] tag#29 CDB: Write(10) 2a 00 00 00 c0 00 00
> 40 00 00
> [  342.298269] blk_update_request: recoverable transport error, dev sdg,
> sector 49152
> [  342.298300] device-mapper: multipath: Failing path 66:176.
> [  342.298486] sd 1:0:0:20: [sdar] tag#16 FAILED Result:
> hostbyte=DID_TRANSPORT_FAILFAST driverbyte=DRIVER_OK
> [  342.298488] sd 1:0:0:20: [sdar] tag#6 FAILED Result:
> hostbyte=DID_TRANSPORT_FAILFAST driverbyte=DRIVER_OK
> [  342.298489] sd 1:0:0:20: [sdar] tag#16 CDB: Write(10) 2a 00 00 2d 40 00 00
> 40 00 00
> [  342.298490] sd 1:0:0:20: [sdar] tag#6 CDB: Write(10) 2a 00 00 04 40 00 00
> 40 00 00
> [  342.298491] blk_update_request: recoverable transport error, dev sdar,
> sector 2965504
> [  342.298492] blk_update_request: recoverable transport error, dev sdar,
> sector 278528
> [  342.298582] sd 1:0:0:9: [sdg] tag#30 FAILED Result:
> hostbyte=DID_TRANSPORT_FAILFAST driverbyte=DRIVER_OK
> [  342.298584] sd 1:0:0:9: [sdg] tag#30 CDB: Write(10) 2a 00 00 01 40 00 00
> 40 00 00
> [  342.298585] blk_update_request: recoverable transport error, dev sdg,
> sector 81920
> [  342.298889] sd 1:0:0:9: [sdg] tag#31 FAILED Result:
> hostbyte=DID_TRANSPORT_FAILFAST driverbyte=DRIVER_OK
> [  342.298890] sd 1:0:0:9: [sdg] tag#31 CDB: Write(10) 2a 00 00 01 c0 00 00
> 40 00 00
> [  342.298891] blk_update_request: recoverable transport error, dev sdg,
> sector 114688
> [  342.298981] sd 1:0:0:20: [sdar] tag#7 FAILED Result:
> hostbyte=DID_TRANSPORT_FAILFAST driverbyte=DRIVER_OK
> [  342.298983] sd 1:0:0:20: [sdar] tag#7 CDB: Write(10) 2a 00 00 04 c0 00 00
> 40 00 00
> [  342.298985] blk_update_request: recoverable transport error, dev sdar,
> sector 311296
> [  342.299004] sd 1:0:0:20: [sdar] tag#17 FAILED Result:
> hostbyte=DID_TRANSPORT_FAILFAST driverbyte=DRIVER_OK
> [  342.299007] sd 1:0:0:20: [sdar] tag#17 CDB: Write(10) 2a 00 00 34 c0 00 00
> 40 00 00
> [  342.299009] blk_update_request: recoverable transport error, dev sdar,
> sector 3457024
> [  342.356353] device-mapper: multipath: Failing path 8:64.
> [  342.356489] device-mapper: multipath: Failing path 8:128.
> [  342.356628] device-mapper: multipath: Failing path 8:160.
> [  342.356699] device-mapper: multipath: Failing path 8:176.
> [  342.356767] device-mapper: multipath: Failing path 8:240.
> [  342.356834] device-mapper: multipath: Failing path 8:208.
> [  342.356900] device-mapper: multipath: Failing path 65:16.
> [  342.356967] device-mapper: multipath: Failing path 65:64.
> [  342.357035] device-mapper: multipath: Failing path 65:96.
> [  342.357103] device-mapper: multipath: Failing path 65:128.
> [  342.357169] device-mapper: multipath: Failing path 65:176.
> [  342.357237] device-mapper: multipath: Failing path 65:208.
> [  342.357303] device-mapper: multipath: Failing path 65:224.
> [  342.357371] device-mapper: multipath: Failing path 66:0.
> [  342.357454] device-mapper: multipath: Failing path 66:32.
> [  342.357521] device-mapper: multipath: Failing path 66:48.
> [  342.357647] device-mapper: multipath: Failing path 66:80.
> [  342.357714] device-mapper: multipath: Failing path 66:112.
> [  342.357781] device-mapper: multipath: Failing path 66:144.
> [  342.357936] device-mapper: multipath: Failing path 66:208.
> [  342.358019] device-mapper: multipath: Failing path 66:240.
> [  342.358115] device-mapper: multipath: Failing path 67:16.
> [  342.358183] device-mapper: multipath: Failing path 67:48.
> [  342.358264] device-mapper: multipath: Failing path 67:80.
> [  342.358359] device-mapper: multipath: Failing path 67:128.
> [  342.358442] device-mapper: multipath: Failing path 67:160.
> [  342.358594] device-mapper: multipath: Failing path 67:224.
> [  342.358671] device-mapper: multipath: Failing path 67:208.
> [  350.157728] scsi host2: ib_srp: reconnect succeeded
> [  350.189605] mlx5_1:dump_cqe:262:(pid 4756): dump error cqe
> [  350.193180] mlx5_1:dump_cqe:262:(pid 1275): dump error cqe
> [  350.193182] 00000000 00000000 00000000 00000000
> [  350.193182] 00000000 00000000 00000000 00000000
> [  350.193183] 00000000 00000000 00000000 00000000
> [  350.193183] 00000000 0f007806 25000035 04f569d0
> [  350.193187] scsi host2: ib_srp: failed FAST REG status memory management
> operation error (6) for CQE ffff8817c6e30078
> [  350.412637] 00000000 00000000 00000000 00000000
> [  350.436431] 00000000 00000000 00000000 00000000
> [  350.461871] 00000000 00000000 00000000 00000000
> [  350.487549] 00000000 0f007806 25000032 000843d0
> 
> Target Log
> 
> Thee events happened after the first failures on the initiator
> 
> [ 1111.029847] ib_srpt Received CM TimeWait exit for ch
> 0x4f6e72000390fe7c7cfe900300726ed3-49.
> [ 1111.078815] ib_srpt Received CM TimeWait exit for ch
> 0x4f6e72000390fe7c7cfe900300726ed3-48.
> [ 1111.127420] ib_srpt Received CM TimeWait exit for ch
> 0x4f6e72000390fe7c7cfe900300726ed3-47.
> [ 1111.175801] ib_srpt Received CM TimeWait exit for ch
> 0x4f6e72000390fe7c7cfe900300726ed3-46.
> [ 1111.223725] ib_srpt Received CM TimeWait exit for ch
> 0x4f6e72000390fe7c7cfe900300726ed3-45.
> [ 1111.271957] ib_srpt Received CM TimeWait exit for ch
> 0x4f6e72000390fe7c7cfe900300726ed3-44.
> [ 1111.319494] ib_srpt Received CM TimeWait exit for ch
> 0x4f6e72000390fe7c7cfe900300726ed3-43.
> [ 1111.365795] ib_srpt Received CM TimeWait exit for ch
> 0x4f6e72000390fe7c7cfe900300726ed3-42.
> 
> --
> To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 

Max

These are the parameters all my tests run with.
Same as always.

[root@localhost modprobe.d]# cat ib_srp.conf
options ib_srp cmd_sg_entries=255 indirect_sg_entries=2048 

I dont set prefer_fr so it defaults to Y

[root@localhost parameters]# cat prefer_fr
Y

I have no settings for mlx5_core, all defaults.

Thanks
Laurence

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH, untested] mlx5: Avoid that mlx5_ib_sg_to_klms() overflows the klms[] array

[Laurence Oberman]

----- Original Message -----
> From: "Laurence Oberman" <loberman-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>
> To: "Max Gurtovoy" <maxg-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
> Cc: "Leon Romanovsky" <leonro-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>, "Bart Van Assche" <bart.vanassche-XdAiOPVOjttBDgjK7y7TUQ@public.gmane.org>, "Doug Ledford"
> <dledford-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>, "Sagi Grimberg" <sagi-NQWnxTmZq1alnMjI0IkVqw@public.gmane.org>, "Israel Rukshin" <israelr-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>,
> linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> Sent: Wednesday, April 26, 2017 9:50:25 AM
> Subject: Re: [PATCH, untested] mlx5: Avoid that mlx5_ib_sg_to_klms() overflows the klms[] array
> 
> 
> 
> ----- Original Message -----
> > From: "Laurence Oberman" <loberman-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>
> > To: "Max Gurtovoy" <maxg-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
> > Cc: "Leon Romanovsky" <leonro-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>, "Bart Van Assche"
> > <bart.vanassche-XdAiOPVOjttBDgjK7y7TUQ@public.gmane.org>, "Doug Ledford"
> > <dledford-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>, "Sagi Grimberg" <sagi-NQWnxTmZq1alnMjI0IkVqw@public.gmane.org>, "Israel Rukshin"
> > <israelr-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>,
> > linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> > Sent: Wednesday, April 26, 2017 9:28:37 AM
> > Subject: Re: [PATCH, untested] mlx5: Avoid that mlx5_ib_sg_to_klms()
> > overflows the klms[] array
> > 
> > 
> > 
> > ----- Original Message -----
> > > From: "Max Gurtovoy" <maxg-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
> > > To: "Laurence Oberman" <loberman-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>
> > > Cc: "Leon Romanovsky" <leonro-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>, "Bart Van Assche"
> > > <bart.vanassche-XdAiOPVOjttBDgjK7y7TUQ@public.gmane.org>, "Doug Ledford"
> > > <dledford-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>, "Sagi Grimberg" <sagi-NQWnxTmZq1alnMjI0IkVqw@public.gmane.org>, "Israel
> > > Rukshin"
> > > <israelr-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>,
> > > linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> > > Sent: Wednesday, April 26, 2017 8:25:30 AM
> > > Subject: Re: [PATCH, untested] mlx5: Avoid that mlx5_ib_sg_to_klms()
> > > overflows the klms[] array
> > > 
> > > 
> > > 
> > > On 4/26/2017 3:18 PM, Laurence Oberman wrote:
> > > >
> > > >
> > > > ----- Original Message -----
> > > >> From: "Laurence Oberman" <loberman-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>
> > > >> To: "Max Gurtovoy" <maxg-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
> > > >> Cc: "Leon Romanovsky" <leonro-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>, "Bart Van Assche"
> > > >> <bart.vanassche-XdAiOPVOjttBDgjK7y7TUQ@public.gmane.org>, "Doug Ledford"
> > > >> <dledford-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>, "Sagi Grimberg" <sagi-NQWnxTmZq1alnMjI0IkVqw@public.gmane.org>, "Israel
> > > >> Rukshin" <israelr-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>,
> > > >> linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> > > >> Sent: Wednesday, April 26, 2017 7:47:37 AM
> > > >> Subject: Re: [PATCH, untested] mlx5: Avoid that mlx5_ib_sg_to_klms()
> > > >> overflows the klms[] array
> > > >>
> > > >>
> > > >>
> > > >> ----- Original Message -----
> > > >>> From: "Max Gurtovoy" <maxg-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
> > > >>> To: "Laurence Oberman" <loberman-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>, "Leon Romanovsky"
> > > >>> <leonro-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
> > > >>> Cc: "Bart Van Assche" <bart.vanassche-XdAiOPVOjttBDgjK7y7TUQ@public.gmane.org>, "Doug Ledford"
> > > >>> <dledford-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>, "Sagi Grimberg"
> > > >>> <sagi-NQWnxTmZq1alnMjI0IkVqw@public.gmane.org>, "Israel Rukshin" <israelr-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>,
> > > >>> linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> > > >>> Sent: Wednesday, April 26, 2017 4:31:57 AM
> > > >>> Subject: Re: [PATCH, untested] mlx5: Avoid that mlx5_ib_sg_to_klms()
> > > >>> overflows the klms[] array
> > > >>>
> > > >>>
> > > >>>
> > > >>> On 4/25/2017 11:37 PM, Laurence Oberman wrote:
> > > >>>>
> > > >>>>
> > > >>>> ----- Original Message -----
> > > >>>>> From: "Leon Romanovsky" <leonro-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
> > > >>>>> To: "Bart Van Assche" <bart.vanassche-XdAiOPVOjttBDgjK7y7TUQ@public.gmane.org>
> > > >>>>> Cc: "Doug Ledford" <dledford-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>, "Max Gurtovoy"
> > > >>>>> <maxg-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>, "Sagi Grimberg" <sagi-NQWnxTmZq1alnMjI0IkVqw@public.gmane.org>,
> > > >>>>> "Israel Rukshin" <israelr-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>, "Laurence Oberman"
> > > >>>>> <loberman-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>, linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> > > >>>>> Sent: Tuesday, April 25, 2017 1:58:49 PM
> > > >>>>> Subject: Re: [PATCH, untested] mlx5: Avoid that
> > > >>>>> mlx5_ib_sg_to_klms()
> > > >>>>> overflows the klms[] array
> > > >>>>>
> > > >>>>> On Mon, Apr 24, 2017 at 03:15:28PM -0700, Bart Van Assche wrote:
> > > >>>>>> ib_map_mr_sg() can pass an SG-list to .map_mr_sg() that is larger
> > > >>>>>> than what fits into a single MR. .map_mr_sg() must not attempt to
> > > >>>>>> map more SG-list elements than what fits into a single MR.
> > > >>>>>> Hence make sure that mlx5_ib_sg_to_klms() does not write outside
> > > >>>>>> the MR klms[] array.
> > > >>>>>>
> > > >>>>>> Fixes: b005d3164713 ("mlx5: Add arbitrary sg list support")
> > > >>>>>> Signed-off-by: Bart Van Assche <bart.vanassche-XdAiOPVOjttBDgjK7y7TUQ@public.gmane.org>
> > > >>>>>> Reviewed-by: Max Gurtovoy <maxg-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
> > > >>>>>> Cc: Sagi Grimberg <sagi-NQWnxTmZq1alnMjI0IkVqw@public.gmane.org>
> > > >>>>>> Cc: Leon Romanovsky <leonro-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
> > > >>>>>> Cc: Israel Rukshin <israelr-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
> > > >>>>>> Cc: <stable-u79uwXL29TY76Z2rM5mHXA@public.gmane.org>
> > > >>>>>> ---
> > > >>>>>>  drivers/infiniband/hw/mlx5/mr.c | 2 +-
> > > >>>>>>  1 file changed, 1 insertion(+), 1 deletion(-)
> > > >>>>>>
> > > >>>>>
> > > >>>>> Bart,
> > > >>>>>
> > > >>>>> Thanks a lot, it indeed looks right.
> > > >>>>> Acked-by: Leon Romanovsky <leonro-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
> > > >>>>>
> > > >>>>> Thanks
> > > >>>>>
> > > >>>>
> > > >>>>
> > > >>>> Hello Bart, Leon, Max and Israel.
> > > >>>>
> > > >>>> I cloned off Barts tree.
> > > >>>>
> > > >>>> git clone https://github.com/bvanassche/linux
> > > >>>> cd linux
> > > >>>> git checkout block-scsi-for-next
> > > >>>>
> > > >>>> I checked all patches were in for this test.
> > > >>>>
> > > >>>> a83e404 IB/srp: Reenable IB_MR_TYPE_SG_GAPS
> > > >>>> dfa5a2b mlx5: Avoid that mlx5_ib_sg_to_klms() overflows the klms[]
> > > >>>> array
> > > >>>> f759c80 mlx5: Fix mlx5_ib_map_mr_sg mr lengt
> > > >>>
> > > >>> Hi,
> > > >>> copying Sagi's request from different thread:
> > > >>>
> > > >>> "
> > > >>> Can you please enable srp_add_one debug:
> > > >>>
> > > >>> echo "func srp_add_one +p" > /sys/kernel/debug/dynamic_debug/control
> > > >>>
> > > >>> In addition apply the following:
> > > >>> --
> > > >>> diff --git a/drivers/infiniband/hw/mlx5/mr.c
> > > >>> b/drivers/infiniband/hw/mlx5/mr.c
> > > >>> index d9c6c0ea750b..040fbc387e4f 100644
> > > >>> --- a/drivers/infiniband/hw/mlx5/mr.c
> > > >>> +++ b/drivers/infiniband/hw/mlx5/mr.c
> > > >>> @@ -1403,6 +1403,8 @@ mlx5_alloc_priv_descs(struct ib_device *device,
> > > >>>          int add_size;
> > > >>>          int ret;
> > > >>>
> > > >>> +       WARN_ON_ONCE(ndescs >
> > > >>> device->attr.max_fast_reg_page_list_len);
> > > >>> +
> > > >>>          add_size = max_t(int, MLX5_UMR_ALIGN -
> > > >>>          ARCH_KMALLOC_MINALIGN,
> > > >>>          0);
> > > >>>
> > > >>>          mr->descs_alloc = kzalloc(size + add_size, GFP_KERNEL);
> > > >>>
> > > >>> "
> > > >>>
> > > >>> Max.
> > > >>>
> > > >>>>
> > > >>>> Built and tested the kernel.
> > > >>>>
> > > >>>> However this issue is not resolved :(
> > > >>>>
> > > >>>> [ 2707.931909] scsi host1: ib_srp: failed RECV status WR flushed (5)
> > > >>>> for
> > > >>>> CQE ffff8817edca86b0
> > > >>>> [ 2708.089806] mlx5_0:dump_cqe:262:(pid 20129): dump error cqe
> > > >>>> [ 2708.121342] 00000000 00000000 00000000 00000000
> > > >>>> [ 2708.147104] 00000000 00000000 00000000 00000000
> > > >>>> [ 2708.172633] 00000000 00000000 00000000 00000000
> > > >>>> [ 2708.198702] 00000000 0f007806 2500002a 14a527d0
> > > >>>> [ 2732.434127] scsi host1: ib_srp: reconnect succeeded
> > > >>>> [ 2733.048023] scsi host1: ib_srp: failed RECV status WR flushed (5)
> > > >>>> for
> > > >>>> CQE ffff8817ed0a9c30
> > > >>>>
> > > >>>> [root@localhost ~]# [ 2746.413277] mlx5_0:dump_cqe:262:(pid 15877):
> > > >>>> dump
> > > >>>> error cqe
> > > >>>> [ 2746.443240] 00000000 00000000 00000000 00000000
> > > >>>> [ 2746.469323] 00000000 00000000 00000000 00000000
> > > >>>> [ 2746.495310] 00000000 00000000 00000000 00000000
> > > >>>> [ 2746.521407] 00000000 0f007806 25000032 003c7ad0
> > > >>>> [ 2752.445899] scsi host1: ib_srp: reconnect succeeded
> > > >>>> [ 2752.481835] scsi host1: ib_srp: failed RECV status WR flushed (5)
> > > >>>> for
> > > >>>> CQE ffff8817ed0a9cf0
> > > >>>> [ 2763.267386] mlx5_0:dump_cqe:262:(pid 15877): dump error cqe
> > > >>>> [ 2763.297826] 00000000 00000000 00000000 00000000
> > > >>>> [ 2763.323352] 00000000 00000000 00000000 00000000
> > > >>>> [ 2763.348722] 00000000 00000000 00000000 00000000
> > > >>>> [ 2763.374681] 00000000 0f007806 2500003a 00084bd0
> > > >>>>
> > > >>>> [root@localhost ~]# [ 2769.385203] fast_io_fail_tmo expired for SRP
> > > >>>> port-1:1 / host1.
> > > >>>> [ 2769.415956] scsi host1: ib_srp: reconnect succeeded
> > > >>>> [ 2769.450258] scsi host1: ib_srp: failed RECV status WR flushed (5)
> > > >>>> for
> > > >>>> CQE ffff8817ed0a9cf0
> > > >>>> [ 2780.064627] mlx5_0:dump_cqe:262:(pid 18771): dump error cqe
> > > >>>> [ 2780.093520] 00000000 00000000 00000000 00000000
> > > >>>> [ 2780.120067] 00000000 00000000 00000000 00000000
> > > >>>> [ 2780.145575] 00000000 00000000 00000000 00000000
> > > >>>> [ 2780.171153] 00000000 0f007806 25000042 000833d0
> > > >>>> [ 2785.923399] scsi host1: ib_srp: reconnect succeeded
> > > >>>> [ 2785.957504] scsi host1: ib_srp: failed RECV status WR flushed (5)
> > > >>>> for
> > > >>>> CQE ffff8817ed0a9cf0
> > > >>>> [ 2796.463426] mlx5_0:dump_cqe:262:(pid 18771): dump error cqe
> > > >>>> [ 2796.495257] 00000000 00000000 00000000 00000000
> > > >>>> [ 2796.521506] 00000000 00000000 00000000 00000000
> > > >>>> [ 2796.547640] 00000000 00000000 00000000 00000000
> > > >>>> [ 2796.573120] 00000000 0f007806 2500004a 00083bd0
> > > >>>> [ 2802.562578] scsi host1: ib_srp: reconnect succeeded
> > > >>>> [ 2802.596880] scsi host1: ib_srp: failed RECV status WR flushed (5)
> > > >>>> for
> > > >>>> CQE ffff8817ed0a9cf0
> > > >>>>
> > > >>>> Regards
> > > >>>> Laurence
> > > >>>>
> > > >>>
> > > >> Doing this now
> > > >> Thanks
> > > >> Laurence
> > > >
> > > > Max
> > > >
> > > > The Patch is not correct.
> > > >
> > > > drivers/infiniband/hw/mlx5/mr.c: In function 'mlx5_alloc_priv_descs':
> > > > drivers/infiniband/hw/mlx5/mr.c:1406:30: error: 'struct ib_device' has
> > > > no
> > > > member named 'attr'
> > > >   WARN_ON_ONCE(ndescs > device->attr.max_fast_reg_page_list_len);
> > > >                               ^
> > > > ./include/asm-generic/bug.h:117:27: note: in definition of macro
> > > > 'WARN_ON_ONCE'
> > > >   int __ret_warn_once = !!(condition);   \
> > > >
> > > > I think you meant to give me
> > > >
> > > > WARN_ON_ONCE(ndescs > ib_device_attr->attr.max_fast_reg_page_list_len);
> > > >
> > > > Can you confirm
> > > 
> > > Hi Laurence,
> > > should be device->attrs.max_fast_reg_page_list_len.
> > > 
> > > please check this one that might solve the issue (on top of everything):
> > > 
> > > 
> > > diff --git a/drivers/infiniband/hw/mlx5/mr.c
> > > b/drivers/infiniband/hw/mlx5/mr.c
> > > index b8f9382..063d116 100644
> > > --- a/drivers/infiniband/hw/mlx5/mr.c
> > > +++ b/drivers/infiniband/hw/mlx5/mr.c
> > > @@ -1559,7 +1559,7 @@ struct ib_mr *mlx5_ib_alloc_mr(struct ib_pd *pd,
> > >                  mr->max_descs = ndescs;
> > >          } else if (mr_type == IB_MR_TYPE_SG_GAPS) {
> > >                  mr->access_mode = MLX5_MKC_ACCESS_MODE_KLMS;
> > > -
> > > +               MLX5_SET(mkc, mkc, translations_octword_size,
> > > ALIGN(max_num_sg + 1, 4));
> > >                  err = mlx5_alloc_priv_descs(pd->device, mr,
> > >                                              ndescs, sizeof(struct
> > > mlx5_klm));
> > >                  if (err)
> > > 
> > > thanks,
> > > Max.
> > > 
> > > >
> > > > Thanks
> > > > Laurence
> > > >
> > > --
> > > To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
> > > the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> > > More majordomo info at  http://vger.kernel.org/majordomo-info.html
> > > 
> > 
> > Hello Max
> > 
> > I have the corrected WARN_ON_ONCE patch and the above patch as well as the
> > rest as it was from Barts tree.
> > 
> > Still fails.
> > 
> > For a baseline I can revert
> > a83e404 IB/srp: Reenable IB_MR_TYPE_SG_GAPS
> > 
> > Then test again to make sure we are starting from a good place.
> > 
> > Initiator log
> > 
> > [  280.481951] scsi host1: ib_srp: failed FAST REG status memory management
> > operation error (6) for CQE ffff8817d9a881b8
> > [  301.149106] scsi host1: ib_srp: reconnect succeeded
> > [  301.280635] scsi host1: ib_srp: failed RECV status WR flushed (5) for
> > CQE
> > ffff8817ed32f2f0
> > [  334.596420] scsi host2: ib_srp: failed RECV status WR flushed (5) for
> > CQE
> > ffff8817c592c970
> > [  334.599689] mlx5_1:dump_cqe:262:(pid 20): dump error cqe
> > [  334.599691] 00000000 00000000 00000000 00000000
> > [  334.599692] 00000000 00000000 00000000 00000000
> > [  334.599692] 00000000 00000000 00000000 00000000
> > [  334.599693] 00000000 0f007806 2500002d 067b48d0
> > [  334.599697] scsi host2: ib_srp: failed FAST REG status memory management
> > operation error (6) for CQE ffff8817c6e30078
> > [  336.117248] mlx5_0:dump_cqe:262:(pid 130): dump error cqe
> > [  336.145840] 00000000 00000000 00000000 00000000
> > [  336.171830] 00000000 00000000 00000000 00000000
> > [  336.197688] 00000000 00000000 00000000 00000000
> > [  336.223720] 00000000 0f007806 25000032 005408d0
> > [  339.712706] fast_io_fail_tmo expired for SRP port-1:1 / host1.
> > [  341.453634] scsi host1: ib_srp: reconnect succeeded
> > [  341.481600] mlx5_0:dump_cqe:262:(pid 130): dump error cqe
> > [  341.482145] scsi host1: ib_srp: failed RECV status WR flushed (5) for
> > CQE
> > ffff8817ecaf6970
> > [  341.559359] 00000000 00000000 00000000 00000000
> > [  341.585397] 00000000 00000000 00000000 00000000
> > [  341.610948] 00000000 00000000 00000000 00000000
> > [  341.637515] 00000000 0f007806 2500003d 000046d0
> > [  342.297598] sd 1:0:0:9: rejecting I/O to offline device
> > [  342.297936] sd 1:0:0:9: [sdg] tag#28 FAILED Result:
> > hostbyte=DID_TRANSPORT_FAILFAST driverbyte=DRIVER_OK
> > [  342.297941] sd 1:0:0:9: [sdg] tag#28 CDB: Write(10) 2a 00 00 00 40 00 00
> > 40 00 00
> > [  342.297943] blk_update_request: recoverable transport error, dev sdg,
> > sector 16384
> > [  342.297951] sd 1:0:0:20: [sdar] tag#5 FAILED Result:
> > hostbyte=DID_TRANSPORT_FAILFAST driverbyte=DRIVER_OK
> > [  342.297952] sd 1:0:0:20: [sdar] tag#15 FAILED Result:
> > hostbyte=DID_TRANSPORT_FAILFAST driverbyte=DRIVER_OK
> > [  342.297956] sd 1:0:0:20: [sdar] tag#5 CDB: Write(10) 2a 00 00 03 c0 00
> > 00
> > 40 00 00
> > [  342.297956] sd 1:0:0:20: [sdar] tag#15 CDB: Write(10) 2a 00 00 2c c0 00
> > 00
> > 40 00 00
> > [  342.297958] blk_update_request: recoverable transport error, dev sdar,
> > sector 245760
> > [  342.297959] blk_update_request: recoverable transport error, dev sdar,
> > sector 2932736
> > [  342.298119] device-mapper: multipath: Failing path 8:96.
> > [  342.298266] sd 1:0:0:9: [sdg] tag#29 FAILED Result:
> > hostbyte=DID_TRANSPORT_FAILFAST driverbyte=DRIVER_OK
> > [  342.298268] sd 1:0:0:9: [sdg] tag#29 CDB: Write(10) 2a 00 00 00 c0 00 00
> > 40 00 00
> > [  342.298269] blk_update_request: recoverable transport error, dev sdg,
> > sector 49152
> > [  342.298300] device-mapper: multipath: Failing path 66:176.
> > [  342.298486] sd 1:0:0:20: [sdar] tag#16 FAILED Result:
> > hostbyte=DID_TRANSPORT_FAILFAST driverbyte=DRIVER_OK
> > [  342.298488] sd 1:0:0:20: [sdar] tag#6 FAILED Result:
> > hostbyte=DID_TRANSPORT_FAILFAST driverbyte=DRIVER_OK
> > [  342.298489] sd 1:0:0:20: [sdar] tag#16 CDB: Write(10) 2a 00 00 2d 40 00
> > 00
> > 40 00 00
> > [  342.298490] sd 1:0:0:20: [sdar] tag#6 CDB: Write(10) 2a 00 00 04 40 00
> > 00
> > 40 00 00
> > [  342.298491] blk_update_request: recoverable transport error, dev sdar,
> > sector 2965504
> > [  342.298492] blk_update_request: recoverable transport error, dev sdar,
> > sector 278528
> > [  342.298582] sd 1:0:0:9: [sdg] tag#30 FAILED Result:
> > hostbyte=DID_TRANSPORT_FAILFAST driverbyte=DRIVER_OK
> > [  342.298584] sd 1:0:0:9: [sdg] tag#30 CDB: Write(10) 2a 00 00 01 40 00 00
> > 40 00 00
> > [  342.298585] blk_update_request: recoverable transport error, dev sdg,
> > sector 81920
> > [  342.298889] sd 1:0:0:9: [sdg] tag#31 FAILED Result:
> > hostbyte=DID_TRANSPORT_FAILFAST driverbyte=DRIVER_OK
> > [  342.298890] sd 1:0:0:9: [sdg] tag#31 CDB: Write(10) 2a 00 00 01 c0 00 00
> > 40 00 00
> > [  342.298891] blk_update_request: recoverable transport error, dev sdg,
> > sector 114688
> > [  342.298981] sd 1:0:0:20: [sdar] tag#7 FAILED Result:
> > hostbyte=DID_TRANSPORT_FAILFAST driverbyte=DRIVER_OK
> > [  342.298983] sd 1:0:0:20: [sdar] tag#7 CDB: Write(10) 2a 00 00 04 c0 00
> > 00
> > 40 00 00
> > [  342.298985] blk_update_request: recoverable transport error, dev sdar,
> > sector 311296
> > [  342.299004] sd 1:0:0:20: [sdar] tag#17 FAILED Result:
> > hostbyte=DID_TRANSPORT_FAILFAST driverbyte=DRIVER_OK
> > [  342.299007] sd 1:0:0:20: [sdar] tag#17 CDB: Write(10) 2a 00 00 34 c0 00
> > 00
> > 40 00 00
> > [  342.299009] blk_update_request: recoverable transport error, dev sdar,
> > sector 3457024
> > [  342.356353] device-mapper: multipath: Failing path 8:64.
> > [  342.356489] device-mapper: multipath: Failing path 8:128.
> > [  342.356628] device-mapper: multipath: Failing path 8:160.
> > [  342.356699] device-mapper: multipath: Failing path 8:176.
> > [  342.356767] device-mapper: multipath: Failing path 8:240.
> > [  342.356834] device-mapper: multipath: Failing path 8:208.
> > [  342.356900] device-mapper: multipath: Failing path 65:16.
> > [  342.356967] device-mapper: multipath: Failing path 65:64.
> > [  342.357035] device-mapper: multipath: Failing path 65:96.
> > [  342.357103] device-mapper: multipath: Failing path 65:128.
> > [  342.357169] device-mapper: multipath: Failing path 65:176.
> > [  342.357237] device-mapper: multipath: Failing path 65:208.
> > [  342.357303] device-mapper: multipath: Failing path 65:224.
> > [  342.357371] device-mapper: multipath: Failing path 66:0.
> > [  342.357454] device-mapper: multipath: Failing path 66:32.
> > [  342.357521] device-mapper: multipath: Failing path 66:48.
> > [  342.357647] device-mapper: multipath: Failing path 66:80.
> > [  342.357714] device-mapper: multipath: Failing path 66:112.
> > [  342.357781] device-mapper: multipath: Failing path 66:144.
> > [  342.357936] device-mapper: multipath: Failing path 66:208.
> > [  342.358019] device-mapper: multipath: Failing path 66:240.
> > [  342.358115] device-mapper: multipath: Failing path 67:16.
> > [  342.358183] device-mapper: multipath: Failing path 67:48.
> > [  342.358264] device-mapper: multipath: Failing path 67:80.
> > [  342.358359] device-mapper: multipath: Failing path 67:128.
> > [  342.358442] device-mapper: multipath: Failing path 67:160.
> > [  342.358594] device-mapper: multipath: Failing path 67:224.
> > [  342.358671] device-mapper: multipath: Failing path 67:208.
> > [  350.157728] scsi host2: ib_srp: reconnect succeeded
> > [  350.189605] mlx5_1:dump_cqe:262:(pid 4756): dump error cqe
> > [  350.193180] mlx5_1:dump_cqe:262:(pid 1275): dump error cqe
> > [  350.193182] 00000000 00000000 00000000 00000000
> > [  350.193182] 00000000 00000000 00000000 00000000
> > [  350.193183] 00000000 00000000 00000000 00000000
> > [  350.193183] 00000000 0f007806 25000035 04f569d0
> > [  350.193187] scsi host2: ib_srp: failed FAST REG status memory management
> > operation error (6) for CQE ffff8817c6e30078
> > [  350.412637] 00000000 00000000 00000000 00000000
> > [  350.436431] 00000000 00000000 00000000 00000000
> > [  350.461871] 00000000 00000000 00000000 00000000
> > [  350.487549] 00000000 0f007806 25000032 000843d0
> > 
> > Target Log
> > 
> > Thee events happened after the first failures on the initiator
> > 
> > [ 1111.029847] ib_srpt Received CM TimeWait exit for ch
> > 0x4f6e72000390fe7c7cfe900300726ed3-49.
> > [ 1111.078815] ib_srpt Received CM TimeWait exit for ch
> > 0x4f6e72000390fe7c7cfe900300726ed3-48.
> > [ 1111.127420] ib_srpt Received CM TimeWait exit for ch
> > 0x4f6e72000390fe7c7cfe900300726ed3-47.
> > [ 1111.175801] ib_srpt Received CM TimeWait exit for ch
> > 0x4f6e72000390fe7c7cfe900300726ed3-46.
> > [ 1111.223725] ib_srpt Received CM TimeWait exit for ch
> > 0x4f6e72000390fe7c7cfe900300726ed3-45.
> > [ 1111.271957] ib_srpt Received CM TimeWait exit for ch
> > 0x4f6e72000390fe7c7cfe900300726ed3-44.
> > [ 1111.319494] ib_srpt Received CM TimeWait exit for ch
> > 0x4f6e72000390fe7c7cfe900300726ed3-43.
> > [ 1111.365795] ib_srpt Received CM TimeWait exit for ch
> > 0x4f6e72000390fe7c7cfe900300726ed3-42.
> > 
> > --
> > To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
> > the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> > More majordomo info at  http://vger.kernel.org/majordomo-info.html
> > 
> 
> Max
> 
> These are the parameters all my tests run with.
> Same as always.
> 
> [root@localhost modprobe.d]# cat ib_srp.conf
> options ib_srp cmd_sg_entries=255 indirect_sg_entries=2048
> 
> I dont set prefer_fr so it defaults to Y
> 
> [root@localhost parameters]# cat prefer_fr
> Y
> 
> I have no settings for mlx5_core, all defaults.
> 
> Thanks
> Laurence
> 
> --
> To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 

Max,

Reverting a83e404 IB/srp: Reenable IB_MR_TYPE_SG_GAPS on the same source tree with all esle applied I am stable.
So clearly we still have issues with IB_MR_TYPE_SG_GAPS.

Thanks
Laurence
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH, untested] mlx5: Avoid that mlx5_ib_sg_to_klms() overflows the klms[] array

[Max Gurtovoy]

On 4/26/2017 6:10 PM, Laurence Oberman wrote:
>
>
> ----- Original Message -----
>> From: "Laurence Oberman" <loberman-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>
>> To: "Max Gurtovoy" <maxg-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
>> Cc: "Leon Romanovsky" <leonro-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>, "Bart Van Assche" <bart.vanassche-XdAiOPVOjttBDgjK7y7TUQ@public.gmane.org>, "Doug Ledford"
>> <dledford-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>, "Sagi Grimberg" <sagi-NQWnxTmZq1alnMjI0IkVqw@public.gmane.org>, "Israel Rukshin" <israelr-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>,
>> linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
>> Sent: Wednesday, April 26, 2017 9:50:25 AM
>> Subject: Re: [PATCH, untested] mlx5: Avoid that mlx5_ib_sg_to_klms() overflows the klms[] array
>>
>>
>>
>> ----- Original Message -----
>>> From: "Laurence Oberman" <loberman-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>
>>> To: "Max Gurtovoy" <maxg-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
>>> Cc: "Leon Romanovsky" <leonro-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>, "Bart Van Assche"
>>> <bart.vanassche-XdAiOPVOjttBDgjK7y7TUQ@public.gmane.org>, "Doug Ledford"
>>> <dledford-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>, "Sagi Grimberg" <sagi-NQWnxTmZq1alnMjI0IkVqw@public.gmane.org>, "Israel Rukshin"
>>> <israelr-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>,
>>> linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
>>> Sent: Wednesday, April 26, 2017 9:28:37 AM
>>> Subject: Re: [PATCH, untested] mlx5: Avoid that mlx5_ib_sg_to_klms()
>>> overflows the klms[] array
>>>
>>>
>>>
>>> ----- Original Message -----
>>>> From: "Max Gurtovoy" <maxg-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
>>>> To: "Laurence Oberman" <loberman-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>
>>>> Cc: "Leon Romanovsky" <leonro-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>, "Bart Van Assche"
>>>> <bart.vanassche-XdAiOPVOjttBDgjK7y7TUQ@public.gmane.org>, "Doug Ledford"
>>>> <dledford-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>, "Sagi Grimberg" <sagi-NQWnxTmZq1alnMjI0IkVqw@public.gmane.org>, "Israel
>>>> Rukshin"
>>>> <israelr-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>,
>>>> linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
>>>> Sent: Wednesday, April 26, 2017 8:25:30 AM
>>>> Subject: Re: [PATCH, untested] mlx5: Avoid that mlx5_ib_sg_to_klms()
>>>> overflows the klms[] array
>>>>
>>>>
>>>>
>>>> On 4/26/2017 3:18 PM, Laurence Oberman wrote:
>>>>>
>>>>>
>>>>> ----- Original Message -----
>>>>>> From: "Laurence Oberman" <loberman-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>
>>>>>> To: "Max Gurtovoy" <maxg-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
>>>>>> Cc: "Leon Romanovsky" <leonro-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>, "Bart Van Assche"
>>>>>> <bart.vanassche-XdAiOPVOjttBDgjK7y7TUQ@public.gmane.org>, "Doug Ledford"
>>>>>> <dledford-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>, "Sagi Grimberg" <sagi-NQWnxTmZq1alnMjI0IkVqw@public.gmane.org>, "Israel
>>>>>> Rukshin" <israelr-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>,
>>>>>> linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
>>>>>> Sent: Wednesday, April 26, 2017 7:47:37 AM
>>>>>> Subject: Re: [PATCH, untested] mlx5: Avoid that mlx5_ib_sg_to_klms()
>>>>>> overflows the klms[] array
>>>>>>
>>>>>>
>>>>>>
>>>>>> ----- Original Message -----
>>>>>>> From: "Max Gurtovoy" <maxg-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
>>>>>>> To: "Laurence Oberman" <loberman-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>, "Leon Romanovsky"
>>>>>>> <leonro-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
>>>>>>> Cc: "Bart Van Assche" <bart.vanassche-XdAiOPVOjttBDgjK7y7TUQ@public.gmane.org>, "Doug Ledford"
>>>>>>> <dledford-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>, "Sagi Grimberg"
>>>>>>> <sagi-NQWnxTmZq1alnMjI0IkVqw@public.gmane.org>, "Israel Rukshin" <israelr-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>,
>>>>>>> linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
>>>>>>> Sent: Wednesday, April 26, 2017 4:31:57 AM
>>>>>>> Subject: Re: [PATCH, untested] mlx5: Avoid that mlx5_ib_sg_to_klms()
>>>>>>> overflows the klms[] array
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On 4/25/2017 11:37 PM, Laurence Oberman wrote:
>>>>>>>>
>>>>>>>>
>>>>>>>> ----- Original Message -----
>>>>>>>>> From: "Leon Romanovsky" <leonro-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
>>>>>>>>> To: "Bart Van Assche" <bart.vanassche-XdAiOPVOjttBDgjK7y7TUQ@public.gmane.org>
>>>>>>>>> Cc: "Doug Ledford" <dledford-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>, "Max Gurtovoy"
>>>>>>>>> <maxg-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>, "Sagi Grimberg" <sagi-NQWnxTmZq1alnMjI0IkVqw@public.gmane.org>,
>>>>>>>>> "Israel Rukshin" <israelr-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>, "Laurence Oberman"
>>>>>>>>> <loberman-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>, linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
>>>>>>>>> Sent: Tuesday, April 25, 2017 1:58:49 PM
>>>>>>>>> Subject: Re: [PATCH, untested] mlx5: Avoid that
>>>>>>>>> mlx5_ib_sg_to_klms()
>>>>>>>>> overflows the klms[] array
>>>>>>>>>
>>>>>>>>> On Mon, Apr 24, 2017 at 03:15:28PM -0700, Bart Van Assche wrote:
>>>>>>>>>> ib_map_mr_sg() can pass an SG-list to .map_mr_sg() that is larger
>>>>>>>>>> than what fits into a single MR. .map_mr_sg() must not attempt to
>>>>>>>>>> map more SG-list elements than what fits into a single MR.
>>>>>>>>>> Hence make sure that mlx5_ib_sg_to_klms() does not write outside
>>>>>>>>>> the MR klms[] array.
>>>>>>>>>>
>>>>>>>>>> Fixes: b005d3164713 ("mlx5: Add arbitrary sg list support")
>>>>>>>>>> Signed-off-by: Bart Van Assche <bart.vanassche-XdAiOPVOjttBDgjK7y7TUQ@public.gmane.org>
>>>>>>>>>> Reviewed-by: Max Gurtovoy <maxg-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
>>>>>>>>>> Cc: Sagi Grimberg <sagi-NQWnxTmZq1alnMjI0IkVqw@public.gmane.org>
>>>>>>>>>> Cc: Leon Romanovsky <leonro-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
>>>>>>>>>> Cc: Israel Rukshin <israelr-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
>>>>>>>>>> Cc: <stable-u79uwXL29TY76Z2rM5mHXA@public.gmane.org>
>>>>>>>>>> ---
>>>>>>>>>>  drivers/infiniband/hw/mlx5/mr.c | 2 +-
>>>>>>>>>>  1 file changed, 1 insertion(+), 1 deletion(-)
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Bart,
>>>>>>>>>
>>>>>>>>> Thanks a lot, it indeed looks right.
>>>>>>>>> Acked-by: Leon Romanovsky <leonro-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
>>>>>>>>>
>>>>>>>>> Thanks
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> Hello Bart, Leon, Max and Israel.
>>>>>>>>
>>>>>>>> I cloned off Barts tree.
>>>>>>>>
>>>>>>>> git clone https://github.com/bvanassche/linux
>>>>>>>> cd linux
>>>>>>>> git checkout block-scsi-for-next
>>>>>>>>
>>>>>>>> I checked all patches were in for this test.
>>>>>>>>
>>>>>>>> a83e404 IB/srp: Reenable IB_MR_TYPE_SG_GAPS
>>>>>>>> dfa5a2b mlx5: Avoid that mlx5_ib_sg_to_klms() overflows the klms[]
>>>>>>>> array
>>>>>>>> f759c80 mlx5: Fix mlx5_ib_map_mr_sg mr lengt
>>>>>>>
>>>>>>> Hi,
>>>>>>> copying Sagi's request from different thread:
>>>>>>>
>>>>>>> "
>>>>>>> Can you please enable srp_add_one debug:
>>>>>>>
>>>>>>> echo "func srp_add_one +p" > /sys/kernel/debug/dynamic_debug/control
>>>>>>>
>>>>>>> In addition apply the following:
>>>>>>> --
>>>>>>> diff --git a/drivers/infiniband/hw/mlx5/mr.c
>>>>>>> b/drivers/infiniband/hw/mlx5/mr.c
>>>>>>> index d9c6c0ea750b..040fbc387e4f 100644
>>>>>>> --- a/drivers/infiniband/hw/mlx5/mr.c
>>>>>>> +++ b/drivers/infiniband/hw/mlx5/mr.c
>>>>>>> @@ -1403,6 +1403,8 @@ mlx5_alloc_priv_descs(struct ib_device *device,
>>>>>>>          int add_size;
>>>>>>>          int ret;
>>>>>>>
>>>>>>> +       WARN_ON_ONCE(ndescs >
>>>>>>> device->attr.max_fast_reg_page_list_len);
>>>>>>> +
>>>>>>>          add_size = max_t(int, MLX5_UMR_ALIGN -
>>>>>>>          ARCH_KMALLOC_MINALIGN,
>>>>>>>          0);
>>>>>>>
>>>>>>>          mr->descs_alloc = kzalloc(size + add_size, GFP_KERNEL);
>>>>>>>
>>>>>>> "
>>>>>>>
>>>>>>> Max.
>>>>>>>
>>>>>>>>
>>>>>>>> Built and tested the kernel.
>>>>>>>>
>>>>>>>> However this issue is not resolved :(
>>>>>>>>
>>>>>>>> [ 2707.931909] scsi host1: ib_srp: failed RECV status WR flushed (5)
>>>>>>>> for
>>>>>>>> CQE ffff8817edca86b0
>>>>>>>> [ 2708.089806] mlx5_0:dump_cqe:262:(pid 20129): dump error cqe
>>>>>>>> [ 2708.121342] 00000000 00000000 00000000 00000000
>>>>>>>> [ 2708.147104] 00000000 00000000 00000000 00000000
>>>>>>>> [ 2708.172633] 00000000 00000000 00000000 00000000
>>>>>>>> [ 2708.198702] 00000000 0f007806 2500002a 14a527d0
>>>>>>>> [ 2732.434127] scsi host1: ib_srp: reconnect succeeded
>>>>>>>> [ 2733.048023] scsi host1: ib_srp: failed RECV status WR flushed (5)
>>>>>>>> for
>>>>>>>> CQE ffff8817ed0a9c30
>>>>>>>>
>>>>>>>> [root@localhost ~]# [ 2746.413277] mlx5_0:dump_cqe:262:(pid 15877):
>>>>>>>> dump
>>>>>>>> error cqe
>>>>>>>> [ 2746.443240] 00000000 00000000 00000000 00000000
>>>>>>>> [ 2746.469323] 00000000 00000000 00000000 00000000
>>>>>>>> [ 2746.495310] 00000000 00000000 00000000 00000000
>>>>>>>> [ 2746.521407] 00000000 0f007806 25000032 003c7ad0
>>>>>>>> [ 2752.445899] scsi host1: ib_srp: reconnect succeeded
>>>>>>>> [ 2752.481835] scsi host1: ib_srp: failed RECV status WR flushed (5)
>>>>>>>> for
>>>>>>>> CQE ffff8817ed0a9cf0
>>>>>>>> [ 2763.267386] mlx5_0:dump_cqe:262:(pid 15877): dump error cqe
>>>>>>>> [ 2763.297826] 00000000 00000000 00000000 00000000
>>>>>>>> [ 2763.323352] 00000000 00000000 00000000 00000000
>>>>>>>> [ 2763.348722] 00000000 00000000 00000000 00000000
>>>>>>>> [ 2763.374681] 00000000 0f007806 2500003a 00084bd0
>>>>>>>>
>>>>>>>> [root@localhost ~]# [ 2769.385203] fast_io_fail_tmo expired for SRP
>>>>>>>> port-1:1 / host1.
>>>>>>>> [ 2769.415956] scsi host1: ib_srp: reconnect succeeded
>>>>>>>> [ 2769.450258] scsi host1: ib_srp: failed RECV status WR flushed (5)
>>>>>>>> for
>>>>>>>> CQE ffff8817ed0a9cf0
>>>>>>>> [ 2780.064627] mlx5_0:dump_cqe:262:(pid 18771): dump error cqe
>>>>>>>> [ 2780.093520] 00000000 00000000 00000000 00000000
>>>>>>>> [ 2780.120067] 00000000 00000000 00000000 00000000
>>>>>>>> [ 2780.145575] 00000000 00000000 00000000 00000000
>>>>>>>> [ 2780.171153] 00000000 0f007806 25000042 000833d0
>>>>>>>> [ 2785.923399] scsi host1: ib_srp: reconnect succeeded
>>>>>>>> [ 2785.957504] scsi host1: ib_srp: failed RECV status WR flushed (5)
>>>>>>>> for
>>>>>>>> CQE ffff8817ed0a9cf0
>>>>>>>> [ 2796.463426] mlx5_0:dump_cqe:262:(pid 18771): dump error cqe
>>>>>>>> [ 2796.495257] 00000000 00000000 00000000 00000000
>>>>>>>> [ 2796.521506] 00000000 00000000 00000000 00000000
>>>>>>>> [ 2796.547640] 00000000 00000000 00000000 00000000
>>>>>>>> [ 2796.573120] 00000000 0f007806 2500004a 00083bd0
>>>>>>>> [ 2802.562578] scsi host1: ib_srp: reconnect succeeded
>>>>>>>> [ 2802.596880] scsi host1: ib_srp: failed RECV status WR flushed (5)
>>>>>>>> for
>>>>>>>> CQE ffff8817ed0a9cf0
>>>>>>>>
>>>>>>>> Regards
>>>>>>>> Laurence
>>>>>>>>
>>>>>>>
>>>>>> Doing this now
>>>>>> Thanks
>>>>>> Laurence
>>>>>
>>>>> Max
>>>>>
>>>>> The Patch is not correct.
>>>>>
>>>>> drivers/infiniband/hw/mlx5/mr.c: In function 'mlx5_alloc_priv_descs':
>>>>> drivers/infiniband/hw/mlx5/mr.c:1406:30: error: 'struct ib_device' has
>>>>> no
>>>>> member named 'attr'
>>>>>   WARN_ON_ONCE(ndescs > device->attr.max_fast_reg_page_list_len);
>>>>>                               ^
>>>>> ./include/asm-generic/bug.h:117:27: note: in definition of macro
>>>>> 'WARN_ON_ONCE'
>>>>>   int __ret_warn_once = !!(condition);   \
>>>>>
>>>>> I think you meant to give me
>>>>>
>>>>> WARN_ON_ONCE(ndescs > ib_device_attr->attr.max_fast_reg_page_list_len);
>>>>>
>>>>> Can you confirm
>>>>
>>>> Hi Laurence,
>>>> should be device->attrs.max_fast_reg_page_list_len.
>>>>
>>>> please check this one that might solve the issue (on top of everything):
>>>>
>>>>
>>>> diff --git a/drivers/infiniband/hw/mlx5/mr.c
>>>> b/drivers/infiniband/hw/mlx5/mr.c
>>>> index b8f9382..063d116 100644
>>>> --- a/drivers/infiniband/hw/mlx5/mr.c
>>>> +++ b/drivers/infiniband/hw/mlx5/mr.c
>>>> @@ -1559,7 +1559,7 @@ struct ib_mr *mlx5_ib_alloc_mr(struct ib_pd *pd,
>>>>                  mr->max_descs = ndescs;
>>>>          } else if (mr_type == IB_MR_TYPE_SG_GAPS) {
>>>>                  mr->access_mode = MLX5_MKC_ACCESS_MODE_KLMS;
>>>> -
>>>> +               MLX5_SET(mkc, mkc, translations_octword_size,
>>>> ALIGN(max_num_sg + 1, 4));
>>>>                  err = mlx5_alloc_priv_descs(pd->device, mr,
>>>>                                              ndescs, sizeof(struct
>>>> mlx5_klm));
>>>>                  if (err)
>>>>
>>>> thanks,
>>>> Max.
>>>>
>>>>>
>>>>> Thanks
>>>>> Laurence
>>>>>
>>>> --
>>>> To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
>>>> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
>>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>>>
>>>
>>> Hello Max
>>>
>>> I have the corrected WARN_ON_ONCE patch and the above patch as well as the
>>> rest as it was from Barts tree.
>>>
>>> Still fails.
>>>
>>> For a baseline I can revert
>>> a83e404 IB/srp: Reenable IB_MR_TYPE_SG_GAPS
>>>
>>> Then test again to make sure we are starting from a good place.
>>>
>>> Initiator log
>>>
>>> [  280.481951] scsi host1: ib_srp: failed FAST REG status memory management
>>> operation error (6) for CQE ffff8817d9a881b8
>>> [  301.149106] scsi host1: ib_srp: reconnect succeeded
>>> [  301.280635] scsi host1: ib_srp: failed RECV status WR flushed (5) for
>>> CQE
>>> ffff8817ed32f2f0
>>> [  334.596420] scsi host2: ib_srp: failed RECV status WR flushed (5) for
>>> CQE
>>> ffff8817c592c970
>>> [  334.599689] mlx5_1:dump_cqe:262:(pid 20): dump error cqe
>>> [  334.599691] 00000000 00000000 00000000 00000000
>>> [  334.599692] 00000000 00000000 00000000 00000000
>>> [  334.599692] 00000000 00000000 00000000 00000000
>>> [  334.599693] 00000000 0f007806 2500002d 067b48d0
>>> [  334.599697] scsi host2: ib_srp: failed FAST REG status memory management
>>> operation error (6) for CQE ffff8817c6e30078
>>> [  336.117248] mlx5_0:dump_cqe:262:(pid 130): dump error cqe
>>> [  336.145840] 00000000 00000000 00000000 00000000
>>> [  336.171830] 00000000 00000000 00000000 00000000
>>> [  336.197688] 00000000 00000000 00000000 00000000
>>> [  336.223720] 00000000 0f007806 25000032 005408d0
>>> [  339.712706] fast_io_fail_tmo expired for SRP port-1:1 / host1.
>>> [  341.453634] scsi host1: ib_srp: reconnect succeeded
>>> [  341.481600] mlx5_0:dump_cqe:262:(pid 130): dump error cqe
>>> [  341.482145] scsi host1: ib_srp: failed RECV status WR flushed (5) for
>>> CQE
>>> ffff8817ecaf6970
>>> [  341.559359] 00000000 00000000 00000000 00000000
>>> [  341.585397] 00000000 00000000 00000000 00000000
>>> [  341.610948] 00000000 00000000 00000000 00000000
>>> [  341.637515] 00000000 0f007806 2500003d 000046d0
>>> [  342.297598] sd 1:0:0:9: rejecting I/O to offline device
>>> [  342.297936] sd 1:0:0:9: [sdg] tag#28 FAILED Result:
>>> hostbyte=DID_TRANSPORT_FAILFAST driverbyte=DRIVER_OK
>>> [  342.297941] sd 1:0:0:9: [sdg] tag#28 CDB: Write(10) 2a 00 00 00 40 00 00
>>> 40 00 00
>>> [  342.297943] blk_update_request: recoverable transport error, dev sdg,
>>> sector 16384
>>> [  342.297951] sd 1:0:0:20: [sdar] tag#5 FAILED Result:
>>> hostbyte=DID_TRANSPORT_FAILFAST driverbyte=DRIVER_OK
>>> [  342.297952] sd 1:0:0:20: [sdar] tag#15 FAILED Result:
>>> hostbyte=DID_TRANSPORT_FAILFAST driverbyte=DRIVER_OK
>>> [  342.297956] sd 1:0:0:20: [sdar] tag#5 CDB: Write(10) 2a 00 00 03 c0 00
>>> 00
>>> 40 00 00
>>> [  342.297956] sd 1:0:0:20: [sdar] tag#15 CDB: Write(10) 2a 00 00 2c c0 00
>>> 00
>>> 40 00 00
>>> [  342.297958] blk_update_request: recoverable transport error, dev sdar,
>>> sector 245760
>>> [  342.297959] blk_update_request: recoverable transport error, dev sdar,
>>> sector 2932736
>>> [  342.298119] device-mapper: multipath: Failing path 8:96.
>>> [  342.298266] sd 1:0:0:9: [sdg] tag#29 FAILED Result:
>>> hostbyte=DID_TRANSPORT_FAILFAST driverbyte=DRIVER_OK
>>> [  342.298268] sd 1:0:0:9: [sdg] tag#29 CDB: Write(10) 2a 00 00 00 c0 00 00
>>> 40 00 00
>>> [  342.298269] blk_update_request: recoverable transport error, dev sdg,
>>> sector 49152
>>> [  342.298300] device-mapper: multipath: Failing path 66:176.
>>> [  342.298486] sd 1:0:0:20: [sdar] tag#16 FAILED Result:
>>> hostbyte=DID_TRANSPORT_FAILFAST driverbyte=DRIVER_OK
>>> [  342.298488] sd 1:0:0:20: [sdar] tag#6 FAILED Result:
>>> hostbyte=DID_TRANSPORT_FAILFAST driverbyte=DRIVER_OK
>>> [  342.298489] sd 1:0:0:20: [sdar] tag#16 CDB: Write(10) 2a 00 00 2d 40 00
>>> 00
>>> 40 00 00
>>> [  342.298490] sd 1:0:0:20: [sdar] tag#6 CDB: Write(10) 2a 00 00 04 40 00
>>> 00
>>> 40 00 00
>>> [  342.298491] blk_update_request: recoverable transport error, dev sdar,
>>> sector 2965504
>>> [  342.298492] blk_update_request: recoverable transport error, dev sdar,
>>> sector 278528
>>> [  342.298582] sd 1:0:0:9: [sdg] tag#30 FAILED Result:
>>> hostbyte=DID_TRANSPORT_FAILFAST driverbyte=DRIVER_OK
>>> [  342.298584] sd 1:0:0:9: [sdg] tag#30 CDB: Write(10) 2a 00 00 01 40 00 00
>>> 40 00 00
>>> [  342.298585] blk_update_request: recoverable transport error, dev sdg,
>>> sector 81920
>>> [  342.298889] sd 1:0:0:9: [sdg] tag#31 FAILED Result:
>>> hostbyte=DID_TRANSPORT_FAILFAST driverbyte=DRIVER_OK
>>> [  342.298890] sd 1:0:0:9: [sdg] tag#31 CDB: Write(10) 2a 00 00 01 c0 00 00
>>> 40 00 00
>>> [  342.298891] blk_update_request: recoverable transport error, dev sdg,
>>> sector 114688
>>> [  342.298981] sd 1:0:0:20: [sdar] tag#7 FAILED Result:
>>> hostbyte=DID_TRANSPORT_FAILFAST driverbyte=DRIVER_OK
>>> [  342.298983] sd 1:0:0:20: [sdar] tag#7 CDB: Write(10) 2a 00 00 04 c0 00
>>> 00
>>> 40 00 00
>>> [  342.298985] blk_update_request: recoverable transport error, dev sdar,
>>> sector 311296
>>> [  342.299004] sd 1:0:0:20: [sdar] tag#17 FAILED Result:
>>> hostbyte=DID_TRANSPORT_FAILFAST driverbyte=DRIVER_OK
>>> [  342.299007] sd 1:0:0:20: [sdar] tag#17 CDB: Write(10) 2a 00 00 34 c0 00
>>> 00
>>> 40 00 00
>>> [  342.299009] blk_update_request: recoverable transport error, dev sdar,
>>> sector 3457024
>>> [  342.356353] device-mapper: multipath: Failing path 8:64.
>>> [  342.356489] device-mapper: multipath: Failing path 8:128.
>>> [  342.356628] device-mapper: multipath: Failing path 8:160.
>>> [  342.356699] device-mapper: multipath: Failing path 8:176.
>>> [  342.356767] device-mapper: multipath: Failing path 8:240.
>>> [  342.356834] device-mapper: multipath: Failing path 8:208.
>>> [  342.356900] device-mapper: multipath: Failing path 65:16.
>>> [  342.356967] device-mapper: multipath: Failing path 65:64.
>>> [  342.357035] device-mapper: multipath: Failing path 65:96.
>>> [  342.357103] device-mapper: multipath: Failing path 65:128.
>>> [  342.357169] device-mapper: multipath: Failing path 65:176.
>>> [  342.357237] device-mapper: multipath: Failing path 65:208.
>>> [  342.357303] device-mapper: multipath: Failing path 65:224.
>>> [  342.357371] device-mapper: multipath: Failing path 66:0.
>>> [  342.357454] device-mapper: multipath: Failing path 66:32.
>>> [  342.357521] device-mapper: multipath: Failing path 66:48.
>>> [  342.357647] device-mapper: multipath: Failing path 66:80.
>>> [  342.357714] device-mapper: multipath: Failing path 66:112.
>>> [  342.357781] device-mapper: multipath: Failing path 66:144.
>>> [  342.357936] device-mapper: multipath: Failing path 66:208.
>>> [  342.358019] device-mapper: multipath: Failing path 66:240.
>>> [  342.358115] device-mapper: multipath: Failing path 67:16.
>>> [  342.358183] device-mapper: multipath: Failing path 67:48.
>>> [  342.358264] device-mapper: multipath: Failing path 67:80.
>>> [  342.358359] device-mapper: multipath: Failing path 67:128.
>>> [  342.358442] device-mapper: multipath: Failing path 67:160.
>>> [  342.358594] device-mapper: multipath: Failing path 67:224.
>>> [  342.358671] device-mapper: multipath: Failing path 67:208.
>>> [  350.157728] scsi host2: ib_srp: reconnect succeeded
>>> [  350.189605] mlx5_1:dump_cqe:262:(pid 4756): dump error cqe
>>> [  350.193180] mlx5_1:dump_cqe:262:(pid 1275): dump error cqe
>>> [  350.193182] 00000000 00000000 00000000 00000000
>>> [  350.193182] 00000000 00000000 00000000 00000000
>>> [  350.193183] 00000000 00000000 00000000 00000000
>>> [  350.193183] 00000000 0f007806 25000035 04f569d0
>>> [  350.193187] scsi host2: ib_srp: failed FAST REG status memory management
>>> operation error (6) for CQE ffff8817c6e30078
>>> [  350.412637] 00000000 00000000 00000000 00000000
>>> [  350.436431] 00000000 00000000 00000000 00000000
>>> [  350.461871] 00000000 00000000 00000000 00000000
>>> [  350.487549] 00000000 0f007806 25000032 000843d0
>>>
>>> Target Log
>>>
>>> Thee events happened after the first failures on the initiator
>>>
>>> [ 1111.029847] ib_srpt Received CM TimeWait exit for ch
>>> 0x4f6e72000390fe7c7cfe900300726ed3-49.
>>> [ 1111.078815] ib_srpt Received CM TimeWait exit for ch
>>> 0x4f6e72000390fe7c7cfe900300726ed3-48.
>>> [ 1111.127420] ib_srpt Received CM TimeWait exit for ch
>>> 0x4f6e72000390fe7c7cfe900300726ed3-47.
>>> [ 1111.175801] ib_srpt Received CM TimeWait exit for ch
>>> 0x4f6e72000390fe7c7cfe900300726ed3-46.
>>> [ 1111.223725] ib_srpt Received CM TimeWait exit for ch
>>> 0x4f6e72000390fe7c7cfe900300726ed3-45.
>>> [ 1111.271957] ib_srpt Received CM TimeWait exit for ch
>>> 0x4f6e72000390fe7c7cfe900300726ed3-44.
>>> [ 1111.319494] ib_srpt Received CM TimeWait exit for ch
>>> 0x4f6e72000390fe7c7cfe900300726ed3-43.
>>> [ 1111.365795] ib_srpt Received CM TimeWait exit for ch
>>> 0x4f6e72000390fe7c7cfe900300726ed3-42.
>>>
>>> --
>>> To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
>>> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>>
>>
>> Max
>>
>> These are the parameters all my tests run with.
>> Same as always.
>>
>> [root@localhost modprobe.d]# cat ib_srp.conf
>> options ib_srp cmd_sg_entries=255 indirect_sg_entries=2048
>>
>> I dont set prefer_fr so it defaults to Y
>>
>> [root@localhost parameters]# cat prefer_fr
>> Y
>>
>> I have no settings for mlx5_core, all defaults.
>>
>> Thanks
>> Laurence
>>
>> --
>> To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
>> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>
>
> Max,
>
> Reverting a83e404 IB/srp: Reenable IB_MR_TYPE_SG_GAPS on the same source tree with all esle applied I am stable.
> So clearly we still have issues with IB_MR_TYPE_SG_GAPS.
>
> Thanks
> Laurence
>

Hi Laurence,
I would like to see the prints that Sagi asked in the srp_add_one 
function (echo "func srp_add_one +p" > 
/sys/kernel/debug/dynamic_debug/control) and also prints from 
srp_create_target (echo "func srp_create_target +p" > 
/sys/kernel/debug/dynamic_debug/control).

another patch can help is:

diff --git a/drivers/infiniband/ulp/srp/ib_srp.c 
b/drivers/infiniband/ulp/srp/ib_srp.c
index cee4626..53a67fd 100644
--- a/drivers/infiniband/ulp/srp/ib_srp.c
+++ b/drivers/infiniband/ulp/srp/ib_srp.c
@@ -3387,6 +3387,10 @@ static ssize_t srp_create_target(struct device *dev,
                              sizeof (struct srp_indirect_buf) +
                              target->cmd_sg_cnt * sizeof (struct 
srp_direct_buf);

+       pr_info("sg_tablesize %u mr_pool_size %u mr_per_cmd %u 
indirect_size %u max_iu_len %u max_sectors %u\n",
+               target->sg_tablesize, target->mr_pool_size, 
target->mr_per_cmd, target->indirect_size,
+               target->max_iu_len, target->scsi_host->max_sectors);
+
         INIT_WORK(&target->tl_err_work, srp_tl_err_work);
         INIT_WORK(&target->remove_work, srp_remove_work);
         spin_lock_init(&target->lock);


please add also the SG_GAPS Reenable commit and let's repro it again.
BTW, how many channels are open ?
can you load ib_srp module with ch_count param changes from 4 to 
#num_cpus and let's see when we get to repro it again.

thanks,
Max.

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH, untested] mlx5: Avoid that mlx5_ib_sg_to_klms() overflows the klms[] array

[Sagi Grimberg]

Looks good Bart,

Reviewed-by: Sagi Grimberg <sagi-NQWnxTmZq1alnMjI0IkVqw@public.gmane.org>
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html