* Re: [PATCH v2 1/8] IB/SRP: Avoid using IB_MR_TYPE_SG_GAPS
[not found] <017955b3-8fd5-40da-8bd5-023bc2f23fb4@email.android.com>
@ 2017-02-15 15:31 ` Bart Van Assche
0 siblings, 0 replies; 38+ messages in thread
From: Bart Van Assche @ 2017-02-15 15:31 UTC (permalink / raw)
To: leonro-VPRAkNaXOzVWk0Htik3J/w, loberman-H+wXaHxf7aLQT0dZR+AlfA
Cc: artemyko-VPRAkNaXOzVWk0Htik3J/w,
linux-rdma-u79uwXL29TY76Z2rM5mHXA,
israelr-VPRAkNaXOzVWk0Htik3J/w,
yuval.shaia-QHcLZuEGTsvQT0dZR+AlfA,
stable-u79uwXL29TY76Z2rM5mHXA, markb-VPRAkNaXOzVWk0Htik3J/w,
maxg-VPRAkNaXOzVWk0Htik3J/w, dledford-H+wXaHxf7aLQT0dZR+AlfA
On Wed, 2017-02-15 at 05:50 -0500, Laurence Oberman wrote:
> Bart only has mlx4.
I'll try to find an mlx5 setup.
Bart.
^ permalink raw reply [flat|nested] 38+ messages in thread
* Re: [PATCH v2 1/8] IB/SRP: Avoid using IB_MR_TYPE_SG_GAPS
@ 2017-02-15 15:31 ` Bart Van Assche
0 siblings, 0 replies; 38+ messages in thread
From: Bart Van Assche @ 2017-02-15 15:31 UTC (permalink / raw)
To: leonro, loberman
Cc: artemyko, linux-rdma, israelr, yuval.shaia, stable, markb, maxg,
dledford
On Wed, 2017-02-15 at 05:50 -0500, Laurence Oberman wrote:
> Bart only has mlx4.
I'll try to find an mlx5 setup.
Bart.
^ permalink raw reply [flat|nested] 38+ messages in thread
* Re: [PATCH v2 1/8] IB/SRP: Avoid using IB_MR_TYPE_SG_GAPS
2017-02-15 15:31 ` Bart Van Assche
@ 2017-02-15 15:34 ` Laurence Oberman
-1 siblings, 0 replies; 38+ messages in thread
From: Laurence Oberman @ 2017-02-15 15:34 UTC (permalink / raw)
To: Bart Van Assche
Cc: leonro-VPRAkNaXOzVWk0Htik3J/w, artemyko-VPRAkNaXOzVWk0Htik3J/w,
linux-rdma-u79uwXL29TY76Z2rM5mHXA,
israelr-VPRAkNaXOzVWk0Htik3J/w, yuval shaia,
stable-u79uwXL29TY76Z2rM5mHXA, markb-VPRAkNaXOzVWk0Htik3J/w,
maxg-VPRAkNaXOzVWk0Htik3J/w, dledford-H+wXaHxf7aLQT0dZR+AlfA
----- Original Message -----
> From: "Bart Van Assche" <Bart.VanAssche-XdAiOPVOjttBDgjK7y7TUQ@public.gmane.org>
> To: leonro-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org, loberman-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org
> Cc: artemyko-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org, linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org, israelr-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org, "yuval shaia" <yuval.shaia-QHcLZuEGTsvQT0dZR+AlfA@public.gmane.org>,
> stable-u79uwXL29TY76Z2rM5mHXA@public.gmane.org, markb-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org, maxg-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org, dledford-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org
> Sent: Wednesday, February 15, 2017 10:31:20 AM
> Subject: Re: [PATCH v2 1/8] IB/SRP: Avoid using IB_MR_TYPE_SG_GAPS
>
> On Wed, 2017-02-15 at 05:50 -0500, Laurence Oberman wrote:
> > Bart only has mlx4.
>
> I'll try to find an mlx5 setup.
>
> Bart.
Hello Bart
Its fine, I tested it already. See later email from me.
I am good to do the mlx5 tests, I have this test bed in place permanently.
Thanks
Laurence
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 38+ messages in thread
* Re: [PATCH v2 1/8] IB/SRP: Avoid using IB_MR_TYPE_SG_GAPS
@ 2017-02-15 15:34 ` Laurence Oberman
0 siblings, 0 replies; 38+ messages in thread
From: Laurence Oberman @ 2017-02-15 15:34 UTC (permalink / raw)
To: Bart Van Assche
Cc: leonro, artemyko, linux-rdma, israelr, yuval shaia, stable,
markb, maxg, dledford
----- Original Message -----
> From: "Bart Van Assche" <Bart.VanAssche@sandisk.com>
> To: leonro@mellanox.com, loberman@redhat.com
> Cc: artemyko@mellanox.com, linux-rdma@vger.kernel.org, israelr@mellanox.com, "yuval shaia" <yuval.shaia@oracle.com>,
> stable@vger.kernel.org, markb@mellanox.com, maxg@mellanox.com, dledford@redhat.com
> Sent: Wednesday, February 15, 2017 10:31:20 AM
> Subject: Re: [PATCH v2 1/8] IB/SRP: Avoid using IB_MR_TYPE_SG_GAPS
>
> On Wed, 2017-02-15 at 05:50 -0500, Laurence Oberman wrote:
> > Bart only has mlx4.
>
> I'll try to find an mlx5 setup.
>
> Bart.
Hello Bart
Its fine, I tested it already. See later email from me.
I am good to do the mlx5 tests, I have this test bed in place permanently.
Thanks
Laurence
^ permalink raw reply [flat|nested] 38+ messages in thread
* Re: [PATCH v2 1/8] IB/SRP: Avoid using IB_MR_TYPE_SG_GAPS
2017-02-16 6:14 ` Leon Romanovsky
@ 2017-02-16 9:11 ` Max Gurtovoy
-1 siblings, 0 replies; 38+ messages in thread
From: Max Gurtovoy @ 2017-02-16 9:11 UTC (permalink / raw)
To: Leon Romanovsky, Sagi Grimberg
Cc: Laurence Oberman, Bart Van Assche, Doug Ledford, linux-rdma,
Israel Rukshin, Mark Bloch, Yuval Shaia, # 4 . 7+
On 2/16/2017 8:14 AM, Leon Romanovsky wrote:
> On Wed, Feb 15, 2017 at 06:55:52PM +0200, Sagi Grimberg wrote:
>>
>>> Started with Linus's tree, applied the change requested by Sagi, built the kernel, rebooted and started the tests.
>>>
>>> Linux ibclient 4.10.0-rc8.sagi+ #1 SMP Wed Feb 15 11:09:44 EST 2017 x86_64 x86_64 x86_64 GNU/Linux
>>>
>>> Very quickly get to this
>>>
>>> [ 180.990285] mlx5_0:dump_cqe:262:(pid 0): dump error cqe
>>> [ 181.016899] 00000000 00000000 00000000 00000000
>>> [ 181.040949] 00000000 00000000 00000000 00000000
>>> [ 181.066960] 00000000 00000000 00000000 00000000
>>> [ 181.092030] 00000000 0f007806 2500002a bf1913d0
>>> [ 181.117254] scsi host2: ib_srp: failed FAST REG status memory management operation error (6) for CQE ffff880bdbe88778
>>> [ 196.288933] fast_io_fail_tmo expired for SRP port-2:1 / host2.
>>> [ 197.090886] scsi host2: ib_srp: reconnect succeeded
>>> [ 197.127628] scsi host2: ib_srp: failed RECV status WR flushed (5) for CQE ffff8817f09b6f30
>>>
>>> So does not help.
>>> I think my and Barts suggestion to revert for now is the best way forward.
>>> I have already tested this in-depth from Bart's tree and its been sent to Doug as V2 of Bart'recent 8 patch series.
>>
>> Yea, probably this is the best way forward.
>>
>> Bart, I think the change I suggested is still needed regardless,
>> do you agree?
>>
>> Max, Leon, is it possible that the max number of klms pr mr is
>> less than what reported in device capabilities for page_list_len?
>
> I hope no and we will check.
> I already asked it, but didn't get any response, and I'll repeat it again.
> ISER has similar code with SG_GAPS, does it work?
Yes, I haven't seen issues with that in iSER.
We need to continue with the debug.
>
>>
>> If so, this means that either:
>> 1. mlx5 needs to expose the minimum between pages and sg elems (sucks)
>> 2. we need yet another capability for SG_GAPS (sucks^2 because the
>> whole point was to make it transparent to the user)
>> 3. mlx5 does not support SG_GAPS (sucks^3 because we now have something
>> thats not supported by any device).
>> --
>> To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 38+ messages in thread
* Re: [PATCH v2 1/8] IB/SRP: Avoid using IB_MR_TYPE_SG_GAPS
@ 2017-02-16 9:11 ` Max Gurtovoy
0 siblings, 0 replies; 38+ messages in thread
From: Max Gurtovoy @ 2017-02-16 9:11 UTC (permalink / raw)
To: Leon Romanovsky, Sagi Grimberg
Cc: Laurence Oberman, Bart Van Assche, Doug Ledford, linux-rdma,
Israel Rukshin, Mark Bloch, Yuval Shaia, # 4 . 7+
On 2/16/2017 8:14 AM, Leon Romanovsky wrote:
> On Wed, Feb 15, 2017 at 06:55:52PM +0200, Sagi Grimberg wrote:
>>
>>> Started with Linus's tree, applied the change requested by Sagi, built the kernel, rebooted and started the tests.
>>>
>>> Linux ibclient 4.10.0-rc8.sagi+ #1 SMP Wed Feb 15 11:09:44 EST 2017 x86_64 x86_64 x86_64 GNU/Linux
>>>
>>> Very quickly get to this
>>>
>>> [ 180.990285] mlx5_0:dump_cqe:262:(pid 0): dump error cqe
>>> [ 181.016899] 00000000 00000000 00000000 00000000
>>> [ 181.040949] 00000000 00000000 00000000 00000000
>>> [ 181.066960] 00000000 00000000 00000000 00000000
>>> [ 181.092030] 00000000 0f007806 2500002a bf1913d0
>>> [ 181.117254] scsi host2: ib_srp: failed FAST REG status memory management operation error (6) for CQE ffff880bdbe88778
>>> [ 196.288933] fast_io_fail_tmo expired for SRP port-2:1 / host2.
>>> [ 197.090886] scsi host2: ib_srp: reconnect succeeded
>>> [ 197.127628] scsi host2: ib_srp: failed RECV status WR flushed (5) for CQE ffff8817f09b6f30
>>>
>>> So does not help.
>>> I think my and Barts suggestion to revert for now is the best way forward.
>>> I have already tested this in-depth from Bart's tree and its been sent to Doug as V2 of Bart'recent 8 patch series.
>>
>> Yea, probably this is the best way forward.
>>
>> Bart, I think the change I suggested is still needed regardless,
>> do you agree?
>>
>> Max, Leon, is it possible that the max number of klms pr mr is
>> less than what reported in device capabilities for page_list_len?
>
> I hope no and we will check.
> I already asked it, but didn't get any response, and I'll repeat it again.
> ISER has similar code with SG_GAPS, does it work?
Yes, I haven't seen issues with that in iSER.
We need to continue with the debug.
>
>>
>> If so, this means that either:
>> 1. mlx5 needs to expose the minimum between pages and sg elems (sucks)
>> 2. we need yet another capability for SG_GAPS (sucks^2 because the
>> whole point was to make it transparent to the user)
>> 3. mlx5 does not support SG_GAPS (sucks^3 because we now have something
>> thats not supported by any device).
>> --
>> To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 38+ messages in thread
* Re: [PATCH v2 1/8] IB/SRP: Avoid using IB_MR_TYPE_SG_GAPS
2017-02-15 16:55 ` Sagi Grimberg
@ 2017-02-16 6:14 ` Leon Romanovsky
2017-02-16 6:14 ` Leon Romanovsky
1 sibling, 0 replies; 38+ messages in thread
From: Leon Romanovsky @ 2017-02-16 6:14 UTC (permalink / raw)
To: Sagi Grimberg
Cc: Laurence Oberman, Bart Van Assche, Doug Ledford, linux-rdma,
Israel Rukshin, Max Gurtovoy, Mark Bloch, Yuval Shaia, # 4 . 7+
[-- Attachment #1: Type: text/plain, Size: 2157 bytes --]
On Wed, Feb 15, 2017 at 06:55:52PM +0200, Sagi Grimberg wrote:
>
> > Started with Linus's tree, applied the change requested by Sagi, built the kernel, rebooted and started the tests.
> >
> > Linux ibclient 4.10.0-rc8.sagi+ #1 SMP Wed Feb 15 11:09:44 EST 2017 x86_64 x86_64 x86_64 GNU/Linux
> >
> > Very quickly get to this
> >
> > [ 180.990285] mlx5_0:dump_cqe:262:(pid 0): dump error cqe
> > [ 181.016899] 00000000 00000000 00000000 00000000
> > [ 181.040949] 00000000 00000000 00000000 00000000
> > [ 181.066960] 00000000 00000000 00000000 00000000
> > [ 181.092030] 00000000 0f007806 2500002a bf1913d0
> > [ 181.117254] scsi host2: ib_srp: failed FAST REG status memory management operation error (6) for CQE ffff880bdbe88778
> > [ 196.288933] fast_io_fail_tmo expired for SRP port-2:1 / host2.
> > [ 197.090886] scsi host2: ib_srp: reconnect succeeded
> > [ 197.127628] scsi host2: ib_srp: failed RECV status WR flushed (5) for CQE ffff8817f09b6f30
> >
> > So does not help.
> > I think my and Barts suggestion to revert for now is the best way forward.
> > I have already tested this in-depth from Bart's tree and its been sent to Doug as V2 of Bart'recent 8 patch series.
>
> Yea, probably this is the best way forward.
>
> Bart, I think the change I suggested is still needed regardless,
> do you agree?
>
> Max, Leon, is it possible that the max number of klms pr mr is
> less than what reported in device capabilities for page_list_len?
I hope no and we will check.
I already asked it, but didn't get any response, and I'll repeat it again.
ISER has similar code with SG_GAPS, does it work?
>
> If so, this means that either:
> 1. mlx5 needs to expose the minimum between pages and sg elems (sucks)
> 2. we need yet another capability for SG_GAPS (sucks^2 because the
> whole point was to make it transparent to the user)
> 3. mlx5 does not support SG_GAPS (sucks^3 because we now have something
> thats not supported by any device).
> --
> To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]
^ permalink raw reply [flat|nested] 38+ messages in thread
* Re: [PATCH v2 1/8] IB/SRP: Avoid using IB_MR_TYPE_SG_GAPS
@ 2017-02-16 6:14 ` Leon Romanovsky
0 siblings, 0 replies; 38+ messages in thread
From: Leon Romanovsky @ 2017-02-16 6:14 UTC (permalink / raw)
To: Sagi Grimberg
Cc: Laurence Oberman, Bart Van Assche, Doug Ledford, linux-rdma,
Israel Rukshin, Max Gurtovoy, Mark Bloch, Yuval Shaia, # 4 . 7+
[-- Attachment #1: Type: text/plain, Size: 2157 bytes --]
On Wed, Feb 15, 2017 at 06:55:52PM +0200, Sagi Grimberg wrote:
>
> > Started with Linus's tree, applied the change requested by Sagi, built the kernel, rebooted and started the tests.
> >
> > Linux ibclient 4.10.0-rc8.sagi+ #1 SMP Wed Feb 15 11:09:44 EST 2017 x86_64 x86_64 x86_64 GNU/Linux
> >
> > Very quickly get to this
> >
> > [ 180.990285] mlx5_0:dump_cqe:262:(pid 0): dump error cqe
> > [ 181.016899] 00000000 00000000 00000000 00000000
> > [ 181.040949] 00000000 00000000 00000000 00000000
> > [ 181.066960] 00000000 00000000 00000000 00000000
> > [ 181.092030] 00000000 0f007806 2500002a bf1913d0
> > [ 181.117254] scsi host2: ib_srp: failed FAST REG status memory management operation error (6) for CQE ffff880bdbe88778
> > [ 196.288933] fast_io_fail_tmo expired for SRP port-2:1 / host2.
> > [ 197.090886] scsi host2: ib_srp: reconnect succeeded
> > [ 197.127628] scsi host2: ib_srp: failed RECV status WR flushed (5) for CQE ffff8817f09b6f30
> >
> > So does not help.
> > I think my and Barts suggestion to revert for now is the best way forward.
> > I have already tested this in-depth from Bart's tree and its been sent to Doug as V2 of Bart'recent 8 patch series.
>
> Yea, probably this is the best way forward.
>
> Bart, I think the change I suggested is still needed regardless,
> do you agree?
>
> Max, Leon, is it possible that the max number of klms pr mr is
> less than what reported in device capabilities for page_list_len?
I hope no and we will check.
I already asked it, but didn't get any response, and I'll repeat it again.
ISER has similar code with SG_GAPS, does it work?
>
> If so, this means that either:
> 1. mlx5 needs to expose the minimum between pages and sg elems (sucks)
> 2. we need yet another capability for SG_GAPS (sucks^2 because the
> whole point was to make it transparent to the user)
> 3. mlx5 does not support SG_GAPS (sucks^3 because we now have something
> thats not supported by any device).
> --
> To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]
^ permalink raw reply [flat|nested] 38+ messages in thread
* Re: [PATCH v2 1/8] IB/SRP: Avoid using IB_MR_TYPE_SG_GAPS
2017-02-15 16:55 ` Sagi Grimberg
@ 2017-02-15 23:49 ` Bart Van Assche
2017-02-16 6:14 ` Leon Romanovsky
1 sibling, 0 replies; 38+ messages in thread
From: Bart Van Assche @ 2017-02-15 23:49 UTC (permalink / raw)
To: sagi, loberman
Cc: leonro, linux-rdma, israelr, yuval.shaia, markb, stable, maxg, dledford
On Wed, 2017-02-15 at 18:55 +0200, Sagi Grimberg wrote:
> Bart, I think the change I suggested is still needed regardless,
> do you agree?
I'm not sure. I have not yet had a close look at that part of the mlx5 driver.
Bart.
^ permalink raw reply [flat|nested] 38+ messages in thread
* Re: [PATCH v2 1/8] IB/SRP: Avoid using IB_MR_TYPE_SG_GAPS
2017-02-15 16:37 ` Laurence Oberman
(?)
@ 2017-02-15 16:55 ` Sagi Grimberg
2017-02-15 23:49 ` Bart Van Assche
2017-02-16 6:14 ` Leon Romanovsky
-1 siblings, 2 replies; 38+ messages in thread
From: Sagi Grimberg @ 2017-02-15 16:55 UTC (permalink / raw)
To: Laurence Oberman
Cc: Bart Van Assche, Doug Ledford, linux-rdma, Israel Rukshin,
Max Gurtovoy, Leon Romanovsky, Mark Bloch, Yuval Shaia, # 4 . 7+
> Started with Linus's tree, applied the change requested by Sagi, built the kernel, rebooted and started the tests.
>
> Linux ibclient 4.10.0-rc8.sagi+ #1 SMP Wed Feb 15 11:09:44 EST 2017 x86_64 x86_64 x86_64 GNU/Linux
>
> Very quickly get to this
>
> [ 180.990285] mlx5_0:dump_cqe:262:(pid 0): dump error cqe
> [ 181.016899] 00000000 00000000 00000000 00000000
> [ 181.040949] 00000000 00000000 00000000 00000000
> [ 181.066960] 00000000 00000000 00000000 00000000
> [ 181.092030] 00000000 0f007806 2500002a bf1913d0
> [ 181.117254] scsi host2: ib_srp: failed FAST REG status memory management operation error (6) for CQE ffff880bdbe88778
> [ 196.288933] fast_io_fail_tmo expired for SRP port-2:1 / host2.
> [ 197.090886] scsi host2: ib_srp: reconnect succeeded
> [ 197.127628] scsi host2: ib_srp: failed RECV status WR flushed (5) for CQE ffff8817f09b6f30
>
> So does not help.
> I think my and Barts suggestion to revert for now is the best way forward.
> I have already tested this in-depth from Bart's tree and its been sent to Doug as V2 of Bart'recent 8 patch series.
Yea, probably this is the best way forward.
Bart, I think the change I suggested is still needed regardless,
do you agree?
Max, Leon, is it possible that the max number of klms pr mr is
less than what reported in device capabilities for page_list_len?
If so, this means that either:
1. mlx5 needs to expose the minimum between pages and sg elems (sucks)
2. we need yet another capability for SG_GAPS (sucks^2 because the
whole point was to make it transparent to the user)
3. mlx5 does not support SG_GAPS (sucks^3 because we now have something
thats not supported by any device).
^ permalink raw reply [flat|nested] 38+ messages in thread
* Re: [PATCH v2 1/8] IB/SRP: Avoid using IB_MR_TYPE_SG_GAPS
2017-02-15 15:38 ` Sagi Grimberg
@ 2017-02-15 16:37 ` Laurence Oberman
-1 siblings, 0 replies; 38+ messages in thread
From: Laurence Oberman @ 2017-02-15 16:37 UTC (permalink / raw)
To: Sagi Grimberg
Cc: Bart Van Assche, Doug Ledford, linux-rdma-u79uwXL29TY76Z2rM5mHXA,
Israel Rukshin, Max Gurtovoy, Leon Romanovsky, Mark Bloch,
Yuval Shaia, # 4 . 7+
----- Original Message -----
> From: "Sagi Grimberg" <sagi-NQWnxTmZq1alnMjI0IkVqw@public.gmane.org>
> To: "Bart Van Assche" <bart.vanassche-XdAiOPVOjttBDgjK7y7TUQ@public.gmane.org>, "Doug Ledford" <dledford-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>
> Cc: linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org, "Israel Rukshin" <israelr-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>, "Max Gurtovoy" <maxg-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>, "Leon
> Romanovsky" <leonro-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>, "Mark Bloch" <markb-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>, "Yuval Shaia" <yuval.shaia-QHcLZuEGTsvQT0dZR+AlfA@public.gmane.org>, "# 4 .
> 7+" <stable-u79uwXL29TY76Z2rM5mHXA@public.gmane.org>
> Sent: Wednesday, February 15, 2017 10:38:06 AM
> Subject: Re: [PATCH v2 1/8] IB/SRP: Avoid using IB_MR_TYPE_SG_GAPS
>
>
> > Tests have shown that the following error message is reported when
> > using SG-GAPS registration with an mlx5 adapter:
> >
> > scsi host1: ib_srp: failed RECV status WR flushed (5) for CQE
> > ffff880bd4270eb0
> > 00000000 00000000 00000000 00000000
> > 00000000 00000000 00000000 00000000
> > 00000000 00000000 00000000 00000000
> > 00000000 0f007806 2500002a ad9fafd1
> > scsi host1: ib_srp: reconnect succeeded
> > mlx5_0:dump_cqe:262:(pid 7369): dump error cqe
> > 00000000 00000000 00000000 00000000
> > 00000000 00000000 00000000 00000000
> > 00000000 00000000 00000000 00000000
> > 00000000 0f007806 25000032 00105dd0
> > scsi host1: ib_srp: failed FAST REG status memory management operation
> > error (6) for CQE ffff880b92860138
> >
> > Hence avoid using SG-GAPS memory registrations. Additionally,
> > always configure the blk_queue_virt_boundary() to avoid to trigger
> > a mapping failure when using adapters that support SG-GAPS (e.g.
> > mlx5).
>
> Hi Guys,
>
> Sorry for addressing this late, but has this failure been investigated?
>
> Max, Israel, what does this error syndrome map to?
>
> Looking at mlx5_ib_sg_to_klms, I think the mr->length is incorrectly
> incremented. Does the following change fix the problem?
> --
> diff --git a/drivers/infiniband/hw/mlx5/mr.c
> b/drivers/infiniband/hw/mlx5/mr.c
> index 8f608debe141..c21c9eee37f6 100644
> --- a/drivers/infiniband/hw/mlx5/mr.c
> +++ b/drivers/infiniband/hw/mlx5/mr.c
> @@ -1832,7 +1832,7 @@ mlx5_ib_sg_to_klms(struct mlx5_ib_mr *mr,
> klms[i].va = cpu_to_be64(sg_dma_address(sg) + sg_offset);
> klms[i].bcount = cpu_to_be32(sg_dma_len(sg) - sg_offset);
> klms[i].key = cpu_to_be32(lkey);
> - mr->ibmr.length += sg_dma_len(sg);
> + mr->ibmr.length += sg_dma_len(sg) - sg_offset;
>
> sg_offset = 0;
> }
> --
> --
> To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
>
Started with Linus's tree, applied the change requested by Sagi, built the kernel, rebooted and started the tests.
Linux ibclient 4.10.0-rc8.sagi+ #1 SMP Wed Feb 15 11:09:44 EST 2017 x86_64 x86_64 x86_64 GNU/Linux
Very quickly get to this
[ 180.990285] mlx5_0:dump_cqe:262:(pid 0): dump error cqe
[ 181.016899] 00000000 00000000 00000000 00000000
[ 181.040949] 00000000 00000000 00000000 00000000
[ 181.066960] 00000000 00000000 00000000 00000000
[ 181.092030] 00000000 0f007806 2500002a bf1913d0
[ 181.117254] scsi host2: ib_srp: failed FAST REG status memory management operation error (6) for CQE ffff880bdbe88778
[ 196.288933] fast_io_fail_tmo expired for SRP port-2:1 / host2.
[ 197.090886] scsi host2: ib_srp: reconnect succeeded
[ 197.127628] scsi host2: ib_srp: failed RECV status WR flushed (5) for CQE ffff8817f09b6f30
So does not help.
I think my and Barts suggestion to revert for now is the best way forward.
I have already tested this in-depth from Bart's tree and its been sent to Doug as V2 of Bart'recent 8 patch series.
Thanks
Laurence
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 38+ messages in thread
* Re: [PATCH v2 1/8] IB/SRP: Avoid using IB_MR_TYPE_SG_GAPS
@ 2017-02-15 16:37 ` Laurence Oberman
0 siblings, 0 replies; 38+ messages in thread
From: Laurence Oberman @ 2017-02-15 16:37 UTC (permalink / raw)
To: Sagi Grimberg
Cc: Bart Van Assche, Doug Ledford, linux-rdma, Israel Rukshin,
Max Gurtovoy, Leon Romanovsky, Mark Bloch, Yuval Shaia, # 4 . 7+
----- Original Message -----
> From: "Sagi Grimberg" <sagi@grimberg.me>
> To: "Bart Van Assche" <bart.vanassche@sandisk.com>, "Doug Ledford" <dledford@redhat.com>
> Cc: linux-rdma@vger.kernel.org, "Israel Rukshin" <israelr@mellanox.com>, "Max Gurtovoy" <maxg@mellanox.com>, "Leon
> Romanovsky" <leonro@mellanox.com>, "Mark Bloch" <markb@mellanox.com>, "Yuval Shaia" <yuval.shaia@oracle.com>, "# 4 .
> 7+" <stable@vger.kernel.org>
> Sent: Wednesday, February 15, 2017 10:38:06 AM
> Subject: Re: [PATCH v2 1/8] IB/SRP: Avoid using IB_MR_TYPE_SG_GAPS
>
>
> > Tests have shown that the following error message is reported when
> > using SG-GAPS registration with an mlx5 adapter:
> >
> > scsi host1: ib_srp: failed RECV status WR flushed (5) for CQE
> > ffff880bd4270eb0
> > 00000000 00000000 00000000 00000000
> > 00000000 00000000 00000000 00000000
> > 00000000 00000000 00000000 00000000
> > 00000000 0f007806 2500002a ad9fafd1
> > scsi host1: ib_srp: reconnect succeeded
> > mlx5_0:dump_cqe:262:(pid 7369): dump error cqe
> > 00000000 00000000 00000000 00000000
> > 00000000 00000000 00000000 00000000
> > 00000000 00000000 00000000 00000000
> > 00000000 0f007806 25000032 00105dd0
> > scsi host1: ib_srp: failed FAST REG status memory management operation
> > error (6) for CQE ffff880b92860138
> >
> > Hence avoid using SG-GAPS memory registrations. Additionally,
> > always configure the blk_queue_virt_boundary() to avoid to trigger
> > a mapping failure when using adapters that support SG-GAPS (e.g.
> > mlx5).
>
> Hi Guys,
>
> Sorry for addressing this late, but has this failure been investigated?
>
> Max, Israel, what does this error syndrome map to?
>
> Looking at mlx5_ib_sg_to_klms, I think the mr->length is incorrectly
> incremented. Does the following change fix the problem?
> --
> diff --git a/drivers/infiniband/hw/mlx5/mr.c
> b/drivers/infiniband/hw/mlx5/mr.c
> index 8f608debe141..c21c9eee37f6 100644
> --- a/drivers/infiniband/hw/mlx5/mr.c
> +++ b/drivers/infiniband/hw/mlx5/mr.c
> @@ -1832,7 +1832,7 @@ mlx5_ib_sg_to_klms(struct mlx5_ib_mr *mr,
> klms[i].va = cpu_to_be64(sg_dma_address(sg) + sg_offset);
> klms[i].bcount = cpu_to_be32(sg_dma_len(sg) - sg_offset);
> klms[i].key = cpu_to_be32(lkey);
> - mr->ibmr.length += sg_dma_len(sg);
> + mr->ibmr.length += sg_dma_len(sg) - sg_offset;
>
> sg_offset = 0;
> }
> --
> --
> To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
>
Started with Linus's tree, applied the change requested by Sagi, built the kernel, rebooted and started the tests.
Linux ibclient 4.10.0-rc8.sagi+ #1 SMP Wed Feb 15 11:09:44 EST 2017 x86_64 x86_64 x86_64 GNU/Linux
Very quickly get to this
[ 180.990285] mlx5_0:dump_cqe:262:(pid 0): dump error cqe
[ 181.016899] 00000000 00000000 00000000 00000000
[ 181.040949] 00000000 00000000 00000000 00000000
[ 181.066960] 00000000 00000000 00000000 00000000
[ 181.092030] 00000000 0f007806 2500002a bf1913d0
[ 181.117254] scsi host2: ib_srp: failed FAST REG status memory management operation error (6) for CQE ffff880bdbe88778
[ 196.288933] fast_io_fail_tmo expired for SRP port-2:1 / host2.
[ 197.090886] scsi host2: ib_srp: reconnect succeeded
[ 197.127628] scsi host2: ib_srp: failed RECV status WR flushed (5) for CQE ffff8817f09b6f30
So does not help.
I think my and Barts suggestion to revert for now is the best way forward.
I have already tested this in-depth from Bart's tree and its been sent to Doug as V2 of Bart'recent 8 patch series.
Thanks
Laurence
^ permalink raw reply [flat|nested] 38+ messages in thread
* Re: [PATCH v2 1/8] IB/SRP: Avoid using IB_MR_TYPE_SG_GAPS
2017-02-15 16:18 ` Max Gurtovoy
@ 2017-02-15 16:30 ` Leon Romanovsky
-1 siblings, 0 replies; 38+ messages in thread
From: Leon Romanovsky @ 2017-02-15 16:30 UTC (permalink / raw)
To: Max Gurtovoy
Cc: Sagi Grimberg, Bart Van Assche, Doug Ledford,
linux-rdma-u79uwXL29TY76Z2rM5mHXA, Israel Rukshin, Mark Bloch,
Yuval Shaia, # 4 . 7+
[-- Attachment #1: Type: text/plain, Size: 2582 bytes --]
On Wed, Feb 15, 2017 at 06:18:02PM +0200, Max Gurtovoy wrote:
>
>
> On 2/15/2017 5:38 PM, Sagi Grimberg wrote:
> >
> > > Tests have shown that the following error message is reported when
> > > using SG-GAPS registration with an mlx5 adapter:
> > >
> > > scsi host1: ib_srp: failed RECV status WR flushed (5) for CQE
> > > ffff880bd4270eb0
> > > 00000000 00000000 00000000 00000000
> > > 00000000 00000000 00000000 00000000
> > > 00000000 00000000 00000000 00000000
> > > 00000000 0f007806 2500002a ad9fafd1
> > > scsi host1: ib_srp: reconnect succeeded
> > > mlx5_0:dump_cqe:262:(pid 7369): dump error cqe
> > > 00000000 00000000 00000000 00000000
> > > 00000000 00000000 00000000 00000000
> > > 00000000 00000000 00000000 00000000
> > > 00000000 0f007806 25000032 00105dd0
> > > scsi host1: ib_srp: failed FAST REG status memory management operation
> > > error (6) for CQE ffff880b92860138
> > >
> > > Hence avoid using SG-GAPS memory registrations. Additionally,
> > > always configure the blk_queue_virt_boundary() to avoid to trigger
> > > a mapping failure when using adapters that support SG-GAPS (e.g.
> > > mlx5).
> >
> > Hi Guys,
> >
> > Sorry for addressing this late, but has this failure been investigated?
> >
> > Max, Israel, what does this error syndrome map to?
>
> Sagi,
> this syndrome says that number of klms to write is bigger than number of
> mtts.
>
> Artemy started investigating it and proposed solution that were tested by
> Laurence.
> Let's see if your fix will help.
No, Artemy's change doesn't fix it.
>
> >
> > Looking at mlx5_ib_sg_to_klms, I think the mr->length is incorrectly
> > incremented. Does the following change fix the problem?
> > --
> > diff --git a/drivers/infiniband/hw/mlx5/mr.c
> > b/drivers/infiniband/hw/mlx5/mr.c
> > index 8f608debe141..c21c9eee37f6 100644
> > --- a/drivers/infiniband/hw/mlx5/mr.c
> > +++ b/drivers/infiniband/hw/mlx5/mr.c
> > @@ -1832,7 +1832,7 @@ mlx5_ib_sg_to_klms(struct mlx5_ib_mr *mr,
> > klms[i].va = cpu_to_be64(sg_dma_address(sg) + sg_offset);
> > klms[i].bcount = cpu_to_be32(sg_dma_len(sg) - sg_offset);
> > klms[i].key = cpu_to_be32(lkey);
> > - mr->ibmr.length += sg_dma_len(sg);
> > + mr->ibmr.length += sg_dma_len(sg) - sg_offset;
> >
> > sg_offset = 0;
> > }
> > --
> --
> To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]
^ permalink raw reply [flat|nested] 38+ messages in thread
* Re: [PATCH v2 1/8] IB/SRP: Avoid using IB_MR_TYPE_SG_GAPS
@ 2017-02-15 16:30 ` Leon Romanovsky
0 siblings, 0 replies; 38+ messages in thread
From: Leon Romanovsky @ 2017-02-15 16:30 UTC (permalink / raw)
To: Max Gurtovoy
Cc: Sagi Grimberg, Bart Van Assche, Doug Ledford, linux-rdma,
Israel Rukshin, Mark Bloch, Yuval Shaia, # 4 . 7+
[-- Attachment #1: Type: text/plain, Size: 2558 bytes --]
On Wed, Feb 15, 2017 at 06:18:02PM +0200, Max Gurtovoy wrote:
>
>
> On 2/15/2017 5:38 PM, Sagi Grimberg wrote:
> >
> > > Tests have shown that the following error message is reported when
> > > using SG-GAPS registration with an mlx5 adapter:
> > >
> > > scsi host1: ib_srp: failed RECV status WR flushed (5) for CQE
> > > ffff880bd4270eb0
> > > 00000000 00000000 00000000 00000000
> > > 00000000 00000000 00000000 00000000
> > > 00000000 00000000 00000000 00000000
> > > 00000000 0f007806 2500002a ad9fafd1
> > > scsi host1: ib_srp: reconnect succeeded
> > > mlx5_0:dump_cqe:262:(pid 7369): dump error cqe
> > > 00000000 00000000 00000000 00000000
> > > 00000000 00000000 00000000 00000000
> > > 00000000 00000000 00000000 00000000
> > > 00000000 0f007806 25000032 00105dd0
> > > scsi host1: ib_srp: failed FAST REG status memory management operation
> > > error (6) for CQE ffff880b92860138
> > >
> > > Hence avoid using SG-GAPS memory registrations. Additionally,
> > > always configure the blk_queue_virt_boundary() to avoid to trigger
> > > a mapping failure when using adapters that support SG-GAPS (e.g.
> > > mlx5).
> >
> > Hi Guys,
> >
> > Sorry for addressing this late, but has this failure been investigated?
> >
> > Max, Israel, what does this error syndrome map to?
>
> Sagi,
> this syndrome says that number of klms to write is bigger than number of
> mtts.
>
> Artemy started investigating it and proposed solution that were tested by
> Laurence.
> Let's see if your fix will help.
No, Artemy's change doesn't fix it.
>
> >
> > Looking at mlx5_ib_sg_to_klms, I think the mr->length is incorrectly
> > incremented. Does the following change fix the problem?
> > --
> > diff --git a/drivers/infiniband/hw/mlx5/mr.c
> > b/drivers/infiniband/hw/mlx5/mr.c
> > index 8f608debe141..c21c9eee37f6 100644
> > --- a/drivers/infiniband/hw/mlx5/mr.c
> > +++ b/drivers/infiniband/hw/mlx5/mr.c
> > @@ -1832,7 +1832,7 @@ mlx5_ib_sg_to_klms(struct mlx5_ib_mr *mr,
> > klms[i].va = cpu_to_be64(sg_dma_address(sg) + sg_offset);
> > klms[i].bcount = cpu_to_be32(sg_dma_len(sg) - sg_offset);
> > klms[i].key = cpu_to_be32(lkey);
> > - mr->ibmr.length += sg_dma_len(sg);
> > + mr->ibmr.length += sg_dma_len(sg) - sg_offset;
> >
> > sg_offset = 0;
> > }
> > --
> --
> To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]
^ permalink raw reply [flat|nested] 38+ messages in thread
* Re: [PATCH v2 1/8] IB/SRP: Avoid using IB_MR_TYPE_SG_GAPS
2017-02-15 16:18 ` Max Gurtovoy
(?)
@ 2017-02-15 16:27 ` Sagi Grimberg
-1 siblings, 0 replies; 38+ messages in thread
From: Sagi Grimberg @ 2017-02-15 16:27 UTC (permalink / raw)
To: Max Gurtovoy, Bart Van Assche, Doug Ledford
Cc: linux-rdma, Israel Rukshin, Leon Romanovsky, Mark Bloch,
Yuval Shaia, # 4 . 7+
>> Hi Guys,
>>
>> Sorry for addressing this late, but has this failure been investigated?
>>
>> Max, Israel, what does this error syndrome map to?
>
> Sagi,
> this syndrome says that number of klms to write is bigger than number of
> mtts.
That is strange, given that we check for it explicitly...
if this is indeed the case, then mlx5 reports wrong
max_fast_reg_page_list_len.
> Artemy started investigating it and proposed solution that were tested
> by Laurence.
> Let's see if your fix will help.
Not sure it will help if the syndrome is what you say it is...
^ permalink raw reply [flat|nested] 38+ messages in thread
* Re: [PATCH v2 1/8] IB/SRP: Avoid using IB_MR_TYPE_SG_GAPS
2017-02-15 15:38 ` Sagi Grimberg
@ 2017-02-15 16:18 ` Max Gurtovoy
-1 siblings, 0 replies; 38+ messages in thread
From: Max Gurtovoy @ 2017-02-15 16:18 UTC (permalink / raw)
To: Sagi Grimberg, Bart Van Assche, Doug Ledford
Cc: linux-rdma-u79uwXL29TY76Z2rM5mHXA, Israel Rukshin,
Leon Romanovsky, Mark Bloch, Yuval Shaia, # 4 . 7+
On 2/15/2017 5:38 PM, Sagi Grimberg wrote:
>
>> Tests have shown that the following error message is reported when
>> using SG-GAPS registration with an mlx5 adapter:
>>
>> scsi host1: ib_srp: failed RECV status WR flushed (5) for CQE
>> ffff880bd4270eb0
>> 00000000 00000000 00000000 00000000
>> 00000000 00000000 00000000 00000000
>> 00000000 00000000 00000000 00000000
>> 00000000 0f007806 2500002a ad9fafd1
>> scsi host1: ib_srp: reconnect succeeded
>> mlx5_0:dump_cqe:262:(pid 7369): dump error cqe
>> 00000000 00000000 00000000 00000000
>> 00000000 00000000 00000000 00000000
>> 00000000 00000000 00000000 00000000
>> 00000000 0f007806 25000032 00105dd0
>> scsi host1: ib_srp: failed FAST REG status memory management operation
>> error (6) for CQE ffff880b92860138
>>
>> Hence avoid using SG-GAPS memory registrations. Additionally,
>> always configure the blk_queue_virt_boundary() to avoid to trigger
>> a mapping failure when using adapters that support SG-GAPS (e.g.
>> mlx5).
>
> Hi Guys,
>
> Sorry for addressing this late, but has this failure been investigated?
>
> Max, Israel, what does this error syndrome map to?
Sagi,
this syndrome says that number of klms to write is bigger than number of
mtts.
Artemy started investigating it and proposed solution that were tested
by Laurence.
Let's see if your fix will help.
>
> Looking at mlx5_ib_sg_to_klms, I think the mr->length is incorrectly
> incremented. Does the following change fix the problem?
> --
> diff --git a/drivers/infiniband/hw/mlx5/mr.c
> b/drivers/infiniband/hw/mlx5/mr.c
> index 8f608debe141..c21c9eee37f6 100644
> --- a/drivers/infiniband/hw/mlx5/mr.c
> +++ b/drivers/infiniband/hw/mlx5/mr.c
> @@ -1832,7 +1832,7 @@ mlx5_ib_sg_to_klms(struct mlx5_ib_mr *mr,
> klms[i].va = cpu_to_be64(sg_dma_address(sg) + sg_offset);
> klms[i].bcount = cpu_to_be32(sg_dma_len(sg) - sg_offset);
> klms[i].key = cpu_to_be32(lkey);
> - mr->ibmr.length += sg_dma_len(sg);
> + mr->ibmr.length += sg_dma_len(sg) - sg_offset;
>
> sg_offset = 0;
> }
> --
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 38+ messages in thread
* Re: [PATCH v2 1/8] IB/SRP: Avoid using IB_MR_TYPE_SG_GAPS
@ 2017-02-15 16:18 ` Max Gurtovoy
0 siblings, 0 replies; 38+ messages in thread
From: Max Gurtovoy @ 2017-02-15 16:18 UTC (permalink / raw)
To: Sagi Grimberg, Bart Van Assche, Doug Ledford
Cc: linux-rdma, Israel Rukshin, Leon Romanovsky, Mark Bloch,
Yuval Shaia, # 4 . 7+
On 2/15/2017 5:38 PM, Sagi Grimberg wrote:
>
>> Tests have shown that the following error message is reported when
>> using SG-GAPS registration with an mlx5 adapter:
>>
>> scsi host1: ib_srp: failed RECV status WR flushed (5) for CQE
>> ffff880bd4270eb0
>> 00000000 00000000 00000000 00000000
>> 00000000 00000000 00000000 00000000
>> 00000000 00000000 00000000 00000000
>> 00000000 0f007806 2500002a ad9fafd1
>> scsi host1: ib_srp: reconnect succeeded
>> mlx5_0:dump_cqe:262:(pid 7369): dump error cqe
>> 00000000 00000000 00000000 00000000
>> 00000000 00000000 00000000 00000000
>> 00000000 00000000 00000000 00000000
>> 00000000 0f007806 25000032 00105dd0
>> scsi host1: ib_srp: failed FAST REG status memory management operation
>> error (6) for CQE ffff880b92860138
>>
>> Hence avoid using SG-GAPS memory registrations. Additionally,
>> always configure the blk_queue_virt_boundary() to avoid to trigger
>> a mapping failure when using adapters that support SG-GAPS (e.g.
>> mlx5).
>
> Hi Guys,
>
> Sorry for addressing this late, but has this failure been investigated?
>
> Max, Israel, what does this error syndrome map to?
Sagi,
this syndrome says that number of klms to write is bigger than number of
mtts.
Artemy started investigating it and proposed solution that were tested
by Laurence.
Let's see if your fix will help.
>
> Looking at mlx5_ib_sg_to_klms, I think the mr->length is incorrectly
> incremented. Does the following change fix the problem?
> --
> diff --git a/drivers/infiniband/hw/mlx5/mr.c
> b/drivers/infiniband/hw/mlx5/mr.c
> index 8f608debe141..c21c9eee37f6 100644
> --- a/drivers/infiniband/hw/mlx5/mr.c
> +++ b/drivers/infiniband/hw/mlx5/mr.c
> @@ -1832,7 +1832,7 @@ mlx5_ib_sg_to_klms(struct mlx5_ib_mr *mr,
> klms[i].va = cpu_to_be64(sg_dma_address(sg) + sg_offset);
> klms[i].bcount = cpu_to_be32(sg_dma_len(sg) - sg_offset);
> klms[i].key = cpu_to_be32(lkey);
> - mr->ibmr.length += sg_dma_len(sg);
> + mr->ibmr.length += sg_dma_len(sg) - sg_offset;
>
> sg_offset = 0;
> }
> --
^ permalink raw reply [flat|nested] 38+ messages in thread
* Re: [PATCH v2 1/8] IB/SRP: Avoid using IB_MR_TYPE_SG_GAPS
2017-02-15 15:38 ` Sagi Grimberg
@ 2017-02-15 15:42 ` Laurence Oberman
-1 siblings, 0 replies; 38+ messages in thread
From: Laurence Oberman @ 2017-02-15 15:42 UTC (permalink / raw)
To: Sagi Grimberg
Cc: Bart Van Assche, Doug Ledford, linux-rdma-u79uwXL29TY76Z2rM5mHXA,
Israel Rukshin, Max Gurtovoy, Leon Romanovsky, Mark Bloch,
Yuval Shaia, # 4 . 7+
----- Original Message -----
> From: "Sagi Grimberg" <sagi-NQWnxTmZq1alnMjI0IkVqw@public.gmane.org>
> To: "Bart Van Assche" <bart.vanassche-XdAiOPVOjttBDgjK7y7TUQ@public.gmane.org>, "Doug Ledford" <dledford-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>
> Cc: linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org, "Israel Rukshin" <israelr-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>, "Max Gurtovoy" <maxg-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>, "Leon
> Romanovsky" <leonro-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>, "Mark Bloch" <markb-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>, "Yuval Shaia" <yuval.shaia-QHcLZuEGTsvQT0dZR+AlfA@public.gmane.org>, "# 4 .
> 7+" <stable-u79uwXL29TY76Z2rM5mHXA@public.gmane.org>
> Sent: Wednesday, February 15, 2017 10:38:06 AM
> Subject: Re: [PATCH v2 1/8] IB/SRP: Avoid using IB_MR_TYPE_SG_GAPS
>
>
> > Tests have shown that the following error message is reported when
> > using SG-GAPS registration with an mlx5 adapter:
> >
> > scsi host1: ib_srp: failed RECV status WR flushed (5) for CQE
> > ffff880bd4270eb0
> > 00000000 00000000 00000000 00000000
> > 00000000 00000000 00000000 00000000
> > 00000000 00000000 00000000 00000000
> > 00000000 0f007806 2500002a ad9fafd1
> > scsi host1: ib_srp: reconnect succeeded
> > mlx5_0:dump_cqe:262:(pid 7369): dump error cqe
> > 00000000 00000000 00000000 00000000
> > 00000000 00000000 00000000 00000000
> > 00000000 00000000 00000000 00000000
> > 00000000 0f007806 25000032 00105dd0
> > scsi host1: ib_srp: failed FAST REG status memory management operation
> > error (6) for CQE ffff880b92860138
> >
> > Hence avoid using SG-GAPS memory registrations. Additionally,
> > always configure the blk_queue_virt_boundary() to avoid to trigger
> > a mapping failure when using adapters that support SG-GAPS (e.g.
> > mlx5).
>
> Hi Guys,
>
> Sorry for addressing this late, but has this failure been investigated?
>
> Max, Israel, what does this error syndrome map to?
>
> Looking at mlx5_ib_sg_to_klms, I think the mr->length is incorrectly
> incremented. Does the following change fix the problem?
> --
> diff --git a/drivers/infiniband/hw/mlx5/mr.c
> b/drivers/infiniband/hw/mlx5/mr.c
> index 8f608debe141..c21c9eee37f6 100644
> --- a/drivers/infiniband/hw/mlx5/mr.c
> +++ b/drivers/infiniband/hw/mlx5/mr.c
> @@ -1832,7 +1832,7 @@ mlx5_ib_sg_to_klms(struct mlx5_ib_mr *mr,
> klms[i].va = cpu_to_be64(sg_dma_address(sg) + sg_offset);
> klms[i].bcount = cpu_to_be32(sg_dma_len(sg) - sg_offset);
> klms[i].key = cpu_to_be32(lkey);
> - mr->ibmr.length += sg_dma_len(sg);
> + mr->ibmr.length += sg_dma_len(sg) - sg_offset;
>
> sg_offset = 0;
> }
> --
> --
> To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
>
Hello Sagi,
I will get this tested.
Thanks
Laurence
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 38+ messages in thread
* Re: [PATCH v2 1/8] IB/SRP: Avoid using IB_MR_TYPE_SG_GAPS
@ 2017-02-15 15:42 ` Laurence Oberman
0 siblings, 0 replies; 38+ messages in thread
From: Laurence Oberman @ 2017-02-15 15:42 UTC (permalink / raw)
To: Sagi Grimberg
Cc: Bart Van Assche, Doug Ledford, linux-rdma, Israel Rukshin,
Max Gurtovoy, Leon Romanovsky, Mark Bloch, Yuval Shaia, # 4 . 7+
----- Original Message -----
> From: "Sagi Grimberg" <sagi@grimberg.me>
> To: "Bart Van Assche" <bart.vanassche@sandisk.com>, "Doug Ledford" <dledford@redhat.com>
> Cc: linux-rdma@vger.kernel.org, "Israel Rukshin" <israelr@mellanox.com>, "Max Gurtovoy" <maxg@mellanox.com>, "Leon
> Romanovsky" <leonro@mellanox.com>, "Mark Bloch" <markb@mellanox.com>, "Yuval Shaia" <yuval.shaia@oracle.com>, "# 4 .
> 7+" <stable@vger.kernel.org>
> Sent: Wednesday, February 15, 2017 10:38:06 AM
> Subject: Re: [PATCH v2 1/8] IB/SRP: Avoid using IB_MR_TYPE_SG_GAPS
>
>
> > Tests have shown that the following error message is reported when
> > using SG-GAPS registration with an mlx5 adapter:
> >
> > scsi host1: ib_srp: failed RECV status WR flushed (5) for CQE
> > ffff880bd4270eb0
> > 00000000 00000000 00000000 00000000
> > 00000000 00000000 00000000 00000000
> > 00000000 00000000 00000000 00000000
> > 00000000 0f007806 2500002a ad9fafd1
> > scsi host1: ib_srp: reconnect succeeded
> > mlx5_0:dump_cqe:262:(pid 7369): dump error cqe
> > 00000000 00000000 00000000 00000000
> > 00000000 00000000 00000000 00000000
> > 00000000 00000000 00000000 00000000
> > 00000000 0f007806 25000032 00105dd0
> > scsi host1: ib_srp: failed FAST REG status memory management operation
> > error (6) for CQE ffff880b92860138
> >
> > Hence avoid using SG-GAPS memory registrations. Additionally,
> > always configure the blk_queue_virt_boundary() to avoid to trigger
> > a mapping failure when using adapters that support SG-GAPS (e.g.
> > mlx5).
>
> Hi Guys,
>
> Sorry for addressing this late, but has this failure been investigated?
>
> Max, Israel, what does this error syndrome map to?
>
> Looking at mlx5_ib_sg_to_klms, I think the mr->length is incorrectly
> incremented. Does the following change fix the problem?
> --
> diff --git a/drivers/infiniband/hw/mlx5/mr.c
> b/drivers/infiniband/hw/mlx5/mr.c
> index 8f608debe141..c21c9eee37f6 100644
> --- a/drivers/infiniband/hw/mlx5/mr.c
> +++ b/drivers/infiniband/hw/mlx5/mr.c
> @@ -1832,7 +1832,7 @@ mlx5_ib_sg_to_klms(struct mlx5_ib_mr *mr,
> klms[i].va = cpu_to_be64(sg_dma_address(sg) + sg_offset);
> klms[i].bcount = cpu_to_be32(sg_dma_len(sg) - sg_offset);
> klms[i].key = cpu_to_be32(lkey);
> - mr->ibmr.length += sg_dma_len(sg);
> + mr->ibmr.length += sg_dma_len(sg) - sg_offset;
>
> sg_offset = 0;
> }
> --
> --
> To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
>
Hello Sagi,
I will get this tested.
Thanks
Laurence
^ permalink raw reply [flat|nested] 38+ messages in thread
* Re: [PATCH v2 1/8] IB/SRP: Avoid using IB_MR_TYPE_SG_GAPS
2017-02-15 14:53 ` Laurence Oberman
[not found] ` <242820990.31706010.1487170436012.JavaMail.zimbra-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>
@ 2017-02-15 15:42 ` Sagi Grimberg
1 sibling, 0 replies; 38+ messages in thread
From: Sagi Grimberg @ 2017-02-15 15:42 UTC (permalink / raw)
To: Laurence Oberman, Leon Romanovsky
Cc: Bart Van Assche, Max Gurtovoy, Doug Ledford, linux-rdma,
Israel Rukshin, Mark Bloch, Yuval Shaia, Artemy Kovalyov,
# 4 . 7+
> Hello Leon
> I pulled your tree, ran git checkout rdma-next
> I applied the patch manually, built the kernel and started the tests.
>
> Pretty soon I ran into
>
> ibclient login: [ 132.640142] mlx5_1:dump_cqe:262:(pid 11417): dump error cqe
> [ 132.640185] scsi host2: ib_srp: failed RECV status WR flushed (5) for CQE ffff880bd86a97b0
> [ 132.718106] 00000000 00000000 00000000 00000000
> [ 132.743616] 00000000 00000000 00000000 00000000
> [ 132.767790] 00000000 00000000 00000000 00000000
> [ 132.793626] 00000000 0f007806 2500002a 8ad015d1
Not surprising given that the ib_map_mr_sg code never visits there...
^ permalink raw reply [flat|nested] 38+ messages in thread
* Re: [PATCH v2 1/8] IB/SRP: Avoid using IB_MR_TYPE_SG_GAPS
2017-02-14 18:56 ` Bart Van Assche
@ 2017-02-15 15:38 ` Sagi Grimberg
-1 siblings, 0 replies; 38+ messages in thread
From: Sagi Grimberg @ 2017-02-15 15:38 UTC (permalink / raw)
To: Bart Van Assche, Doug Ledford
Cc: linux-rdma, Israel Rukshin, Max Gurtovoy, Leon Romanovsky,
Mark Bloch, Yuval Shaia, # 4 . 7+
> Tests have shown that the following error message is reported when
> using SG-GAPS registration with an mlx5 adapter:
>
> scsi host1: ib_srp: failed RECV status WR flushed (5) for CQE ffff880bd4270eb0
> 00000000 00000000 00000000 00000000
> 00000000 00000000 00000000 00000000
> 00000000 00000000 00000000 00000000
> 00000000 0f007806 2500002a ad9fafd1
> scsi host1: ib_srp: reconnect succeeded
> mlx5_0:dump_cqe:262:(pid 7369): dump error cqe
> 00000000 00000000 00000000 00000000
> 00000000 00000000 00000000 00000000
> 00000000 00000000 00000000 00000000
> 00000000 0f007806 25000032 00105dd0
> scsi host1: ib_srp: failed FAST REG status memory management operation error (6) for CQE ffff880b92860138
>
> Hence avoid using SG-GAPS memory registrations. Additionally,
> always configure the blk_queue_virt_boundary() to avoid to trigger
> a mapping failure when using adapters that support SG-GAPS (e.g.
> mlx5).
Hi Guys,
Sorry for addressing this late, but has this failure been investigated?
Max, Israel, what does this error syndrome map to?
Looking at mlx5_ib_sg_to_klms, I think the mr->length is incorrectly
incremented. Does the following change fix the problem?
--
diff --git a/drivers/infiniband/hw/mlx5/mr.c
b/drivers/infiniband/hw/mlx5/mr.c
index 8f608debe141..c21c9eee37f6 100644
--- a/drivers/infiniband/hw/mlx5/mr.c
+++ b/drivers/infiniband/hw/mlx5/mr.c
@@ -1832,7 +1832,7 @@ mlx5_ib_sg_to_klms(struct mlx5_ib_mr *mr,
klms[i].va = cpu_to_be64(sg_dma_address(sg) + sg_offset);
klms[i].bcount = cpu_to_be32(sg_dma_len(sg) - sg_offset);
klms[i].key = cpu_to_be32(lkey);
- mr->ibmr.length += sg_dma_len(sg);
+ mr->ibmr.length += sg_dma_len(sg) - sg_offset;
sg_offset = 0;
}
^ permalink raw reply related [flat|nested] 38+ messages in thread
* Re: [PATCH v2 1/8] IB/SRP: Avoid using IB_MR_TYPE_SG_GAPS
@ 2017-02-15 15:38 ` Sagi Grimberg
0 siblings, 0 replies; 38+ messages in thread
From: Sagi Grimberg @ 2017-02-15 15:38 UTC (permalink / raw)
To: Bart Van Assche, Doug Ledford
Cc: linux-rdma, Israel Rukshin, Max Gurtovoy, Leon Romanovsky,
Mark Bloch, Yuval Shaia, # 4 . 7+
> Tests have shown that the following error message is reported when
> using SG-GAPS registration with an mlx5 adapter:
>
> scsi host1: ib_srp: failed RECV status WR flushed (5) for CQE ffff880bd4270eb0
> 00000000 00000000 00000000 00000000
> 00000000 00000000 00000000 00000000
> 00000000 00000000 00000000 00000000
> 00000000 0f007806 2500002a ad9fafd1
> scsi host1: ib_srp: reconnect succeeded
> mlx5_0:dump_cqe:262:(pid 7369): dump error cqe
> 00000000 00000000 00000000 00000000
> 00000000 00000000 00000000 00000000
> 00000000 00000000 00000000 00000000
> 00000000 0f007806 25000032 00105dd0
> scsi host1: ib_srp: failed FAST REG status memory management operation error (6) for CQE ffff880b92860138
>
> Hence avoid using SG-GAPS memory registrations. Additionally,
> always configure the blk_queue_virt_boundary() to avoid to trigger
> a mapping failure when using adapters that support SG-GAPS (e.g.
> mlx5).
Hi Guys,
Sorry for addressing this late, but has this failure been investigated?
Max, Israel, what does this error syndrome map to?
Looking at mlx5_ib_sg_to_klms, I think the mr->length is incorrectly
incremented. Does the following change fix the problem?
--
diff --git a/drivers/infiniband/hw/mlx5/mr.c
b/drivers/infiniband/hw/mlx5/mr.c
index 8f608debe141..c21c9eee37f6 100644
--- a/drivers/infiniband/hw/mlx5/mr.c
+++ b/drivers/infiniband/hw/mlx5/mr.c
@@ -1832,7 +1832,7 @@ mlx5_ib_sg_to_klms(struct mlx5_ib_mr *mr,
klms[i].va = cpu_to_be64(sg_dma_address(sg) + sg_offset);
klms[i].bcount = cpu_to_be32(sg_dma_len(sg) - sg_offset);
klms[i].key = cpu_to_be32(lkey);
- mr->ibmr.length += sg_dma_len(sg);
+ mr->ibmr.length += sg_dma_len(sg) - sg_offset;
sg_offset = 0;
}
--
^ permalink raw reply related [flat|nested] 38+ messages in thread
* Re: [PATCH v2 1/8] IB/SRP: Avoid using IB_MR_TYPE_SG_GAPS
2017-02-15 14:53 ` Laurence Oberman
@ 2017-02-15 15:18 ` Leon Romanovsky
2017-02-15 15:42 ` Sagi Grimberg
1 sibling, 0 replies; 38+ messages in thread
From: Leon Romanovsky @ 2017-02-15 15:18 UTC (permalink / raw)
To: Laurence Oberman
Cc: Bart Van Assche, Max Gurtovoy, Doug Ledford,
linux-rdma-u79uwXL29TY76Z2rM5mHXA, Israel Rukshin, Mark Bloch,
Yuval Shaia, Artemy Kovalyov, # 4 . 7+
[-- Attachment #1: Type: text/plain, Size: 12548 bytes --]
On Wed, Feb 15, 2017 at 09:53:56AM -0500, Laurence Oberman wrote:
>
>
> ----- Original Message -----
> > From: "Leon Romanovsky" <leonro-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
> > To: "Laurence Oberman" <loberman-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>
> > Cc: "Bart Van Assche" <bart.vanassche-XdAiOPVOjttBDgjK7y7TUQ@public.gmane.org>, "Max Gurtovoy" <maxg-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>, "Doug Ledford"
> > <dledford-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>, linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org, "Israel Rukshin" <israelr-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>, "Mark Bloch"
> > <markb-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>, "Yuval Shaia" <yuval.shaia-QHcLZuEGTsvQT0dZR+AlfA@public.gmane.org>, "Artemy Kovalyov" <artemyko-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>, "# 4 . 7+"
> > <stable-u79uwXL29TY76Z2rM5mHXA@public.gmane.org>
> > Sent: Wednesday, February 15, 2017 8:47:07 AM
> > Subject: Re: [PATCH v2 1/8] IB/SRP: Avoid using IB_MR_TYPE_SG_GAPS
> >
> > On Wed, Feb 15, 2017 at 08:43:38AM -0500, Laurence Oberman wrote:
> > >
> > >
> > > ----- Original Message -----
> > > > From: "Laurence Oberman" <loberman-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>
> > > > To: "Leon Romanovsky" <leonro-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
> > > > Cc: "Bart Van Assche" <bart.vanassche-XdAiOPVOjttBDgjK7y7TUQ@public.gmane.org>, "Max Gurtovoy"
> > > > <maxg-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>, "Doug Ledford"
> > > > <dledford-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>, linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org, "Israel Rukshin"
> > > > <israelr-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>, "Mark Bloch"
> > > > <markb-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>, "Yuval Shaia" <yuval.shaia-QHcLZuEGTsvQT0dZR+AlfA@public.gmane.org>, "Artemy
> > > > Kovalyov" <artemyko-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>, "# 4 . 7+"
> > > > <stable-u79uwXL29TY76Z2rM5mHXA@public.gmane.org>
> > > > Sent: Wednesday, February 15, 2017 8:33:35 AM
> > > > Subject: Re: [PATCH v2 1/8] IB/SRP: Avoid using IB_MR_TYPE_SG_GAPS
> > > >
> > > >
> > > >
> > > > ----- Original Message -----
> > > > > From: "Leon Romanovsky" <leonro-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
> > > > > To: "Bart Van Assche" <bart.vanassche-XdAiOPVOjttBDgjK7y7TUQ@public.gmane.org>, "Max Gurtovoy"
> > > > > <maxg-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
> > > > > Cc: "Doug Ledford" <dledford-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>, linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org,
> > > > > "Israel Rukshin" <israelr-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>, "Mark
> > > > > Bloch" <markb-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>, "Yuval Shaia" <yuval.shaia-QHcLZuEGTsvQT0dZR+AlfA@public.gmane.org>,
> > > > > "Artemy Kovalyov" <artemyko-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>, "# 4
> > > > > . 7+" <stable-u79uwXL29TY76Z2rM5mHXA@public.gmane.org>
> > > > > Sent: Wednesday, February 15, 2017 3:19:45 AM
> > > > > Subject: Re: [PATCH v2 1/8] IB/SRP: Avoid using IB_MR_TYPE_SG_GAPS
> > > > >
> > > > > On Wed, Feb 15, 2017 at 09:14:49AM +0200, Leon Romanovsky wrote:
> > > > > > On Tue, Feb 14, 2017 at 10:56:29AM -0800, Bart Van Assche wrote:
> > > > > > > Tests have shown that the following error message is reported when
> > > > > > > using SG-GAPS registration with an mlx5 adapter:
> > > > > > >
> > > > > > > scsi host1: ib_srp: failed RECV status WR flushed (5) for CQE
> > > > > > > ffff880bd4270eb0
> > > > > > > 00000000 00000000 00000000 00000000
> > > > > > > 00000000 00000000 00000000 00000000
> > > > > > > 00000000 00000000 00000000 00000000
> > > > > > > 00000000 0f007806 2500002a ad9fafd1
> > > > > > > scsi host1: ib_srp: reconnect succeeded
> > > > > > > mlx5_0:dump_cqe:262:(pid 7369): dump error cqe
> > > > > > > 00000000 00000000 00000000 00000000
> > > > > > > 00000000 00000000 00000000 00000000
> > > > > > > 00000000 00000000 00000000 00000000
> > > > > > > 00000000 0f007806 25000032 00105dd0
> > > > > > > scsi host1: ib_srp: failed FAST REG status memory management
> > > > > > > operation
> > > > > > > error (6) for CQE ffff880b92860138
> > > > > > >
> > > > > > > Hence avoid using SG-GAPS memory registrations. Additionally,
> > > > > > > always configure the blk_queue_virt_boundary() to avoid to trigger
> > > > > > > a mapping failure when using adapters that support SG-GAPS (e.g.
> > > > > > > mlx5).
> > > > > >
> > > > > > According to the error dump, we have an issue with max_page_list_len
> > > > > > supplied and/or
> > > > > > internal calculations from that value to the UMR byte count.
> > > > >
> > > > > Hi Bart,
> > > > >
> > > > > Do you mind to try your test on my branch rdma-next [1] with the
> > > > > following
> > > > > fixup?
> > > > >
> > > > > diff --git a/drivers/infiniband/hw/mlx5/mr.c
> > > > > b/drivers/infiniband/hw/mlx5/mr.c
> > > > > index 3c1f483d003f..3e59dce10d5e 100644
> > > > > --- a/drivers/infiniband/hw/mlx5/mr.c
> > > > > +++ b/drivers/infiniband/hw/mlx5/mr.c
> > > > > @@ -1045,8 +1045,9 @@ int mlx5_ib_update_xlt(struct mlx5_ib_mr *mr, u64
> > > > > idx,
> > > > > int npages,
> > > > > for (pages_mapped = 0;
> > > > > pages_mapped < pages_to_map && !err;
> > > > > pages_mapped += pages_iter, idx += pages_iter) {
> > > > > + npages = min_t(int, pages_iter, pages_to_map - pages_mapped);
> > > > > dma_sync_single_for_cpu(ddev, dma, size, DMA_TO_DEVICE);
> > > > > - npages = populate_xlt(mr, idx, pages_iter, xlt,
> > > > > + npages = populate_xlt(mr, idx, npages, xlt,
> > > > > page_shift, size, flags);
> > > > >
> > > > > dma_sync_single_for_device(ddev, dma, size, DMA_TO_DEVICE);
> > > > >
> > > > > [1]
> > > > > https://git.kernel.org/cgit/linux/kernel/git/leon/linux-rdma.git/log/?h=rdma-next
> > > > >
> > > > > Thanks
> > > > >
> > > >
> > > > Hello Leon
> > > > Replied earlier but I dont know if my reply made it.
> > > > I will have to test this.
> > > >
> > > > is this repo
> > > > https://git.kernel.org/cgit/linux/kernel/git/leon/linux-rdma.git/log/?h=rdma-next
> > > > already patched with the change you want.
> > > > If not can I just take the patch and apply to my earlier tree based just
> > > > on
> > > > Linus's tree where I reverted the patch.
> > > >
> > > > Thanks
> > > > Laurence
> > > > --
> > > > To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
> > > > the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> > > > More majordomo info at http://vger.kernel.org/majordomo-info.html
> > > >
> > >
> > > I guess not, I looked at applying this patch to Linus's tree but the
> > > function
> > > mlx5_ib_update_xlt() is not in there yet.
> >
> > This is why I asked to use my tree, there is a chance that wrong
> > calculation was before we introduced mlx5_ib_update_xlt() function.
> >
> > >
> > > I will see if I can get your tree staged and test this for you.
> >
> > Thanks a lot.
> >
> > >
> > > Thanks
> > > Laurence
> > > --
> > > To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
> > > the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> > > More majordomo info at http://vger.kernel.org/majordomo-info.html
> >
>
> Hello Leon
> I pulled your tree, ran git checkout rdma-next
> I applied the patch manually, built the kernel and started the tests.
>
> Pretty soon I ran into
>
> ibclient login: [ 132.640142] mlx5_1:dump_cqe:262:(pid 11417): dump error cqe
> [ 132.640185] scsi host2: ib_srp: failed RECV status WR flushed (5) for CQE ffff880bd86a97b0
> [ 132.718106] 00000000 00000000 00000000 00000000
> [ 132.743616] 00000000 00000000 00000000 00000000
> [ 132.767790] 00000000 00000000 00000000 00000000
> [ 132.793626] 00000000 0f007806 2500002a 8ad015d1
> [ 136.181512] systemd-readahead[701]: open(/var/tmp/dracut.ZR8CYG/initramfs/usr/bin/loginctl) failed: Too many levels of symbolic links
> [ 136.250779] systemd-readahead[701]: open(/var/tmp/dracut.ZR8CYG/initramfs/usr/lib/systemd/system/dracut-emergency.service) failed: Too many levels of symbolic links
> [ 147.791028] scsi host2: ib_srp: reconnect succeeded
> [ 147.827012] scsi host2: ib_srp: failed RECV status WR flushed (5) for CQE ffff880bf5e71330
> [ 162.908764] scsi host2: ib_srp: reconnect succeeded
> [ 162.944244] scsi host2: ib_srp: failed RECV status WR flushed (5) for CQE ffff880bf5e71330
> [ 166.409523] scsi host1: ib_srp: failed RECV status WR flushed (5) for CQE ffff8817f23ec570
> [ 177.292496] scsi host2: ib_srp: reconnect succeeded
> [ 177.334396] mlx5_1:dump_cqe:262:(pid 0): dump error cqe
> [ 177.363793] 00000000 00000000 00000000 00000000
> [ 177.388434] 00000000 00000000 00000000 00000000
> [ 177.413918] 00000000 00000000 00000000 00000000
> [ 177.438911] 00000000 0f007806 25000042 00102dd0
> [ 177.465048] scsi host2: ib_srp: failed FAST REG status memory management operation error (6) for CQE ffff880bd7c90138
> [ 181.386124] scsi host1: ib_srp: reconnect succeeded
> [ 181.422892] scsi host1: ib_srp: failed RECV status WR flushed (5) for CQE ffff8817f23ec5b0
> [ 192.827678] fast_io_fail_tmo expired for SRP port-2:1 / host2.
> [ 193.230036] scsi host2: ib_srp: reconnect succeeded
> [ 193.329290] mlx5_1:dump_cqe:262:(pid 0): dump error cqe
> [ 193.356245] 00000000 00000000 00000000 00000000
> [ 193.379488] 00000000 00000000 00000000 00000000
> [ 193.404853] 00000000 00000000 00000000 00000000
> [ 193.429633] 00000000 0f007806 2500004a 006c46d0
> [ 193.455183] scsi host2: ib_srp: failed FAST REG status memory management operation error (6) for CQE ffff880bd7c906f8
> [ 196.178111] scsi host1: ib_srp: reconnect succeeded
> [ 196.215197] scsi host1: ib_srp: failed RECV status WR flushed (5) for CQE ffff8817f23ec5b0
> [ 208.277396] scsi host2: ib_srp: reconnect succeeded
> [ 208.318454] mlx5_1:dump_cqe:262:(pid 0): dump error cqe
> [ 208.349012] 00000000 00000000 00000000 00000000
> [ 208.374873] 00000000 00000000 00000000 00000000
> [ 208.401070] 00000000 00000000 00000000 00000000
> [ 208.426873] 00000000 0f007806 25000052 00103dd0
> [ 208.452788] scsi host2: ib_srp: failed FAST REG status memory management operation error (6) for CQE ffff880bd7c90138
> [ 211.066945] scsi host1: ib_srp: reconnect succeeded
> [ 211.101645] scsi host1: ib_srp: failed RECV status WR flushed (5) for CQE ffff8817f23ec5b0
> [ 223.296126] scsi host2: ib_srp: reconnect succeeded
> [ 223.396340] mlx5_1:dump_cqe:262:(pid 0): dump error cqe
> [ 223.423310] 00000000 00000000 00000000 00000000
> [ 223.448590] 00000000 00000000 00000000 00000000
> [ 223.473520] 00000000 00000000 00000000 00000000
> [ 223.499008] 00000000 0f007806 2500005a 006c56d0
> [ 223.524787] scsi host2: ib_srp: failed FAST REG status memory management operation error (6) for CQE ffff880bd7c906f8
> [ 225.463312] scsi host1: ib_srp: reconnect succeeded
> [ 225.499915] scsi host1: ib_srp: failed RECV status WR flushed (5) for CQE ffff8817f23ec5b0
> [ 238.912126] fast_io_fail_tmo expired for SRP port-2:1 / host2.
> [ 239.254388] scsi host2: ib_srp: reconnect succeeded
> [ 239.291843] scsi host2: ib_srp: failed RECV status WR flushed (5) for CQE ffff880bf5e712f0
> [ 240.619914] scsi host1: ib_srp: reconnect succeeded
> [ 240.654445] scsi host1: ib_srp: failed RECV status WR flushed (5) for CQE ffff8817f23ec5b0
> [ 254.620679] scsi host2: ib_srp: reconnect succeeded
> [ 254.658021] scsi host2: ib_srp: failed RECV status WR flushed (5) for CQE ffff880bf5e712f0
> [ 255.798616] scsi host1: ib_srp: reconnect succeeded
> [ 255.832897] scsi host1: ib_srp: failed RECV status WR flushed (5) for CQE ffff8817f23ec5b0
> [ 269.228783] scsi host2: ib_srp: reconnect succeeded
> [ 269.272055] mlx5_1:dump_cqe:262:(pid 0): dump error cqe
> [ 269.300655] 00000000 00000000 00000000 00000000
> [ 269.325885] 00000000 00000000 00000000 00000000
> [ 269.350457] 00000000 00000000 00000000 00000000
> [ 269.375757] 00000000 0f007806 25000072 005847d0
> [ 269.402013] scsi host2: ib_srp: failed FAST REG status memory management operation error (6) for CQE ffff880bd7c905b8
> [ 269.856464] scsi host1: ib_srp: reconnect succeeded
> [ 269.893203] scsi host1: ib_srp: failed RECV status WR flushed (5) for CQE ffff8817f23ec5b0
>
Thanks a lot for your effort, we tried fast path to debug, but it didn't
work :(.
> Thanks
> Laurence
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]
^ permalink raw reply [flat|nested] 38+ messages in thread
* Re: [PATCH v2 1/8] IB/SRP: Avoid using IB_MR_TYPE_SG_GAPS
@ 2017-02-15 15:18 ` Leon Romanovsky
0 siblings, 0 replies; 38+ messages in thread
From: Leon Romanovsky @ 2017-02-15 15:18 UTC (permalink / raw)
To: Laurence Oberman
Cc: Bart Van Assche, Max Gurtovoy, Doug Ledford, linux-rdma,
Israel Rukshin, Mark Bloch, Yuval Shaia, Artemy Kovalyov,
# 4 . 7+
[-- Attachment #1: Type: text/plain, Size: 11611 bytes --]
On Wed, Feb 15, 2017 at 09:53:56AM -0500, Laurence Oberman wrote:
>
>
> ----- Original Message -----
> > From: "Leon Romanovsky" <leonro@mellanox.com>
> > To: "Laurence Oberman" <loberman@redhat.com>
> > Cc: "Bart Van Assche" <bart.vanassche@sandisk.com>, "Max Gurtovoy" <maxg@mellanox.com>, "Doug Ledford"
> > <dledford@redhat.com>, linux-rdma@vger.kernel.org, "Israel Rukshin" <israelr@mellanox.com>, "Mark Bloch"
> > <markb@mellanox.com>, "Yuval Shaia" <yuval.shaia@oracle.com>, "Artemy Kovalyov" <artemyko@mellanox.com>, "# 4 . 7+"
> > <stable@vger.kernel.org>
> > Sent: Wednesday, February 15, 2017 8:47:07 AM
> > Subject: Re: [PATCH v2 1/8] IB/SRP: Avoid using IB_MR_TYPE_SG_GAPS
> >
> > On Wed, Feb 15, 2017 at 08:43:38AM -0500, Laurence Oberman wrote:
> > >
> > >
> > > ----- Original Message -----
> > > > From: "Laurence Oberman" <loberman@redhat.com>
> > > > To: "Leon Romanovsky" <leonro@mellanox.com>
> > > > Cc: "Bart Van Assche" <bart.vanassche@sandisk.com>, "Max Gurtovoy"
> > > > <maxg@mellanox.com>, "Doug Ledford"
> > > > <dledford@redhat.com>, linux-rdma@vger.kernel.org, "Israel Rukshin"
> > > > <israelr@mellanox.com>, "Mark Bloch"
> > > > <markb@mellanox.com>, "Yuval Shaia" <yuval.shaia@oracle.com>, "Artemy
> > > > Kovalyov" <artemyko@mellanox.com>, "# 4 . 7+"
> > > > <stable@vger.kernel.org>
> > > > Sent: Wednesday, February 15, 2017 8:33:35 AM
> > > > Subject: Re: [PATCH v2 1/8] IB/SRP: Avoid using IB_MR_TYPE_SG_GAPS
> > > >
> > > >
> > > >
> > > > ----- Original Message -----
> > > > > From: "Leon Romanovsky" <leonro@mellanox.com>
> > > > > To: "Bart Van Assche" <bart.vanassche@sandisk.com>, "Max Gurtovoy"
> > > > > <maxg@mellanox.com>
> > > > > Cc: "Doug Ledford" <dledford@redhat.com>, linux-rdma@vger.kernel.org,
> > > > > "Israel Rukshin" <israelr@mellanox.com>, "Mark
> > > > > Bloch" <markb@mellanox.com>, "Yuval Shaia" <yuval.shaia@oracle.com>,
> > > > > "Artemy Kovalyov" <artemyko@mellanox.com>, "# 4
> > > > > . 7+" <stable@vger.kernel.org>
> > > > > Sent: Wednesday, February 15, 2017 3:19:45 AM
> > > > > Subject: Re: [PATCH v2 1/8] IB/SRP: Avoid using IB_MR_TYPE_SG_GAPS
> > > > >
> > > > > On Wed, Feb 15, 2017 at 09:14:49AM +0200, Leon Romanovsky wrote:
> > > > > > On Tue, Feb 14, 2017 at 10:56:29AM -0800, Bart Van Assche wrote:
> > > > > > > Tests have shown that the following error message is reported when
> > > > > > > using SG-GAPS registration with an mlx5 adapter:
> > > > > > >
> > > > > > > scsi host1: ib_srp: failed RECV status WR flushed (5) for CQE
> > > > > > > ffff880bd4270eb0
> > > > > > > 00000000 00000000 00000000 00000000
> > > > > > > 00000000 00000000 00000000 00000000
> > > > > > > 00000000 00000000 00000000 00000000
> > > > > > > 00000000 0f007806 2500002a ad9fafd1
> > > > > > > scsi host1: ib_srp: reconnect succeeded
> > > > > > > mlx5_0:dump_cqe:262:(pid 7369): dump error cqe
> > > > > > > 00000000 00000000 00000000 00000000
> > > > > > > 00000000 00000000 00000000 00000000
> > > > > > > 00000000 00000000 00000000 00000000
> > > > > > > 00000000 0f007806 25000032 00105dd0
> > > > > > > scsi host1: ib_srp: failed FAST REG status memory management
> > > > > > > operation
> > > > > > > error (6) for CQE ffff880b92860138
> > > > > > >
> > > > > > > Hence avoid using SG-GAPS memory registrations. Additionally,
> > > > > > > always configure the blk_queue_virt_boundary() to avoid to trigger
> > > > > > > a mapping failure when using adapters that support SG-GAPS (e.g.
> > > > > > > mlx5).
> > > > > >
> > > > > > According to the error dump, we have an issue with max_page_list_len
> > > > > > supplied and/or
> > > > > > internal calculations from that value to the UMR byte count.
> > > > >
> > > > > Hi Bart,
> > > > >
> > > > > Do you mind to try your test on my branch rdma-next [1] with the
> > > > > following
> > > > > fixup?
> > > > >
> > > > > diff --git a/drivers/infiniband/hw/mlx5/mr.c
> > > > > b/drivers/infiniband/hw/mlx5/mr.c
> > > > > index 3c1f483d003f..3e59dce10d5e 100644
> > > > > --- a/drivers/infiniband/hw/mlx5/mr.c
> > > > > +++ b/drivers/infiniband/hw/mlx5/mr.c
> > > > > @@ -1045,8 +1045,9 @@ int mlx5_ib_update_xlt(struct mlx5_ib_mr *mr, u64
> > > > > idx,
> > > > > int npages,
> > > > > for (pages_mapped = 0;
> > > > > pages_mapped < pages_to_map && !err;
> > > > > pages_mapped += pages_iter, idx += pages_iter) {
> > > > > + npages = min_t(int, pages_iter, pages_to_map - pages_mapped);
> > > > > dma_sync_single_for_cpu(ddev, dma, size, DMA_TO_DEVICE);
> > > > > - npages = populate_xlt(mr, idx, pages_iter, xlt,
> > > > > + npages = populate_xlt(mr, idx, npages, xlt,
> > > > > page_shift, size, flags);
> > > > >
> > > > > dma_sync_single_for_device(ddev, dma, size, DMA_TO_DEVICE);
> > > > >
> > > > > [1]
> > > > > https://git.kernel.org/cgit/linux/kernel/git/leon/linux-rdma.git/log/?h=rdma-next
> > > > >
> > > > > Thanks
> > > > >
> > > >
> > > > Hello Leon
> > > > Replied earlier but I dont know if my reply made it.
> > > > I will have to test this.
> > > >
> > > > is this repo
> > > > https://git.kernel.org/cgit/linux/kernel/git/leon/linux-rdma.git/log/?h=rdma-next
> > > > already patched with the change you want.
> > > > If not can I just take the patch and apply to my earlier tree based just
> > > > on
> > > > Linus's tree where I reverted the patch.
> > > >
> > > > Thanks
> > > > Laurence
> > > > --
> > > > To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
> > > > the body of a message to majordomo@vger.kernel.org
> > > > More majordomo info at http://vger.kernel.org/majordomo-info.html
> > > >
> > >
> > > I guess not, I looked at applying this patch to Linus's tree but the
> > > function
> > > mlx5_ib_update_xlt() is not in there yet.
> >
> > This is why I asked to use my tree, there is a chance that wrong
> > calculation was before we introduced mlx5_ib_update_xlt() function.
> >
> > >
> > > I will see if I can get your tree staged and test this for you.
> >
> > Thanks a lot.
> >
> > >
> > > Thanks
> > > Laurence
> > > --
> > > To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
> > > the body of a message to majordomo@vger.kernel.org
> > > More majordomo info at http://vger.kernel.org/majordomo-info.html
> >
>
> Hello Leon
> I pulled your tree, ran git checkout rdma-next
> I applied the patch manually, built the kernel and started the tests.
>
> Pretty soon I ran into
>
> ibclient login: [ 132.640142] mlx5_1:dump_cqe:262:(pid 11417): dump error cqe
> [ 132.640185] scsi host2: ib_srp: failed RECV status WR flushed (5) for CQE ffff880bd86a97b0
> [ 132.718106] 00000000 00000000 00000000 00000000
> [ 132.743616] 00000000 00000000 00000000 00000000
> [ 132.767790] 00000000 00000000 00000000 00000000
> [ 132.793626] 00000000 0f007806 2500002a 8ad015d1
> [ 136.181512] systemd-readahead[701]: open(/var/tmp/dracut.ZR8CYG/initramfs/usr/bin/loginctl) failed: Too many levels of symbolic links
> [ 136.250779] systemd-readahead[701]: open(/var/tmp/dracut.ZR8CYG/initramfs/usr/lib/systemd/system/dracut-emergency.service) failed: Too many levels of symbolic links
> [ 147.791028] scsi host2: ib_srp: reconnect succeeded
> [ 147.827012] scsi host2: ib_srp: failed RECV status WR flushed (5) for CQE ffff880bf5e71330
> [ 162.908764] scsi host2: ib_srp: reconnect succeeded
> [ 162.944244] scsi host2: ib_srp: failed RECV status WR flushed (5) for CQE ffff880bf5e71330
> [ 166.409523] scsi host1: ib_srp: failed RECV status WR flushed (5) for CQE ffff8817f23ec570
> [ 177.292496] scsi host2: ib_srp: reconnect succeeded
> [ 177.334396] mlx5_1:dump_cqe:262:(pid 0): dump error cqe
> [ 177.363793] 00000000 00000000 00000000 00000000
> [ 177.388434] 00000000 00000000 00000000 00000000
> [ 177.413918] 00000000 00000000 00000000 00000000
> [ 177.438911] 00000000 0f007806 25000042 00102dd0
> [ 177.465048] scsi host2: ib_srp: failed FAST REG status memory management operation error (6) for CQE ffff880bd7c90138
> [ 181.386124] scsi host1: ib_srp: reconnect succeeded
> [ 181.422892] scsi host1: ib_srp: failed RECV status WR flushed (5) for CQE ffff8817f23ec5b0
> [ 192.827678] fast_io_fail_tmo expired for SRP port-2:1 / host2.
> [ 193.230036] scsi host2: ib_srp: reconnect succeeded
> [ 193.329290] mlx5_1:dump_cqe:262:(pid 0): dump error cqe
> [ 193.356245] 00000000 00000000 00000000 00000000
> [ 193.379488] 00000000 00000000 00000000 00000000
> [ 193.404853] 00000000 00000000 00000000 00000000
> [ 193.429633] 00000000 0f007806 2500004a 006c46d0
> [ 193.455183] scsi host2: ib_srp: failed FAST REG status memory management operation error (6) for CQE ffff880bd7c906f8
> [ 196.178111] scsi host1: ib_srp: reconnect succeeded
> [ 196.215197] scsi host1: ib_srp: failed RECV status WR flushed (5) for CQE ffff8817f23ec5b0
> [ 208.277396] scsi host2: ib_srp: reconnect succeeded
> [ 208.318454] mlx5_1:dump_cqe:262:(pid 0): dump error cqe
> [ 208.349012] 00000000 00000000 00000000 00000000
> [ 208.374873] 00000000 00000000 00000000 00000000
> [ 208.401070] 00000000 00000000 00000000 00000000
> [ 208.426873] 00000000 0f007806 25000052 00103dd0
> [ 208.452788] scsi host2: ib_srp: failed FAST REG status memory management operation error (6) for CQE ffff880bd7c90138
> [ 211.066945] scsi host1: ib_srp: reconnect succeeded
> [ 211.101645] scsi host1: ib_srp: failed RECV status WR flushed (5) for CQE ffff8817f23ec5b0
> [ 223.296126] scsi host2: ib_srp: reconnect succeeded
> [ 223.396340] mlx5_1:dump_cqe:262:(pid 0): dump error cqe
> [ 223.423310] 00000000 00000000 00000000 00000000
> [ 223.448590] 00000000 00000000 00000000 00000000
> [ 223.473520] 00000000 00000000 00000000 00000000
> [ 223.499008] 00000000 0f007806 2500005a 006c56d0
> [ 223.524787] scsi host2: ib_srp: failed FAST REG status memory management operation error (6) for CQE ffff880bd7c906f8
> [ 225.463312] scsi host1: ib_srp: reconnect succeeded
> [ 225.499915] scsi host1: ib_srp: failed RECV status WR flushed (5) for CQE ffff8817f23ec5b0
> [ 238.912126] fast_io_fail_tmo expired for SRP port-2:1 / host2.
> [ 239.254388] scsi host2: ib_srp: reconnect succeeded
> [ 239.291843] scsi host2: ib_srp: failed RECV status WR flushed (5) for CQE ffff880bf5e712f0
> [ 240.619914] scsi host1: ib_srp: reconnect succeeded
> [ 240.654445] scsi host1: ib_srp: failed RECV status WR flushed (5) for CQE ffff8817f23ec5b0
> [ 254.620679] scsi host2: ib_srp: reconnect succeeded
> [ 254.658021] scsi host2: ib_srp: failed RECV status WR flushed (5) for CQE ffff880bf5e712f0
> [ 255.798616] scsi host1: ib_srp: reconnect succeeded
> [ 255.832897] scsi host1: ib_srp: failed RECV status WR flushed (5) for CQE ffff8817f23ec5b0
> [ 269.228783] scsi host2: ib_srp: reconnect succeeded
> [ 269.272055] mlx5_1:dump_cqe:262:(pid 0): dump error cqe
> [ 269.300655] 00000000 00000000 00000000 00000000
> [ 269.325885] 00000000 00000000 00000000 00000000
> [ 269.350457] 00000000 00000000 00000000 00000000
> [ 269.375757] 00000000 0f007806 25000072 005847d0
> [ 269.402013] scsi host2: ib_srp: failed FAST REG status memory management operation error (6) for CQE ffff880bd7c905b8
> [ 269.856464] scsi host1: ib_srp: reconnect succeeded
> [ 269.893203] scsi host1: ib_srp: failed RECV status WR flushed (5) for CQE ffff8817f23ec5b0
>
Thanks a lot for your effort, we tried fast path to debug, but it didn't
work :(.
> Thanks
> Laurence
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]
^ permalink raw reply [flat|nested] 38+ messages in thread
* Re: [PATCH v2 1/8] IB/SRP: Avoid using IB_MR_TYPE_SG_GAPS
2017-02-15 13:47 ` Leon Romanovsky
(?)
@ 2017-02-15 14:53 ` Laurence Oberman
[not found] ` <242820990.31706010.1487170436012.JavaMail.zimbra-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>
2017-02-15 15:42 ` Sagi Grimberg
-1 siblings, 2 replies; 38+ messages in thread
From: Laurence Oberman @ 2017-02-15 14:53 UTC (permalink / raw)
To: Leon Romanovsky
Cc: Bart Van Assche, Max Gurtovoy, Doug Ledford, linux-rdma,
Israel Rukshin, Mark Bloch, Yuval Shaia, Artemy Kovalyov,
# 4 . 7+
----- Original Message -----
> From: "Leon Romanovsky" <leonro@mellanox.com>
> To: "Laurence Oberman" <loberman@redhat.com>
> Cc: "Bart Van Assche" <bart.vanassche@sandisk.com>, "Max Gurtovoy" <maxg@mellanox.com>, "Doug Ledford"
> <dledford@redhat.com>, linux-rdma@vger.kernel.org, "Israel Rukshin" <israelr@mellanox.com>, "Mark Bloch"
> <markb@mellanox.com>, "Yuval Shaia" <yuval.shaia@oracle.com>, "Artemy Kovalyov" <artemyko@mellanox.com>, "# 4 . 7+"
> <stable@vger.kernel.org>
> Sent: Wednesday, February 15, 2017 8:47:07 AM
> Subject: Re: [PATCH v2 1/8] IB/SRP: Avoid using IB_MR_TYPE_SG_GAPS
>
> On Wed, Feb 15, 2017 at 08:43:38AM -0500, Laurence Oberman wrote:
> >
> >
> > ----- Original Message -----
> > > From: "Laurence Oberman" <loberman@redhat.com>
> > > To: "Leon Romanovsky" <leonro@mellanox.com>
> > > Cc: "Bart Van Assche" <bart.vanassche@sandisk.com>, "Max Gurtovoy"
> > > <maxg@mellanox.com>, "Doug Ledford"
> > > <dledford@redhat.com>, linux-rdma@vger.kernel.org, "Israel Rukshin"
> > > <israelr@mellanox.com>, "Mark Bloch"
> > > <markb@mellanox.com>, "Yuval Shaia" <yuval.shaia@oracle.com>, "Artemy
> > > Kovalyov" <artemyko@mellanox.com>, "# 4 . 7+"
> > > <stable@vger.kernel.org>
> > > Sent: Wednesday, February 15, 2017 8:33:35 AM
> > > Subject: Re: [PATCH v2 1/8] IB/SRP: Avoid using IB_MR_TYPE_SG_GAPS
> > >
> > >
> > >
> > > ----- Original Message -----
> > > > From: "Leon Romanovsky" <leonro@mellanox.com>
> > > > To: "Bart Van Assche" <bart.vanassche@sandisk.com>, "Max Gurtovoy"
> > > > <maxg@mellanox.com>
> > > > Cc: "Doug Ledford" <dledford@redhat.com>, linux-rdma@vger.kernel.org,
> > > > "Israel Rukshin" <israelr@mellanox.com>, "Mark
> > > > Bloch" <markb@mellanox.com>, "Yuval Shaia" <yuval.shaia@oracle.com>,
> > > > "Artemy Kovalyov" <artemyko@mellanox.com>, "# 4
> > > > . 7+" <stable@vger.kernel.org>
> > > > Sent: Wednesday, February 15, 2017 3:19:45 AM
> > > > Subject: Re: [PATCH v2 1/8] IB/SRP: Avoid using IB_MR_TYPE_SG_GAPS
> > > >
> > > > On Wed, Feb 15, 2017 at 09:14:49AM +0200, Leon Romanovsky wrote:
> > > > > On Tue, Feb 14, 2017 at 10:56:29AM -0800, Bart Van Assche wrote:
> > > > > > Tests have shown that the following error message is reported when
> > > > > > using SG-GAPS registration with an mlx5 adapter:
> > > > > >
> > > > > > scsi host1: ib_srp: failed RECV status WR flushed (5) for CQE
> > > > > > ffff880bd4270eb0
> > > > > > 00000000 00000000 00000000 00000000
> > > > > > 00000000 00000000 00000000 00000000
> > > > > > 00000000 00000000 00000000 00000000
> > > > > > 00000000 0f007806 2500002a ad9fafd1
> > > > > > scsi host1: ib_srp: reconnect succeeded
> > > > > > mlx5_0:dump_cqe:262:(pid 7369): dump error cqe
> > > > > > 00000000 00000000 00000000 00000000
> > > > > > 00000000 00000000 00000000 00000000
> > > > > > 00000000 00000000 00000000 00000000
> > > > > > 00000000 0f007806 25000032 00105dd0
> > > > > > scsi host1: ib_srp: failed FAST REG status memory management
> > > > > > operation
> > > > > > error (6) for CQE ffff880b92860138
> > > > > >
> > > > > > Hence avoid using SG-GAPS memory registrations. Additionally,
> > > > > > always configure the blk_queue_virt_boundary() to avoid to trigger
> > > > > > a mapping failure when using adapters that support SG-GAPS (e.g.
> > > > > > mlx5).
> > > > >
> > > > > According to the error dump, we have an issue with max_page_list_len
> > > > > supplied and/or
> > > > > internal calculations from that value to the UMR byte count.
> > > >
> > > > Hi Bart,
> > > >
> > > > Do you mind to try your test on my branch rdma-next [1] with the
> > > > following
> > > > fixup?
> > > >
> > > > diff --git a/drivers/infiniband/hw/mlx5/mr.c
> > > > b/drivers/infiniband/hw/mlx5/mr.c
> > > > index 3c1f483d003f..3e59dce10d5e 100644
> > > > --- a/drivers/infiniband/hw/mlx5/mr.c
> > > > +++ b/drivers/infiniband/hw/mlx5/mr.c
> > > > @@ -1045,8 +1045,9 @@ int mlx5_ib_update_xlt(struct mlx5_ib_mr *mr, u64
> > > > idx,
> > > > int npages,
> > > > for (pages_mapped = 0;
> > > > pages_mapped < pages_to_map && !err;
> > > > pages_mapped += pages_iter, idx += pages_iter) {
> > > > + npages = min_t(int, pages_iter, pages_to_map - pages_mapped);
> > > > dma_sync_single_for_cpu(ddev, dma, size, DMA_TO_DEVICE);
> > > > - npages = populate_xlt(mr, idx, pages_iter, xlt,
> > > > + npages = populate_xlt(mr, idx, npages, xlt,
> > > > page_shift, size, flags);
> > > >
> > > > dma_sync_single_for_device(ddev, dma, size, DMA_TO_DEVICE);
> > > >
> > > > [1]
> > > > https://git.kernel.org/cgit/linux/kernel/git/leon/linux-rdma.git/log/?h=rdma-next
> > > >
> > > > Thanks
> > > >
> > >
> > > Hello Leon
> > > Replied earlier but I dont know if my reply made it.
> > > I will have to test this.
> > >
> > > is this repo
> > > https://git.kernel.org/cgit/linux/kernel/git/leon/linux-rdma.git/log/?h=rdma-next
> > > already patched with the change you want.
> > > If not can I just take the patch and apply to my earlier tree based just
> > > on
> > > Linus's tree where I reverted the patch.
> > >
> > > Thanks
> > > Laurence
> > > --
> > > To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
> > > the body of a message to majordomo@vger.kernel.org
> > > More majordomo info at http://vger.kernel.org/majordomo-info.html
> > >
> >
> > I guess not, I looked at applying this patch to Linus's tree but the
> > function
> > mlx5_ib_update_xlt() is not in there yet.
>
> This is why I asked to use my tree, there is a chance that wrong
> calculation was before we introduced mlx5_ib_update_xlt() function.
>
> >
> > I will see if I can get your tree staged and test this for you.
>
> Thanks a lot.
>
> >
> > Thanks
> > Laurence
> > --
> > To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
> > the body of a message to majordomo@vger.kernel.org
> > More majordomo info at http://vger.kernel.org/majordomo-info.html
>
Hello Leon
I pulled your tree, ran git checkout rdma-next
I applied the patch manually, built the kernel and started the tests.
Pretty soon I ran into
ibclient login: [ 132.640142] mlx5_1:dump_cqe:262:(pid 11417): dump error cqe
[ 132.640185] scsi host2: ib_srp: failed RECV status WR flushed (5) for CQE ffff880bd86a97b0
[ 132.718106] 00000000 00000000 00000000 00000000
[ 132.743616] 00000000 00000000 00000000 00000000
[ 132.767790] 00000000 00000000 00000000 00000000
[ 132.793626] 00000000 0f007806 2500002a 8ad015d1
[ 136.181512] systemd-readahead[701]: open(/var/tmp/dracut.ZR8CYG/initramfs/usr/bin/loginctl) failed: Too many levels of symbolic links
[ 136.250779] systemd-readahead[701]: open(/var/tmp/dracut.ZR8CYG/initramfs/usr/lib/systemd/system/dracut-emergency.service) failed: Too many levels of symbolic links
[ 147.791028] scsi host2: ib_srp: reconnect succeeded
[ 147.827012] scsi host2: ib_srp: failed RECV status WR flushed (5) for CQE ffff880bf5e71330
[ 162.908764] scsi host2: ib_srp: reconnect succeeded
[ 162.944244] scsi host2: ib_srp: failed RECV status WR flushed (5) for CQE ffff880bf5e71330
[ 166.409523] scsi host1: ib_srp: failed RECV status WR flushed (5) for CQE ffff8817f23ec570
[ 177.292496] scsi host2: ib_srp: reconnect succeeded
[ 177.334396] mlx5_1:dump_cqe:262:(pid 0): dump error cqe
[ 177.363793] 00000000 00000000 00000000 00000000
[ 177.388434] 00000000 00000000 00000000 00000000
[ 177.413918] 00000000 00000000 00000000 00000000
[ 177.438911] 00000000 0f007806 25000042 00102dd0
[ 177.465048] scsi host2: ib_srp: failed FAST REG status memory management operation error (6) for CQE ffff880bd7c90138
[ 181.386124] scsi host1: ib_srp: reconnect succeeded
[ 181.422892] scsi host1: ib_srp: failed RECV status WR flushed (5) for CQE ffff8817f23ec5b0
[ 192.827678] fast_io_fail_tmo expired for SRP port-2:1 / host2.
[ 193.230036] scsi host2: ib_srp: reconnect succeeded
[ 193.329290] mlx5_1:dump_cqe:262:(pid 0): dump error cqe
[ 193.356245] 00000000 00000000 00000000 00000000
[ 193.379488] 00000000 00000000 00000000 00000000
[ 193.404853] 00000000 00000000 00000000 00000000
[ 193.429633] 00000000 0f007806 2500004a 006c46d0
[ 193.455183] scsi host2: ib_srp: failed FAST REG status memory management operation error (6) for CQE ffff880bd7c906f8
[ 196.178111] scsi host1: ib_srp: reconnect succeeded
[ 196.215197] scsi host1: ib_srp: failed RECV status WR flushed (5) for CQE ffff8817f23ec5b0
[ 208.277396] scsi host2: ib_srp: reconnect succeeded
[ 208.318454] mlx5_1:dump_cqe:262:(pid 0): dump error cqe
[ 208.349012] 00000000 00000000 00000000 00000000
[ 208.374873] 00000000 00000000 00000000 00000000
[ 208.401070] 00000000 00000000 00000000 00000000
[ 208.426873] 00000000 0f007806 25000052 00103dd0
[ 208.452788] scsi host2: ib_srp: failed FAST REG status memory management operation error (6) for CQE ffff880bd7c90138
[ 211.066945] scsi host1: ib_srp: reconnect succeeded
[ 211.101645] scsi host1: ib_srp: failed RECV status WR flushed (5) for CQE ffff8817f23ec5b0
[ 223.296126] scsi host2: ib_srp: reconnect succeeded
[ 223.396340] mlx5_1:dump_cqe:262:(pid 0): dump error cqe
[ 223.423310] 00000000 00000000 00000000 00000000
[ 223.448590] 00000000 00000000 00000000 00000000
[ 223.473520] 00000000 00000000 00000000 00000000
[ 223.499008] 00000000 0f007806 2500005a 006c56d0
[ 223.524787] scsi host2: ib_srp: failed FAST REG status memory management operation error (6) for CQE ffff880bd7c906f8
[ 225.463312] scsi host1: ib_srp: reconnect succeeded
[ 225.499915] scsi host1: ib_srp: failed RECV status WR flushed (5) for CQE ffff8817f23ec5b0
[ 238.912126] fast_io_fail_tmo expired for SRP port-2:1 / host2.
[ 239.254388] scsi host2: ib_srp: reconnect succeeded
[ 239.291843] scsi host2: ib_srp: failed RECV status WR flushed (5) for CQE ffff880bf5e712f0
[ 240.619914] scsi host1: ib_srp: reconnect succeeded
[ 240.654445] scsi host1: ib_srp: failed RECV status WR flushed (5) for CQE ffff8817f23ec5b0
[ 254.620679] scsi host2: ib_srp: reconnect succeeded
[ 254.658021] scsi host2: ib_srp: failed RECV status WR flushed (5) for CQE ffff880bf5e712f0
[ 255.798616] scsi host1: ib_srp: reconnect succeeded
[ 255.832897] scsi host1: ib_srp: failed RECV status WR flushed (5) for CQE ffff8817f23ec5b0
[ 269.228783] scsi host2: ib_srp: reconnect succeeded
[ 269.272055] mlx5_1:dump_cqe:262:(pid 0): dump error cqe
[ 269.300655] 00000000 00000000 00000000 00000000
[ 269.325885] 00000000 00000000 00000000 00000000
[ 269.350457] 00000000 00000000 00000000 00000000
[ 269.375757] 00000000 0f007806 25000072 005847d0
[ 269.402013] scsi host2: ib_srp: failed FAST REG status memory management operation error (6) for CQE ffff880bd7c905b8
[ 269.856464] scsi host1: ib_srp: reconnect succeeded
[ 269.893203] scsi host1: ib_srp: failed RECV status WR flushed (5) for CQE ffff8817f23ec5b0
Thanks
Laurence
^ permalink raw reply [flat|nested] 38+ messages in thread
* Re: [PATCH v2 1/8] IB/SRP: Avoid using IB_MR_TYPE_SG_GAPS
2017-02-15 13:43 ` Laurence Oberman
@ 2017-02-15 13:47 ` Leon Romanovsky
0 siblings, 0 replies; 38+ messages in thread
From: Leon Romanovsky @ 2017-02-15 13:47 UTC (permalink / raw)
To: Laurence Oberman
Cc: Bart Van Assche, Max Gurtovoy, Doug Ledford, linux-rdma,
Israel Rukshin, Mark Bloch, Yuval Shaia, Artemy Kovalyov,
# 4 . 7+
[-- Attachment #1: Type: text/plain, Size: 5018 bytes --]
On Wed, Feb 15, 2017 at 08:43:38AM -0500, Laurence Oberman wrote:
>
>
> ----- Original Message -----
> > From: "Laurence Oberman" <loberman@redhat.com>
> > To: "Leon Romanovsky" <leonro@mellanox.com>
> > Cc: "Bart Van Assche" <bart.vanassche@sandisk.com>, "Max Gurtovoy" <maxg@mellanox.com>, "Doug Ledford"
> > <dledford@redhat.com>, linux-rdma@vger.kernel.org, "Israel Rukshin" <israelr@mellanox.com>, "Mark Bloch"
> > <markb@mellanox.com>, "Yuval Shaia" <yuval.shaia@oracle.com>, "Artemy Kovalyov" <artemyko@mellanox.com>, "# 4 . 7+"
> > <stable@vger.kernel.org>
> > Sent: Wednesday, February 15, 2017 8:33:35 AM
> > Subject: Re: [PATCH v2 1/8] IB/SRP: Avoid using IB_MR_TYPE_SG_GAPS
> >
> >
> >
> > ----- Original Message -----
> > > From: "Leon Romanovsky" <leonro@mellanox.com>
> > > To: "Bart Van Assche" <bart.vanassche@sandisk.com>, "Max Gurtovoy"
> > > <maxg@mellanox.com>
> > > Cc: "Doug Ledford" <dledford@redhat.com>, linux-rdma@vger.kernel.org,
> > > "Israel Rukshin" <israelr@mellanox.com>, "Mark
> > > Bloch" <markb@mellanox.com>, "Yuval Shaia" <yuval.shaia@oracle.com>,
> > > "Artemy Kovalyov" <artemyko@mellanox.com>, "# 4
> > > . 7+" <stable@vger.kernel.org>
> > > Sent: Wednesday, February 15, 2017 3:19:45 AM
> > > Subject: Re: [PATCH v2 1/8] IB/SRP: Avoid using IB_MR_TYPE_SG_GAPS
> > >
> > > On Wed, Feb 15, 2017 at 09:14:49AM +0200, Leon Romanovsky wrote:
> > > > On Tue, Feb 14, 2017 at 10:56:29AM -0800, Bart Van Assche wrote:
> > > > > Tests have shown that the following error message is reported when
> > > > > using SG-GAPS registration with an mlx5 adapter:
> > > > >
> > > > > scsi host1: ib_srp: failed RECV status WR flushed (5) for CQE
> > > > > ffff880bd4270eb0
> > > > > 00000000 00000000 00000000 00000000
> > > > > 00000000 00000000 00000000 00000000
> > > > > 00000000 00000000 00000000 00000000
> > > > > 00000000 0f007806 2500002a ad9fafd1
> > > > > scsi host1: ib_srp: reconnect succeeded
> > > > > mlx5_0:dump_cqe:262:(pid 7369): dump error cqe
> > > > > 00000000 00000000 00000000 00000000
> > > > > 00000000 00000000 00000000 00000000
> > > > > 00000000 00000000 00000000 00000000
> > > > > 00000000 0f007806 25000032 00105dd0
> > > > > scsi host1: ib_srp: failed FAST REG status memory management operation
> > > > > error (6) for CQE ffff880b92860138
> > > > >
> > > > > Hence avoid using SG-GAPS memory registrations. Additionally,
> > > > > always configure the blk_queue_virt_boundary() to avoid to trigger
> > > > > a mapping failure when using adapters that support SG-GAPS (e.g.
> > > > > mlx5).
> > > >
> > > > According to the error dump, we have an issue with max_page_list_len
> > > > supplied and/or
> > > > internal calculations from that value to the UMR byte count.
> > >
> > > Hi Bart,
> > >
> > > Do you mind to try your test on my branch rdma-next [1] with the following
> > > fixup?
> > >
> > > diff --git a/drivers/infiniband/hw/mlx5/mr.c
> > > b/drivers/infiniband/hw/mlx5/mr.c
> > > index 3c1f483d003f..3e59dce10d5e 100644
> > > --- a/drivers/infiniband/hw/mlx5/mr.c
> > > +++ b/drivers/infiniband/hw/mlx5/mr.c
> > > @@ -1045,8 +1045,9 @@ int mlx5_ib_update_xlt(struct mlx5_ib_mr *mr, u64
> > > idx,
> > > int npages,
> > > for (pages_mapped = 0;
> > > pages_mapped < pages_to_map && !err;
> > > pages_mapped += pages_iter, idx += pages_iter) {
> > > + npages = min_t(int, pages_iter, pages_to_map - pages_mapped);
> > > dma_sync_single_for_cpu(ddev, dma, size, DMA_TO_DEVICE);
> > > - npages = populate_xlt(mr, idx, pages_iter, xlt,
> > > + npages = populate_xlt(mr, idx, npages, xlt,
> > > page_shift, size, flags);
> > >
> > > dma_sync_single_for_device(ddev, dma, size, DMA_TO_DEVICE);
> > >
> > > [1]
> > > https://git.kernel.org/cgit/linux/kernel/git/leon/linux-rdma.git/log/?h=rdma-next
> > >
> > > Thanks
> > >
> >
> > Hello Leon
> > Replied earlier but I dont know if my reply made it.
> > I will have to test this.
> >
> > is this repo
> > https://git.kernel.org/cgit/linux/kernel/git/leon/linux-rdma.git/log/?h=rdma-next
> > already patched with the change you want.
> > If not can I just take the patch and apply to my earlier tree based just on
> > Linus's tree where I reverted the patch.
> >
> > Thanks
> > Laurence
> > --
> > To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
> > the body of a message to majordomo@vger.kernel.org
> > More majordomo info at http://vger.kernel.org/majordomo-info.html
> >
>
> I guess not, I looked at applying this patch to Linus's tree but the function
> mlx5_ib_update_xlt() is not in there yet.
This is why I asked to use my tree, there is a chance that wrong
calculation was before we introduced mlx5_ib_update_xlt() function.
>
> I will see if I can get your tree staged and test this for you.
Thanks a lot.
>
> Thanks
> Laurence
> --
> To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]
^ permalink raw reply [flat|nested] 38+ messages in thread
* Re: [PATCH v2 1/8] IB/SRP: Avoid using IB_MR_TYPE_SG_GAPS
@ 2017-02-15 13:47 ` Leon Romanovsky
0 siblings, 0 replies; 38+ messages in thread
From: Leon Romanovsky @ 2017-02-15 13:47 UTC (permalink / raw)
To: Laurence Oberman
Cc: Bart Van Assche, Max Gurtovoy, Doug Ledford, linux-rdma,
Israel Rukshin, Mark Bloch, Yuval Shaia, Artemy Kovalyov,
# 4 . 7+
[-- Attachment #1: Type: text/plain, Size: 5018 bytes --]
On Wed, Feb 15, 2017 at 08:43:38AM -0500, Laurence Oberman wrote:
>
>
> ----- Original Message -----
> > From: "Laurence Oberman" <loberman@redhat.com>
> > To: "Leon Romanovsky" <leonro@mellanox.com>
> > Cc: "Bart Van Assche" <bart.vanassche@sandisk.com>, "Max Gurtovoy" <maxg@mellanox.com>, "Doug Ledford"
> > <dledford@redhat.com>, linux-rdma@vger.kernel.org, "Israel Rukshin" <israelr@mellanox.com>, "Mark Bloch"
> > <markb@mellanox.com>, "Yuval Shaia" <yuval.shaia@oracle.com>, "Artemy Kovalyov" <artemyko@mellanox.com>, "# 4 . 7+"
> > <stable@vger.kernel.org>
> > Sent: Wednesday, February 15, 2017 8:33:35 AM
> > Subject: Re: [PATCH v2 1/8] IB/SRP: Avoid using IB_MR_TYPE_SG_GAPS
> >
> >
> >
> > ----- Original Message -----
> > > From: "Leon Romanovsky" <leonro@mellanox.com>
> > > To: "Bart Van Assche" <bart.vanassche@sandisk.com>, "Max Gurtovoy"
> > > <maxg@mellanox.com>
> > > Cc: "Doug Ledford" <dledford@redhat.com>, linux-rdma@vger.kernel.org,
> > > "Israel Rukshin" <israelr@mellanox.com>, "Mark
> > > Bloch" <markb@mellanox.com>, "Yuval Shaia" <yuval.shaia@oracle.com>,
> > > "Artemy Kovalyov" <artemyko@mellanox.com>, "# 4
> > > . 7+" <stable@vger.kernel.org>
> > > Sent: Wednesday, February 15, 2017 3:19:45 AM
> > > Subject: Re: [PATCH v2 1/8] IB/SRP: Avoid using IB_MR_TYPE_SG_GAPS
> > >
> > > On Wed, Feb 15, 2017 at 09:14:49AM +0200, Leon Romanovsky wrote:
> > > > On Tue, Feb 14, 2017 at 10:56:29AM -0800, Bart Van Assche wrote:
> > > > > Tests have shown that the following error message is reported when
> > > > > using SG-GAPS registration with an mlx5 adapter:
> > > > >
> > > > > scsi host1: ib_srp: failed RECV status WR flushed (5) for CQE
> > > > > ffff880bd4270eb0
> > > > > 00000000 00000000 00000000 00000000
> > > > > 00000000 00000000 00000000 00000000
> > > > > 00000000 00000000 00000000 00000000
> > > > > 00000000 0f007806 2500002a ad9fafd1
> > > > > scsi host1: ib_srp: reconnect succeeded
> > > > > mlx5_0:dump_cqe:262:(pid 7369): dump error cqe
> > > > > 00000000 00000000 00000000 00000000
> > > > > 00000000 00000000 00000000 00000000
> > > > > 00000000 00000000 00000000 00000000
> > > > > 00000000 0f007806 25000032 00105dd0
> > > > > scsi host1: ib_srp: failed FAST REG status memory management operation
> > > > > error (6) for CQE ffff880b92860138
> > > > >
> > > > > Hence avoid using SG-GAPS memory registrations. Additionally,
> > > > > always configure the blk_queue_virt_boundary() to avoid to trigger
> > > > > a mapping failure when using adapters that support SG-GAPS (e.g.
> > > > > mlx5).
> > > >
> > > > According to the error dump, we have an issue with max_page_list_len
> > > > supplied and/or
> > > > internal calculations from that value to the UMR byte count.
> > >
> > > Hi Bart,
> > >
> > > Do you mind to try your test on my branch rdma-next [1] with the following
> > > fixup?
> > >
> > > diff --git a/drivers/infiniband/hw/mlx5/mr.c
> > > b/drivers/infiniband/hw/mlx5/mr.c
> > > index 3c1f483d003f..3e59dce10d5e 100644
> > > --- a/drivers/infiniband/hw/mlx5/mr.c
> > > +++ b/drivers/infiniband/hw/mlx5/mr.c
> > > @@ -1045,8 +1045,9 @@ int mlx5_ib_update_xlt(struct mlx5_ib_mr *mr, u64
> > > idx,
> > > int npages,
> > > for (pages_mapped = 0;
> > > pages_mapped < pages_to_map && !err;
> > > pages_mapped += pages_iter, idx += pages_iter) {
> > > + npages = min_t(int, pages_iter, pages_to_map - pages_mapped);
> > > dma_sync_single_for_cpu(ddev, dma, size, DMA_TO_DEVICE);
> > > - npages = populate_xlt(mr, idx, pages_iter, xlt,
> > > + npages = populate_xlt(mr, idx, npages, xlt,
> > > page_shift, size, flags);
> > >
> > > dma_sync_single_for_device(ddev, dma, size, DMA_TO_DEVICE);
> > >
> > > [1]
> > > https://git.kernel.org/cgit/linux/kernel/git/leon/linux-rdma.git/log/?h=rdma-next
> > >
> > > Thanks
> > >
> >
> > Hello Leon
> > Replied earlier but I dont know if my reply made it.
> > I will have to test this.
> >
> > is this repo
> > https://git.kernel.org/cgit/linux/kernel/git/leon/linux-rdma.git/log/?h=rdma-next
> > already patched with the change you want.
> > If not can I just take the patch and apply to my earlier tree based just on
> > Linus's tree where I reverted the patch.
> >
> > Thanks
> > Laurence
> > --
> > To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
> > the body of a message to majordomo@vger.kernel.org
> > More majordomo info at http://vger.kernel.org/majordomo-info.html
> >
>
> I guess not, I looked at applying this patch to Linus's tree but the function
> mlx5_ib_update_xlt() is not in there yet.
This is why I asked to use my tree, there is a chance that wrong
calculation was before we introduced mlx5_ib_update_xlt() function.
>
> I will see if I can get your tree staged and test this for you.
Thanks a lot.
>
> Thanks
> Laurence
> --
> To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]
^ permalink raw reply [flat|nested] 38+ messages in thread
* Re: [PATCH v2 1/8] IB/SRP: Avoid using IB_MR_TYPE_SG_GAPS
2017-02-15 13:33 ` Laurence Oberman
(?)
@ 2017-02-15 13:43 ` Laurence Oberman
2017-02-15 13:47 ` Leon Romanovsky
-1 siblings, 1 reply; 38+ messages in thread
From: Laurence Oberman @ 2017-02-15 13:43 UTC (permalink / raw)
To: Leon Romanovsky
Cc: Bart Van Assche, Max Gurtovoy, Doug Ledford, linux-rdma,
Israel Rukshin, Mark Bloch, Yuval Shaia, Artemy Kovalyov,
# 4 . 7+
----- Original Message -----
> From: "Laurence Oberman" <loberman@redhat.com>
> To: "Leon Romanovsky" <leonro@mellanox.com>
> Cc: "Bart Van Assche" <bart.vanassche@sandisk.com>, "Max Gurtovoy" <maxg@mellanox.com>, "Doug Ledford"
> <dledford@redhat.com>, linux-rdma@vger.kernel.org, "Israel Rukshin" <israelr@mellanox.com>, "Mark Bloch"
> <markb@mellanox.com>, "Yuval Shaia" <yuval.shaia@oracle.com>, "Artemy Kovalyov" <artemyko@mellanox.com>, "# 4 . 7+"
> <stable@vger.kernel.org>
> Sent: Wednesday, February 15, 2017 8:33:35 AM
> Subject: Re: [PATCH v2 1/8] IB/SRP: Avoid using IB_MR_TYPE_SG_GAPS
>
>
>
> ----- Original Message -----
> > From: "Leon Romanovsky" <leonro@mellanox.com>
> > To: "Bart Van Assche" <bart.vanassche@sandisk.com>, "Max Gurtovoy"
> > <maxg@mellanox.com>
> > Cc: "Doug Ledford" <dledford@redhat.com>, linux-rdma@vger.kernel.org,
> > "Israel Rukshin" <israelr@mellanox.com>, "Mark
> > Bloch" <markb@mellanox.com>, "Yuval Shaia" <yuval.shaia@oracle.com>,
> > "Artemy Kovalyov" <artemyko@mellanox.com>, "# 4
> > . 7+" <stable@vger.kernel.org>
> > Sent: Wednesday, February 15, 2017 3:19:45 AM
> > Subject: Re: [PATCH v2 1/8] IB/SRP: Avoid using IB_MR_TYPE_SG_GAPS
> >
> > On Wed, Feb 15, 2017 at 09:14:49AM +0200, Leon Romanovsky wrote:
> > > On Tue, Feb 14, 2017 at 10:56:29AM -0800, Bart Van Assche wrote:
> > > > Tests have shown that the following error message is reported when
> > > > using SG-GAPS registration with an mlx5 adapter:
> > > >
> > > > scsi host1: ib_srp: failed RECV status WR flushed (5) for CQE
> > > > ffff880bd4270eb0
> > > > 00000000 00000000 00000000 00000000
> > > > 00000000 00000000 00000000 00000000
> > > > 00000000 00000000 00000000 00000000
> > > > 00000000 0f007806 2500002a ad9fafd1
> > > > scsi host1: ib_srp: reconnect succeeded
> > > > mlx5_0:dump_cqe:262:(pid 7369): dump error cqe
> > > > 00000000 00000000 00000000 00000000
> > > > 00000000 00000000 00000000 00000000
> > > > 00000000 00000000 00000000 00000000
> > > > 00000000 0f007806 25000032 00105dd0
> > > > scsi host1: ib_srp: failed FAST REG status memory management operation
> > > > error (6) for CQE ffff880b92860138
> > > >
> > > > Hence avoid using SG-GAPS memory registrations. Additionally,
> > > > always configure the blk_queue_virt_boundary() to avoid to trigger
> > > > a mapping failure when using adapters that support SG-GAPS (e.g.
> > > > mlx5).
> > >
> > > According to the error dump, we have an issue with max_page_list_len
> > > supplied and/or
> > > internal calculations from that value to the UMR byte count.
> >
> > Hi Bart,
> >
> > Do you mind to try your test on my branch rdma-next [1] with the following
> > fixup?
> >
> > diff --git a/drivers/infiniband/hw/mlx5/mr.c
> > b/drivers/infiniband/hw/mlx5/mr.c
> > index 3c1f483d003f..3e59dce10d5e 100644
> > --- a/drivers/infiniband/hw/mlx5/mr.c
> > +++ b/drivers/infiniband/hw/mlx5/mr.c
> > @@ -1045,8 +1045,9 @@ int mlx5_ib_update_xlt(struct mlx5_ib_mr *mr, u64
> > idx,
> > int npages,
> > for (pages_mapped = 0;
> > pages_mapped < pages_to_map && !err;
> > pages_mapped += pages_iter, idx += pages_iter) {
> > + npages = min_t(int, pages_iter, pages_to_map - pages_mapped);
> > dma_sync_single_for_cpu(ddev, dma, size, DMA_TO_DEVICE);
> > - npages = populate_xlt(mr, idx, pages_iter, xlt,
> > + npages = populate_xlt(mr, idx, npages, xlt,
> > page_shift, size, flags);
> >
> > dma_sync_single_for_device(ddev, dma, size, DMA_TO_DEVICE);
> >
> > [1]
> > https://git.kernel.org/cgit/linux/kernel/git/leon/linux-rdma.git/log/?h=rdma-next
> >
> > Thanks
> >
>
> Hello Leon
> Replied earlier but I dont know if my reply made it.
> I will have to test this.
>
> is this repo
> https://git.kernel.org/cgit/linux/kernel/git/leon/linux-rdma.git/log/?h=rdma-next
> already patched with the change you want.
> If not can I just take the patch and apply to my earlier tree based just on
> Linus's tree where I reverted the patch.
>
> Thanks
> Laurence
> --
> To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
>
I guess not, I looked at applying this patch to Linus's tree but the function
mlx5_ib_update_xlt() is not in there yet.
I will see if I can get your tree staged and test this for you.
Thanks
Laurence
^ permalink raw reply [flat|nested] 38+ messages in thread
* Re: [PATCH v2 1/8] IB/SRP: Avoid using IB_MR_TYPE_SG_GAPS
2017-02-15 8:19 ` Leon Romanovsky
@ 2017-02-15 13:33 ` Laurence Oberman
-1 siblings, 0 replies; 38+ messages in thread
From: Laurence Oberman @ 2017-02-15 13:33 UTC (permalink / raw)
To: Leon Romanovsky
Cc: Bart Van Assche, Max Gurtovoy, Doug Ledford,
linux-rdma-u79uwXL29TY76Z2rM5mHXA, Israel Rukshin, Mark Bloch,
Yuval Shaia, Artemy Kovalyov, # 4 . 7+
----- Original Message -----
> From: "Leon Romanovsky" <leonro-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
> To: "Bart Van Assche" <bart.vanassche-XdAiOPVOjttBDgjK7y7TUQ@public.gmane.org>, "Max Gurtovoy" <maxg-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
> Cc: "Doug Ledford" <dledford-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>, linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org, "Israel Rukshin" <israelr-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>, "Mark
> Bloch" <markb-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>, "Yuval Shaia" <yuval.shaia-QHcLZuEGTsvQT0dZR+AlfA@public.gmane.org>, "Artemy Kovalyov" <artemyko-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>, "# 4
> . 7+" <stable-u79uwXL29TY76Z2rM5mHXA@public.gmane.org>
> Sent: Wednesday, February 15, 2017 3:19:45 AM
> Subject: Re: [PATCH v2 1/8] IB/SRP: Avoid using IB_MR_TYPE_SG_GAPS
>
> On Wed, Feb 15, 2017 at 09:14:49AM +0200, Leon Romanovsky wrote:
> > On Tue, Feb 14, 2017 at 10:56:29AM -0800, Bart Van Assche wrote:
> > > Tests have shown that the following error message is reported when
> > > using SG-GAPS registration with an mlx5 adapter:
> > >
> > > scsi host1: ib_srp: failed RECV status WR flushed (5) for CQE
> > > ffff880bd4270eb0
> > > 00000000 00000000 00000000 00000000
> > > 00000000 00000000 00000000 00000000
> > > 00000000 00000000 00000000 00000000
> > > 00000000 0f007806 2500002a ad9fafd1
> > > scsi host1: ib_srp: reconnect succeeded
> > > mlx5_0:dump_cqe:262:(pid 7369): dump error cqe
> > > 00000000 00000000 00000000 00000000
> > > 00000000 00000000 00000000 00000000
> > > 00000000 00000000 00000000 00000000
> > > 00000000 0f007806 25000032 00105dd0
> > > scsi host1: ib_srp: failed FAST REG status memory management operation
> > > error (6) for CQE ffff880b92860138
> > >
> > > Hence avoid using SG-GAPS memory registrations. Additionally,
> > > always configure the blk_queue_virt_boundary() to avoid to trigger
> > > a mapping failure when using adapters that support SG-GAPS (e.g.
> > > mlx5).
> >
> > According to the error dump, we have an issue with max_page_list_len
> > supplied and/or
> > internal calculations from that value to the UMR byte count.
>
> Hi Bart,
>
> Do you mind to try your test on my branch rdma-next [1] with the following
> fixup?
>
> diff --git a/drivers/infiniband/hw/mlx5/mr.c
> b/drivers/infiniband/hw/mlx5/mr.c
> index 3c1f483d003f..3e59dce10d5e 100644
> --- a/drivers/infiniband/hw/mlx5/mr.c
> +++ b/drivers/infiniband/hw/mlx5/mr.c
> @@ -1045,8 +1045,9 @@ int mlx5_ib_update_xlt(struct mlx5_ib_mr *mr, u64 idx,
> int npages,
> for (pages_mapped = 0;
> pages_mapped < pages_to_map && !err;
> pages_mapped += pages_iter, idx += pages_iter) {
> + npages = min_t(int, pages_iter, pages_to_map - pages_mapped);
> dma_sync_single_for_cpu(ddev, dma, size, DMA_TO_DEVICE);
> - npages = populate_xlt(mr, idx, pages_iter, xlt,
> + npages = populate_xlt(mr, idx, npages, xlt,
> page_shift, size, flags);
>
> dma_sync_single_for_device(ddev, dma, size, DMA_TO_DEVICE);
>
> [1]
> https://git.kernel.org/cgit/linux/kernel/git/leon/linux-rdma.git/log/?h=rdma-next
>
> Thanks
>
Hello Leon
Replied earlier but I dont know if my reply made it.
I will have to test this.
is this repo https://git.kernel.org/cgit/linux/kernel/git/leon/linux-rdma.git/log/?h=rdma-next already patched with the change you want.
If not can I just take the patch and apply to my earlier tree based just on Linus's tree where I reverted the patch.
Thanks
Laurence
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 38+ messages in thread
* Re: [PATCH v2 1/8] IB/SRP: Avoid using IB_MR_TYPE_SG_GAPS
@ 2017-02-15 13:33 ` Laurence Oberman
0 siblings, 0 replies; 38+ messages in thread
From: Laurence Oberman @ 2017-02-15 13:33 UTC (permalink / raw)
To: Leon Romanovsky
Cc: Bart Van Assche, Max Gurtovoy, Doug Ledford, linux-rdma,
Israel Rukshin, Mark Bloch, Yuval Shaia, Artemy Kovalyov,
# 4 . 7+
----- Original Message -----
> From: "Leon Romanovsky" <leonro@mellanox.com>
> To: "Bart Van Assche" <bart.vanassche@sandisk.com>, "Max Gurtovoy" <maxg@mellanox.com>
> Cc: "Doug Ledford" <dledford@redhat.com>, linux-rdma@vger.kernel.org, "Israel Rukshin" <israelr@mellanox.com>, "Mark
> Bloch" <markb@mellanox.com>, "Yuval Shaia" <yuval.shaia@oracle.com>, "Artemy Kovalyov" <artemyko@mellanox.com>, "# 4
> . 7+" <stable@vger.kernel.org>
> Sent: Wednesday, February 15, 2017 3:19:45 AM
> Subject: Re: [PATCH v2 1/8] IB/SRP: Avoid using IB_MR_TYPE_SG_GAPS
>
> On Wed, Feb 15, 2017 at 09:14:49AM +0200, Leon Romanovsky wrote:
> > On Tue, Feb 14, 2017 at 10:56:29AM -0800, Bart Van Assche wrote:
> > > Tests have shown that the following error message is reported when
> > > using SG-GAPS registration with an mlx5 adapter:
> > >
> > > scsi host1: ib_srp: failed RECV status WR flushed (5) for CQE
> > > ffff880bd4270eb0
> > > 00000000 00000000 00000000 00000000
> > > 00000000 00000000 00000000 00000000
> > > 00000000 00000000 00000000 00000000
> > > 00000000 0f007806 2500002a ad9fafd1
> > > scsi host1: ib_srp: reconnect succeeded
> > > mlx5_0:dump_cqe:262:(pid 7369): dump error cqe
> > > 00000000 00000000 00000000 00000000
> > > 00000000 00000000 00000000 00000000
> > > 00000000 00000000 00000000 00000000
> > > 00000000 0f007806 25000032 00105dd0
> > > scsi host1: ib_srp: failed FAST REG status memory management operation
> > > error (6) for CQE ffff880b92860138
> > >
> > > Hence avoid using SG-GAPS memory registrations. Additionally,
> > > always configure the blk_queue_virt_boundary() to avoid to trigger
> > > a mapping failure when using adapters that support SG-GAPS (e.g.
> > > mlx5).
> >
> > According to the error dump, we have an issue with max_page_list_len
> > supplied and/or
> > internal calculations from that value to the UMR byte count.
>
> Hi Bart,
>
> Do you mind to try your test on my branch rdma-next [1] with the following
> fixup?
>
> diff --git a/drivers/infiniband/hw/mlx5/mr.c
> b/drivers/infiniband/hw/mlx5/mr.c
> index 3c1f483d003f..3e59dce10d5e 100644
> --- a/drivers/infiniband/hw/mlx5/mr.c
> +++ b/drivers/infiniband/hw/mlx5/mr.c
> @@ -1045,8 +1045,9 @@ int mlx5_ib_update_xlt(struct mlx5_ib_mr *mr, u64 idx,
> int npages,
> for (pages_mapped = 0;
> pages_mapped < pages_to_map && !err;
> pages_mapped += pages_iter, idx += pages_iter) {
> + npages = min_t(int, pages_iter, pages_to_map - pages_mapped);
> dma_sync_single_for_cpu(ddev, dma, size, DMA_TO_DEVICE);
> - npages = populate_xlt(mr, idx, pages_iter, xlt,
> + npages = populate_xlt(mr, idx, npages, xlt,
> page_shift, size, flags);
>
> dma_sync_single_for_device(ddev, dma, size, DMA_TO_DEVICE);
>
> [1]
> https://git.kernel.org/cgit/linux/kernel/git/leon/linux-rdma.git/log/?h=rdma-next
>
> Thanks
>
Hello Leon
Replied earlier but I dont know if my reply made it.
I will have to test this.
is this repo https://git.kernel.org/cgit/linux/kernel/git/leon/linux-rdma.git/log/?h=rdma-next already patched with the change you want.
If not can I just take the patch and apply to my earlier tree based just on Linus's tree where I reverted the patch.
Thanks
Laurence
^ permalink raw reply [flat|nested] 38+ messages in thread
* Re: [PATCH v2 1/8] IB/SRP: Avoid using IB_MR_TYPE_SG_GAPS
2017-02-15 7:14 ` Leon Romanovsky
@ 2017-02-15 8:19 ` Leon Romanovsky
-1 siblings, 0 replies; 38+ messages in thread
From: Leon Romanovsky @ 2017-02-15 8:19 UTC (permalink / raw)
To: Bart Van Assche, Max Gurtovoy
Cc: Doug Ledford, linux-rdma, Israel Rukshin, Mark Bloch,
Yuval Shaia, Artemy Kovalyov, # 4 . 7+
[-- Attachment #1: Type: text/plain, Size: 2163 bytes --]
On Wed, Feb 15, 2017 at 09:14:49AM +0200, Leon Romanovsky wrote:
> On Tue, Feb 14, 2017 at 10:56:29AM -0800, Bart Van Assche wrote:
> > Tests have shown that the following error message is reported when
> > using SG-GAPS registration with an mlx5 adapter:
> >
> > scsi host1: ib_srp: failed RECV status WR flushed (5) for CQE ffff880bd4270eb0
> > 00000000 00000000 00000000 00000000
> > 00000000 00000000 00000000 00000000
> > 00000000 00000000 00000000 00000000
> > 00000000 0f007806 2500002a ad9fafd1
> > scsi host1: ib_srp: reconnect succeeded
> > mlx5_0:dump_cqe:262:(pid 7369): dump error cqe
> > 00000000 00000000 00000000 00000000
> > 00000000 00000000 00000000 00000000
> > 00000000 00000000 00000000 00000000
> > 00000000 0f007806 25000032 00105dd0
> > scsi host1: ib_srp: failed FAST REG status memory management operation error (6) for CQE ffff880b92860138
> >
> > Hence avoid using SG-GAPS memory registrations. Additionally,
> > always configure the blk_queue_virt_boundary() to avoid to trigger
> > a mapping failure when using adapters that support SG-GAPS (e.g.
> > mlx5).
>
> According to the error dump, we have an issue with max_page_list_len supplied and/or
> internal calculations from that value to the UMR byte count.
Hi Bart,
Do you mind to try your test on my branch rdma-next [1] with the following
fixup?
diff --git a/drivers/infiniband/hw/mlx5/mr.c b/drivers/infiniband/hw/mlx5/mr.c
index 3c1f483d003f..3e59dce10d5e 100644
--- a/drivers/infiniband/hw/mlx5/mr.c
+++ b/drivers/infiniband/hw/mlx5/mr.c
@@ -1045,8 +1045,9 @@ int mlx5_ib_update_xlt(struct mlx5_ib_mr *mr, u64 idx, int npages,
for (pages_mapped = 0;
pages_mapped < pages_to_map && !err;
pages_mapped += pages_iter, idx += pages_iter) {
+ npages = min_t(int, pages_iter, pages_to_map - pages_mapped);
dma_sync_single_for_cpu(ddev, dma, size, DMA_TO_DEVICE);
- npages = populate_xlt(mr, idx, pages_iter, xlt,
+ npages = populate_xlt(mr, idx, npages, xlt,
page_shift, size, flags);
dma_sync_single_for_device(ddev, dma, size, DMA_TO_DEVICE);
[1] https://git.kernel.org/cgit/linux/kernel/git/leon/linux-rdma.git/log/?h=rdma-next
Thanks
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]
^ permalink raw reply related [flat|nested] 38+ messages in thread
* Re: [PATCH v2 1/8] IB/SRP: Avoid using IB_MR_TYPE_SG_GAPS
@ 2017-02-15 8:19 ` Leon Romanovsky
0 siblings, 0 replies; 38+ messages in thread
From: Leon Romanovsky @ 2017-02-15 8:19 UTC (permalink / raw)
To: Bart Van Assche, Max Gurtovoy
Cc: Doug Ledford, linux-rdma, Israel Rukshin, Mark Bloch,
Yuval Shaia, Artemy Kovalyov, # 4 . 7+
[-- Attachment #1: Type: text/plain, Size: 2163 bytes --]
On Wed, Feb 15, 2017 at 09:14:49AM +0200, Leon Romanovsky wrote:
> On Tue, Feb 14, 2017 at 10:56:29AM -0800, Bart Van Assche wrote:
> > Tests have shown that the following error message is reported when
> > using SG-GAPS registration with an mlx5 adapter:
> >
> > scsi host1: ib_srp: failed RECV status WR flushed (5) for CQE ffff880bd4270eb0
> > 00000000 00000000 00000000 00000000
> > 00000000 00000000 00000000 00000000
> > 00000000 00000000 00000000 00000000
> > 00000000 0f007806 2500002a ad9fafd1
> > scsi host1: ib_srp: reconnect succeeded
> > mlx5_0:dump_cqe:262:(pid 7369): dump error cqe
> > 00000000 00000000 00000000 00000000
> > 00000000 00000000 00000000 00000000
> > 00000000 00000000 00000000 00000000
> > 00000000 0f007806 25000032 00105dd0
> > scsi host1: ib_srp: failed FAST REG status memory management operation error (6) for CQE ffff880b92860138
> >
> > Hence avoid using SG-GAPS memory registrations. Additionally,
> > always configure the blk_queue_virt_boundary() to avoid to trigger
> > a mapping failure when using adapters that support SG-GAPS (e.g.
> > mlx5).
>
> According to the error dump, we have an issue with max_page_list_len supplied and/or
> internal calculations from that value to the UMR byte count.
Hi Bart,
Do you mind to try your test on my branch rdma-next [1] with the following
fixup?
diff --git a/drivers/infiniband/hw/mlx5/mr.c b/drivers/infiniband/hw/mlx5/mr.c
index 3c1f483d003f..3e59dce10d5e 100644
--- a/drivers/infiniband/hw/mlx5/mr.c
+++ b/drivers/infiniband/hw/mlx5/mr.c
@@ -1045,8 +1045,9 @@ int mlx5_ib_update_xlt(struct mlx5_ib_mr *mr, u64 idx, int npages,
for (pages_mapped = 0;
pages_mapped < pages_to_map && !err;
pages_mapped += pages_iter, idx += pages_iter) {
+ npages = min_t(int, pages_iter, pages_to_map - pages_mapped);
dma_sync_single_for_cpu(ddev, dma, size, DMA_TO_DEVICE);
- npages = populate_xlt(mr, idx, pages_iter, xlt,
+ npages = populate_xlt(mr, idx, npages, xlt,
page_shift, size, flags);
dma_sync_single_for_device(ddev, dma, size, DMA_TO_DEVICE);
[1] https://git.kernel.org/cgit/linux/kernel/git/leon/linux-rdma.git/log/?h=rdma-next
Thanks
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]
^ permalink raw reply related [flat|nested] 38+ messages in thread
* Re: [PATCH v2 1/8] IB/SRP: Avoid using IB_MR_TYPE_SG_GAPS
2017-02-14 18:56 ` Bart Van Assche
@ 2017-02-15 7:14 ` Leon Romanovsky
-1 siblings, 0 replies; 38+ messages in thread
From: Leon Romanovsky @ 2017-02-15 7:14 UTC (permalink / raw)
To: Bart Van Assche, Max Gurtovoy
Cc: Doug Ledford, linux-rdma-u79uwXL29TY76Z2rM5mHXA, Israel Rukshin,
Mark Bloch, Yuval Shaia, Artemy Kovalyov, # 4 . 7+
[-- Attachment #1: Type: text/plain, Size: 3655 bytes --]
On Tue, Feb 14, 2017 at 10:56:29AM -0800, Bart Van Assche wrote:
> Tests have shown that the following error message is reported when
> using SG-GAPS registration with an mlx5 adapter:
>
> scsi host1: ib_srp: failed RECV status WR flushed (5) for CQE ffff880bd4270eb0
> 00000000 00000000 00000000 00000000
> 00000000 00000000 00000000 00000000
> 00000000 00000000 00000000 00000000
> 00000000 0f007806 2500002a ad9fafd1
> scsi host1: ib_srp: reconnect succeeded
> mlx5_0:dump_cqe:262:(pid 7369): dump error cqe
> 00000000 00000000 00000000 00000000
> 00000000 00000000 00000000 00000000
> 00000000 00000000 00000000 00000000
> 00000000 0f007806 25000032 00105dd0
> scsi host1: ib_srp: failed FAST REG status memory management operation error (6) for CQE ffff880b92860138
>
> Hence avoid using SG-GAPS memory registrations. Additionally,
> always configure the blk_queue_virt_boundary() to avoid to trigger
> a mapping failure when using adapters that support SG-GAPS (e.g.
> mlx5).
According to the error dump, we have an issue with max_page_list_len supplied and/or
internal calculations from that value to the UMR byte count.
I assume that iser works with SG_GAPS.
>
> Fixes: commit ad8e66b4a801 ("IB/srp: fix mr allocation when the device supports sg gaps")
> Fixes: commit 509c5f33f4f6 ("IB/srp: Prevent mapping failures")
> Reported-by: Laurence Oberman <loberman-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>
> Signed-off-by: Bart Van Assche <bart.vanassche-XdAiOPVOjttBDgjK7y7TUQ@public.gmane.org>
> Cc: Israel Rukshin <israelr-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
> Cc: Max Gurtovoy <maxg-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
> Cc: Leon Romanovsky <leonro-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
> Cc: Mark Bloch <markb-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
> Cc: Yuval Shaia <yuval.shaia-QHcLZuEGTsvQT0dZR+AlfA@public.gmane.org>
> Cc: <stable-u79uwXL29TY76Z2rM5mHXA@public.gmane.org> # 4.7+
> ---
> drivers/infiniband/ulp/srp/ib_srp.c | 11 +++--------
> 1 file changed, 3 insertions(+), 8 deletions(-)
>
> diff --git a/drivers/infiniband/ulp/srp/ib_srp.c b/drivers/infiniband/ulp/srp/ib_srp.c
> index 79bf48477ddb..07e3613e4798 100644
> --- a/drivers/infiniband/ulp/srp/ib_srp.c
> +++ b/drivers/infiniband/ulp/srp/ib_srp.c
> @@ -371,7 +371,6 @@ static struct srp_fr_pool *srp_create_fr_pool(struct ib_device *device,
> struct srp_fr_desc *d;
> struct ib_mr *mr;
> int i, ret = -EINVAL;
> - enum ib_mr_type mr_type;
>
> if (pool_size <= 0)
> goto err;
> @@ -385,13 +384,9 @@ static struct srp_fr_pool *srp_create_fr_pool(struct ib_device *device,
> spin_lock_init(&pool->lock);
> INIT_LIST_HEAD(&pool->free_list);
>
> - if (device->attrs.device_cap_flags & IB_DEVICE_SG_GAPS_REG)
> - mr_type = IB_MR_TYPE_SG_GAPS;
> - else
> - mr_type = IB_MR_TYPE_MEM_REG;
> -
> for (i = 0, d = &pool->desc[0]; i < pool->size; i++, d++) {
> - mr = ib_alloc_mr(pd, mr_type, max_page_list_len);
> + mr = ib_alloc_mr(pd, IB_MR_TYPE_MEM_REG,
> + max_page_list_len);
> if (IS_ERR(mr)) {
> ret = PTR_ERR(mr);
> if (ret == -ENOMEM)
> @@ -2666,7 +2661,7 @@ static int srp_slave_alloc(struct scsi_device *sdev)
> struct srp_device *srp_dev = target->srp_host->srp_dev;
> struct ib_device *ibdev = srp_dev->dev;
>
> - if (!(ibdev->attrs.device_cap_flags & IB_DEVICE_SG_GAPS_REG))
> + if (true)
> blk_queue_virt_boundary(sdev->request_queue,
> ~srp_dev->mr_page_mask);
>
> --
> 2.11.0
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]
^ permalink raw reply [flat|nested] 38+ messages in thread
* Re: [PATCH v2 1/8] IB/SRP: Avoid using IB_MR_TYPE_SG_GAPS
@ 2017-02-15 7:14 ` Leon Romanovsky
0 siblings, 0 replies; 38+ messages in thread
From: Leon Romanovsky @ 2017-02-15 7:14 UTC (permalink / raw)
To: Bart Van Assche, Max Gurtovoy
Cc: Doug Ledford, linux-rdma, Israel Rukshin, Mark Bloch,
Yuval Shaia, Artemy Kovalyov, # 4 . 7+
[-- Attachment #1: Type: text/plain, Size: 3413 bytes --]
On Tue, Feb 14, 2017 at 10:56:29AM -0800, Bart Van Assche wrote:
> Tests have shown that the following error message is reported when
> using SG-GAPS registration with an mlx5 adapter:
>
> scsi host1: ib_srp: failed RECV status WR flushed (5) for CQE ffff880bd4270eb0
> 00000000 00000000 00000000 00000000
> 00000000 00000000 00000000 00000000
> 00000000 00000000 00000000 00000000
> 00000000 0f007806 2500002a ad9fafd1
> scsi host1: ib_srp: reconnect succeeded
> mlx5_0:dump_cqe:262:(pid 7369): dump error cqe
> 00000000 00000000 00000000 00000000
> 00000000 00000000 00000000 00000000
> 00000000 00000000 00000000 00000000
> 00000000 0f007806 25000032 00105dd0
> scsi host1: ib_srp: failed FAST REG status memory management operation error (6) for CQE ffff880b92860138
>
> Hence avoid using SG-GAPS memory registrations. Additionally,
> always configure the blk_queue_virt_boundary() to avoid to trigger
> a mapping failure when using adapters that support SG-GAPS (e.g.
> mlx5).
According to the error dump, we have an issue with max_page_list_len supplied and/or
internal calculations from that value to the UMR byte count.
I assume that iser works with SG_GAPS.
>
> Fixes: commit ad8e66b4a801 ("IB/srp: fix mr allocation when the device supports sg gaps")
> Fixes: commit 509c5f33f4f6 ("IB/srp: Prevent mapping failures")
> Reported-by: Laurence Oberman <loberman@redhat.com>
> Signed-off-by: Bart Van Assche <bart.vanassche@sandisk.com>
> Cc: Israel Rukshin <israelr@mellanox.com>
> Cc: Max Gurtovoy <maxg@mellanox.com>
> Cc: Leon Romanovsky <leonro@mellanox.com>
> Cc: Mark Bloch <markb@mellanox.com>
> Cc: Yuval Shaia <yuval.shaia@oracle.com>
> Cc: <stable@vger.kernel.org> # 4.7+
> ---
> drivers/infiniband/ulp/srp/ib_srp.c | 11 +++--------
> 1 file changed, 3 insertions(+), 8 deletions(-)
>
> diff --git a/drivers/infiniband/ulp/srp/ib_srp.c b/drivers/infiniband/ulp/srp/ib_srp.c
> index 79bf48477ddb..07e3613e4798 100644
> --- a/drivers/infiniband/ulp/srp/ib_srp.c
> +++ b/drivers/infiniband/ulp/srp/ib_srp.c
> @@ -371,7 +371,6 @@ static struct srp_fr_pool *srp_create_fr_pool(struct ib_device *device,
> struct srp_fr_desc *d;
> struct ib_mr *mr;
> int i, ret = -EINVAL;
> - enum ib_mr_type mr_type;
>
> if (pool_size <= 0)
> goto err;
> @@ -385,13 +384,9 @@ static struct srp_fr_pool *srp_create_fr_pool(struct ib_device *device,
> spin_lock_init(&pool->lock);
> INIT_LIST_HEAD(&pool->free_list);
>
> - if (device->attrs.device_cap_flags & IB_DEVICE_SG_GAPS_REG)
> - mr_type = IB_MR_TYPE_SG_GAPS;
> - else
> - mr_type = IB_MR_TYPE_MEM_REG;
> -
> for (i = 0, d = &pool->desc[0]; i < pool->size; i++, d++) {
> - mr = ib_alloc_mr(pd, mr_type, max_page_list_len);
> + mr = ib_alloc_mr(pd, IB_MR_TYPE_MEM_REG,
> + max_page_list_len);
> if (IS_ERR(mr)) {
> ret = PTR_ERR(mr);
> if (ret == -ENOMEM)
> @@ -2666,7 +2661,7 @@ static int srp_slave_alloc(struct scsi_device *sdev)
> struct srp_device *srp_dev = target->srp_host->srp_dev;
> struct ib_device *ibdev = srp_dev->dev;
>
> - if (!(ibdev->attrs.device_cap_flags & IB_DEVICE_SG_GAPS_REG))
> + if (true)
> blk_queue_virt_boundary(sdev->request_queue,
> ~srp_dev->mr_page_mask);
>
> --
> 2.11.0
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]
^ permalink raw reply [flat|nested] 38+ messages in thread
* Re: [PATCH v2 1/8] IB/SRP: Avoid using IB_MR_TYPE_SG_GAPS
2017-02-14 18:56 ` Bart Van Assche
@ 2017-02-15 0:22 ` Bart Van Assche
-1 siblings, 0 replies; 38+ messages in thread
From: Bart Van Assche @ 2017-02-15 0:22 UTC (permalink / raw)
To: dledford-H+wXaHxf7aLQT0dZR+AlfA
Cc: yuval.shaia-QHcLZuEGTsvQT0dZR+AlfA, maxg-VPRAkNaXOzVWk0Htik3J/w,
israelr-VPRAkNaXOzVWk0Htik3J/w,
linux-rdma-u79uwXL29TY76Z2rM5mHXA, leonro-VPRAkNaXOzVWk0Htik3J/w,
stable-u79uwXL29TY76Z2rM5mHXA, markb-VPRAkNaXOzVWk0Htik3J/w
On Tue, 2017-02-14 at 10:56 -0800, Bart Van Assche wrote:
> @@ -2666,7 +2661,7 @@ static int srp_slave_alloc(struct scsi_device *sdev)
> struct srp_device *srp_dev = target->srp_host->srp_dev;
> struct ib_device *ibdev = srp_dev->dev;
>
> - if (!(ibdev->attrs.device_cap_flags & IB_DEVICE_SG_GAPS_REG))
> + if (true)
> blk_queue_virt_boundary(sdev->request_queue,
> ~srp_dev->mr_page_mask);
Hello Doug,
This patch introduces a new warning when building with W=1, namely that
variable ibdev is not used. The build bot figured this out. Do you want me
to address this?
Thanks,
Bart.--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 38+ messages in thread
* Re: [PATCH v2 1/8] IB/SRP: Avoid using IB_MR_TYPE_SG_GAPS
@ 2017-02-15 0:22 ` Bart Van Assche
0 siblings, 0 replies; 38+ messages in thread
From: Bart Van Assche @ 2017-02-15 0:22 UTC (permalink / raw)
To: dledford; +Cc: yuval.shaia, maxg, israelr, linux-rdma, leonro, stable, markb
On Tue, 2017-02-14 at 10:56 -0800, Bart Van Assche wrote:
> @@ -2666,7 +2661,7 @@ static int srp_slave_alloc(struct scsi_device *sdev)
> struct srp_device *srp_dev = target->srp_host->srp_dev;
> struct ib_device *ibdev = srp_dev->dev;
>
> - if (!(ibdev->attrs.device_cap_flags & IB_DEVICE_SG_GAPS_REG))
> + if (true)
> blk_queue_virt_boundary(sdev->request_queue,
> ~srp_dev->mr_page_mask);
Hello Doug,
This patch introduces a new warning when building with W=1, namely that
variable ibdev is not used. The build bot figured this out. Do you want me
to address this?
Thanks,
Bart.
^ permalink raw reply [flat|nested] 38+ messages in thread
* [PATCH v2 1/8] IB/SRP: Avoid using IB_MR_TYPE_SG_GAPS
2017-02-14 18:56 [PATCH v2 0/8] IB/srp bug fixes Bart Van Assche
@ 2017-02-14 18:56 ` Bart Van Assche
0 siblings, 0 replies; 38+ messages in thread
From: Bart Van Assche @ 2017-02-14 18:56 UTC (permalink / raw)
To: Doug Ledford
Cc: linux-rdma-u79uwXL29TY76Z2rM5mHXA, Bart Van Assche,
Israel Rukshin, Max Gurtovoy, Leon Romanovsky, Mark Bloch,
Yuval Shaia, # 4 . 7+
Tests have shown that the following error message is reported when
using SG-GAPS registration with an mlx5 adapter:
scsi host1: ib_srp: failed RECV status WR flushed (5) for CQE ffff880bd4270eb0
00000000 00000000 00000000 00000000
00000000 00000000 00000000 00000000
00000000 00000000 00000000 00000000
00000000 0f007806 2500002a ad9fafd1
scsi host1: ib_srp: reconnect succeeded
mlx5_0:dump_cqe:262:(pid 7369): dump error cqe
00000000 00000000 00000000 00000000
00000000 00000000 00000000 00000000
00000000 00000000 00000000 00000000
00000000 0f007806 25000032 00105dd0
scsi host1: ib_srp: failed FAST REG status memory management operation error (6) for CQE ffff880b92860138
Hence avoid using SG-GAPS memory registrations. Additionally,
always configure the blk_queue_virt_boundary() to avoid to trigger
a mapping failure when using adapters that support SG-GAPS (e.g.
mlx5).
Fixes: commit ad8e66b4a801 ("IB/srp: fix mr allocation when the device supports sg gaps")
Fixes: commit 509c5f33f4f6 ("IB/srp: Prevent mapping failures")
Reported-by: Laurence Oberman <loberman-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>
Signed-off-by: Bart Van Assche <bart.vanassche-XdAiOPVOjttBDgjK7y7TUQ@public.gmane.org>
Cc: Israel Rukshin <israelr-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
Cc: Max Gurtovoy <maxg-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
Cc: Leon Romanovsky <leonro-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
Cc: Mark Bloch <markb-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
Cc: Yuval Shaia <yuval.shaia-QHcLZuEGTsvQT0dZR+AlfA@public.gmane.org>
Cc: <stable-u79uwXL29TY76Z2rM5mHXA@public.gmane.org> # 4.7+
---
drivers/infiniband/ulp/srp/ib_srp.c | 11 +++--------
1 file changed, 3 insertions(+), 8 deletions(-)
diff --git a/drivers/infiniband/ulp/srp/ib_srp.c b/drivers/infiniband/ulp/srp/ib_srp.c
index 79bf48477ddb..07e3613e4798 100644
--- a/drivers/infiniband/ulp/srp/ib_srp.c
+++ b/drivers/infiniband/ulp/srp/ib_srp.c
@@ -371,7 +371,6 @@ static struct srp_fr_pool *srp_create_fr_pool(struct ib_device *device,
struct srp_fr_desc *d;
struct ib_mr *mr;
int i, ret = -EINVAL;
- enum ib_mr_type mr_type;
if (pool_size <= 0)
goto err;
@@ -385,13 +384,9 @@ static struct srp_fr_pool *srp_create_fr_pool(struct ib_device *device,
spin_lock_init(&pool->lock);
INIT_LIST_HEAD(&pool->free_list);
- if (device->attrs.device_cap_flags & IB_DEVICE_SG_GAPS_REG)
- mr_type = IB_MR_TYPE_SG_GAPS;
- else
- mr_type = IB_MR_TYPE_MEM_REG;
-
for (i = 0, d = &pool->desc[0]; i < pool->size; i++, d++) {
- mr = ib_alloc_mr(pd, mr_type, max_page_list_len);
+ mr = ib_alloc_mr(pd, IB_MR_TYPE_MEM_REG,
+ max_page_list_len);
if (IS_ERR(mr)) {
ret = PTR_ERR(mr);
if (ret == -ENOMEM)
@@ -2666,7 +2661,7 @@ static int srp_slave_alloc(struct scsi_device *sdev)
struct srp_device *srp_dev = target->srp_host->srp_dev;
struct ib_device *ibdev = srp_dev->dev;
- if (!(ibdev->attrs.device_cap_flags & IB_DEVICE_SG_GAPS_REG))
+ if (true)
blk_queue_virt_boundary(sdev->request_queue,
~srp_dev->mr_page_mask);
--
2.11.0
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply related [flat|nested] 38+ messages in thread
* [PATCH v2 1/8] IB/SRP: Avoid using IB_MR_TYPE_SG_GAPS
@ 2017-02-14 18:56 ` Bart Van Assche
0 siblings, 0 replies; 38+ messages in thread
From: Bart Van Assche @ 2017-02-14 18:56 UTC (permalink / raw)
To: Doug Ledford
Cc: linux-rdma, Bart Van Assche, Israel Rukshin, Max Gurtovoy,
Leon Romanovsky, Mark Bloch, Yuval Shaia, # 4 . 7+
Tests have shown that the following error message is reported when
using SG-GAPS registration with an mlx5 adapter:
scsi host1: ib_srp: failed RECV status WR flushed (5) for CQE ffff880bd4270eb0
00000000 00000000 00000000 00000000
00000000 00000000 00000000 00000000
00000000 00000000 00000000 00000000
00000000 0f007806 2500002a ad9fafd1
scsi host1: ib_srp: reconnect succeeded
mlx5_0:dump_cqe:262:(pid 7369): dump error cqe
00000000 00000000 00000000 00000000
00000000 00000000 00000000 00000000
00000000 00000000 00000000 00000000
00000000 0f007806 25000032 00105dd0
scsi host1: ib_srp: failed FAST REG status memory management operation error (6) for CQE ffff880b92860138
Hence avoid using SG-GAPS memory registrations. Additionally,
always configure the blk_queue_virt_boundary() to avoid to trigger
a mapping failure when using adapters that support SG-GAPS (e.g.
mlx5).
Fixes: commit ad8e66b4a801 ("IB/srp: fix mr allocation when the device supports sg gaps")
Fixes: commit 509c5f33f4f6 ("IB/srp: Prevent mapping failures")
Reported-by: Laurence Oberman <loberman@redhat.com>
Signed-off-by: Bart Van Assche <bart.vanassche@sandisk.com>
Cc: Israel Rukshin <israelr@mellanox.com>
Cc: Max Gurtovoy <maxg@mellanox.com>
Cc: Leon Romanovsky <leonro@mellanox.com>
Cc: Mark Bloch <markb@mellanox.com>
Cc: Yuval Shaia <yuval.shaia@oracle.com>
Cc: <stable@vger.kernel.org> # 4.7+
---
drivers/infiniband/ulp/srp/ib_srp.c | 11 +++--------
1 file changed, 3 insertions(+), 8 deletions(-)
diff --git a/drivers/infiniband/ulp/srp/ib_srp.c b/drivers/infiniband/ulp/srp/ib_srp.c
index 79bf48477ddb..07e3613e4798 100644
--- a/drivers/infiniband/ulp/srp/ib_srp.c
+++ b/drivers/infiniband/ulp/srp/ib_srp.c
@@ -371,7 +371,6 @@ static struct srp_fr_pool *srp_create_fr_pool(struct ib_device *device,
struct srp_fr_desc *d;
struct ib_mr *mr;
int i, ret = -EINVAL;
- enum ib_mr_type mr_type;
if (pool_size <= 0)
goto err;
@@ -385,13 +384,9 @@ static struct srp_fr_pool *srp_create_fr_pool(struct ib_device *device,
spin_lock_init(&pool->lock);
INIT_LIST_HEAD(&pool->free_list);
- if (device->attrs.device_cap_flags & IB_DEVICE_SG_GAPS_REG)
- mr_type = IB_MR_TYPE_SG_GAPS;
- else
- mr_type = IB_MR_TYPE_MEM_REG;
-
for (i = 0, d = &pool->desc[0]; i < pool->size; i++, d++) {
- mr = ib_alloc_mr(pd, mr_type, max_page_list_len);
+ mr = ib_alloc_mr(pd, IB_MR_TYPE_MEM_REG,
+ max_page_list_len);
if (IS_ERR(mr)) {
ret = PTR_ERR(mr);
if (ret == -ENOMEM)
@@ -2666,7 +2661,7 @@ static int srp_slave_alloc(struct scsi_device *sdev)
struct srp_device *srp_dev = target->srp_host->srp_dev;
struct ib_device *ibdev = srp_dev->dev;
- if (!(ibdev->attrs.device_cap_flags & IB_DEVICE_SG_GAPS_REG))
+ if (true)
blk_queue_virt_boundary(sdev->request_queue,
~srp_dev->mr_page_mask);
--
2.11.0
^ permalink raw reply related [flat|nested] 38+ messages in thread
end of thread, other threads:[~2017-02-16 9:11 UTC | newest]
Thread overview: 38+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
[not found] <017955b3-8fd5-40da-8bd5-023bc2f23fb4@email.android.com>
[not found] ` <017955b3-8fd5-40da-8bd5-023bc2f23fb4-2ueSQiBKiTY7tOexoI0I+QC/G2K4zDHf@public.gmane.org>
2017-02-15 15:31 ` [PATCH v2 1/8] IB/SRP: Avoid using IB_MR_TYPE_SG_GAPS Bart Van Assche
2017-02-15 15:31 ` Bart Van Assche
[not found] ` <1487172663.2990.5.camel-XdAiOPVOjttBDgjK7y7TUQ@public.gmane.org>
2017-02-15 15:34 ` Laurence Oberman
2017-02-15 15:34 ` Laurence Oberman
2017-02-14 18:56 [PATCH v2 0/8] IB/srp bug fixes Bart Van Assche
[not found] ` <20170214185636.29250-1-bart.vanassche-XdAiOPVOjttBDgjK7y7TUQ@public.gmane.org>
2017-02-14 18:56 ` [PATCH v2 1/8] IB/SRP: Avoid using IB_MR_TYPE_SG_GAPS Bart Van Assche
2017-02-14 18:56 ` Bart Van Assche
[not found] ` <20170214185636.29250-2-bart.vanassche-XdAiOPVOjttBDgjK7y7TUQ@public.gmane.org>
2017-02-15 0:22 ` Bart Van Assche
2017-02-15 0:22 ` Bart Van Assche
2017-02-15 7:14 ` Leon Romanovsky
2017-02-15 7:14 ` Leon Romanovsky
2017-02-15 8:19 ` Leon Romanovsky
2017-02-15 8:19 ` Leon Romanovsky
[not found] ` <20170215081945.GP6989-U/DQcQFIOTAAJjI8aNfphQ@public.gmane.org>
2017-02-15 13:33 ` Laurence Oberman
2017-02-15 13:33 ` Laurence Oberman
2017-02-15 13:43 ` Laurence Oberman
2017-02-15 13:47 ` Leon Romanovsky
2017-02-15 13:47 ` Leon Romanovsky
2017-02-15 14:53 ` Laurence Oberman
[not found] ` <242820990.31706010.1487170436012.JavaMail.zimbra-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>
2017-02-15 15:18 ` Leon Romanovsky
2017-02-15 15:18 ` Leon Romanovsky
2017-02-15 15:42 ` Sagi Grimberg
2017-02-15 15:38 ` Sagi Grimberg
2017-02-15 15:38 ` Sagi Grimberg
[not found] ` <cebcaeae-94a6-de82-cfc8-ce055b273836-NQWnxTmZq1alnMjI0IkVqw@public.gmane.org>
2017-02-15 15:42 ` Laurence Oberman
2017-02-15 15:42 ` Laurence Oberman
2017-02-15 16:18 ` Max Gurtovoy
2017-02-15 16:18 ` Max Gurtovoy
2017-02-15 16:27 ` Sagi Grimberg
[not found] ` <0514bb01-95cf-c10a-b883-494f149845f3-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
2017-02-15 16:30 ` Leon Romanovsky
2017-02-15 16:30 ` Leon Romanovsky
2017-02-15 16:37 ` Laurence Oberman
2017-02-15 16:37 ` Laurence Oberman
2017-02-15 16:55 ` Sagi Grimberg
2017-02-15 23:49 ` Bart Van Assche
2017-02-16 6:14 ` Leon Romanovsky
2017-02-16 6:14 ` Leon Romanovsky
2017-02-16 9:11 ` Max Gurtovoy
2017-02-16 9:11 ` Max Gurtovoy
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.