Status of RXE/Soft-RoCE driver?

Status of RXE/Soft-RoCE driver?

[Robert LeBlanc]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

What is the status of getting RXE included in the mainline kernel? I
saw a post from Kamal Heib on 22 Sep 2015 with a patchset for
inclusion [0], but there didn't seem to be any discussion about it. I
diffed the kernel tree with the RXE tree on GitHub using the
rxe_submission_v8 branch [1], but there was over 10 million lines in
the patch file for the same kernel version. I'd rather patch into the
vanilla kernel tree so that I can see the differences rather than just
building the tree that someone else provides.

I will probably go through all 32 patches anyway, but thought I would
prod here about the status as it can be very helpful for
compatibility.

[0] http://www.spinics.net/lists/linux-rdma/msg28829.html
[1] https://github.com/SoftRoCE/rxe-dev/tree/rxe_submission_v8

Thanks,
- ----------------
Robert LeBlanc
PGP Fingerprint 79A2 9CA4 6CC4 45DD A904  C70E E654 3BB2 FA62 B9F1
-----BEGIN PGP SIGNATURE-----
Version: Mailvelope v1.3.6
Comment: https://www.mailvelope.com

wsFcBAEBCAAQBQJXIUPOCRDmVDuy+mK58QAA+noQALVF1CpD+oqlMR2lgRT1
zFgy5PeKLjX+nmqZExuvd0uu2kVw/JNJUnsvUGG0083/HT68Qm0XuVzZMoPX
XQq5VoyuqsapEX2nEGhT2Wy+w6hSvxpVn6xGde11v6RQ9Lq/u1yHGxUFmWGf
BwHD3G4/pGgJg5dG/BbfxUYE4w+TNl4FpGVAeZOPsCRCVehxQsgU7LNPWyo+
pWcn3CY1AAIgUB+ZLKASsQq5a7/jbuYKFJCdC0i56dCiv8RHdGxTLRGzquLy
QOR0sUhxDhUAKS1iwX/UdOXvCm6iWOesF5XWS19311ENPZREgaGtrcCNkXTM
cDQi1eQU0BB2j2ZIsFMhTh1mD5NxKNTOWDptexEpa0VYFLxs1nnkajhm854d
YZXXqSOnKtAkatjEXKRXWzJImgVcPl7FnWY6va+VKt/YNpZ4WzekL+V73Xly
5at18zou4DpZG8OYkg1hTW13X/1NYlOctwdu7qYSj1MAuJ0DHcoV/UAg8/SQ
L+27Hf71svtsLvDqwN7+uDj3mg74Iu6s5jBhjwYFNU6aeQKOZzllAjOczT+Z
bY+1MGSGPY3Tt7Y5BunTLsp63O5VPcWBk64P5TE+RDoXYAzfan1Ok0h1Vinm
x3n6yWY23mlTSflVtzF+z+97Dq6VoSmKDUVhz/yB4Ehu6r54N5bcT25ohDUB
L9IZ
=u+48
-----END PGP SIGNATURE-----
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Status of RXE/Soft-RoCE driver?

[Sagi Grimberg]

> What is the status of getting RXE included in the mainline kernel? I
> saw a post from Kamal Heib on 22 Sep 2015 with a patchset for
> inclusion [0], but there didn't seem to be any discussion about it. I
> diffed the kernel tree with the RXE tree on GitHub using the
> rxe_submission_v8 branch [1], but there was over 10 million lines in
> the patch file for the same kernel version. I'd rather patch into the
> vanilla kernel tree so that I can see the differences rather than just
> building the tree that someone else provides.
>
> I will probably go through all 32 patches anyway, but thought I would
> prod here about the status as it can be very helpful for
> compatibility.

FWIW, I'd love to see ib_rxe upstream asap. In fact, I even have
some patches to contribute for it.

Cheers,
Sagi.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Status of RXE/Soft-RoCE driver?

[Leon Romanovsky]

[-- Attachment #1: Type: text/plain, Size: 1200 bytes --]

On Thu, Apr 28, 2016 at 02:17:29PM +0300, Sagi Grimberg wrote:
> 
> >What is the status of getting RXE included in the mainline kernel? I
> >saw a post from Kamal Heib on 22 Sep 2015 with a patchset for
> >inclusion [0], but there didn't seem to be any discussion about it. I
> >diffed the kernel tree with the RXE tree on GitHub using the
> >rxe_submission_v8 branch [1], but there was over 10 million lines in
> >the patch file for the same kernel version. I'd rather patch into the
> >vanilla kernel tree so that I can see the differences rather than just
> >building the tree that someone else provides.
> >
> >I will probably go through all 32 patches anyway, but thought I would
> >prod here about the status as it can be very helpful for
> >compatibility.
> 
> FWIW, I'd love to see ib_rxe upstream asap. In fact, I even have
> some patches to contribute for it.

We are planning on submitting it in very near future.

> 
> Cheers,
> Sagi.
> --
> To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: Status of RXE/Soft-RoCE driver?

[Or Gerlitz]

On Thu, Apr 28, 2016 at 2:56 PM, Leon Romanovsky <leon-DgEjT+Ai2ygdnm+yROfE0A@public.gmane.org> wrote:
> On Thu, Apr 28, 2016 at 02:17:29PM +0300, Sagi Grimberg wrote:

>> FWIW, I'd love to see ib_rxe upstream asap. In fact, I even have
>> some patches to contribute for it.
>
> We are planning on submitting it in very near future.

So.. lets get this going, Linus made a comment [1] that we might not
go up even till rc7, which means there very little time left, Moni?

On a related note, is there ANY patch merged for 4.7? Doug, are we
again going to have have it in
all-patches-merged-on-the-same-night-10-days-after-the-merge-window-opened
manner?

Or.

[1] https://lkml.org/lkml/2016/4/24/177
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Status of RXE/Soft-RoCE driver?

[Robert LeBlanc]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

Cool, thanks for the update.
-----BEGIN PGP SIGNATURE-----
Version: Mailvelope v1.3.6
Comment: https://www.mailvelope.com

wsFcBAEBCAAQBQJXIjDMCRDmVDuy+mK58QAAjcMP/jVRTBSRTJNkhGkGCiL5
HbRJUsslKXp61BqL4LnLQ26ELq+YPVvG/ds/7wKAaaQrWv8RZyb+L38J0tI6
s9C25Bo+PwheTJHmnTjrL4m0MiEWsiKGHCkc/x98YMla//UHoooE1X1QtJ4w
pGNcuF4rgCIoZs8Q8Gd2cIuZWgZUhHNK5TEv866YVXcH1YxkVCKFqen3JD0/
8gtcPv9uw0Try4q584TUjZEEw/7SSy+FAcYDsF84ttUvalwWNcGX/1KR9tf+
UVokNxuqpJNEsMqyF+UYg9xFCiBdIbR2KDRd7UVtnJLnfbJd4IFOejxV6x/g
MEHFnzckK1pwIT24wEOUedyMELbbPJOfzbw2YzIZH6HAfLi/2a92vlAaW4sy
rVAEDBbC+DuXIxhY4DdTw63S4ZS1Tphl8DJmd/PBLMlu9vFS7lay1pZsQ3bQ
j7PWhb8h+a2us74xHUfHxd4AoFbRfm5s5EYnlZH/9ho8HiY9vubK3R8ID+ND
qYTxLHDSkdccw+AsE+vTXoAFl9vx4nF8F2XQcXlMFjlZ9dmgrVjDIgLbQogb
cQI1CunsqwLcYxaPl6W6JLWlEENwagdoSfTDjSZQud8MEQ/LTKipAab2tdmK
vD3M9HXyb7oKWrdDfcAhSe/TGcMk8RfaVlPKq3FTVTyhcUQyhKIoNb3iKXoX
PDCa
=Zzq9
-----END PGP SIGNATURE-----
----------------
Robert LeBlanc
PGP Fingerprint 79A2 9CA4 6CC4 45DD A904  C70E E654 3BB2 FA62 B9F1


On Thu, Apr 28, 2016 at 5:56 AM, Leon Romanovsky <leon-DgEjT+Ai2ygdnm+yROfE0A@public.gmane.org> wrote:
> On Thu, Apr 28, 2016 at 02:17:29PM +0300, Sagi Grimberg wrote:
>>
>> >What is the status of getting RXE included in the mainline kernel? I
>> >saw a post from Kamal Heib on 22 Sep 2015 with a patchset for
>> >inclusion [0], but there didn't seem to be any discussion about it. I
>> >diffed the kernel tree with the RXE tree on GitHub using the
>> >rxe_submission_v8 branch [1], but there was over 10 million lines in
>> >the patch file for the same kernel version. I'd rather patch into the
>> >vanilla kernel tree so that I can see the differences rather than just
>> >building the tree that someone else provides.
>> >
>> >I will probably go through all 32 patches anyway, but thought I would
>> >prod here about the status as it can be very helpful for
>> >compatibility.
>>
>> FWIW, I'd love to see ib_rxe upstream asap. In fact, I even have
>> some patches to contribute for it.
>
> We are planning on submitting it in very near future.
>
>>
>> Cheers,
>> Sagi.
>> --
>> To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
>> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Status of RXE/Soft-RoCE driver?

[Robert LeBlanc]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

Looks like quite a bit has changed especially with
b7d3e0a94fe128912bbebf0ae68551c85fd2d429 causing those patches to not
apply cleanly. I'm trying to get this to work on 4.5+, but having
trouble getting this all figured out (not being a programmer and not
extremely familiar with the inner workings of RDMA. If someone has a
newer patch set, I'd like to do some testing.
-----BEGIN PGP SIGNATURE-----
Version: Mailvelope v1.3.6
Comment: https://www.mailvelope.com

wsFcBAEBCAAQBQJXIp8mCRDmVDuy+mK58QAAAwgP+gPby0h66AIYDfzCXv7v
s+enVrSSCynFxnRALlH5BJjdOZFOaaPQ+K3S9d4w6y8vvszaAdWKJmtabtk2
jBBQigz7sUI9LjP6Te1VV7e1tjL/b2C/7ieNWCXuZUIQe11ImiaMCYWwwClI
OsBHYyZTvyW+VjMTNYwmBTUL5mI4yt5MUP0Izj3ZUeQmuhRxqRm3w7ywSjGX
+8lmFU7Z/oCA8SWIJX5yY0QYfNOLeh/cMPVvXgZwKPnRqqsnO33bGT842UJU
aQE1YPuLsqFsLTnaVPgqEJAuo2loYdOpCKgIj4MlDu/QSXiOlfBqVDw5DU+c
Ldj4nIUq4OOdnm9D0IcbgFxSHV17iRhmgaH8oCd3S4lQCtV8FGmS3AN8pVjq
UMIFIibchEwBI+Q/Xt4Nc00ccU3Cup3jvpWRZw4FXz9GiQQSUPAsdZLMT+Io
9dDA7hLCgdeWe8pIjeyqfB8nWxWKppu01Dk4Yyxd08KSoYFa0Lj2wgGOyJqp
RJ3LxI8IX7eKTgk7Vf/IqorEyka9/KpDNgLxxT2nL6EFKArb626DFgWK+YLG
91gocctxebiGEZ6mDAPdMCEYvEyY5KyG1d02Chyl3uiFqJWZMFSwGr+S1Bqa
WelDLBvjp8xWiqFduZMWv1Z2RibCbh2McsfNXDI5eSU4JyPGhTUBtFddU1Cz
LcMQ
=8ZMY
-----END PGP SIGNATURE-----
----------------
Robert LeBlanc
PGP Fingerprint 79A2 9CA4 6CC4 45DD A904  C70E E654 3BB2 FA62 B9F1


On Thu, Apr 28, 2016 at 9:48 AM, Robert LeBlanc <robert-4JaGZRWAfWbajFs6igw21g@public.gmane.org> wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA256
>
> Cool, thanks for the update.
> -----BEGIN PGP SIGNATURE-----
> Version: Mailvelope v1.3.6
> Comment: https://www.mailvelope.com
>
> wsFcBAEBCAAQBQJXIjDMCRDmVDuy+mK58QAAjcMP/jVRTBSRTJNkhGkGCiL5
> HbRJUsslKXp61BqL4LnLQ26ELq+YPVvG/ds/7wKAaaQrWv8RZyb+L38J0tI6
> s9C25Bo+PwheTJHmnTjrL4m0MiEWsiKGHCkc/x98YMla//UHoooE1X1QtJ4w
> pGNcuF4rgCIoZs8Q8Gd2cIuZWgZUhHNK5TEv866YVXcH1YxkVCKFqen3JD0/
> 8gtcPv9uw0Try4q584TUjZEEw/7SSy+FAcYDsF84ttUvalwWNcGX/1KR9tf+
> UVokNxuqpJNEsMqyF+UYg9xFCiBdIbR2KDRd7UVtnJLnfbJd4IFOejxV6x/g
> MEHFnzckK1pwIT24wEOUedyMELbbPJOfzbw2YzIZH6HAfLi/2a92vlAaW4sy
> rVAEDBbC+DuXIxhY4DdTw63S4ZS1Tphl8DJmd/PBLMlu9vFS7lay1pZsQ3bQ
> j7PWhb8h+a2us74xHUfHxd4AoFbRfm5s5EYnlZH/9ho8HiY9vubK3R8ID+ND
> qYTxLHDSkdccw+AsE+vTXoAFl9vx4nF8F2XQcXlMFjlZ9dmgrVjDIgLbQogb
> cQI1CunsqwLcYxaPl6W6JLWlEENwagdoSfTDjSZQud8MEQ/LTKipAab2tdmK
> vD3M9HXyb7oKWrdDfcAhSe/TGcMk8RfaVlPKq3FTVTyhcUQyhKIoNb3iKXoX
> PDCa
> =Zzq9
> -----END PGP SIGNATURE-----
> ----------------
> Robert LeBlanc
> PGP Fingerprint 79A2 9CA4 6CC4 45DD A904  C70E E654 3BB2 FA62 B9F1
>
>
> On Thu, Apr 28, 2016 at 5:56 AM, Leon Romanovsky <leon-DgEjT+Ai2ygdnm+yROfE0A@public.gmane.org> wrote:
>> On Thu, Apr 28, 2016 at 02:17:29PM +0300, Sagi Grimberg wrote:
>>>
>>> >What is the status of getting RXE included in the mainline kernel? I
>>> >saw a post from Kamal Heib on 22 Sep 2015 with a patchset for
>>> >inclusion [0], but there didn't seem to be any discussion about it. I
>>> >diffed the kernel tree with the RXE tree on GitHub using the
>>> >rxe_submission_v8 branch [1], but there was over 10 million lines in
>>> >the patch file for the same kernel version. I'd rather patch into the
>>> >vanilla kernel tree so that I can see the differences rather than just
>>> >building the tree that someone else provides.
>>> >
>>> >I will probably go through all 32 patches anyway, but thought I would
>>> >prod here about the status as it can be very helpful for
>>> >compatibility.
>>>
>>> FWIW, I'd love to see ib_rxe upstream asap. In fact, I even have
>>> some patches to contribute for it.
>>
>> We are planning on submitting it in very near future.
>>
>>>
>>> Cheers,
>>> Sagi.
>>> --
>>> To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
>>> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Status of RXE/Soft-RoCE driver?

[Holger Hoffstätte]

On Thu, 28 Apr 2016 17:39:23 -0600, Robert LeBlanc wrote:

> Looks like quite a bit has changed especially with
> b7d3e0a94fe128912bbebf0ae68551c85fd2d429 causing those patches to not
> apply cleanly. I'm trying to get this to work on 4.5+, but having
> trouble getting this all figured out (not being a programmer and not
> extremely familiar with the inner workings of RDMA. If someone has a
> newer patch set, I'd like to do some testing.

I have no idea whether it works, but the rxe_submission_v8 branch in
the Github upstream repo (https://github.com/SoftRoCE/rxe-dev) looks
up-to-date and is based on top of kernel 4.5, so that will probably be
the easiest path to get anywhere. I haven't tried myself yet, so good
luck. :)

-h

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Status of RXE/Soft-RoCE driver?

[Doug Ledford]

[-- Attachment #1: Type: text/plain, Size: 1421 bytes --]

On 04/28/2016 10:01 AM, Or Gerlitz wrote:
> On Thu, Apr 28, 2016 at 2:56 PM, Leon Romanovsky <leon-DgEjT+Ai2ygdnm+yROfE0A@public.gmane.org> wrote:
>> On Thu, Apr 28, 2016 at 02:17:29PM +0300, Sagi Grimberg wrote:
> 
>>> FWIW, I'd love to see ib_rxe upstream asap. In fact, I even have
>>> some patches to contribute for it.
>>
>> We are planning on submitting it in very near future.
> 
> So.. lets get this going, Linus made a comment [1] that we might not
> go up even till rc7, which means there very little time left, Moni?
> 
> On a related note, is there ANY patch merged for 4.7? Doug, are we
> again going to have have it in
> all-patches-merged-on-the-same-night-10-days-after-the-merge-window-opened
> manner?

I have topic branches started.  The generic RDMA READ/WRITE API is
almost ready to go once Christoph answers the last few things.  That's
really the only one in queue right now that's big in terms of core
changes, most of the rest of the stuff is all driver or ULP specific.
The RSS patches haven't got much review yet.  The SELinux patches are in
RFC stage, and IMO should be shelved until verbs 2.0 API is settled so
we only have one API to apply it too,  There doesn't appear to be too
much else on the list that would likely effect anyone's patch submissions.

-- 
Doug Ledford <dledford-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>
              GPG KeyID: 0E572FDD



[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 884 bytes --]

Re: Status of RXE/Soft-RoCE driver?

[Leon Romanovsky]

[-- Attachment #1: Type: text/plain, Size: 1772 bytes --]

On Thu, Apr 28, 2016 at 10:09:15PM -0400, Doug Ledford wrote:
> On 04/28/2016 10:01 AM, Or Gerlitz wrote:
> > On Thu, Apr 28, 2016 at 2:56 PM, Leon Romanovsky <leon-DgEjT+Ai2ygdnm+yROfE0A@public.gmane.org> wrote:
> >> On Thu, Apr 28, 2016 at 02:17:29PM +0300, Sagi Grimberg wrote:
> > 
> >>> FWIW, I'd love to see ib_rxe upstream asap. In fact, I even have
> >>> some patches to contribute for it.
> >>
> >> We are planning on submitting it in very near future.
> > 
> > So.. lets get this going, Linus made a comment [1] that we might not
> > go up even till rc7, which means there very little time left, Moni?
> > 
> > On a related note, is there ANY patch merged for 4.7? Doug, are we
> > again going to have have it in
> > all-patches-merged-on-the-same-night-10-days-after-the-merge-window-opened
> > manner?
> 
> I have topic branches started.  The generic RDMA READ/WRITE API is
> almost ready to go once Christoph answers the last few things.  That's
> really the only one in queue right now that's big in terms of core
> changes, most of the rest of the stuff is all driver or ULP specific.
> The RSS patches haven't got much review yet.

Doug,
These patches are long time here in mailing list and they were reviewed as
all other patches in this mailing list.

> The SELinux patches are in RFC stage, and IMO should be shelved until
> verbs 2.0 API is settled so we only have one API to apply it too,

SELinux code adds hooks which are deep in IB/core code and doesn't
play with ABI at all. This is unrelated to ABI change initiative which
anyhow will be required to support existing IBTA specification.

> -- 
> Doug Ledford <dledford-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>
>               GPG KeyID: 0E572FDD
> 
> 



[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: Status of RXE/Soft-RoCE driver?

[Doug Ledford]

[-- Attachment #1: Type: text/plain, Size: 3086 bytes --]

On 04/29/2016 01:17 AM, Leon Romanovsky wrote:
> On Thu, Apr 28, 2016 at 10:09:15PM -0400, Doug Ledford wrote:
>> On 04/28/2016 10:01 AM, Or Gerlitz wrote:
>>> On Thu, Apr 28, 2016 at 2:56 PM, Leon Romanovsky <leon-DgEjT+Ai2ygdnm+yROfE0A@public.gmane.org> wrote:
>>>> On Thu, Apr 28, 2016 at 02:17:29PM +0300, Sagi Grimberg wrote:
>>>
>>>>> FWIW, I'd love to see ib_rxe upstream asap. In fact, I even have
>>>>> some patches to contribute for it.
>>>>
>>>> We are planning on submitting it in very near future.
>>>
>>> So.. lets get this going, Linus made a comment [1] that we might not
>>> go up even till rc7, which means there very little time left, Moni?
>>>
>>> On a related note, is there ANY patch merged for 4.7? Doug, are we
>>> again going to have have it in
>>> all-patches-merged-on-the-same-night-10-days-after-the-merge-window-opened
>>> manner?
>>
>> I have topic branches started.  The generic RDMA READ/WRITE API is
>> almost ready to go once Christoph answers the last few things.  That's
>> really the only one in queue right now that's big in terms of core
>> changes, most of the rest of the stuff is all driver or ULP specific.
>> The RSS patches haven't got much review yet.
> 
> Doug,
> These patches are long time here in mailing list and they were reviewed as
> all other patches in this mailing list.

That's not true.  The RSS patches in particular tend to get no review.
No one appears to care about them.  V1 was posted way back last year and
no one had anything to say about them.  I don't know when V2 was, it's
no longer in my mailbox.  Then v3 has been on the list for 12 days and
again, no one has responded.  No review at all does not mean
automatically accepted.

My slowness on the patches is that you are doing this in an obviously
device specific way.  The only device that will support RSS this way is
mlx5 devices.  But, I'm wondering if there isn't a more generic way to
do this that can be done in the core or in the driver during WQE
processing.  Maybe we can scale performance by having a more
multithreaded process approach without resorting to firmware specific
effects.  If that were the case, that would be my preferred way to go.
Until I have the spare time to investigate if this approach is possible,
it leaves these patches in limbo land.

>> The SELinux patches are in RFC stage, and IMO should be shelved until
>> verbs 2.0 API is settled so we only have one API to apply it too,
> 
> SELinux code adds hooks which are deep in IB/core code and doesn't
> play with ABI at all.

They snoop ABI.  If they ABI changes, they would possibly need to be
changed in order to continue to be able to read the elements.

 This is unrelated to ABI change initiative which
> anyhow will be required to support existing IBTA specification.
> 
>> -- 
>> Doug Ledford <dledford-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>
>>               GPG KeyID: 0E572FDD
>>
>>
> 
> 


-- 
Doug Ledford <dledford-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>
              GPG KeyID: 0E572FDD



[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 884 bytes --]

Re: Status of RXE/Soft-RoCE driver?

[Leon Romanovsky]

[-- Attachment #1: Type: text/plain, Size: 3721 bytes --]

On Fri, Apr 29, 2016 at 11:06:46AM -0400, Doug Ledford wrote:
> On 04/29/2016 01:17 AM, Leon Romanovsky wrote:
> > On Thu, Apr 28, 2016 at 10:09:15PM -0400, Doug Ledford wrote:
> >> On 04/28/2016 10:01 AM, Or Gerlitz wrote:
> >>> On Thu, Apr 28, 2016 at 2:56 PM, Leon Romanovsky <leon-DgEjT+Ai2ygdnm+yROfE0A@public.gmane.org> wrote:
> >>>> On Thu, Apr 28, 2016 at 02:17:29PM +0300, Sagi Grimberg wrote:
> >>>
> >>>>> FWIW, I'd love to see ib_rxe upstream asap. In fact, I even have
> >>>>> some patches to contribute for it.
> >>>>
> >>>> We are planning on submitting it in very near future.
> >>>
> >>> So.. lets get this going, Linus made a comment [1] that we might not
> >>> go up even till rc7, which means there very little time left, Moni?
> >>>
> >>> On a related note, is there ANY patch merged for 4.7? Doug, are we
> >>> again going to have have it in
> >>> all-patches-merged-on-the-same-night-10-days-after-the-merge-window-opened
> >>> manner?
> >>
> >> I have topic branches started.  The generic RDMA READ/WRITE API is
> >> almost ready to go once Christoph answers the last few things.  That's
> >> really the only one in queue right now that's big in terms of core
> >> changes, most of the rest of the stuff is all driver or ULP specific.
> >> The RSS patches haven't got much review yet.
> > 
> > Doug,
> > These patches are long time here in mailing list and they were reviewed as
> > all other patches in this mailing list.
> 
> That's not true.  The RSS patches in particular tend to get no review.
> No one appears to care about them.  V1 was posted way back last year and
> no one had anything to say about them.  I don't know when V2 was, it's
> no longer in my mailbox.  Then v3 has been on the list for 12 days and
> again, no one has responded.  No review at all does not mean
> automatically accepted.

I'm a little bit confused here, do I need to stop reviewing other code?
So it won't be counted as "interesting".

For example, there is no many interest in hns, NES and i40iw code, what
do we do here? Maybe we should stop to accept it too?

> 
> My slowness on the patches is that you are doing this in an obviously
> device specific way.  The only device that will support RSS this way is
> mlx5 devices.  But, I'm wondering if there isn't a more generic way to
> do this that can be done in the core or in the driver during WQE
> processing.  Maybe we can scale performance by having a more
> multithreaded process approach without resorting to firmware specific
> effects.  If that were the case, that would be my preferred way to go.
> Until I have the spare time to investigate if this approach is possible,
> it leaves these patches in limbo land.

What did you stop from express your point before?

> 
> >> The SELinux patches are in RFC stage, and IMO should be shelved until
> >> verbs 2.0 API is settled so we only have one API to apply it too,
> > 
> > SELinux code adds hooks which are deep in IB/core code and doesn't
> > play with ABI at all.
> 
> They snoop ABI.  If they ABI changes, they would possibly need to be
> changed in order to continue to be able to read the elements.

Why is it different from other IB/core code? Once it will be needed, it will be
updated together with other code.

> 
>  This is unrelated to ABI change initiative which
> > anyhow will be required to support existing IBTA specification.
> > 
> >> -- 
> >> Doug Ledford <dledford-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>
> >>               GPG KeyID: 0E572FDD
> >>
> >>
> > 
> > 
> 
> 
> -- 
> Doug Ledford <dledford-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>
>               GPG KeyID: 0E572FDD
> 
> 



[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: Status of RXE/Soft-RoCE driver?

[Robert LeBlanc]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

Holger,

Apparently I wasn't properly subscribed to the list (fixed now). I
started by looking at the rxe-dev GitHub repo and copying out the
appropriate files and patching the other files. There was still some
tweaking needed to get it compiled and I then got lost in the
differences between the patches submitted to linux-rdma and what was
in the repo. Then the 4.5 kernel in the repo (possibly based on
linux-next or something) was very different from 4.5 in Linus' repo so
I started going down trying to patch the linux-rdma into that tree.
I'll go back to what I was doing originally. Thanks for the reply.
-----BEGIN PGP SIGNATURE-----
Version: Mailvelope v1.3.6
Comment: https://www.mailvelope.com

wsFcBAEBCAAQBQJXI5NTCRDmVDuy+mK58QAAA7YP/RR7TRIkCtji0aaLOCoV
1dPQcFocY0krCDJ0epWxx2JYqogWIfx4ldu/zhESqOjAjeLyLzgsv+Yi5hYm
5vxpg/5FaGf9hd0EUpnbUroqOgqREG82Wk8QFtZ++ccnyr0dGbKqI75O8q7X
jrSexAfjvUD9kUp7A01e2CWppa0lt3jPD/IHkLzA5MXZl6XPsmiO8X7pJHg+
Lx86VTQ0dL1FuGxNoiH7voNJVFpG+LaMSFZQkYB63wB92iWDDyJoi2XiCG5j
AvELc0sY5Km6d9PUxa/4b6mhJlZgeOUIHfXfID2iduHNn3XIPr8sGTehLMdM
6M2/Ag4fWcRfBFn7xyD/ZPd2YTidOq4zgnrs0VzvqOe2EhQHsikIFn1FkqbM
hg0XrrJE03LLEIB6+cEdJrQQf3cn6qNHLzKR1d/LSCyaPDn4XiAdARPex/+d
mHgqGHTd+XJRrmPvftOidfQr3Y8qoegSIP5kur61w5ucvTsAMZ/xoxjZlCVE
4J8HNvyFZtvaj1qmABgwyT/I2jeGzW1dQ7dXTQQia/VmBkvctSzkik9QyCzk
1xJQaHRul8MPycGcpDjuCmbkzK2dLTEM16uQvEwCon+F1xDd4/9hONX9amP3
IY/rDyA48J/Y1HK3CvzfwtQTySr4YFy+5UlaCwONaUNR6aBGxnTdWm2e1+pA
n/lt
=JPji
-----END PGP SIGNATURE-----
----------------
Robert LeBlanc
PGP Fingerprint 79A2 9CA4 6CC4 45DD A904  C70E E654 3BB2 FA62 B9F1


On Thu, Apr 28, 2016 at 5:39 PM, Robert LeBlanc <robert-4JaGZRWAfWbajFs6igw21g@public.gmane.org> wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA256
>
> Looks like quite a bit has changed especially with
> b7d3e0a94fe128912bbebf0ae68551c85fd2d429 causing those patches to not
> apply cleanly. I'm trying to get this to work on 4.5+, but having
> trouble getting this all figured out (not being a programmer and not
> extremely familiar with the inner workings of RDMA. If someone has a
> newer patch set, I'd like to do some testing.
> -----BEGIN PGP SIGNATURE-----
> Version: Mailvelope v1.3.6
> Comment: https://www.mailvelope.com
>
> wsFcBAEBCAAQBQJXIp8mCRDmVDuy+mK58QAAAwgP+gPby0h66AIYDfzCXv7v
> s+enVrSSCynFxnRALlH5BJjdOZFOaaPQ+K3S9d4w6y8vvszaAdWKJmtabtk2
> jBBQigz7sUI9LjP6Te1VV7e1tjL/b2C/7ieNWCXuZUIQe11ImiaMCYWwwClI
> OsBHYyZTvyW+VjMTNYwmBTUL5mI4yt5MUP0Izj3ZUeQmuhRxqRm3w7ywSjGX
> +8lmFU7Z/oCA8SWIJX5yY0QYfNOLeh/cMPVvXgZwKPnRqqsnO33bGT842UJU
> aQE1YPuLsqFsLTnaVPgqEJAuo2loYdOpCKgIj4MlDu/QSXiOlfBqVDw5DU+c
> Ldj4nIUq4OOdnm9D0IcbgFxSHV17iRhmgaH8oCd3S4lQCtV8FGmS3AN8pVjq
> UMIFIibchEwBI+Q/Xt4Nc00ccU3Cup3jvpWRZw4FXz9GiQQSUPAsdZLMT+Io
> 9dDA7hLCgdeWe8pIjeyqfB8nWxWKppu01Dk4Yyxd08KSoYFa0Lj2wgGOyJqp
> RJ3LxI8IX7eKTgk7Vf/IqorEyka9/KpDNgLxxT2nL6EFKArb626DFgWK+YLG
> 91gocctxebiGEZ6mDAPdMCEYvEyY5KyG1d02Chyl3uiFqJWZMFSwGr+S1Bqa
> WelDLBvjp8xWiqFduZMWv1Z2RibCbh2McsfNXDI5eSU4JyPGhTUBtFddU1Cz
> LcMQ
> =8ZMY
> -----END PGP SIGNATURE-----
> ----------------
> Robert LeBlanc
> PGP Fingerprint 79A2 9CA4 6CC4 45DD A904  C70E E654 3BB2 FA62 B9F1
>
>
> On Thu, Apr 28, 2016 at 9:48 AM, Robert LeBlanc <robert-4JaGZRWAfWbajFs6igw21g@public.gmane.org> wrote:
>> -----BEGIN PGP SIGNED MESSAGE-----
>> Hash: SHA256
>>
>> Cool, thanks for the update.
>> -----BEGIN PGP SIGNATURE-----
>> Version: Mailvelope v1.3.6
>> Comment: https://www.mailvelope.com
>>
>> wsFcBAEBCAAQBQJXIjDMCRDmVDuy+mK58QAAjcMP/jVRTBSRTJNkhGkGCiL5
>> HbRJUsslKXp61BqL4LnLQ26ELq+YPVvG/ds/7wKAaaQrWv8RZyb+L38J0tI6
>> s9C25Bo+PwheTJHmnTjrL4m0MiEWsiKGHCkc/x98YMla//UHoooE1X1QtJ4w
>> pGNcuF4rgCIoZs8Q8Gd2cIuZWgZUhHNK5TEv866YVXcH1YxkVCKFqen3JD0/
>> 8gtcPv9uw0Try4q584TUjZEEw/7SSy+FAcYDsF84ttUvalwWNcGX/1KR9tf+
>> UVokNxuqpJNEsMqyF+UYg9xFCiBdIbR2KDRd7UVtnJLnfbJd4IFOejxV6x/g
>> MEHFnzckK1pwIT24wEOUedyMELbbPJOfzbw2YzIZH6HAfLi/2a92vlAaW4sy
>> rVAEDBbC+DuXIxhY4DdTw63S4ZS1Tphl8DJmd/PBLMlu9vFS7lay1pZsQ3bQ
>> j7PWhb8h+a2us74xHUfHxd4AoFbRfm5s5EYnlZH/9ho8HiY9vubK3R8ID+ND
>> qYTxLHDSkdccw+AsE+vTXoAFl9vx4nF8F2XQcXlMFjlZ9dmgrVjDIgLbQogb
>> cQI1CunsqwLcYxaPl6W6JLWlEENwagdoSfTDjSZQud8MEQ/LTKipAab2tdmK
>> vD3M9HXyb7oKWrdDfcAhSe/TGcMk8RfaVlPKq3FTVTyhcUQyhKIoNb3iKXoX
>> PDCa
>> =Zzq9
>> -----END PGP SIGNATURE-----
>> ----------------
>> Robert LeBlanc
>> PGP Fingerprint 79A2 9CA4 6CC4 45DD A904  C70E E654 3BB2 FA62 B9F1
>>
>>
>> On Thu, Apr 28, 2016 at 5:56 AM, Leon Romanovsky <leon-DgEjT+Ai2ygdnm+yROfE0A@public.gmane.org> wrote:
>>> On Thu, Apr 28, 2016 at 02:17:29PM +0300, Sagi Grimberg wrote:
>>>>
>>>> >What is the status of getting RXE included in the mainline kernel? I
>>>> >saw a post from Kamal Heib on 22 Sep 2015 with a patchset for
>>>> >inclusion [0], but there didn't seem to be any discussion about it. I
>>>> >diffed the kernel tree with the RXE tree on GitHub using the
>>>> >rxe_submission_v8 branch [1], but there was over 10 million lines in
>>>> >the patch file for the same kernel version. I'd rather patch into the
>>>> >vanilla kernel tree so that I can see the differences rather than just
>>>> >building the tree that someone else provides.
>>>> >
>>>> >I will probably go through all 32 patches anyway, but thought I would
>>>> >prod here about the status as it can be very helpful for
>>>> >compatibility.
>>>>
>>>> FWIW, I'd love to see ib_rxe upstream asap. In fact, I even have
>>>> some patches to contribute for it.
>>>
>>> We are planning on submitting it in very near future.
>>>
>>>>
>>>> Cheers,
>>>> Sagi.
>>>> --
>>>> To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
>>>> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
>>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Status of RXE/Soft-RoCE driver?

[Jason Gunthorpe]

On Fri, Apr 29, 2016 at 11:06:46AM -0400, Doug Ledford wrote:

> > These patches are long time here in mailing list and they were reviewed as
> > all other patches in this mailing list.
> 
> That's not true.  The RSS patches in particular tend to get no review.
> No one appears to care about them.  V1 was posted way back last year and
> no one had anything to say about them.  I don't know when V2 was, it's
> no longer in my mailbox.  Then v3 has been on the list for 12 days and
> again, no one has responded.  No review at all does not mean
> automatically accepted.

During the Collab summit I thought we reached a rough consensus that
these sorts of uAPI changes to the common multi-vendor API would go
through a more rigorous process and those who are proposing them would
actively seek a multi-vendor sign off instead of simply dumping them
on the list. Liran nominated himself and the OFVWG to act as a
forum. I recommend we let that process get going and do it's work.

Further, I also thought we reached a rough consensus that we'd shunt
some of this stuff to the driver specific uAPI channel to reduce this
continual churn on the common multi-vendor API for currently
driver-specific features. Drivers can implement their unique features
and sanely export them through the kernel without so much pain. The
uAPI 2.0 concepts from all parties have been heavily influenced by
this idea.

Certainly, if patches cannot attract any review or interest from the
community as part of the core API then I'd say that is an excellent
sign they should be shunted to a driver specific channel.

There is more flexability in user space to make mistakes with the
libibverbs/etc API, and more options to provide access to these kind
of services. For instance I really wonder if integrating into
libibverbs is really the right home to trial a broad range of brand
new UD offload focused features.

> >> verbs 2.0 API is settled so we only have one API to apply it too,
> > 
> > SELinux code adds hooks which are deep in IB/core code and doesn't
> > play with ABI at all.
> 
> They snoop ABI.  If they ABI changes, they would possibly need to be
> changed in order to continue to be able to read the elements.

Agree, if we go to a object based uAPI then there may be better ways
to make use of selinux right at the uapi layer.

Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Status of RXE/Soft-RoCE driver?

[Leon Romanovsky]

[-- Attachment #1: Type: text/plain, Size: 3111 bytes --]

On Fri, Apr 29, 2016 at 02:54:34PM -0600, Jason Gunthorpe wrote:
> On Fri, Apr 29, 2016 at 11:06:46AM -0400, Doug Ledford wrote:
> 
> > > These patches are long time here in mailing list and they were reviewed as
> > > all other patches in this mailing list.
> > 
> > That's not true.  The RSS patches in particular tend to get no review.
> > No one appears to care about them.  V1 was posted way back last year and
> > no one had anything to say about them.  I don't know when V2 was, it's
> > no longer in my mailbox.  Then v3 has been on the list for 12 days and
> > again, no one has responded.  No review at all does not mean
> > automatically accepted.
> 
> During the Collab summit I thought we reached a rough consensus that
> these sorts of uAPI changes to the common multi-vendor API would go
> through a more rigorous process and those who are proposing them would
> actively seek a multi-vendor sign off instead of simply dumping them
> on the list. Liran nominated himself and the OFVWG to act as a
> forum. I recommend we let that process get going and do it's work.
> 
> Further, I also thought we reached a rough consensus that we'd shunt
> some of this stuff to the driver specific uAPI channel to reduce this
> continual churn on the common multi-vendor API for currently
> driver-specific features. Drivers can implement their unique features
> and sanely export them through the kernel without so much pain. The
> uAPI 2.0 concepts from all parties have been heavily influenced by
> this idea.
> 
> Certainly, if patches cannot attract any review or interest from the
> community as part of the core API then I'd say that is an excellent
> sign they should be shunted to a driver specific channel.
> 
> There is more flexability in user space to make mistakes with the
> libibverbs/etc API, and more options to provide access to these kind
> of services. For instance I really wonder if integrating into
> libibverbs is really the right home to trial a broad range of brand
> new UD offload focused features.

Jason,
It is **respin** of patches which were with us for **more than year**, it is
enough time to all participants and maintainer to say their word.

> 
> > >> verbs 2.0 API is settled so we only have one API to apply it too,
> > > 
> > > SELinux code adds hooks which are deep in IB/core code and doesn't
> > > play with ABI at all.
> > 
> > They snoop ABI.  If they ABI changes, they would possibly need to be
> > changed in order to continue to be able to read the elements.
> 
> Agree, if we go to a object based uAPI then there may be better ways
> to make use of selinux right at the uapi layer.

From user perspective, it will be the same, so please name me technical
reason do not proceed with them now.

SELinux patches are contributed by open source company who is actively
supporting IB stack by code and by following IBTA specification.

As I said before and continue to say again, we will invest time and effort to
update all needed pieces of our code to support new ABI whenever it comes.

> 
> Jason

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: Status of RXE/Soft-RoCE driver?

[Yishai Hadas]

On 4/30/2016 10:28 AM, Leon Romanovsky wrote:
> On Fri, Apr 29, 2016 at 02:54:34PM -0600, Jason Gunthorpe wrote:
>> On Fri, Apr 29, 2016 at 11:06:46AM -0400, Doug Ledford wrote:
>>
>>>> These patches are long time here in mailing list and they were reviewed as
>>>> all other patches in this mailing list.
>>>
>>> That's not true.  The RSS patches in particular tend to get no review.
>>> No one appears to care about them.  V1 was posted way back last year and
>>> no one had anything to say about them.  I don't know when V2 was, it's
>>> no longer in my mailbox.  Then v3 has been on the list for 12 days and
>>> again, no one has responded.  No review at all does not mean
>>> automatically accepted.
>>
>> During the Collab summit I thought we reached a rough consensus that
>> these sorts of uAPI changes to the common multi-vendor API would go
>> through a more rigorous process and those who are proposing them would
>> actively seek a multi-vendor sign off instead of simply dumping them
>> on the list. Liran nominated himself and the OFVWG to act as a
>> forum. I recommend we let that process get going and do it's work.
>>
>> Further, I also thought we reached a rough consensus that we'd shunt
>> some of this stuff to the driver specific uAPI channel to reduce this
>> continual churn on the common multi-vendor API for currently
>> driver-specific features. Drivers can implement their unique features
>> and sanely export them through the kernel without so much pain. The
>> uAPI 2.0 concepts from all parties have been heavily influenced by
>> this idea.
>>
>> Certainly, if patches cannot attract any review or interest from the
>> community as part of the core API then I'd say that is an excellent
>> sign they should be shunted to a driver specific channel.
>>
>> There is more flexability in user space to make mistakes with the
>> libibverbs/etc API, and more options to provide access to these kind
>> of services. For instance I really wonder if integrating into
>> libibverbs is really the right home to trial a broad range of brand
>> new UD offload focused features.
>
> Jason,
> It is **respin** of patches which were with us for **more than year**, it is
> enough time to all participants and maintainer to say their word.
>

Hi Doug,

We've gone through the following stages for this RSS patch set already:
1) First series of patch set for RSS verbs was submitted at 8 May 2012: 
http://www.spinics.net/lists/linux-rdma/msg11746.html.
2) After *long* discussions with the community, and after addressing the 
community comments both on the mailing list and f2f discussions at OFA 
developers workshop at 2012,  we built an abstract and generic new API.
3) In order to get wider acceptance to the new proposal *before* we 
implement, we've posted an RFC at Mon, 11 May 2015, a year (!) ago
4) Additionally, The API was presented by Liran and AlexV from Mellanox 
in OFAWG meeting at Jun 9 2015. Link to the invite: 
http://lists.openfabrics.org/pipermail/ofvwg/2015-June/000077.html, Link 
to the presentation: 
http://lists.openfabrics.org/pipermail/ofvwg/attachments/20150609/30f73a1d/attachment-0001.pdf
5) Following discussion over the mailing list and responding to all 
feedback we've received, presentation at OFVWG we moved forward and 
invested large effort to implement and posted the patch series at Oct 18 
2015: https://www.mail-archive.com/linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org/msg28611.html
6) The new API was presented once more by Tzahi from Mellanox at last 
OFA 2016 workshop in the summit as part of the User Mode Ethernet Verbs 
presentation: 
https://www.openfabrics.org/images/eventpresos/2016presentations/205EthernetVerbs.pdf
7) Following more discussion at the 2016 workshop we were asked to 
resubmit V3 patch-set. We did so at 17 April 2016: 
http://www.spinics.net/lists/linux-rdma/msg35119.html

The above stages are way over the latest collaboration summit agreed 
guidelines for submitting new functionality and enriching the API. There 
is not technical feedback on the table that we left behind and did not 
address to make the API vendor agnostic and abstract.

Please accept.
Yishai
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Status of RXE/Soft-RoCE driver?

[Jason Gunthorpe]

On Mon, May 02, 2016 at 01:03:12PM +0300, Yishai Hadas wrote:
 
> The above stages are way over the latest collaboration summit agreed
> guidelines for submitting new functionality and enriching the API.

The key take away from the collab discussion was the changes to the
common uAPI needs *multi-vendor* support.

All I see is silence from everyone, and a bunch of patches that 
nobody wants to review. That isn't good enough.

This is especially critical for RSS because it adds all sorts of new
verbs objects that need hardware mappings! I see no formal
specification for this, no IBTA/IETF support, and no voting.

The other take away from the Collab discussion is that there is a
reasonable opinion that linux-rdma is not the right forum to decide if
major changes to the multi-vendor common-verbs APIs are
OK. I don't know if we reached a consensus on this or not.

Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: Status of RXE/Soft-RoCE driver?

[Hefty, Sean]

> During the Collab summit I thought we reached a rough consensus that
> these sorts of uAPI changes to the common multi-vendor API would go
> through a more rigorous process and those who are proposing them would
> actively seek a multi-vendor sign off instead of simply dumping them
> on the list.

I agree that this is the right approach to follow.  A Linux maintainer should not be in the position of having to decide on which features merit a uAPI change.

> Further, I also thought we reached a rough consensus that we'd shunt
> some of this stuff to the driver specific uAPI channel to reduce this
> continual churn on the common multi-vendor API for currently
> driver-specific features. Drivers can implement their unique features
> and sanely export them through the kernel without so much pain. The
> uAPI 2.0 concepts from all parties have been heavily influenced by
> this idea.
> 
> Certainly, if patches cannot attract any review or interest from the
> community as part of the core API then I'd say that is an excellent
> sign they should be shunted to a driver specific channel.

This is an excellent point.  The lack of interest is likely a very good sign that this is a driver specific feature.

I'm not sure why there seems to be objection to calling out driver specific features as driver specific and exposing them as such.  It seems like the quickest way to get features into the hands of prospective users.

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Status of RXE/Soft-RoCE driver?

[Christoph Hellwig]

On Mon, May 02, 2016 at 11:57:09AM -0600, Jason Gunthorpe wrote:
> The other take away from the Collab discussion is that there is a
> reasonable opinion that linux-rdma is not the right forum to decide if
> major changes to the multi-vendor common-verbs APIs are
> OK. I don't know if we reached a consensus on this or not.

Who had that discussion, and why do you believe it's viable?  In fact
I'd like to state that any other forum than linux-rdma simply is not
credible, just like the relevant list is the only credible place to
discuss the ABIs for any other Linux subsystem.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Status of RXE/Soft-RoCE driver?

[Jason Gunthorpe]

On Mon, May 02, 2016 at 11:31:51AM -0700, Christoph Hellwig wrote:
> On Mon, May 02, 2016 at 11:57:09AM -0600, Jason Gunthorpe wrote:
> > The other take away from the Collab discussion is that there is a
> > reasonable opinion that linux-rdma is not the right forum to decide if
> > major changes to the multi-vendor common-verbs APIs are
> > OK. I don't know if we reached a consensus on this or not.
> 
> Who had that discussion, and why do you believe it's viable?  In fact
> I'd like to state that any other forum than linux-rdma simply is not
> credible, just like the relevant list is the only credible place to
> discuss the ABIs for any other Linux subsystem.

I would say mostly from people looking at it from a hardware design
perspective.

The unique issue with verbs is that nearly all the APIs match directly
to some feature in silicon, and cross-vendor silicon focused agreements
are rarely done on Linux focused mailing lists. Particularly when
these features are expected to be multi-operating system.

That is my fundamental concern every time a uAPI change comes up:
These proposed API changes *directly* require other vendors to
implement very specific things in their sillicon. This is not a
software-only discussion, as the majority of other Linux uAPI things
are.

* And I specifically separate the nitty gritty details of the API/ABI
from the overarching ideas: eg, introducing a dis-aggregated QP
concept with WQ objects, and defining it how that interacts with
everything else.

Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: Status of RXE/Soft-RoCE driver?

[Hefty, Sean]

> > The other take away from the Collab discussion is that there is a
> > reasonable opinion that linux-rdma is not the right forum to decide if
> > major changes to the multi-vendor common-verbs APIs are
> > OK. I don't know if we reached a consensus on this or not.
> 
> Who had that discussion, and why do you believe it's viable?  In fact
> I'd like to state that any other forum than linux-rdma simply is not
> credible, just like the relevant list is the only credible place to
> discuss the ABIs for any other Linux subsystem.

The problem is that the linux-rdma stack tries to expose vendor-specific, transport-specific, hardware-implementation specific features to user space under the guise of being a common API.  And many of the items being exported are not defined by any standards body.  Guidance about what's sane needs to come from somewhere other than vendor X posted a patch to change the common API because it suits the latest model of their HW.

No one is suggesting that the linux-rdma developers do not own the ABI or kernel components, or that some external entity (like OFA) has any authority to dictate what needs to be done.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Status of RXE/Soft-RoCE driver?

[Jason Gunthorpe]

On Mon, May 02, 2016 at 11:31:51AM -0700, Christoph Hellwig wrote:
> On Mon, May 02, 2016 at 11:57:09AM -0600, Jason Gunthorpe wrote:
> > The other take away from the Collab discussion is that there is a
> > reasonable opinion that linux-rdma is not the right forum to decide if
> > major changes to the multi-vendor common-verbs APIs are
> > OK. I don't know if we reached a consensus on this or not.
> 
> Who had that discussion, and why do you believe it's viable?  In fact
> I'd like to state that any other forum than linux-rdma simply is not
> credible, just like the relevant list is the only credible place to
> discuss the ABIs for any other Linux subsystem.

Right, and we have been asking around if any other Linux communities
face that same challenge.

Sean identified GPU (DRM). Do you know of others?

Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Status of RXE/Soft-RoCE driver?

[Leon Romanovsky]

[-- Attachment #1: Type: text/plain, Size: 2846 bytes --]

On Mon, May 02, 2016 at 06:29:38PM +0000, Hefty, Sean wrote:
> > During the Collab summit I thought we reached a rough consensus that
> > these sorts of uAPI changes to the common multi-vendor API would go
> > through a more rigorous process and those who are proposing them would
> > actively seek a multi-vendor sign off instead of simply dumping them
> > on the list.
> 
> I agree that this is the right approach to follow.  A Linux maintainer should not be in the position of having to decide on which features merit a uAPI change.

In this specific case, all guides were followed.

> 
> > Further, I also thought we reached a rough consensus that we'd shunt
> > some of this stuff to the driver specific uAPI channel to reduce this
> > continual churn on the common multi-vendor API for currently
> > driver-specific features. Drivers can implement their unique features
> > and sanely export them through the kernel without so much pain. The
> > uAPI 2.0 concepts from all parties have been heavily influenced by
> > this idea.
> > 
> > Certainly, if patches cannot attract any review or interest from the
> > community as part of the core API then I'd say that is an excellent
> > sign they should be shunted to a driver specific channel.
> 
> This is an excellent point.  The lack of interest is likely a very good sign that this is a driver specific feature.

I don't agree with that point, the lack of discussion doesn't mean that
this is driver specific feature. In linux-rdma community, there is a small
amount of developers focused on their products and maintainer who doesn't
express his opinion.

There is general agreement that Mellanox is a leading provider of IB solutions
and as a leader it's expected from the company to propose new exciting
features which are valuable for the community (general API) and for the
customers.

In this particular case (RSS), we followed **all** possible guidelines of
presenting, discussing and adjusting. It was done in **all** available
forums (linux-rdma, OFA and OFVWG).

We did it openly without any hiding under "proprietary umbrella", as
other companies did during review feedback.

> 
> I'm not sure why there seems to be objection to calling out driver specific features as driver specific and exposing them as such.  It seems like the quickest way to get features into the hands of prospective users.

We want it beneficial to all participants without proprietary stuff.
We don't want to see vendor interfaces proposed and implemented without
any review as it was for several drivers in RDMA world.

> 
> --
> To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: Status of RXE/Soft-RoCE driver?

[Robert LeBlanc]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

I was able to pull those files from the repo and build it against
4.6-rc5 just fine. I was able to do some simple tests and verify that
things seemed to work ok. I've been working to get Soft-iWARP building
on the same kernel (that hasn't seen some love in a while). Once I get
that done, I'll be doing some more extensive testing.

Thanks for the help.
-----BEGIN PGP SIGNATURE-----
Version: Mailvelope v1.3.6
Comment: https://www.mailvelope.com

wsFcBAEBCAAQBQJXLNoQCRDmVDuy+mK58QAAuNcP/3DW9K2L1yIcMceG6EvH
wxyywSqMM6+YzJeNMhV0CgJwSuS3IRpogffujrqemR7yPYmqKTxcRKFItKZy
3KAdD7VMUIS9vUSgjCPTcRTONNQpUOgaDguvHf1u+CTnNLp/C5x2La2d5vYQ
0X5TI739MF3coN82vq6mcfWIcOIhfnVWYCsMR28v3nTZOoOjFc0DfIER59FM
7ThvMyHxYxMmKfUIqnztLTpFPA6iQUxSg6et0cu9DpIKCxvdBwhf39c4vnUW
jexEZzPdFB2pOGhkxL4icQ2Z4/MorohbfU77lkohG/ZnwScDtOhx4QKLNonb
vuchowZkFfsSQ7Cjt6TJeaeA7pISwm/3P+WX9H0O3PfVb3qESwMZ3NOG+seM
Y8To4OknDV0Y4ZXQiSjd39qBL8aoh2dYVAkxFwJBLbv1ttYMbq9WmnDmHHoy
L65l8YL8eh4E30lORif4HD5OnffCL+h3rYDYVDBUqOQlacoDFRR7tniJBaxH
fYhWZHO08PH0zDJTB51XLsX/icP6tDjpIFS7ct4/XGVZIOOOAFLs5uH++nzD
zYONK/olCk3e/thBwHWIRhZuShnEQxgPCMTrE4b4nf1X7F9BTm2nvVTkIvJP
y7/YU+zY/qsZhg1KYKUeY3A7dgp4XqAD/GIyzNQ01KgCUQVuLXOPlFOh7miC
wYvu
=1pGt
-----END PGP SIGNATURE-----
----------------
Robert LeBlanc
PGP Fingerprint 79A2 9CA4 6CC4 45DD A904  C70E E654 3BB2 FA62 B9F1


On Fri, Apr 29, 2016 at 11:01 AM, Robert LeBlanc <robert-4JaGZRWAfWbajFs6igw21g@public.gmane.org> wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA256
>
> Holger,
>
> Apparently I wasn't properly subscribed to the list (fixed now). I
> started by looking at the rxe-dev GitHub repo and copying out the
> appropriate files and patching the other files. There was still some
> tweaking needed to get it compiled and I then got lost in the
> differences between the patches submitted to linux-rdma and what was
> in the repo. Then the 4.5 kernel in the repo (possibly based on
> linux-next or something) was very different from 4.5 in Linus' repo so
> I started going down trying to patch the linux-rdma into that tree.
> I'll go back to what I was doing originally. Thanks for the reply.
> -----BEGIN PGP SIGNATURE-----
> Version: Mailvelope v1.3.6
> Comment: https://www.mailvelope.com
>
> wsFcBAEBCAAQBQJXI5NTCRDmVDuy+mK58QAAA7YP/RR7TRIkCtji0aaLOCoV
> 1dPQcFocY0krCDJ0epWxx2JYqogWIfx4ldu/zhESqOjAjeLyLzgsv+Yi5hYm
> 5vxpg/5FaGf9hd0EUpnbUroqOgqREG82Wk8QFtZ++ccnyr0dGbKqI75O8q7X
> jrSexAfjvUD9kUp7A01e2CWppa0lt3jPD/IHkLzA5MXZl6XPsmiO8X7pJHg+
> Lx86VTQ0dL1FuGxNoiH7voNJVFpG+LaMSFZQkYB63wB92iWDDyJoi2XiCG5j
> AvELc0sY5Km6d9PUxa/4b6mhJlZgeOUIHfXfID2iduHNn3XIPr8sGTehLMdM
> 6M2/Ag4fWcRfBFn7xyD/ZPd2YTidOq4zgnrs0VzvqOe2EhQHsikIFn1FkqbM
> hg0XrrJE03LLEIB6+cEdJrQQf3cn6qNHLzKR1d/LSCyaPDn4XiAdARPex/+d
> mHgqGHTd+XJRrmPvftOidfQr3Y8qoegSIP5kur61w5ucvTsAMZ/xoxjZlCVE
> 4J8HNvyFZtvaj1qmABgwyT/I2jeGzW1dQ7dXTQQia/VmBkvctSzkik9QyCzk
> 1xJQaHRul8MPycGcpDjuCmbkzK2dLTEM16uQvEwCon+F1xDd4/9hONX9amP3
> IY/rDyA48J/Y1HK3CvzfwtQTySr4YFy+5UlaCwONaUNR6aBGxnTdWm2e1+pA
> n/lt
> =JPji
> -----END PGP SIGNATURE-----
> ----------------
> Robert LeBlanc
> PGP Fingerprint 79A2 9CA4 6CC4 45DD A904  C70E E654 3BB2 FA62 B9F1
>
>
> On Thu, Apr 28, 2016 at 5:39 PM, Robert LeBlanc <robert-4JaGZRWAfWbajFs6igw21g@public.gmane.org> wrote:
>> -----BEGIN PGP SIGNED MESSAGE-----
>> Hash: SHA256
>>
>> Looks like quite a bit has changed especially with
>> b7d3e0a94fe128912bbebf0ae68551c85fd2d429 causing those patches to not
>> apply cleanly. I'm trying to get this to work on 4.5+, but having
>> trouble getting this all figured out (not being a programmer and not
>> extremely familiar with the inner workings of RDMA. If someone has a
>> newer patch set, I'd like to do some testing.
>> -----BEGIN PGP SIGNATURE-----
>> Version: Mailvelope v1.3.6
>> Comment: https://www.mailvelope.com
>>
>> wsFcBAEBCAAQBQJXIp8mCRDmVDuy+mK58QAAAwgP+gPby0h66AIYDfzCXv7v
>> s+enVrSSCynFxnRALlH5BJjdOZFOaaPQ+K3S9d4w6y8vvszaAdWKJmtabtk2
>> jBBQigz7sUI9LjP6Te1VV7e1tjL/b2C/7ieNWCXuZUIQe11ImiaMCYWwwClI
>> OsBHYyZTvyW+VjMTNYwmBTUL5mI4yt5MUP0Izj3ZUeQmuhRxqRm3w7ywSjGX
>> +8lmFU7Z/oCA8SWIJX5yY0QYfNOLeh/cMPVvXgZwKPnRqqsnO33bGT842UJU
>> aQE1YPuLsqFsLTnaVPgqEJAuo2loYdOpCKgIj4MlDu/QSXiOlfBqVDw5DU+c
>> Ldj4nIUq4OOdnm9D0IcbgFxSHV17iRhmgaH8oCd3S4lQCtV8FGmS3AN8pVjq
>> UMIFIibchEwBI+Q/Xt4Nc00ccU3Cup3jvpWRZw4FXz9GiQQSUPAsdZLMT+Io
>> 9dDA7hLCgdeWe8pIjeyqfB8nWxWKppu01Dk4Yyxd08KSoYFa0Lj2wgGOyJqp
>> RJ3LxI8IX7eKTgk7Vf/IqorEyka9/KpDNgLxxT2nL6EFKArb626DFgWK+YLG
>> 91gocctxebiGEZ6mDAPdMCEYvEyY5KyG1d02Chyl3uiFqJWZMFSwGr+S1Bqa
>> WelDLBvjp8xWiqFduZMWv1Z2RibCbh2McsfNXDI5eSU4JyPGhTUBtFddU1Cz
>> LcMQ
>> =8ZMY
>> -----END PGP SIGNATURE-----
>> ----------------
>> Robert LeBlanc
>> PGP Fingerprint 79A2 9CA4 6CC4 45DD A904  C70E E654 3BB2 FA62 B9F1
>>
>>
>> On Thu, Apr 28, 2016 at 9:48 AM, Robert LeBlanc <robert-4JaGZRWAfWbajFs6igw21g@public.gmane.org> wrote:
>>> -----BEGIN PGP SIGNED MESSAGE-----
>>> Hash: SHA256
>>>
>>> Cool, thanks for the update.
>>> -----BEGIN PGP SIGNATURE-----
>>> Version: Mailvelope v1.3.6
>>> Comment: https://www.mailvelope.com
>>>
>>> wsFcBAEBCAAQBQJXIjDMCRDmVDuy+mK58QAAjcMP/jVRTBSRTJNkhGkGCiL5
>>> HbRJUsslKXp61BqL4LnLQ26ELq+YPVvG/ds/7wKAaaQrWv8RZyb+L38J0tI6
>>> s9C25Bo+PwheTJHmnTjrL4m0MiEWsiKGHCkc/x98YMla//UHoooE1X1QtJ4w
>>> pGNcuF4rgCIoZs8Q8Gd2cIuZWgZUhHNK5TEv866YVXcH1YxkVCKFqen3JD0/
>>> 8gtcPv9uw0Try4q584TUjZEEw/7SSy+FAcYDsF84ttUvalwWNcGX/1KR9tf+
>>> UVokNxuqpJNEsMqyF+UYg9xFCiBdIbR2KDRd7UVtnJLnfbJd4IFOejxV6x/g
>>> MEHFnzckK1pwIT24wEOUedyMELbbPJOfzbw2YzIZH6HAfLi/2a92vlAaW4sy
>>> rVAEDBbC+DuXIxhY4DdTw63S4ZS1Tphl8DJmd/PBLMlu9vFS7lay1pZsQ3bQ
>>> j7PWhb8h+a2us74xHUfHxd4AoFbRfm5s5EYnlZH/9ho8HiY9vubK3R8ID+ND
>>> qYTxLHDSkdccw+AsE+vTXoAFl9vx4nF8F2XQcXlMFjlZ9dmgrVjDIgLbQogb
>>> cQI1CunsqwLcYxaPl6W6JLWlEENwagdoSfTDjSZQud8MEQ/LTKipAab2tdmK
>>> vD3M9HXyb7oKWrdDfcAhSe/TGcMk8RfaVlPKq3FTVTyhcUQyhKIoNb3iKXoX
>>> PDCa
>>> =Zzq9
>>> -----END PGP SIGNATURE-----
>>> ----------------
>>> Robert LeBlanc
>>> PGP Fingerprint 79A2 9CA4 6CC4 45DD A904  C70E E654 3BB2 FA62 B9F1
>>>
>>>
>>> On Thu, Apr 28, 2016 at 5:56 AM, Leon Romanovsky <leon-DgEjT+Ai2ygdnm+yROfE0A@public.gmane.org> wrote:
>>>> On Thu, Apr 28, 2016 at 02:17:29PM +0300, Sagi Grimberg wrote:
>>>>>
>>>>> >What is the status of getting RXE included in the mainline kernel? I
>>>>> >saw a post from Kamal Heib on 22 Sep 2015 with a patchset for
>>>>> >inclusion [0], but there didn't seem to be any discussion about it. I
>>>>> >diffed the kernel tree with the RXE tree on GitHub using the
>>>>> >rxe_submission_v8 branch [1], but there was over 10 million lines in
>>>>> >the patch file for the same kernel version. I'd rather patch into the
>>>>> >vanilla kernel tree so that I can see the differences rather than just
>>>>> >building the tree that someone else provides.
>>>>> >
>>>>> >I will probably go through all 32 patches anyway, but thought I would
>>>>> >prod here about the status as it can be very helpful for
>>>>> >compatibility.
>>>>>
>>>>> FWIW, I'd love to see ib_rxe upstream asap. In fact, I even have
>>>>> some patches to contribute for it.
>>>>
>>>> We are planning on submitting it in very near future.
>>>>
>>>>>
>>>>> Cheers,
>>>>> Sagi.
>>>>> --
>>>>> To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
>>>>> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
>>>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html