dedicated list for rdma userspace programming

dedicated list for rdma userspace programming

[Vasiliy Tolstov]

Hi! Does this mailing list suitable for questions about rdma userspace
programming or i need to find another list?

My question is - how userspace libraries communicate with kernel
devices which provided by kernel infiniband drivers? This is character
devices and userspace libraries read/write to it or userspace library
do some ioctls?

Where i can find info about it?

-- 
Vasiliy Tolstov,
e-mail: v.tolstov-+9FY0jupvH6HXe+LvDLADg@public.gmane.org
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: dedicated list for rdma userspace programming

[Leon Romanovsky]

[-- Attachment #1: Type: text/plain, Size: 1211 bytes --]

On Mon, Apr 24, 2017 at 12:58:53PM +0300, Vasiliy Tolstov wrote:
> Hi! Does this mailing list suitable for questions about rdma userspace
> programming or i need to find another list?

This is the right list.

>
> My question is - how userspace libraries communicate with kernel
> devices which provided by kernel infiniband drivers? This is character
> devices and userspace libraries read/write to it or userspace library
> do some ioctls?

There are number of char devices exported by infiniband subsystem
(/dev/infiniband/*) and the communication is performed with read/write
interfaces to uverbs file [1].

Right now, we are in process of migration from these interfaces to ioctl
based, see Matan's latest RFC [2].

[1] https://github.com/linux-rdma/rdma-core/blob/master/Documentation/libibverbs.md
[2] http://marc.info/?l=linux-rdma&m=149261526916544&w=2

>
> Where i can find info about it?
>
> --
> Vasiliy Tolstov,
> e-mail: v.tolstov-+9FY0jupvH6HXe+LvDLADg@public.gmane.org
> --
> To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: dedicated list for rdma userspace programming

[Vasiliy Tolstov]

2017-04-24 13:42 GMT+03:00 Leon Romanovsky <leon-DgEjT+Ai2ygdnm+yROfE0A@public.gmane.org>:
> There are number of char devices exported by infiniband subsystem
> (/dev/infiniband/*) and the communication is performed with read/write
> interfaces to uverbs file [1].
>
> Right now, we are in process of migration from these interfaces to ioctl
> based, see Matan's latest RFC [2].


Where i can find tree with commits from [2] ?
And how much work needs to be done to get this work upstream? (1
month, 6 months, years...) ?
If i want to try working with uverbs without C librarary, where i can
find some info about format of messages reader and writed to uverbs
device?

-- 
Vasiliy Tolstov,
e-mail: v.tolstov-+9FY0jupvH6HXe+LvDLADg@public.gmane.org
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: dedicated list for rdma userspace programming

[Leon Romanovsky]

[-- Attachment #1: Type: text/plain, Size: 1065 bytes --]

On Mon, Apr 24, 2017 at 02:07:14PM +0300, Vasiliy Tolstov wrote:
> 2017-04-24 13:42 GMT+03:00 Leon Romanovsky <leon-DgEjT+Ai2ygdnm+yROfE0A@public.gmane.org>:
> > There are number of char devices exported by infiniband subsystem
> > (/dev/infiniband/*) and the communication is performed with read/write
> > interfaces to uverbs file [1].
> >
> > Right now, we are in process of migration from these interfaces to ioctl
> > based, see Matan's latest RFC [2].
>
>
> Where i can find tree with commits from [2] ?

https://github.com/matanb10/linux tag: abi-devel-latest

> And how much work needs to be done to get this work upstream? (1
> month, 6 months, years...) ?

At least one kernel cycle, and after transition read/write will be
obsolete and will be dropped.

> If i want to try working with uverbs without C librarary, where i can
> find some info about format of messages reader and writed to uverbs
> device?

Did you see Documentation/infiniband/user_verbs.txt?

Thanks
>
> --
> Vasiliy Tolstov,
> e-mail: v.tolstov-+9FY0jupvH6HXe+LvDLADg@public.gmane.org

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: dedicated list for rdma userspace programming

[Vasiliy Tolstov]

2017-04-24 14:47 GMT+03:00 Leon Romanovsky <leon-DgEjT+Ai2ygdnm+yROfE0A@public.gmane.org>:
> On Mon, Apr 24, 2017 at 02:07:14PM +0300, Vasiliy Tolstov wrote:
>> 2017-04-24 13:42 GMT+03:00 Leon Romanovsky <leon-DgEjT+Ai2ygdnm+yROfE0A@public.gmane.org>:
>> > There are number of char devices exported by infiniband subsystem
>> > (/dev/infiniband/*) and the communication is performed with read/write
>> > interfaces to uverbs file [1].
>> >
>> > Right now, we are in process of migration from these interfaces to ioctl
>> > based, see Matan's latest RFC [2].
>>
>>
>> Where i can find tree with commits from [2] ?
>
> https://github.com/matanb10/linux tag: abi-devel-latest

Thanks!

>
>> And how much work needs to be done to get this work upstream? (1
>> month, 6 months, years...) ?
>
> At least one kernel cycle, and after transition read/write will be
> obsolete and will be dropped.
>

So 4.12-4.13 ?

>> If i want to try working with uverbs without C librarary, where i can
>> find some info about format of messages reader and writed to uverbs
>> device?
>
> Did you see Documentation/infiniband/user_verbs.txt?


https://www.kernel.org/doc/Documentation/infiniband/user_verbs.txt as
i see this nothing says about format of data writed/readed...

-- 
Vasiliy Tolstov,
e-mail: v.tolstov-+9FY0jupvH6HXe+LvDLADg@public.gmane.org
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: dedicated list for rdma userspace programming

[Leon Romanovsky]

[-- Attachment #1: Type: text/plain, Size: 1968 bytes --]

On Mon, Apr 24, 2017 at 02:58:57PM +0300, Vasiliy Tolstov wrote:
> 2017-04-24 14:47 GMT+03:00 Leon Romanovsky <leon-DgEjT+Ai2ygdnm+yROfE0A@public.gmane.org>:
> > On Mon, Apr 24, 2017 at 02:07:14PM +0300, Vasiliy Tolstov wrote:
> >> 2017-04-24 13:42 GMT+03:00 Leon Romanovsky <leon-DgEjT+Ai2ygdnm+yROfE0A@public.gmane.org>:
> >> > There are number of char devices exported by infiniband subsystem
> >> > (/dev/infiniband/*) and the communication is performed with read/write
> >> > interfaces to uverbs file [1].
> >> >
> >> > Right now, we are in process of migration from these interfaces to ioctl
> >> > based, see Matan's latest RFC [2].
> >>
> >>
> >> Where i can find tree with commits from [2] ?
> >
> > https://github.com/matanb10/linux tag: abi-devel-latest
>
> Thanks!
>
> >
> >> And how much work needs to be done to get this work upstream? (1
> >> month, 6 months, years...) ?
> >
> > At least one kernel cycle, and after transition read/write will be
> > obsolete and will be dropped.
> >
>
> So 4.12-4.13 ?

4.12 is almost closed. We are in rc8 now and merge window will be next
week, so it will be 4.13-4.14 (optimistic).

>
> >> If i want to try working with uverbs without C librarary, where i can
> >> find some info about format of messages reader and writed to uverbs
> >> device?
> >
> > Did you see Documentation/infiniband/user_verbs.txt?
>
>
> https://www.kernel.org/doc/Documentation/infiniband/user_verbs.txt as
> i see this nothing says about format of data writed/readed...
>

See in include/uapi/rdma/ib_user_verbs.h, User fills struct from that
file, and kernel performs copy_from_user to extract the data. That's
all.

Thanks


> --
> Vasiliy Tolstov,
> e-mail: v.tolstov-+9FY0jupvH6HXe+LvDLADg@public.gmane.org
> --
> To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: dedicated list for rdma userspace programming

[Vasiliy Tolstov]

2017-04-24 15:34 GMT+03:00 Leon Romanovsky <leon-DgEjT+Ai2ygdnm+yROfE0A@public.gmane.org>:
>> >> And how much work needs to be done to get this work upstream? (1
>> >> month, 6 months, years...) ?
>> >
>> > At least one kernel cycle, and after transition read/write will be
>> > obsolete and will be dropped.
>> >
>>
>> So 4.12-4.13 ?
>
> 4.12 is almost closed. We are in rc8 now and merge window will be next
> week, so it will be 4.13-4.14 (optimistic).
>

Fine. So older interfaces will be deprecated and removed in next few
cycles. I think i can now create something that used old api and do
transition later (i'm use fedora)

>>
>> >> If i want to try working with uverbs without C librarary, where i can
>> >> find some info about format of messages reader and writed to uverbs
>> >> device?
>> >
>> > Did you see Documentation/infiniband/user_verbs.txt?
>>
>>
>> https://www.kernel.org/doc/Documentation/infiniband/user_verbs.txt as
>> i see this nothing says about format of data writed/readed...
>>
>
> See in include/uapi/rdma/ib_user_verbs.h, User fills struct from that
> file, and kernel performs copy_from_user to extract the data. That's
> all.

Thanks! If i need to establish communication between two nodes and
send/recv data (and get notification that data can be readed) does
this file  enough to do that?
(i'm plan to try create plain go binding, now only for fun, create
generator for this C struct/enums to plain go types)


-- 
Vasiliy Tolstov,
e-mail: v.tolstov-+9FY0jupvH6HXe+LvDLADg@public.gmane.org
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: dedicated list for rdma userspace programming

[Leon Romanovsky]

[-- Attachment #1: Type: text/plain, Size: 2005 bytes --]

On Mon, Apr 24, 2017 at 03:45:26PM +0300, Vasiliy Tolstov wrote:
> 2017-04-24 15:34 GMT+03:00 Leon Romanovsky <leon-DgEjT+Ai2ygdnm+yROfE0A@public.gmane.org>:
> >> >> And how much work needs to be done to get this work upstream? (1
> >> >> month, 6 months, years...) ?
> >> >
> >> > At least one kernel cycle, and after transition read/write will be
> >> > obsolete and will be dropped.
> >> >
> >>
> >> So 4.12-4.13 ?
> >
> > 4.12 is almost closed. We are in rc8 now and merge window will be next
> > week, so it will be 4.13-4.14 (optimistic).
> >
>
> Fine. So older interfaces will be deprecated and removed in next few
> cycles. I think i can now create something that used old api and do
> transition later (i'm use fedora)

Yes, conversion should be possible, Matan and Jason invest a lot of time
and effort to be sure that conversion will be as minimal as possible.

>
> >>
> >> >> If i want to try working with uverbs without C librarary, where i can
> >> >> find some info about format of messages reader and writed to uverbs
> >> >> device?
> >> >
> >> > Did you see Documentation/infiniband/user_verbs.txt?
> >>
> >>
> >> https://www.kernel.org/doc/Documentation/infiniband/user_verbs.txt as
> >> i see this nothing says about format of data writed/readed...
> >>
> >
> > See in include/uapi/rdma/ib_user_verbs.h, User fills struct from that
> > file, and kernel performs copy_from_user to extract the data. That's
> > all.
>
> Thanks! If i need to establish communication between two nodes and
> send/recv data (and get notification that data can be readed) does
> this file  enough to do that?

Yes, if performance is not critical to you.
Otherwise, the best solution will be to take a look on implementation
of data transfer (various post_send functions) in rdma-core library.

> (i'm plan to try create plain go binding, now only for fun, create
> generator for this C struct/enums to plain go types)


>
>
> --
> Vasiliy Tolstov,
> e-mail: v.tolstov-+9FY0jupvH6HXe+LvDLADg@public.gmane.org

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: dedicated list for rdma userspace programming

[Vasiliy Tolstov]

2017-04-24 21:13 GMT+03:00 Leon Romanovsky <leon-DgEjT+Ai2ygdnm+yROfE0A@public.gmane.org>:
> Yes, if performance is not critical to you.
> Otherwise, the best solution will be to take a look on implementation
> of data transfer (various post_send functions) in rdma-core library.


Hm.. performance is critical for me... How can i create binding for it
without C deps.

-- 
Vasiliy Tolstov,
e-mail: v.tolstov-+9FY0jupvH6HXe+LvDLADg@public.gmane.org
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: dedicated list for rdma userspace programming

[Jason Gunthorpe]

On Mon, Apr 24, 2017 at 09:26:17PM +0300, Vasiliy Tolstov wrote:
> 2017-04-24 21:13 GMT+03:00 Leon Romanovsky <leon-DgEjT+Ai2ygdnm+yROfE0A@public.gmane.org>:
> > Yes, if performance is not critical to you.
> > Otherwise, the best solution will be to take a look on implementation
> > of data transfer (various post_send functions) in rdma-core library.
> 
> 
> Hm.. performance is critical for me... How can i create binding for it
> without C deps.

You would have to re-write the user space verbs provider for your
hardware in a non-C language.

What you are asking is enormously non-trivial, rdma is 'special'
because it has a user space driver component that directly
communicates with the hardware, for performance.

I'm not even sure one could implement a non-performant verbs using
just the kernel API without a driver specific helper. The kernel API
does not cover things like CQ parsing, for instance? At least nobody
has ever done it...

Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: dedicated list for rdma userspace programming

[Vasiliy Tolstov]

2017-04-24 21:31 GMT+03:00 Jason Gunthorpe <jgunthorpe-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org>:
> You would have to re-write the user space verbs provider for your
> hardware in a non-C language.
>

That is bad =(

> What you are asking is enormously non-trivial, rdma is 'special'
> because it has a user space driver component that directly
> communicates with the hardware, for performance.
>
> I'm not even sure one could implement a non-performant verbs using
> just the kernel API without a driver specific helper. The kernel API
> does not cover things like CQ parsing, for instance? At least nobody
> has ever done it...

Hm. For example some times ago i'm use srp/scst that utilize only
kernel space code.
I don't know any performance issues with it (but may be it presented..).


-- 
Vasiliy Tolstov,
e-mail: v.tolstov-+9FY0jupvH6HXe+LvDLADg@public.gmane.org
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: dedicated list for rdma userspace programming

[Jason Gunthorpe]

On Mon, Apr 24, 2017 at 10:49:00PM +0300, Vasiliy Tolstov wrote:
> Hm. For example some times ago i'm use srp/scst that utilize only
> kernel space code.
> I don't know any performance issues with it (but may be it presented..).

Sorry, I ment the kernel API to userspace. The internal kernal API is
internally complete of course.

Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: dedicated list for rdma userspace programming

[Vasiliy Tolstov]

2017-04-24 23:10 GMT+03:00 Jason Gunthorpe <jgunthorpe-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org>:
> Sorry, I ment the kernel API to userspace. The internal kernal API is
> internally complete of course.


=(. SO as i understand i need to know how mellanox userspace library
interract with kernel and hardware ?

-- 
Vasiliy Tolstov,
e-mail: v.tolstov-+9FY0jupvH6HXe+LvDLADg@public.gmane.org
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: dedicated list for rdma userspace programming

[Hefty, Sean]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain; charset="utf-8", Size: 586 bytes --]

> =(. SO as i understand i need to know how mellanox userspace library
> interract with kernel and hardware ?

Yes - this is true for any RDMA hardware.  You may be able to reverse engineer it by looking at the user space code for whichever hardware you're interested in.  Alternatively, you could always take the existing code and re-use most of it, but provide optimized versions of targeted calls, such as post send/recv, cq read, etc.

- Sean
N‹§²æìr¸›yúèšØb²X¬¶Ç§vØ^–)Þº{.nÇ+‰·¥Š{±­ÙšŠ{ayº\x1dʇڙë,j\a­¢f£¢·hš‹»öì\x17/oSc¾™Ú³9˜uÀ¦æå‰È&jw¨®\x03(­éšŽŠÝ¢j"ú\x1a¶^[m§ÿïêäz¹Þ–Šàþf£¢·hšˆ§~ˆmš

Re: dedicated list for rdma userspace programming

[Jason Gunthorpe]

On Tue, Apr 25, 2017 at 02:29:58PM +0300, Vasiliy Tolstov wrote:
> 2017-04-24 23:10 GMT+03:00 Jason Gunthorpe <jgunthorpe-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org>:
> > Sorry, I ment the kernel API to userspace. The internal kernal API is
> > internally complete of course.
> 
> 
> =(. SO as i understand i need to know how mellanox userspace library
> interract with kernel and hardware ?

and also how to implement user space access to hardware in your new
language, this includes something like the C11 multi-threaded memory
model, unlocked atomics, and processor specific memory barriers (that
even C requires inline asm for)

You may find implementing the above at all will always require some
kind of C calls out..

Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: dedicated list for rdma userspace programming

[Vasiliy Tolstov]

2017-04-25 20:39 GMT+03:00 Jason Gunthorpe <jgunthorpe-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org>:
> and also how to implement user space access to hardware in your new
> language, this includes something like the C11 multi-threaded memory
> model, unlocked atomics, and processor specific memory barriers (that
> even C requires inline asm for)
>
> You may find implementing the above at all will always require some
> kind of C calls out..


=( pkay i'm try to stuck at C code now. And if i have some problems,
try to solve it

-- 
Vasiliy Tolstov,
e-mail: v.tolstov-+9FY0jupvH6HXe+LvDLADg@public.gmane.org
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: dedicated list for rdma userspace programming

[Leon Romanovsky]

[-- Attachment #1: Type: text/plain, Size: 739 bytes --]

On Tue, Apr 25, 2017 at 02:29:58PM +0300, Vasiliy Tolstov wrote:
> 2017-04-24 23:10 GMT+03:00 Jason Gunthorpe <jgunthorpe-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org>:
> > Sorry, I ment the kernel API to userspace. The internal kernal API is
> > internally complete of course.
>
>
> =(. SO as i understand i need to know how mellanox userspace library
> interract with kernel and hardware ?

Despite the fact that this guide [1] lacks RDMA part, the concepts and
software interfaces can help you understand (reverse engineering)
Mellanox HW.

[1] http://www.mellanox.com/related-docs/user_manuals/Ethernet_Adapters_Programming_Manual.pdf

Thanks

>
> --
> Vasiliy Tolstov,
> e-mail: v.tolstov-+9FY0jupvH6HXe+LvDLADg@public.gmane.org

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]