Infiniband Shared Memory Segment

Infiniband Shared Memory Segment

[Greg I Kerr]

I sent this e-mail to the OFED ewg list but I figured someone on the
Linux mailing list might know about this as well, since it's fairly
low level details.

I was hoping someone could answer a few questions for me, or point to
where the answers might be. I dug through the formal IBA specification
V1.2 but could not find the relevant information.

If I run an Infiniband program, say the ibv_rc_pingpong 'test'
program, and look in /proc/<pid>/fd the following file descriptors
appear:

lrwx------ 1 kerrg kerrg 64 Jan 26 13:02 3 -> /dev/infiniband/uverbs0
lr-x------ 1 kerrg kerrg 64 Jan 26 13:02 4 -> infinibandevent:

And if I look in /proc/<pid>/maps:
2b6f3379f000-2b6f337a0000 -w-s 00000000 00:11 7797
 /dev/infiniband/uverbs0

I'm assuming those are file descriptors which are mapped to the HCA
and used to send commands to it. I noticed that the cmd_fd field of
the ibv_context data structure pointed to to fd 3.

What kind of data might be located within the memory pointed to by the
FD? I.e. Could there be configuration information for the connection
and/or HCA? Could there be the data waiting to be sent (although I
don't believe the HCA buffers)? Would it just be the command sent to
the HCA?

Thanks,

Greg Kerr
Northeastern University
High Performance Computing Lab
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: Infiniband Shared Memory Segment

[Hefty, Sean]

> What kind of data might be located within the memory pointed to by the
> FD? I.e. Could there be configuration information for the connection
> and/or HCA? Could there be the data waiting to be sent (although I
> don't believe the HCA buffers)? Would it just be the command sent to
> the HCA?

A write to the uverbs file issues a command to the lower driver, for example, to allocate a QP or CQ.  Reads are not supported.  A read to the event file will retrieve information about an asynchronous event (e.g. port up/down).  Writes to that file are not handled.

- Sean
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Infiniband Shared Memory Segment

[Roland Dreier]

> What kind of data might be located within the memory pointed to by the
> FD? I.e. Could there be configuration information for the connection
> and/or HCA? Could there be the data waiting to be sent (although I
> don't believe the HCA buffers)? Would it just be the command sent to
> the HCA?

I'm not sure what you mean by "memory pointed to by the FD".  File
descriptors don't point to memory, do they?

/dev/infiniband/uverbs0 is used for two things: userspace does write()
on it to tell the kernel to do various control path things like creating QPs,
modifying QPs, etc.  Also userpace does mmap() on it to get access to
a device-specific region; on many adapters it is used to map PCI BAR
space directly into the userspace process for kernel bypass on data path
operations.

The event file is used for the kernel to return async events to userspace.

 - R.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html