[PATCH rdma-next 0/7] Introduce KABIs to query UCONTEXT, PD and MR properties

[PATCH rdma-next 0/7] Introduce KABIs to query UCONTEXT, PD and MR properties

[Leon Romanovsky]

From: Leon Romanovsky <leonro@mellanox.com>

From Yishai,

This series introduce KABIs to query existing UCONTEXT, PD and MR for their
properties.

By those KABIs a user space application which owns the command FD of the device
can retrieve and rebuild the above objects despite that it wasn't their creator.
As of that, PDs and MRs can be shared by any process who owns the command FD of
their creator.

This functionality enables to utilize and optimize some application flows, few
examples below.

Any solution which is a single business logic based on multi-process design
needs this. Example include NGINX, with TCP load balancing, sharing the RSS
indirection table with RQ per process. HPC frameworks with multi-rank (process)
solution on single hosts. UCX can share IB resources using the shared PD and
can help dispatch data to multiple processes/MR's in single RDMA operation.
Also, there are use cases when a primary processes registered a large shared
memory range, and each worker process spawned will create a private QP on the
shared PD, and use the shared MR to save the registration time per-process.

The matching verbs APIs were introduced in the mailing list by the below RFC [1].

In addition, the series enables CQ ioctl commands by default as this
functionality is fully mature for a long time and sets IOVA on IB MR in the uverbs
layer to let all drivers have it.

[1] https://patchwork.kernel.org/patch/11540665/

Yishai

Yishai Hadas (7):
  IB/uverbs: Enable CQ ioctl commands by default
  IB/uverbs: Set IOVA on IB MR in uverbs layer
  IB/uverbs: Expose KABI to query ucontext
  RDMA/mlx5: Refactor mlx5_ib_alloc_ucontext() response
  RDMA/mlx5: Implement the query ucontext functionality
  RDMA/mlx5: Introduce KABI to query PD attributes
  IB/uverbs: Expose KABI to query MR

 drivers/infiniband/Kconfig                    |   8 -
 drivers/infiniband/core/device.c              |   1 +
 drivers/infiniband/core/uverbs_cmd.c          |   4 +
 drivers/infiniband/core/uverbs_ioctl.c        |   1 +
 drivers/infiniband/core/uverbs_std_types_cq.c |   3 -
 .../infiniband/core/uverbs_std_types_device.c |  41 +++-
 drivers/infiniband/core/uverbs_std_types_mr.c |  50 +++-
 drivers/infiniband/hw/cxgb4/mem.c             |   1 -
 drivers/infiniband/hw/mlx4/mr.c               |   1 -
 drivers/infiniband/hw/mlx5/Makefile           |   3 +-
 drivers/infiniband/hw/mlx5/main.c             | 232 ++++++++++--------
 drivers/infiniband/hw/mlx5/mlx5_ib.h          |   2 +
 drivers/infiniband/hw/mlx5/std_types.c        |  45 ++++
 include/rdma/ib_verbs.h                       |   4 +
 include/uapi/rdma/ib_user_ioctl_cmds.h        |  15 ++
 include/uapi/rdma/mlx5_user_ioctl_cmds.h      |  14 ++
 16 files changed, 306 insertions(+), 119 deletions(-)
 create mode 100644 drivers/infiniband/hw/mlx5/std_types.c

--
2.26.2

Re: [PATCH rdma-next 0/7] Introduce KABIs to query UCONTEXT, PD and MR properties

[Christoph Hellwig]

I think you are talking about UABIs (which in linux we actually call
uapis).

Re: [PATCH rdma-next 0/7] Introduce KABIs to query UCONTEXT, PD and MR properties

[Leon Romanovsky]

On Wed, Jun 17, 2020 at 01:29:16AM -0700, Christoph Hellwig wrote:
> I think you are talking about UABIs (which in linux we actually call
> uapis).

Yes, I used Yishai's cover letter as is.

>

Re: [PATCH rdma-next 0/7] Introduce KABIs to query UCONTEXT, PD and MR properties

[Christoph Hellwig]

On Wed, Jun 17, 2020 at 11:31:38AM +0300, Leon Romanovsky wrote:
> On Wed, Jun 17, 2020 at 01:29:16AM -0700, Christoph Hellwig wrote:
> > I think you are talking about UABIs (which in linux we actually call
> > uapis).
> 
> Yes, I used Yishai's cover letter as is.

Why can't he just posted his patches himeself?  And if you forward it
you could actually add value by fixing up obvious issues :)

Re: [PATCH rdma-next 0/7] Introduce KABIs to query UCONTEXT, PD and MR properties

[Leon Romanovsky]

On Wed, Jun 17, 2020 at 01:34:50AM -0700, Christoph Hellwig wrote:
> On Wed, Jun 17, 2020 at 11:31:38AM +0300, Leon Romanovsky wrote:
> > On Wed, Jun 17, 2020 at 01:29:16AM -0700, Christoph Hellwig wrote:
> > > I think you are talking about UABIs (which in linux we actually call
> > > uapis).
> >
> > Yes, I used Yishai's cover letter as is.
>
> Why can't he just posted his patches himeself?  And if you forward it
> you could actually add value by fixing up obvious issues :)

To say that I'm not adding any value, you need to see patches before
they arrive to me and after they sent to the mailing list.

Thanks

Re: [PATCH rdma-next 0/7] Introduce KABIs to query UCONTEXT, PD and MR properties

[Yishai Hadas]

On 6/17/2020 11:34 AM, Christoph Hellwig wrote:
> On Wed, Jun 17, 2020 at 11:31:38AM +0300, Leon Romanovsky wrote:
>> On Wed, Jun 17, 2020 at 01:29:16AM -0700, Christoph Hellwig wrote:
>>> I think you are talking about UABIs (which in linux we actually call
>>> uapis).
>>
>> Yes, I used Yishai's cover letter as is.
> 

We used in the past the "kABI" notation in the rdma sub system when we 
referred to the ioctl interface that kernel exposes to user space.

See the below [1] from Jason and the below [2] mentioned by Linus as of 
Doug's pull request.

However,
We may better refers to UAPI to make things clearer.

[1] https://patchwork.kernel.org/patch/10226945/
[2] commit 004e390d8133b96485e1ab9af5351c2db4300c63
Merge: 24180a6008ac f45765872e7a
Author: Linus Torvalds <torvalds@linux-foundation.org>
Date:   Thu Feb 22 11:57:39 2018 -0800

     Merge tag 'for-linus' of 
git://git.kernel.org/pub/scm/linux/kernel/git/rdma/rdma

     Pull rdma fixes from Doug Ledford:
      "Nothing in this is overly interesting, it's mostly your garden 
variety
       fixes.

       There was some work in this merge cycle around the new ioctl kABI, so
       there are fixes in here related to that (probably with more to come).


Yishai

Re: [PATCH rdma-next 0/7] Introduce KABIs to query UCONTEXT, PD and MR properties

[Jason Gunthorpe]

On Wed, Jun 17, 2020 at 12:20:53PM +0300, Yishai Hadas wrote:
> On 6/17/2020 11:34 AM, Christoph Hellwig wrote:
> > On Wed, Jun 17, 2020 at 11:31:38AM +0300, Leon Romanovsky wrote:
> > > On Wed, Jun 17, 2020 at 01:29:16AM -0700, Christoph Hellwig wrote:
> > > > I think you are talking about UABIs (which in linux we actually call
> > > > uapis).
> > > 
> > > Yes, I used Yishai's cover letter as is.
> > 
> 
> We used in the past the "kABI" notation in the rdma sub system when we
> referred to the ioctl interface that kernel exposes to user space.

It is true, somewhere along the way it got to be called the "ioctl
kabi" which is confusing and wrong, we should move away from that
language.

Jason