On Mon, Apr 01, 2019 at 11:32:21AM -0700, Jakub Kicinski wrote:
> On Mon, 1 Apr 2019 09:28:16 +0300, Leon Romanovsky wrote:
> > On Sun, Mar 31, 2019 at 02:06:33PM -0700, Jakub Kicinski wrote:
> > > On Sun, 31 Mar 2019 19:44:42 +0300, Leon Romanovsky wrote:
> > > > From: Leon Romanovsky <leonro@mellanox.com>
> > > >
> > > > From Ariel,
> > > >
> > > > This series of patches adds user space managed steering infrastructure
> > > > to the mlx5_ib driver.
> > > >
> > > > User space managed steering requires the means to access a dedicated
> > > > memory space that is used by the device to store the packet steering
> > > > and header modification tables and rules in order to manage them directly
> > > > without the device's firmware involvement. This dedicated memory is part
> > > > of the ICM memory space.
> > > >
> > > > The changes are introducing the mlx5_ib API to allocate, deallocate and
> > > > register this dedicated SW ICM memory via the existing device memory API
> > > > using a private attribute which specifies the memory type.
> > > >
> > > > The allocated memory itself is not IO mapped and user can only access it
> > > > using remote RDMA operations.
> > > >
> > > > In addition, the series exposed the ICM address of the receive transport
> > > > interface (TIR) of Raw Packet and RSS QPs to user since they are required
> > > > to properly create and insert steering rules that direct flows to these QPs.
> > >
> > > Hi Leon!  Out of curiosity what protocols are you guys steering here?
> > > Is this some form of RoCE or IB?  Is there a good resource to read up on
> > > the DEVX stuff? (I presume this series is related?)
> >
> > mlx5 devices steering operates on packets without relation to specific
> > protocol. The decision to steer is based on HW capabilities and user
> > configuration for matching specific fields. Such configuration is done
> > through kernel and through DEVX.
> >
> > General description of DEVX can be found in manuals of libmlx5.
> > https://github.com/linux-rdma/rdma-core/blob/master/providers/mlx5/man/mlx5dv_devx_obj_create.3.md
> > https://github.com/linux-rdma/rdma-core/blob/master/providers/mlx5/man/
>
> Thanks for those, looks quite powerful.  What's your primary use case
> here?  Is there some flow steering and encapsulation going on in RDMA
> world?

Usual use case for that is in RoCE fabric, but as I said it is not
limited to that. For example, flow steering is used in libpcap [1]
and encapsulation is used for efficient VM to VM communication over
different hosts (I assume exactly as in netdev world).

In regarding to this patch set, the combination of this steering
functionality with DEVX allows us to speedup greatly buildup of rules
by doing it in powerful host CPU instead of internal to NIC CPU.

[1]
https://github.com/the-tcpdump-group/libpcap/commit/f5f1484a931851d2d3b67d6033aedea8da5acb42

Thanks