Re: [PATCH net-next 00/19] Mellanox, mlx5 sub function support

[Jakub Kicinski <jakub.kicinski@netronome.com>]

On Thu, 7 Nov 2019 20:52:29 +0000, Parav Pandit wrote:
> > On Thu,  7 Nov 2019 10:04:48 -0600, Parav Pandit wrote:  
> > > Mellanox sub function capability allows users to create several
> > > hundreds of networking and/or rdma devices without depending on PCI SR-  
> > IOV support.
> > 
> > You call the new port type "sub function" but the devlink port flavour is mdev.
> >   
> Sub function is the internal driver structure. The abstract entity at user and stack level is mdev.
> Hence the port flavour is mdev.

FWIW I agree mdev as flavour seems like the right choice.

> > As I'm sure you remember you nacked my patches exposing NFP's PCI sub
> > functions which are just regions of the BAR without any mdev capability. Am I
> > in the clear to repost those now? Jiri?
> >   
> For sure I didn't nack it. :-)

Well, maybe the word "nack" wasn't exactly used :)

> What I remember discussing offline/mailing list is 
> (a) exposing mdev/sub fuctions as devlink sub ports is not so good abstraction
> (b) user creating/deleting eswitch sub ports would be hard to fit in the whole usage model

Okay, so I can repost the "basic" sub functions?

> > > Overview:
> > > ---------
> > > Mellanox ConnectX sub functions are exposed to user as a mediated
> > > device (mdev) [2] as discussed in RFC [3] and further during
> > > netdevconf0x13 at [4].
> > >
> > > mlx5 mediated device (mdev) enables users to create multiple
> > > netdevices and/or RDMA devices from single PCI function.
> > >
> > > Each mdev maps to a mlx5 sub function.
> > > mlx5 sub function is similar to PCI VF. However it doesn't have its
> > > own PCI function and MSI-X vectors.
> > >
> > > mlx5 mdevs share common PCI resources such as PCI BAR region, MSI-X
> > > interrupts.
> > >
> > > Each mdev has its own window in the PCI BAR region, which is
> > > accessible only to that mdev and applications using it.
> > >
> > > Each mlx5 sub function has its own resource namespace for RDMA resources.
> > >
> > > mdevs are supported when eswitch mode of the devlink instance is in
> > > switchdev mode described in devlink documentation [5].  
> > 
> > So presumably the mdevs don't spawn their own devlink instance today, but
> > once mapped via VIRTIO to a VM they will create one?
> >   
> mdev doesn't spawn the devlink instance today when mdev is created by user, like PCI.
> When PCI bus driver enumerates and creates PCI device, there isn't a devlink instance for it.
> 
> But, mdev's devlink instance is created when mlx5_core driver binds to the mdev device.
> (again similar to PCI, when mlx5_core driver binds to PCI, its devlink instance is created ).
> 
> I should have put the example in patch-15 which creates/deletes devlink instance of mdev.
> I will revise the commit log of patch-15 to include that.
> Good point.

Thanks.

> > It could be useful to specify.
> >   
> Yes, its certainly useful. I missed to put the example in commit log of patch-15.
> 
> > > Network side:
> > > - By default the netdevice and the rdma device of mlx5 mdev cannot
> > > send or receive any packets over the network or to any other mlx5 mdev.  
> > 
> > Does this mean the frames don't fall back to the repr by default?  
> Probably I wasn't clear.
> What I wanted to say is, that frames transmitted by mdev's netdevice and rdma devices don't go to network.
> These frames goes to representor device.
> User must configure representor to send/receive/steer traffic to mdev.

👍