Re: [RFC] current devlink extension plan for NICs

[Jakub Kicinski <kuba@kernel.org>]

On Fri, 27 Mar 2020 19:45:53 +0000 Saeed Mahameed wrote:
> On Fri, 2020-03-27 at 12:10 -0700, Jakub Kicinski wrote:
> > On Fri, 27 Mar 2020 11:49:10 -0700 Samudrala, Sridhar wrote:  
> > > On 3/27/2020 9:38 AM, Jakub Kicinski wrote:  
> > > > On Fri, 27 Mar 2020 08:47:36 +0100 Jiri Pirko wrote:    
> > > > > > So the queues, interrupts, and other resources are also part
> > > > > > of the slice then?    
> > > > > 
> > > > > Yep, that seems to make sense.
> > > > >    
> > > > > > How do slice parameters like rate apply to NVMe?    
> > > > > 
> > > > > Not really.
> > > > >    
> > > > > > Are ports always ethernet? and slices also cover endpoints
> > > > > > with
> > > > > > transport stack offloaded to the NIC?    
> > > > > 
> > > > > devlink_port now can be either "ethernet" or "infiniband".
> > > > > Perhaps,
> > > > > there can be port type "nve" which would contain only some of
> > > > > the
> > > > > config options and would not have a representor "netdev/ibdev"
> > > > > linked.
> > > > > I don't know.    
> > > > 
> > > > I honestly find it hard to understand what that slice abstraction
> > > > is,
> > > > and which things belong to slices and which to PCI ports (or why
> > > > we even
> > > > have them).    
> > > 
> > > Looks like slice is a new term for sub function and we can consider
> > > this
> > > as a VMDQ VSI(intel terminology) or even a Queue group of a VSI.
> > > 
> > > Today we expose VMDQ VSI via offloaded MACVLAN. This mechanism
> > > should 
> > > allow us to expose it as a separate netdev.  
> > 
> > Kinda. Looks like with the new APIs you guys will definitely be able
> > to
> > expose VMDQ as a full(er) device, and if memory serves me well that's
> > what you wanted initially.
> 
> VMDQ is just a steering based isolated set of rx tx rings pointed at by
> a dumb steering rule in the HW .. i am not sure we can just wrap them
> in their own vendor specific netdev and just call it a slice..
> 
> from what i understand, a real slice is a full isolated HW pipeline
> with its own HW resources and HW based isolation, a slice rings/hw
> resources can never be shared between different slices, just like a vf,
> but without the pcie virtual function back-end..
> 
> Why would you need a devlink slice instance for something that has only
> rx/tx rings attributes ? if we are going with such design, then i guess
> a simple rdma app with a pair of QPs can call itself a slice .. 

Ack, I'm not sure where Intel is, but I'd hope since VMDq in its
"just a bunch of queues with dumb steering" form was created in
igb/ixgbe days, 2 generations of HW later its not just that..

> We need a clear-cut definition of what a Sub-function slice is.. this
> RFC doesn't seem to address that clearly.

Definitely. I'd say we need a clear definition of (a) what a
sub-functions is, and (b) what a slice is.

> > But the sub-functions are just a subset of slices, PF and VFs also
> > have a slice associated with them.. And all those things have a port,
> > too.
> >   
> 
> PFs/VFs, might have more than one port sometimes .. 

Like I said below, right? So in that case do you think they should have
multiple slices, too, or slice per port?

> > > > With devices like NFP and Mellanox CX3 which have one PCI PF
> > > > maybe it
> > > > would have made sense to have a slice that covers multiple ports,
> > > > but
> > > > it seems the proposal is to have port to slice mapping be 1:1.
> > > > And rate
> > > > in those devices should still be per port not per slice.
> > > > 
> > > > But this keeps coming back, and since you guys are doing all the
> > > > work,
> > > > if you really really need it..