Re: [RFC] current devlink extension plan for NICs

[Jakub Kicinski <kuba@kernel.org>]

On Mon, 30 Mar 2020 07:48:39 +0000 Parav Pandit wrote:
> On 3/27/2020 10:08 PM, Jakub Kicinski wrote:
> > On Fri, 27 Mar 2020 08:47:36 +0100 Jiri Pirko wrote:  
> >>> So the queues, interrupts, and other resources are also part 
> >>> of the slice then?    
> >>
> >> Yep, that seems to make sense.
> >>  
> >>> How do slice parameters like rate apply to NVMe?    
> >>
> >> Not really.
> >>  
> >>> Are ports always ethernet? and slices also cover endpoints with
> >>> transport stack offloaded to the NIC?    
> >>
> >> devlink_port now can be either "ethernet" or "infiniband". Perhaps,
> >> there can be port type "nve" which would contain only some of the
> >> config options and would not have a representor "netdev/ibdev" linked.
> >> I don't know.  
> > 
> > I honestly find it hard to understand what that slice abstraction is,
> > and which things belong to slices and which to PCI ports (or why we even
> > have them).
> >   
> In an alternative, devlink port can be overloaded/retrofit to do all
> things that slice desires to do.

I wouldn't say retrofitted, in my mind port has always been a port of 
a device.

Jiri explained to me that to Mellanox port is port of a eswitch, not
port of a device. While to me (/Netronome) it was any way to send or
receive data to/from the device.

Now I understand why to you nvme doesn't fit the port abstraction.

> For that matter representor netdev can be overloaded/extended to do what
> slice desire to do (instead of devlink port).

Right, in my mental model representor _is_ a port of the eswitch, so
repr would not make sense to me.

> Can you please explain why you think devlink port should be overloaded
> instead of netdev or any other kernel object?
> Do you have an example of such overloaded functionality of a kernel object?
> Like why macvlan and vlan drivers are not combined to in single driver
> object? Why teaming and bonding driver are combined in single driver
> object?...

I think it's not overloading, but the fact that we started with
different definitions. We (me and you) tried adding the PCIe ports
around the same time, I guess we should have dug into the details
right away.

> User should be able to create, configure, deploy, delete a 'portion of
> the device' with/without eswitch.

Right, to me ports are of the device, not eswitch.

> We shouldn't be starting with restrictive/narrow view of devlink port.
> 
> Internally with Jiri and others, we also explored the possibility to
> have 'mgmtvf', 'mgmtpf',  'mgmtsf' port flavours by overloading port to
> do all things as that of slice.
> It wasn't elegant enough. Why not create right object?

We just need clear definitions of what goes where. We already have
params etc. hanging off the ports, including irq/sriov stuff. But in
slice model those don't belong there :S

In fact very little belongs to the port in that model. So why have
PCI ports in the first place?

> Additionally devlink port object doesn't go through the same state
> machine as that what slice has to go through.
> So its weird that some devlink port has state machine and some doesn't.

You mean for VFs? I think you can add the states to the API.

> > With devices like NFP and Mellanox CX3 which have one PCI PF maybe it
> > would have made sense to have a slice that covers multiple ports, but
> > it seems the proposal is to have port to slice mapping be 1:1. And rate
> > in those devices should still be per port not per slice.
> >   
> Slice can have multiple ports. slice object doesn't restrict it. User
> can always split the port for a device, if device support it.

Okay, so slices are not 1:1 with ports, then?  Is it any:any?

> > But this keeps coming back, and since you guys are doing all the work,
> > if you really really need it..