Re: [RFC] current devlink extension plan for NICs

From: Jiri Pirko <jiri@resnulli.us>
To: Jakub Kicinski <kuba@kernel.org>
Cc: netdev@vger.kernel.org, davem@davemloft.net, parav@mellanox.com,
	yuvalav@mellanox.com, jgg@ziepe.ca, saeedm@mellanox.com,
	leon@kernel.org, andrew.gospodarek@broadcom.com,
	michael.chan@broadcom.com, moshe@mellanox.com, ayal@mellanox.com,
	eranbe@mellanox.com, vladbu@mellanox.com, kliteyn@mellanox.com,
	dchickles@marvell.com, sburla@marvell.com, fmanlunas@marvell.com,
	tariqt@mellanox.com, oss-drivers@netronome.com,
	snelson@pensando.io, drivers@pensando.io, aelior@marvell.com,
	GR-everest-linux-l2@marvell.com, grygorii.strashko@ti.com,
	mlxsw@mellanox.com, idosch@mellanox.com, markz@mellanox.com,
	jacob.e.keller@intel.com, valex@mellanox.com,
	linyunsheng@huawei.com, lihong.yang@intel.com,
	vikas.gupta@broadcom.com, magnus.karlsson@intel.com
Subject: Re: [RFC] current devlink extension plan for NICs
Date: Fri, 27 Mar 2020 08:47:36 +0100	[thread overview]
Message-ID: <20200327074736.GJ11304@nanopsycho.orion> (raw)
In-Reply-To: <20200326133001.1b2694c9@kicinski-fedora-pc1c0hjn.dhcp.thefacebook.com>

Thu, Mar 26, 2020 at 09:30:01PM CET, kuba@kernel.org wrote:
>On Thu, 26 Mar 2020 15:51:46 +0100 Jiri Pirko wrote:
>> Thu, Mar 26, 2020 at 03:47:09PM CET, jiri@resnulli.us wrote:
>> >>> >> >> $ devlink slice show
>> >>> >> >> pci/0000:06:00.0/0: flavour physical pfnum 0 port 0 state active
>> >>> >> >> pci/0000:06:00.0/1: flavour physical pfnum 1 port 1 state active
>> >>> >> >> pci/0000:06:00.0/2: flavour pcivf pfnum 0 vfnum 0 port 2 hw_addr 10:22:33:44:55:66 state active
>> >>> >> >> pci/0000:06:00.0/3: flavour pcivf pfnum 0 vfnum 1 port 3 hw_addr aa:bb:cc:dd:ee:ff state active
>> >>> >> >> pci/0000:06:00.0/4: flavour pcivf pfnum 1 vfnum 0 port 4 hw_addr 10:22:33:44:55:88 state active
>> >>> >> >> pci/0000:06:00.0/5: flavour pcivf pfnum 1 vfnum 1 port 5 hw_addr 10:22:33:44:55:99 state active
>> >>> >> >> pci/0000:06:00.0/6: flavour pcivf pfnum 1 vfnum 2      
>> >>> >> >
>> >>> >> >What are slices?      
>> >>> >> 
>> >>> >> Slice is basically a piece of ASIC. pf/vf/sf. They serve for
>> >>> >> configuration of the "other side of the wire". Like the mac. Hypervizor
>> >>> >> admin can use the slite to set the mac address of a VF which is in the
>> >>> >> virtual machine. Basically this should be a replacement of "ip vf"
>> >>> >> command.    
>> >>> >
>> >>> >I lost my mail archive but didn't we already have a long thread with
>> >>> >Parav about this?    
>> >>> 
>> >>> I believe so.  
>> >>
>> >>Oh, well. I still don't see the need for it :( If it's one to one with
>> >>ports why add another API, and have to do some cross linking to get
>> >>from one to the other?
>> >>
>> >>I'd much rather resources hanging off the port.  
>> >
>> >Yeah, I was originally saying exactly the same as you do. However, there
>> >might be slices that are not related to any port. Like NVE. Port does
>> >not make sense in that world. It is just a slice of device.
>> >Do we want to model those as "ports" too? Maybe. What do you think?  
>> 
>> Also, the slice is to model "the other side of the wire":
>> 
>> eswitch - devlink_port ...... slice
>> 
>> If we have it under devlink port, it would probably
>> have to be nested object to have the clean cut.
>
>So the queues, interrupts, and other resources are also part 
>of the slice then?

Yep, that seems to make sense.

>
>How do slice parameters like rate apply to NVMe?

Not really.

>
>Are ports always ethernet? and slices also cover endpoints with
>transport stack offloaded to the NIC?

devlink_port now can be either "ethernet" or "infiniband". Perhaps,
there can be port type "nve" which would contain only some of the
config options and would not have a representor "netdev/ibdev" linked.
I don't know.