RE: [PATCH net-next v2 00/14] Add mlx5 subfunction support

From: Parav Pandit <parav@nvidia.com>
To: "Samudrala, Sridhar" <sridhar.samudrala@intel.com>,
	"saeed@kernel.org" <saeed@kernel.org>,
	"David S. Miller" <davem@davemloft.net>,
	Jakub Kicinski <kuba@kernel.org>,
	Jason Gunthorpe <jgg@nvidia.com>
Cc: Leon Romanovsky <leonro@nvidia.com>,
	"netdev@vger.kernel.org" <netdev@vger.kernel.org>,
	"linux-rdma@vger.kernel.org" <linux-rdma@vger.kernel.org>,
	David Ahern <dsahern@kernel.org>,
	Jacob Keller <jacob.e.keller@intel.com>,
	"david.m.ertman@intel.com" <david.m.ertman@intel.com>,
	"dan.j.williams@intel.com" <dan.j.williams@intel.com>,
	"kiran.patil@intel.com" <kiran.patil@intel.com>,
	"gregkh@linuxfoundation.org" <gregkh@linuxfoundation.org>
Subject: RE: [PATCH net-next v2 00/14] Add mlx5 subfunction support
Date: Fri, 11 Dec 2020 08:33:54 +0000	[thread overview]
Message-ID: <BY5PR12MB43227784BB34D929CA64E315DCCA0@BY5PR12MB4322.namprd12.prod.outlook.com> (raw)
In-Reply-To: <f720c3fb-7401-4f3a-7de2-25309c2570f5@intel.com>

> From: Samudrala, Sridhar <sridhar.samudrala@intel.com>
> Sent: Friday, December 11, 2020 9:42 AM
> 
> On 12/8/2020 11:29 PM, saeed@kernel.org wrote:
> > From: Parav Pandit <parav@nvidia.com>

> > Subfunction provide the same functionality as SRIOV but in a very
> > lightweight manner, please see the thorough and detailed documentation
> > from Parav below, in the commit messages and the Networking
> > documentation patches at the end of this series.
> 
> What is the mechanism for assigning these subfunctions to VMs?
> OR is this only targeted for container usecases at this time?
> 
Currently subfunction cannot be assigned to VM as_is.
Some more vfio_pci style software may be developed in future to map subfunction auxiliary device to the VM.

> >
> > Add a devlink port of subfunction flaovur:
> > $ devlink port add pci/0000:06:00.0 flavour pcisf pfnum 0 sfnum 88
> Is there any requirement that subfunctions can be created only when
> eswitch mode is set to switchdev?
> I think we should not restrict this functionality without switchdev mode .
> 
It is not restricted. We discussed this before at [3].

> After this step, i guess an auxiliary device is created on the auxiliary bus and a
> devlink port.
> Does "devlink port show" show this port and can we list the auxiliary device.
Yes and yes.
Below command will show the the auxiliary device.
Auxiliary device is listed in detail in the patch_7 at [4] when its created.
$ devlink dev show auxiliary/mlx5_core.sf.4/
More below.

> > Configure mac address of the port function:
> > $ devlink port function set ens2f0npf0sf88 hw_addr 00:00:00:00:88:88
> What is ens2f0npf0sf88? Is this the port representer netdev? 
Yes, it is representor netdev associated with the devlink port.

> I think we should allow setting this by passing the devlink port.
Absolutely. It is. Every devlink port is identified by a unique port index.
So
$ devlink port show pci/0000:06:00.0/<devlink_port_index>  will show it.

It is captured in detailed example in the commit log of the patch_7 that adds it at [4].
Also present in the Documentation of mlx5.rst patch_14 at [5].

I just used the representor netdev example as it was intuitive to view the world from eswitch side.
Bu yes, instead of netdev port index is already supported natively.

> 
> What about other attributes like number of queues, interrupt vectors and
> port capabilities etc? Can we add other attributes via this interface?
> >
We believe that capabilities of the function should be controlled using the port function set command.
At the moment only mac address can be configured.
Number of queues is a resource so devlink resource is more suitable interface.

> > Now activate the function:
> > $ devlink port function set ens2f0npf0sf88 state active
> Is the subfunction netdev created after this step?
Yes.
> I thought there was a step to bind the auxiliary device to the driver.
Yes. User can always bind/unbind auxiliary driver from the auxiliary device.
Currently auxiliary bus do not have option to disable autoprobe (per device).
This is something to be extended in future so that user can select how a subfunction device to be used in the host system.

> How does the probe routine for the auxiliary device get invoked?
> >
When the subfunction auxiliary device is placed on the auxiliary bus, driver core invokes the registered driver probe routine.
Please refer to patch _7 at [4]. It is similar to how a pci device is probed.

[3] https://lore.kernel.org/netdev/BY5PR12MB43225AA5A5E42E76C03F645BDC3F0@BY5PR12MB4322.namprd12.prod.outlook.com/
[4] https://lore.kernel.org/netdev/20201209072934.1272819-4-saeed@kernel.org/
[5] https://lore.kernel.org/netdev/20201209072934.1272819-15-saeed@kernel.org/