RE: [net-next v5 03/15] devlink: Introduce PCI SF port flavour and port attribute

From: Parav Pandit <parav@nvidia.com>
To: Jakub Kicinski <kuba@kernel.org>, Saeed Mahameed <saeed@kernel.org>
Cc: "David S. Miller" <davem@davemloft.net>,
	Jason Gunthorpe <jgg@nvidia.com>,
	Leon Romanovsky <leonro@nvidia.com>,
	"netdev@vger.kernel.org" <netdev@vger.kernel.org>,
	"linux-rdma@vger.kernel.org" <linux-rdma@vger.kernel.org>,
	David Ahern <dsahern@kernel.org>,
	Jacob Keller <jacob.e.keller@intel.com>,
	Sridhar Samudrala <sridhar.samudrala@intel.com>,
	"david.m.ertman@intel.com" <david.m.ertman@intel.com>,
	"dan.j.williams@intel.com" <dan.j.williams@intel.com>,
	"kiran.patil@intel.com" <kiran.patil@intel.com>,
	"gregkh@linuxfoundation.org" <gregkh@linuxfoundation.org>,
	Jiri Pirko <jiri@nvidia.com>, Vu Pham <vuhuong@nvidia.com>,
	Saeed Mahameed <saeedm@nvidia.com>
Subject: RE: [net-next v5 03/15] devlink: Introduce PCI SF port flavour and port attribute
Date: Wed, 16 Dec 2020 03:42:51 +0000	[thread overview]
Message-ID: <BY5PR12MB432268C16D118BC435C0EF5CDCC50@BY5PR12MB4322.namprd12.prod.outlook.com> (raw)
In-Reply-To: <20201215152740.0b3ed376@kicinski-fedora-pc1c0hjn.dhcp.thefacebook.com>


> From: Jakub Kicinski <kuba@kernel.org>
> Sent: Wednesday, December 16, 2020 4:58 AM
> 
> On Tue, 15 Dec 2020 01:03:46 -0800 Saeed Mahameed wrote:
> > + *	devlink_port_attrs_pci_sf_set - Set PCI SF port attributes
> > + *
> > + *	@devlink_port: devlink port
> > + *	@controller: associated controller number for the devlink port
> instance
> > + *	@pf: associated PF for the devlink port instance
> > + *	@sf: associated SF of a PF for the devlink port instance
> > + *	@external: indicates if the port is for an external controller
> > + */
> > +void devlink_port_attrs_pci_sf_set(struct devlink_port *devlink_port, u32
> controller,
> > +				   u16 pf, u32 sf, bool external) {
> > +	struct devlink_port_attrs *attrs = &devlink_port->attrs;
> > +	int ret;
> > +
> > +	if (WARN_ON(devlink_port->registered))
> > +		return;
> > +	ret = __devlink_port_attrs_set(devlink_port,
> DEVLINK_PORT_FLAVOUR_PCI_SF);
> > +	if (ret)
> > +		return;
> > +	attrs->pci_sf.controller = controller;
> > +	attrs->pci_sf.pf = pf;
> > +	attrs->pci_sf.sf = sf;
> > +	attrs->pci_sf.external = external;
> > +}
> > +EXPORT_SYMBOL_GPL(devlink_port_attrs_pci_sf_set);
> 
> So subfunctions don't have a VF id but they may have a controller?
>
Right. SF can be on external controller.
 
> Can you tell us more about the use cases and deployment models you're
> intending to support? Let's not add attributes and info which will go unused.
> 
External will be used the same way how it is used for PF and VF.

> How are SFs supposed to be used with SmartNICs? Are you assuming single
> domain of control?
No. it is not assumed. SF can be deployed from smartnic to external host.
A user has to pass appropriate controller number, pf number attributes during creation time.

> It seems that the way the industry is moving the major
> use case for SmartNICs is bare metal.
> 
> I always assumed nested eswitches when thinking about SmartNICs, what
> are you intending to do?
>
Mlx5 doesn't support nested eswitch. SF can be deployed on the external controller PCI function.
But this interface neither limited nor enforcing nested or flat eswitch.
 
> What are your plans for enabling this feature in user space project?
Do you mean K8s plugin or iproute2? Can you please tell us what user space project?
If iproute2, will send the iproute2 patchset like other patchset pointing to kernel uapi headers..