RE: [PATCH net-next 00/19] Mellanox, mlx5 sub function support

From: Parav Pandit <parav@mellanox.com>
To: Jiri Pirko <jiri@resnulli.us>,
	Jakub Kicinski <jakub.kicinski@netronome.com>
Cc: "alex.williamson@redhat.com" <alex.williamson@redhat.com>,
	"davem@davemloft.net" <davem@davemloft.net>,
	"kvm@vger.kernel.org" <kvm@vger.kernel.org>,
	"netdev@vger.kernel.org" <netdev@vger.kernel.org>,
	Saeed Mahameed <saeedm@mellanox.com>,
	"kwankhede@nvidia.com" <kwankhede@nvidia.com>,
	"leon@kernel.org" <leon@kernel.org>,
	"cohuck@redhat.com" <cohuck@redhat.com>,
	Jiri Pirko <jiri@mellanox.com>,
	"linux-rdma@vger.kernel.org" <linux-rdma@vger.kernel.org>,
	Or Gerlitz <gerlitz.or@gmail.com>
Subject: RE: [PATCH net-next 00/19] Mellanox, mlx5 sub function support
Date: Fri, 8 Nov 2019 20:40:46 +0000	[thread overview]
Message-ID: <AM0PR05MB48666B8475EF12809FE31059D17B0@AM0PR05MB4866.eurprd05.prod.outlook.com> (raw)
In-Reply-To: <20191108194118.GY6990@nanopsycho>


> -----Original Message-----
> From: Jiri Pirko <jiri@resnulli.us>
> 
> Fri, Nov 08, 2019 at 08:06:40PM CET, jakub.kicinski@netronome.com wrote:
> >On Fri, 8 Nov 2019 13:12:33 +0100, Jiri Pirko wrote:
> >> Thu, Nov 07, 2019 at 09:32:34PM CET, jakub.kicinski@netronome.com
> wrote:
> >> >On Thu,  7 Nov 2019 10:04:48 -0600, Parav Pandit wrote:
> >> >> Mellanox sub function capability allows users to create several
> >> >> hundreds of networking and/or rdma devices without depending on PCI
> SR-IOV support.
> >> >
> >> >You call the new port type "sub function" but the devlink port
> >> >flavour is mdev.
> >> >
> >> >As I'm sure you remember you nacked my patches exposing NFP's PCI
> >> >sub functions which are just regions of the BAR without any mdev
> >> >capability. Am I in the clear to repost those now? Jiri?
> >>
> >> Well question is, if it makes sense to have SFs without having them
> >> as mdev? I mean, we discussed the modelling thoroughtly and
> >> eventually we realized that in order to model this correctly, we need SFs on
> "a bus".
> >> Originally we were thinking about custom bus, but mdev is already
> >> there to handle this.
> >
> >But the "main/real" port is not a mdev in your case. NFP is like mlx4.
> >It has one PCI PF for multiple ports.
> 
> I don't see how relevant the number of PFs-vs-uplink_ports is.
> 
> 
> >
> >> Our SFs are also just regions of the BAR, same thing as you have.
> >>
> >> Can't you do the same for nfp SFs?
> >> Then the "mdev" flavour is enough for all.
> >
> >Absolutely not.
> >
> >Why not make the main device of mlx5 a mdev, too, if that's acceptable.
> >There's (a) long precedence for multiple ports on one PCI PF in
> >networking devices, (b) plenty deployed software which depend on the
> >main devices hanging off the PCI PF directly.
> >
> >The point of mdevs is being able to sign them to VFs or run DPDK on
> >them (map to user space).
> >
> >For normal devices existing sysfs hierarchy were one device has
> >multiple children of a certain class, without a bus and a separate
> >driver is perfectly fine. Do you think we should also slice all serial
> >chips into mdevs if they have multiple lines.
> >
> >Exactly as I predicted much confusion about what's being achieved here,
> >heh :)
> 
> Please let me understand how your device is different.
> Originally Parav didn't want to have mlx5 subfunctions as mdev. He wanted to
> have them tight to the same pci device as the pf. No difference from what you
> describe you want.
> However while we thought about how to fit things in, how to
> handle na phys_port_name, how to see things in sysfs we came up with an idea
> of a dedicated bus. We took it upstream and people suggested to use mdev bus
> for this.
> 
You are right. We considered multiple ports approach, followed by subdevices and mfd.
Around that time mdev was being proposed that can address current and future VM/userspace usecases using one way to lifecycle the devices.

> Parav, please correct me if I'm wrong but I don't think where is a plan to push
> SFs into VM or to userspace as Jakub expects, right?
With this series - certainly not.
In future, if mdev to be used by via vfio/VM framework, why should we prevent it (ofcourse after implementing necessary isolation method)?