Re: [PATCH net-next 00/19] Mellanox, mlx5 sub function support

From: Jason Gunthorpe <jgg@ziepe.ca>
To: Jakub Kicinski <jakub.kicinski@netronome.com>
Cc: Parav Pandit <parav@mellanox.com>,
	Alex Williamson <alex.williamson@redhat.com>,
	Jiri Pirko <jiri@resnulli.us>, David M <david.m.ertman@intel.com>,
	"gregkh@linuxfoundation.org" <gregkh@linuxfoundation.org>,
	"davem@davemloft.net" <davem@davemloft.net>,
	"kvm@vger.kernel.org" <kvm@vger.kernel.org>,
	"netdev@vger.kernel.org" <netdev@vger.kernel.org>,
	Saeed Mahameed <saeedm@mellanox.com>,
	"kwankhede@nvidia.com" <kwankhede@nvidia.com>,
	"leon@kernel.org" <leon@kernel.org>,
	"cohuck@redhat.com" <cohuck@redhat.com>,
	Jiri Pirko <jiri@mellanox.com>,
	"linux-rdma@vger.kernel.org" <linux-rdma@vger.kernel.org>,
	Or Gerlitz <gerlitz.or@gmail.com>,
	"Jason Wang (jasowang@redhat.com)" <jasowang@redhat.com>
Subject: Re: [PATCH net-next 00/19] Mellanox, mlx5 sub function support
Date: Sun, 10 Nov 2019 15:04:17 -0400	[thread overview]
Message-ID: <20191110190417.GD31761@ziepe.ca> (raw)
In-Reply-To: <20191109094103.739033a3@cakuba>

On Sat, Nov 09, 2019 at 09:41:03AM -0800, Jakub Kicinski wrote:
> On Fri, 8 Nov 2019 20:57:08 -0400, Jason Gunthorpe wrote:
> > On Fri, Nov 08, 2019 at 10:48:31PM +0000, Parav Pandit wrote:
> > > We should be creating 3 different buses, instead of mdev bus being de-multiplexer of that?
> > > 
> > > Hence, depending the device flavour specified, create such device on right bus?
> > > 
> > > For example,
> > > $ devlink create subdev pci/0000:05:00.0 flavour virtio name foo subdev_id 1
> > > $ devlink create subdev pci/0000:05:00.0 flavour mdev <uuid> subdev_id 2
> > > $ devlink create subdev pci/0000:05:00.0 flavour mlx5 id 1 subdev_id 3  
> > 
> > I like the idea of specifying what kind of interface you want at sub
> > device creation time. It fits the driver model pretty well and doesn't
> > require abusing the vfio mdev for binding to a netdev driver.
> 
> Aren't the HW resources spun out in all three cases exactly identical?

Exactly? No, not really. The only constant is that some chunk of the
BAR is dedicated to this subedv. The BAR is flexible, so a BAR chunk
configured for virtio is not going to support mlx5 mode.

Aside from that, there are other differences ie - mlx5 does not need a
dedicated set of MSI-X's while other modes do. There are fewer MSI-X's
than SF's, so managing this is important for the admin.

Even in modes which are very similar, like mlx5 vs mdev-vfio, the HW
still has to be configured to provide global DMA isolation on the NIC
for vfio as the IOMMU cannot be involved. This is extra overhead and
should not be activated unless vfio is being used.

.. and finally the driver core does not support a
'multiple-inheritance' like idea, so we can't have a 'foo_device' that
is three different things.

So somehow the 'flavour' of the 'struct device' has to be exposed to
userspace, and it is best if this is done at device creation time so
the BAR region and HW can be setup once and we don't have to have
complex reconfiguration flows.

Jason