Re: [PATCH net-next 00/19] Mellanox, mlx5 sub function support

From: Jason Gunthorpe <jgg@ziepe.ca>
To: Parav Pandit <parav@mellanox.com>
Cc: Jakub Kicinski <jakub.kicinski@netronome.com>,
	Jiri Pirko <jiri@resnulli.us>, David M <david.m.ertman@intel.com>,
	"gregkh@linuxfoundation.org" <gregkh@linuxfoundation.org>,
	"alex.williamson@redhat.com" <alex.williamson@redhat.com>,
	"davem@davemloft.net" <davem@davemloft.net>,
	"kvm@vger.kernel.org" <kvm@vger.kernel.org>,
	"netdev@vger.kernel.org" <netdev@vger.kernel.org>,
	Saeed Mahameed <saeedm@mellanox.com>,
	"kwankhede@nvidia.com" <kwankhede@nvidia.com>,
	"leon@kernel.org" <leon@kernel.org>,
	"cohuck@redhat.com" <cohuck@redhat.com>,
	Jiri Pirko <jiri@mellanox.com>,
	"linux-rdma@vger.kernel.org" <linux-rdma@vger.kernel.org>,
	Or Gerlitz <gerlitz.or@gmail.com>
Subject: Re: [PATCH net-next 00/19] Mellanox, mlx5 sub function support
Date: Fri, 8 Nov 2019 16:32:09 -0400	[thread overview]
Message-ID: <20191108203209.GF10956@ziepe.ca> (raw)
In-Reply-To: <AM0PR05MB4866299C3AE2448C8226DC00D17B0@AM0PR05MB4866.eurprd05.prod.outlook.com>

On Fri, Nov 08, 2019 at 08:20:43PM +0000, Parav Pandit wrote:
> 
> 
> > From: Jason Gunthorpe <jgg@ziepe.ca>
> > On Fri, Nov 08, 2019 at 11:12:38AM -0800, Jakub Kicinski wrote:
> > > On Fri, 8 Nov 2019 15:40:22 +0000, Parav Pandit wrote:
> > > > > The new intel driver has been having a very similar discussion
> > > > > about how to model their 'multi function device' ie to bind RDMA
> > > > > and other drivers to a shared PCI function, and I think that discussion
> > settled on adding a new bus?
> > > > >
> > > > > Really these things are all very similar, it would be nice to have
> > > > > a clear methodology on how to use the device core if a single PCI
> > > > > device is split by software into multiple different functional
> > > > > units and attached to different driver instances.
> > > > >
> > > > > Currently there is alot of hacking in this area.. And a consistent
> > > > > scheme might resolve the ugliness with the dma_ops wrappers.
> > > > >
> > > > > We already have the 'mfd' stuff to support splitting platform
> > > > > devices, maybe we need to create a 'pci-mfd' to support splitting PCI
> > devices?
> > > > >
> > > > > I'm not really clear how mfd and mdev relate, I always thought
> > > > > mdev was strongly linked to vfio.
> > > > >
> > > >
> > > > Mdev at beginning was strongly linked to vfio, but as I mentioned
> > > > above it is addressing more use case.
> > > >
> > > > I observed that discussion, but was not sure of extending mdev further.
> > > >
> > > > One way to do for Intel drivers to do is after series [9].
> > > > Where PCI driver says, MDEV_CLASS_ID_I40_FOO RDMA driver
> > > > mdev_register_driver(), matches on it and does the probe().
> > >
> > > Yup, FWIW to me the benefit of reusing mdevs for the Intel case vs
> > > muddying the purpose of mdevs is not a clear trade off.
> > 
> > IMHO, mdev has amdev_parent_ops structure clearly intended to link it to vfio,
> > so using a mdev for something not related to vfio seems like a poor choice.
> > 
> Splitting mdev_parent_ops{} is already in works for larger use case in series [1] for virtio.
> 
> [1] https://patchwork.kernel.org/patch/11233127/

Weird. So what is mdev actually providing and what does it represent
if the entire driver facing API surface is under a union?

This smells a lot like it is re-implementing a bus.. AFAIK bus is
supposed to represent the in-kernel API the struct device presents to
drivers.

Jason