RE: [PATCH net-next 00/19] Mellanox, mlx5 sub function support

[Parav Pandit <parav@mellanox.com>]

Hi Jason,

> -----Original Message-----
> From: Jason Gunthorpe <jgg@ziepe.ca>
> 
> On Fri, Nov 08, 2019 at 02:52:10PM -0700, Alex Williamson wrote:
> > > >
> > > > Unless there's some opposition, I'm intended to queue this for v5.5:
> > > >
> > > > https://www.spinics.net/lists/kvm/msg199613.html
> > > >
> > > > mdev has started out as tied to vfio, but at it's core, it's just
> > > > a device life cycle infrastructure with callbacks between bus
> > > > drivers and vendor devices.  If virtio is on the wrong path with
> > > > the above series, please speak up.  Thanks,
> > >
> > > Well, I think Greg just objected pretty strongly.
> > >
> > > IMHO it is wrong to turn mdev into some API multiplexor. That is
> > > what the driver core already does and AFAIK your bus type is
> > > supposed to represent your API contract to your drivers.
> > >
> > > Since the bus type is ABI, 'mdev' is really all about vfio I guess?
> > >
> > > Maybe mdev should grow by factoring the special GUID life cycle
> > > stuff into a helper library that can make it simpler to build proper
> > > API specific bus's using that lifecycle model? ie the virtio I saw
> > > proposed should probably be a mdev-virtio bus type providing this
> > > new virtio API contract using a 'struct mdev_virtio'?
> >
> > I see, the bus:API contract is more clear when we're talking about
> > physical buses and physical devices following a hardware
> > specification.
> 
> Well, I don't think it matters, this is a software contract inside the kernel
> between the 'struct foo_device' (as provided by the foo_bus) and the 'struct
> foo_driver'
> 
> This contract is certainly easier to define when a HW specification dictates
> basically how it works.
> 
> > But if we take PCI for example, each PCI device has it's own internal
> > API that operates on the bus API.  PCI bus drivers match devices based
> > on vendor and device ID, which defines that internal API, not the bus
> > API.
> 
> Yes, this matching is part of the API contract between the bus and device
> driver.
> 
> But all of the pci_* functions that accept a 'struct pci_device *' are also part of
> this API contract toward the driver.
> 
> > The bus API is pretty thin when we're talking virtual devices and
> > virtual buses though.  The bus "API" is essentially that lifecycle
> > management, so I'm having a bit of a hard time differentiating this
> 
> But Parav just pointed out to a virtio SW API that had something like
> 20 API entry points.
> 
> > instead?"  Essentially for virtual devices, we're dictating a bus per
> > device type, whereas it seemed like a reasonable idea at the time to
> 
> Well, what does a driver binding to a virtual device need to know?
> 
> The virtual device API should provide all of that information.
> 
> I think things like vfio and virtio APIs are very reasonable bus types. virtio in
> particular has a published 'hw-like' specification with some good layers that
> can build a bus API.
> 
> Not so sure about the very HW specific things like the Intel driver and these SFs.
> These will really only ever bind to one driver and seem to have no
> commonalities.
> 
> For those we either create a bus per driver-specific proprietary API (feels kind
> of wrong) or we have a generic bus essentially for managed multi-function
> hardware that uses a simple 'void *hw_data' as the driver API and some
> matching logic to support that.
> 
Its certainly important to use generic bus approach overall at kernel level so that every vendor doesn't define that own devlink flavor, id scheme, udev naming method, PM and so on.
(It is not just bus definition).

Coming to hw_data part, even if this subdev (vendor) bus is created, it can still exactly follow your foo_device example.

In fact my first published RFC [1] and its specific patch [2] was doing that.

probe() routine in series [1] didn't have PCI like struct subdev *, because I wanted to use the core's generic probe(),
However it still correct because probe() can reach out to foo_device using container_of().
And hence the *hw_data is also resolved.

So struct looks like,

struct subdev {
	struct device device;
	/* resource range */
	/* num of irq vectors */
	const char *hw_addr;
	[..]
};

struct mlx5_subdev {
	struct subdev device;
	[..];
};

I request to reconsider RFC [1] for multi-function SFs use with extension of device flavour as 'virtio', 'mlx5' etc in [3].

[1] https://lkml.org/lkml/2019/3/1/19
[2] https://lore.kernel.org/patchwork/patch/1046997/#1238851
[3] https://lkml.org/lkml/2019/3/1/25

> > create a common virtual device bus, but maybe it went into the weeds
> > when trying to figure out how device drivers match to devices on that
> > bus and actually interact with them.
> 
> I think it is important to focus on the the SW API the 'struct foo_device' is
> supposed to provide toward the driver that binds to it.
> 
> It should be a sensible API covering some well defined area..
> 
> Jason