Re: [PATCH net-next 4/8] devlink: allow subports on devlink PCI ports

From: Jason Gunthorpe <jgg@mellanox.com>
To: Jakub Kicinski <jakub.kicinski@netronome.com>
Cc: Jiri Pirko <jiri@resnulli.us>,
	"davem@davemloft.net" <davem@davemloft.net>,
	"oss-drivers@netronome.com" <oss-drivers@netronome.com>,
	"netdev@vger.kernel.org" <netdev@vger.kernel.org>,
	Parav Pandit <parav@mellanox.com>
Subject: Re: [PATCH net-next 4/8] devlink: allow subports on devlink PCI ports
Date: Mon, 4 Mar 2019 16:15:14 +0000	[thread overview]
Message-ID: <20190304161510.GO8627@mellanox.com> (raw)
In-Reply-To: <20190227103000.6ea6f7c0@cakuba.netronome.com>

On Wed, Feb 27, 2019 at 10:30:00AM -0800, Jakub Kicinski wrote:
> On Wed, 27 Feb 2019 13:37:53 +0100, Jiri Pirko wrote:
> > Tue, Feb 26, 2019 at 07:24:32PM CET, jakub.kicinski@netronome.com wrote:
> > >PCI endpoint corresponds to a PCI device, but such device
> > >can have one more more logical device ports associated with it.
> > >We need a way to distinguish those. Add a PCI subport in the
> > >dumps and print the info in phys_port_name appropriately.
> > >
> > >This is not equivalent to port splitting, there is no split
> > >group. It's just a way of representing multiple netdevs on
> > >a single PCI function.
> > >
> > >Note that the quality of being multiport pertains only to
> > >the PCI function itself. A PF having multiple netdevs does
> > >not mean that its VFs will also have multiple, or that VFs
> > >are associated with any particular port of a multiport VF.
> > 
> > We've been discussing the problem of subport (we call it "subfunction"
> > or "SF") for some time internally. Turned out, this is probably harder
> > task to model. Please prove me wrong.
> > 
> > The nature of VF makes it a logically separate entity. It has a separate
> > PCI address, it should therefore have a separate devlink instance.
> > You can pass it through to VM, then the same devlink instance should be
> > created inside the VM and disappear from the host.
> 
> Depends what a devlink instance represents :/  On one hand you may want
> to create an instance for a VF to allow it to spawn soft ports, on the
> other you may want to group multiple functions together.
> 
> IOW if devlink instance is for an ASIC, there should be one per device
> per host.  

Don't we already have devlink instances for every mlx5 physical port
and VF as they are unique PCI functions?

> You guys come from the RDMA side of the world, with which I'm less
> familiar, and the soft bus + spawning devices seems to be a popular
> design there.  Could you describe the advantages of that model for 
> the sake of the netdev-only folks? :)

I don't think we do this in RDMA at all yet, or maybe I'm not sure
what you are thinking of?

The forward looking discussion is mainly to create something like
macvlan that can be offloaded, so we have something like a 'rdma
offload HW context' for each 'rdma macvlan'

.. and that 'rdma offload HW context' has all the same knobs as an
offload context for a VF that would normally be tied to a unique PCI
BDF (via devlink). But in this case there is no unique BDF.

From another angle this is broadly similar to the scalable IOV stuff,
but without placing requirements on the host IOMMU to implement it.

Jason