RE: [RFC] current devlink extension plan for NICs

From: Parav Pandit <parav@mellanox.com>
To: Jiri Pirko <jiri@resnulli.us>, Jakub Kicinski <kuba@kernel.org>
Cc: "netdev@vger.kernel.org" <netdev@vger.kernel.org>,
	"davem@davemloft.net" <davem@davemloft.net>,
	Yuval Avnery <yuvalav@mellanox.com>,
	"jgg@ziepe.ca" <jgg@ziepe.ca>,
	Saeed Mahameed <saeedm@mellanox.com>,
	"leon@kernel.org" <leon@kernel.org>,
	"andrew.gospodarek@broadcom.com" <andrew.gospodarek@broadcom.com>,
	"michael.chan@broadcom.com" <michael.chan@broadcom.com>,
	Moshe Shemesh <moshe@mellanox.com>, Aya Levin <ayal@mellanox.com>,
	Eran Ben Elisha <eranbe@mellanox.com>,
	Vlad Buslov <vladbu@mellanox.com>,
	Yevgeny Kliteynik <kliteyn@mellanox.com>,
	"dchickles@marvell.com" <dchickles@marvell.com>,
	"sburla@marvell.com" <sburla@marvell.com>,
	"fmanlunas@marvell.com" <fmanlunas@marvell.com>,
	Tariq Toukan <tariqt@mellanox.com>,
	"oss-drivers@netronome.com" <oss-drivers@netronome.com>,
	"snelson@pensando.io" <snelson@pensando.io>,
	"drivers@pensando.io" <drivers@pensando.io>,
	"aelior@marvell.com" <aelior@marvell.com>,
	"GR-everest-linux-l2@marvell.com"
	<GR-everest-linux-l2@marvell.com>,
	"grygorii.strashko@ti.com" <grygorii.strashko@ti.com>,
	mlxsw <mlxsw@mellanox.com>, Ido Schimmel <idosch@mellanox.com>,
	Mark Zhang <markz@mellanox.com>,
	"jacob.e.keller@intel.com" <jacob.e.keller@intel.com>,
	Alex Vesker <valex@mellanox.com>,
	"linyunsheng@huawei.com" <linyunsheng@huawei.com>,
	"lihong.yang@intel.com" <lihong.yang@intel.com>,
	"vikas.gupta@broadcom.com" <vikas.gupta@broadcom.com>,
	"magnus.karlsson@intel.com" <magnus.karlsson@intel.com>
Subject: RE: [RFC] current devlink extension plan for NICs
Date: Wed, 8 Apr 2020 05:10:50 +0000	[thread overview]
Message-ID: <AM0PR05MB4866B04EE343A12A83F1B4DAD1C00@AM0PR05MB4866.eurprd05.prod.outlook.com> (raw)
In-Reply-To: <20200402061643.GB2199@nanopsycho.orion>

> From: Jiri Pirko <jiri@resnulli.us>
> Sent: Thursday, April 2, 2020 11:47 AM
> 
> Wed, Apr 01, 2020 at 10:12:31PM CEST, kuba@kernel.org wrote:
> >On Wed, 1 Apr 2020 07:32:46 +0000 Parav Pandit wrote:
> >> > From: Jakub Kicinski <kuba@kernel.org>
> >> > Sent: Tuesday, March 31, 2020 11:03 PM
> >> >
> >> > On Tue, 31 Mar 2020 07:45:51 +0000 Parav Pandit wrote:
> >> > > > In fact very little belongs to the port in that model. So why
> >> > > > have PCI ports in the first place?
> >> > > >
> >> > > for few reasons.
> >> > > 1. PCI ports are establishing the relationship between eswitch
> >> > > port and its representor netdevice.
> >> > > Relying on plain netdev name doesn't work in certain pci topology
> >> > > where netdev name exceeds 15 characters.
> >> > > 2. health reporters can be at port level.
> >> >
> >> > Why? The health reporters we have not AFAIK are for FW and for
> >> > queues hanging. Aren't queues on the slice and FW on the device?
> >> There are multiple heath reporters per object.
> >> There are per q health reporters on the representor queues (and
> >> representors are attached to devlink port). Can someone can have
> >> representor netdev for an eswitch port without devlink port? No,
> >> net/core/devlink.c cross verify this and do WARN_ON. So devlink port
> >> for eswitch are linked to representors and are needed. Their
> >> existence is not a replacement for representing 'portion of the
> >> device'.
> >
> >I don't understand what you're trying to say. My question was why are
> >queues not on the "slice"? If PCIe resources are on the slice, then so
> >should be the health reporters.
> 
> I agree. These should be attached to the slice.
>
Representor netdev has txq and rxq.
Health reporters for this queue are attached to the txq and rxq.

Txq/rxq related health reporters can be linked to a slice, if that is what you meant.

> 
> >
> >> > > 3. In future at eswitch pci port, I will be adding dpipe support
> >> > > for the internal flow tables done by the driver.
> >> > > 4. There were inconsistency among vendor drivers in using/abusing
> >> > > phys_port_name of the eswitch ports. This is consolidated via
> >> > > devlink port in core. This provides consistent view among all
> >> > > vendor drivers.
> >> > >
> >> > > So PCI eswitch side ports are useful regardless of slice.
> >> > >
> >> > > >> Additionally devlink port object doesn't go through the same
> >> > > >> state machine as that what slice has to go through.
> >> > > >> So its weird that some devlink port has state machine and some
> >> > > >> doesn't.
> >> > > >
> >> > > > You mean for VFs? I think you can add the states to the API.
> >> > > >
> >> > > As we agreed above that eswitch side objects (devlink port and
> >> > > representor netdev) should not be used for 'portion of device',
> >> >
> >> > We haven't agreed, I just explained how we differ.
> >>
> >> You mentioned that " Right, in my mental model representor _is_ a
> >> port of the eswitch, so repr would not make sense to me."
> >>
> >> With that I infer that 'any object that is directly and _always_
> >> linked to eswitch and represents an eswitch port is out of question,
> >> this includes devlink port of eswitch and netdev representor. Hence,
> >> the comment 'we agree conceptually' to not involve devlink port of
> >> eswitch and representor netdev to represent 'portion of the device'.
> >
> >I disagree, repr is one to one with eswitch port. Just because repr is
> >associated with a devlink port doesn't mean devlink port must be
> >associated with a repr or a netdev.