Re: [RFC] current devlink extension plan for NICs

From: Jason Gunthorpe <jgg@ziepe.ca>
To: Jakub Kicinski <kuba@kernel.org>
Cc: Parav Pandit <parav@mellanox.com>, Jiri Pirko <jiri@resnulli.us>,
	"netdev@vger.kernel.org" <netdev@vger.kernel.org>,
	"davem@davemloft.net" <davem@davemloft.net>,
	Yuval Avnery <yuvalav@mellanox.com>,
	Saeed Mahameed <saeedm@mellanox.com>,
	"leon@kernel.org" <leon@kernel.org>,
	"andrew.gospodarek@broadcom.com" <andrew.gospodarek@broadcom.com>,
	"michael.chan@broadcom.com" <michael.chan@broadcom.com>,
	Moshe Shemesh <moshe@mellanox.com>, Aya Levin <ayal@mellanox.com>,
	Eran Ben Elisha <eranbe@mellanox.com>,
	Vlad Buslov <vladbu@mellanox.com>,
	Yevgeny Kliteynik <kliteyn@mellanox.com>,
	"dchickles@marvell.com" <dchickles@marvell.com>,
	"sburla@marvell.com" <sburla@marvell.com>,
	"fmanlunas@marvell.com" <fmanlunas@marvell.com>,
	Tariq Toukan <tariqt@mellanox.com>,
	"oss-drivers@netronome.com" <oss-drivers@netronome.com>,
	"snelson@pensando.io" <snelson@pensando.io>,
	"drivers@pensando.io" <drivers@pensando.io>,
	"aelior@marvell.com" <aelior@marvell.com>,
	"GR-everest-linux-l2@marvell.com"
	<GR-everest-linux-l2@marvell.com>,
	"grygorii.strashko@ti.com" <grygorii.strashko@ti.com>,
	mlxsw <mlxsw@mellanox.com>, Ido Schimmel <idosch@mellanox.com>,
	Mark Zhang <markz@mellanox.com>,
	"jacob.e.keller@intel.com" <jacob.e.keller@intel.com>,
	Alex Vesker <valex@mellanox.com>,
	"linyunsheng@huawei.com" <linyunsheng@huawei.com>,
	"lihong.yang@intel.com" <lihong.yang@intel.com>,
	"vikas.gupta@broadcom.com" <vikas.gupta@broadcom.com>,
	"magnus.karlsson@intel.com" <magnus.karlsson@intel.com>
Subject: Re: [RFC] current devlink extension plan for NICs
Date: Tue, 24 Mar 2020 10:43:57 -0300	[thread overview]
Message-ID: <20200324134357.GJ20941@ziepe.ca> (raw)
In-Reply-To: <20200323204116.7c2f6e46@kicinski-fedora-pc1c0hjn.dhcp.thefacebook.com>

On Mon, Mar 23, 2020 at 08:41:16PM -0700, Jakub Kicinski wrote:
> On Mon, 23 Mar 2020 19:50:09 -0300 Jason Gunthorpe wrote:
> > On Mon, Mar 23, 2020 at 12:31:16PM -0700, Jakub Kicinski wrote:
> > 
> > > Right, that is the point. It's the host admin that wants the new
> > > entity, so if possible it'd be better if they could just ask for it 
> > > via devlink rather than some cloud API. Not that I'm completely opposed
> > > to a cloud API - just seems unnecessary here.  
> > 
> > The cloud API provides all the permissions checks and security
> > elements. It cannot be avoided.
> 
> Ack, the question is just who consults the cloud API, the Host or the
> SmartNIC (latter would abstract differences between cloud APIs).

I feel it is more natural to use the native cloud API. It will always
have the right feature set and authorization scheme. If we try to make
a lowest common denominator in devlink then it will probably be a poor
match, and possibly very complicated.

We have to support push auto-creation anyhow to make the 'device
pre-exists at boot time' case work right.

> > If you try to do it as you say then it is weird. You have to use the
> > cloud API to authorize the VM to touch a certain network, then the VM
> > has to somehow take that network ID and use devlink to get a netdev
> > for it. And the cloud side has to protect against a hostile VM sending
> > garbage along this communication channel.
> 
> I don't understand how the VM needs to know the network ID, quite the
> opposite, the Network ID should be gettable/settable by the hypervisor/
> /PF.

I don't follow, if I have thousands of vlans in my cloud world I need
some way to label them all. If the VM is going to ask for a specific
vlan to be plugged into it, then it needs to ask with the right label.

> > vs simply host plugging in the correct network fully operational when
> > the cloud API connects the VM to the network.
> 
> That means the user has to pre-allocate the device ID, or query the
> cloud API after the device is created about its attributes (in the case
> of two interfaces being requested simultaneously).

I suppose it would use MAC address matching, the cloud creation action
will indicate the MAC and the netdev will appear with that MAC when it
is ready

I think we could do a better job passing some kind of meta-information
into the guest..

Jason