Re: [RFC] current devlink extension plan for NICs

From: Jiri Pirko <jiri@resnulli.us>
To: Jakub Kicinski <kuba@kernel.org>
Cc: netdev@vger.kernel.org, davem@davemloft.net, parav@mellanox.com,
	yuvalav@mellanox.com, jgg@ziepe.ca, saeedm@mellanox.com,
	leon@kernel.org, andrew.gospodarek@broadcom.com,
	michael.chan@broadcom.com, moshe@mellanox.com, ayal@mellanox.com,
	eranbe@mellanox.com, vladbu@mellanox.com, kliteyn@mellanox.com,
	dchickles@marvell.com, sburla@marvell.com, fmanlunas@marvell.com,
	tariqt@mellanox.com, oss-drivers@netronome.com,
	snelson@pensando.io, drivers@pensando.io, aelior@marvell.com,
	GR-everest-linux-l2@marvell.com, grygorii.strashko@ti.com,
	mlxsw@mellanox.com, idosch@mellanox.com, markz@mellanox.com,
	jacob.e.keller@intel.com, valex@mellanox.com,
	linyunsheng@huawei.com, lihong.yang@intel.com,
	vikas.gupta@broadcom.com, magnus.karlsson@intel.com
Subject: Re: [RFC] current devlink extension plan for NICs
Date: Thu, 26 Mar 2020 15:43:10 +0100	[thread overview]
Message-ID: <20200326144310.GV11304@nanopsycho.orion> (raw)
In-Reply-To: <20200323122123.2a3ff20f@kicinski-fedora-pc1c0hjn.dhcp.thefacebook.com>

>> >> Now the PF itself can have a "nested eswitch" to manage. The "parent
>> >> eswitch" where the PF was created would only see one leg to the "nested
>> >> eswitch".
>> >> 
>> >> This "nested eswitch management" might or might not be required. Depends
>> >> on a usecare. The question was, how to configure that I as a user
>> >> want this or not.  
>> >
>> >Ack. I'm extending your question. I think the question is not only who
>> >controls the eswitch but also which PFs share the eswitch.  
>> 
>> Yes.
>> 
>> >
>> >I think eswitch is just one capability, but SmartNIC will want to
>> >control which ports see what capabilities in general. crypto offloads
>> >and such.
>> >
>> >I presume in your model if host controls eswitch the smartNIC sees just  
>> 
>> host may control the "nested eswitch" in the SmartNIC case.
>
>By nested eswitch you mean eswitch between ports of the same Host, or
>within one PF? Then SmartNIC may switch between PFs or multiple hosts?

In general, each pf can manage a switch and have another pf underneath
which may manage the nested switch. This may go to more than 2 levels in
theory.

It is basically an independent switch with uplink to the higher switch.

It can be on the same host, or a different host. Does not matter.

>
>> >what what comes out of Hosts single "uplink"? What if SmartNIC wants
>> >the host to be able to control the forwarding but not loose the ability
>> >to tap the VF to VF traffic?  
>> 
>> You mean that the VF representors would be in both SmartNIC host and
>> host? I don't know how that could work. I think it has to be either
>> there or there.
>
>That'd certainly be easier. Without representors we can't even check
>traffic stats, though. SmartNIC may want to see the resource
>utilization of ports even if it doesn't see the ports. That's just a
>theoretical divagation, I don't think it's required.

Hmm, it would be really off. I don't think it would be possible to use
them both to do "write" config. Maybe only "read" for showing some stats
or something. Each would have netdev representor? One may be used to
send data and second not? That would be hell to maintain and understand :/