Re: [RFC] current devlink extension plan for NICs

[Parav Pandit <parav@mellanox.com>]

On 3/24/2020 1:01 AM, Jakub Kicinski wrote:
> On Sat, 21 Mar 2020 09:07:30 +0000 Parav Pandit wrote:
>>> I see so you want the creation to be controlled by the same entity that
>>> controls the eswitch..
>>>
>>> To me the creation should be on the side that actually needs/will use
>>> the new port. And if it's not eswitch manager then eswitch manager
>>> needs to ack it.
>>>  
>>
>> There are few reasons to create them on eswitch manager system as below.
>>
>> 1. Creation and deletion on one system and synchronizing it with eswitch
>> system requires multiple back-n-forth calls between two systems.
>>
>> 2. When this happens, system where its created, doesn't know when is the
>> right time to provision to a VM or to a application.
>> udev/systemd/Network Manager and others such software might already
>> start initializing it doing DHCP but its switch side is not yet ready.
> 
> Networking software can deal with link down..
> 
Serving a half cooked device to an application is just not going to
work. It is not just link status.
A typical desired flow is:

1. create device
2. configure mac address
3. configure its rate limits
4. setup policy, encap/decap settings via tc offloads etc
5. bring up the link via rep
6. activate the device and attach it to application

Often administrator wants to assign/do (2), even though user is free to
change it later on.

This can only work if user system and eswitch has secure channel
established which often is just not available.

In other use case user system to define networkId is not trusted.
In this case there is some arbitrary/undefined on wait time at user host
to know that 2 to 6 are done.

vs doing step-1 to 6 on eswitch side by trusted entity and attaching it
to system where it is desired to use is elegant, secure.

networkID is read only for the system where this is deployed.

A application/vm/container may have one or more such devices that needs
to be present for the life of it regardless of its link status.

link down => detach device from container/vm
link up => attach device from container/vm is not right.
Hence port link status doesn't drive device status.

>> So it is desired to make sure that once device is fully
>> ready/configured, its activated.
>>
>> 3. Additionally it doesn't follow mirror sequence during deletion when
>> created on host.
> 
> Why so? Surely host needs to request deletion, otherwise container
> given only an SF could be cut off?
> 
creation from user system,
(a) create device
(b) configure device
(c) synchronous kick to create rep on other system (involving sw)

deletion from user system should be,
(d) synchronous kick to delete rep on other system (involving sw)
(e) unconfig the device
(f) delete the device

To achieve this mirror a sw synchronization is needed, not just with device.

Even if this is achieved somehow, it doesn't address the issue of
untrusted user system not having privilege to create the device with
given NetworkID.

>> 4. eswitch administrator simply doesn't have direct access to the system
>> where this device is used. So it just cannot be created there by eswitch
>> administrator.
> 
> Right, that is the point. It's the host admin that wants the new
> entity, so if possible it'd be better if they could just ask for it 
> via devlink rather than some cloud API. Not that I'm completely opposed
> to a cloud API - just seems unnecessary here.
> 

Flow is:
trusted_administator->cloud_api->smartnic->devlink_create,config,deploy->get_device_on_user_system.

untrusted_user_system->query_network_id->attach_to_container/vm/application.