RE: [PATCH RESEND iproute2-next] devlink: Add optional controller user input

[Parav Pandit <parav@nvidia.com>]

> From: Yunsheng Lin <linyunsheng@huawei.com>
> Sent: Monday, June 7, 2021 9:01 AM
> 
> On 2021/6/6 15:10, Parav Pandit wrote:
> > Hi Yunsheng,
> >
> >> From: Yunsheng Lin <linyunsheng@huawei.com>
> >> Sent: Friday, June 4, 2021 7:05 AM
> >>
> >> On 2021/6/3 19:19, Parav Pandit wrote:
> >>> A user optionally provides the external controller number when user
> >>> wants to create devlink port for the external controller.
> >>
> >> Hi, Parav
> >>    I was planing to use controller id to solve the devlink instance
> >> representing problem for multi-function which shares common resource
> >> in the same ASIC, see [1].
> >>
> >> It seems the controller id used here is to differentiate the function
> >> used in different host?
> >>
> > That’s correct. Controller holds one or more PCI functions (PF,VF,SF).
> 
> I am not sure I understand the exact usage of controller and why controller id
> is in "devlink_port_*_attrs".
> 
> Let's consider a simplified case where there is two PF(supposing both have
> VF enabled), and each PF has different controller and each PF corresponds to
> a different physical port(Or it is about multi-host case multi PF may sharing
> the same physical port?):
Typically single host with two PFs have their own physical ports.
Multi-host with two PFs, on each host they share respective physical ports.

Single host:
Pf0.physical_port = p0
Pf1.physical_port = p1.

Multi-host (two) host setup
H1.pf0.phyical_port = p0.
H1.pf1.phyical_port = p1.
H2.pf0.phyical_port = p0.
H2.pf1.phyical_port = p1.

> 1. I suppose each PF has it's devlink instance for mlx case(I suppose each
>    VF can not have it's own devlink instance for VF shares the same physical
>    port with PF, right?).
VF and SF ports are of flavour VIRTUAL.

> 2. each PF's devlink instance has three types of port, which is
>    FLAVOUR_PHYSICAL, FLAVOUR_PCI_PF and FLAVOUR_PCI_VF(supposing I
> understand
>    port flavour correctly).
> 
FLAVOUR_PCI_{PF,VF,SF} belongs to eswitch (representor) side on switchdev device.

> If I understand above correctly, all ports in the same devlink instance should
> have the same controller id, right? If yes, why not put the controller id in the
> devlink instance?
Need not be. All PCI_{PF,VF,SF} can have controller id different for different controllers.
Usually each multi-host is a different controller. 
Refer to this diagram [1] and detailed description.

> 
> > In your case if there is single devlink instance representing ASIC, it is better
> to have health reporters under this single instance.
> >
> > Devlink parameters do not span multiple devlink instance.
> 
> Yes, that is what I try to do: shared status/parameters in devlink instance,
> physical port specific status/parameters in devlink port instance.
> 
> > So if you need to control devlink instance parameters of each function
> byitself, you likely need devlink instance for each.
> > And still continue to have ASIC wide health reporters under single instance
> that represents whole ASIC.
> 
> I do not think each function need a devlink instance if there is a devlink
> instance representing a whole ASIC, using the devlink port instance to
> represent the function seems enough?
'devlink port function's is equivalent of hypervisor/nicvisor entity controlled by the network/sysadmin.
While devlink instance of a given PF,VF,SF is managed by the user of such function itself.
For example when a VF is mapped to a VM, devlink instance of this VF resides in the VM managed by the guest VM.

Before this VF is given to VM, usually hypervisor/nicvisor admin programs this function (such as mac address) explained in [3].
So that a given VM always gets the same mac address regardless of which VF {a or b).

[1] https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/Documentation/networking/devlink/devlink-port.rst?h=v5.13-rc5#n72
[2] https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/Documentation/networking/devlink/devlink-port.rst?h=v5.13-rc5#n60
[3] https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/Documentation/networking/devlink/devlink-port.rst?h=v5.13-rc5#n110

> 
> >
> >> 1. https://lkml.org/lkml/2021/5/31/296