RE: [PATCH net-next v2 4/7] devlink: allow subports on devlink PCI ports

[Parav Pandit <parav@mellanox.com>]

> -----Original Message-----
> From: Samudrala, Sridhar <sridhar.samudrala@intel.com>
> Sent: Friday, March 15, 2019 12:58 AM
> To: Parav Pandit <parav@mellanox.com>; Jakub Kicinski
> <jakub.kicinski@netronome.com>
> Cc: Jiri Pirko <jiri@resnulli.us>; davem@davemloft.net;
> netdev@vger.kernel.org; oss-drivers@netronome.com
> Subject: Re: [PATCH net-next v2 4/7] devlink: allow subports on devlink PCI
> ports
> 
> 
> On 3/14/2019 7:40 PM, Parav Pandit wrote:
> >
> >
> >> -----Original Message-----
> >> From: Samudrala, Sridhar <sridhar.samudrala@intel.com>
> >> Sent: Thursday, March 14, 2019 9:16 PM
> >> To: Parav Pandit <parav@mellanox.com>; Jakub Kicinski
> >> <jakub.kicinski@netronome.com>
> >> Cc: Jiri Pirko <jiri@resnulli.us>; davem@davemloft.net;
> >> netdev@vger.kernel.org; oss-drivers@netronome.com
> >> Subject: Re: [PATCH net-next v2 4/7] devlink: allow subports on
> >> devlink PCI ports
> >>
> >>
> >>
> >> On 3/14/2019 6:28 PM, Parav Pandit wrote:
> >>>
> >>>
> >>>> -----Original Message-----
> >>>> From: Jakub Kicinski <jakub.kicinski@netronome.com>
> >>>> Sent: Thursday, March 14, 2019 6:39 PM
> >>>> To: Parav Pandit <parav@mellanox.com>
> >>>> Cc: Jiri Pirko <jiri@resnulli.us>; davem@davemloft.net;
> >>>> netdev@vger.kernel.org; oss-drivers@netronome.com
> >>>> Subject: Re: [PATCH net-next v2 4/7] devlink: allow subports on
> >>>> devlink PCI ports
> >>>>
> >>>> On Thu, 14 Mar 2019 22:35:36 +0000, Parav Pandit wrote:
> >>>>>>> Then instances of flavour pci_vf are going to appear in the same
> >>>>>>> devlink instance. Those are the switch ports:
> >>>>>>> pci/0000:05:00.0/10002: type eth netdev enp5s0npf0pf0s0
> >>>>>>>                           flavour pci_vf pf 0 vf 0
> >>>>>>>                           switch_id 00154d130d2f peer
> >>>>>>> pci/0000:05:10.1/0
> >>>>>>> pci/0000:05:00.0/10003: type eth netdev enp5s0npf0pf0s0
> >>>>>>>                           flavour pci_vf pf 0 vf 0 subport 1
> >>>>>>>                           switch_id 00154d130d2f peer
> >>>>>>> pci/0000:05:10.1/1
> >>>>>>>
> >>>>>>> With that, peers are going to appear too, and those are the
> >>>>>>> actual VF/VF
> >>>>>>> subport:
> >>>>>>> pci/0000:05:10.1/0: type eth netdev ??? flavour pci_vf_host
> >>>>>>>                       peer pci/0000:05:00.0/10002
> >>>>>>> pci/0000:05:10.1/1: type eth netdev ??? flavour pci_vf_host
> >>>>>>>                       peer pci/0000:05:00.0/10003
> >>>>>>>
> >>>>>>> Later you can push this VF along with all subports to VM. So in
> >>>>>>> VM, you are going to see the VF like this:
> >>>>>>> $ devlink dev
> >>>>>>> pci/0000:00:08.0
> >>>>>>> $ devlink port
> >>>>>>> pci/0000:00:08.0/0: type eth netdev ??? flavour pci_vf_host
> >>>>>>> pci/0000:00:08.0/1: type eth netdev ??? flavour pci_vf_host
> >>>>>>>
> >>>>>>> And back to your question of how are they connected in eswitch.
> >>>>>>> That is totally up to the original user John who did the creation.
> >>>>>>> He is in charge of the eswitch on baremetal, he would configure
> >>>>>>> the forwarding however he likes.
> >>>>>>
> >>>>>> Ack, so I think you're saying VM has to communicate to the cloud
> >>>>>> environment to have this provisioned using some service API, not
> >>>>>> a kernel API.  That's what I wanted to confirm.
> >>>>>>
> >>>>>> I don't see any benefit to having the "host ports" under devlink,
> >>>>>> as such I think it's a matter of preference.
> >>>>>
> >>>>> We need 'host ports' to configure parameters of this host port
> >>>>> which is not exposed by the rep-netdev.
> >>>>> Such as mac address.
> >>>>
> >>>> Please look at the quote of what Jiri wrote above - the host port
> >>>> gets passed to the VM, you can't use it as a handle to set the MAC.
> >>>>
> >>>> The way to set the MAC remains:
> >>>>
> >>>> # devlink port set pci/0000:05:00.0/10002 peer mac_addr
> >>>> 00:11:22:33:44:55
> >>>>
> >>> Even though it can be done, I think this is wrong model to program
> >> hostport mac address using eswitch port.
> >>> All devlink objects are control objects, so what is passed to VM is
> >>> what is
> >> represented by devlink.
> >>> VF in the VM will anyway create its devlink object.
> >>> What is wrong in programming hostport?
> >>> It gives a very clear view to users of topology and objects.
> >>
> >> The VF or any subport MAC address should be configured by the
> >> orchestration layer that is running on the hypervisor and when a VF
> >> is assigned to a VF, the host port is not visible to the hypervisor.
> > What prevents  creation of hostport due to which is not visible?
> > Hostport is control port to program host side of parameters.
> > It should be created when user wants to program the parameters.
> >
> > Model is really straight forward.
> > Program host port params using hostport object.
> > Program switchport params using rep-netdev.
> 
> IIUC, Jiri/Jakub are proposing creation of 2 devlink objects for each port -
> host facing ports and switch facing ports. This is in addition to the netdevs
> that are created today.
> 
I am not proposing any different.
I am proposing only two changes.
1. control hostport params via referring hostport (not via indirect peer)
2. flavour should not be vf/pf, flavour should be hostport, switchport.
Because switch is flat and agnostic of pf/vf/mdev.

> Are you suggesting that all the devlink objects should be visible only at the
> hypervisor layer?
> 
Of course not.

Ports and params controlled by hypervisor should be exposed at hypervisor/eswitch wherever its parent devlink instance exist.
Ports which should be visible inside a VM should be exposed inside a VM.
So for a given VF,

If eswitch is at hypervisor level,
$ devlink port show
pci/0000:05:00.0/10002 eth netdev flavour switchport switch_id 00154d130d2f peer pci/0000:05:10.1/0
pci/0000:05:10.1/0 eth netdev flavour hostport switch_id 00154d130d2f peer pci/0000:05:00.0/10002

where VF is enumerated,
$ devlink port show
pci/0000:05:10.1/0 eth netdev flavour hostport
This is because unprivileged VF doesn't have visibility to eswitch and its links.

> I think the terminology need to be defined clearly so that we are all on the
> same page.
> 
> >
> >> Currently we have ndo_set_vf_mac_addr api that works with PF netdev,
> >> but i think we are trying to move away from that API and do all the
> >> configuration via the port representor netdevs.
> > This is fine rep-netdev represents eswitch port.
> > You normally don't go to switch to program host port params.
> >
> >> As the mac address cannot be configured using this netdev, i think
> >> Jakub is suggesting creating a devlink opject for each port
> >> representor and use that interface to set peer mac address.
> >
> > I understand but is convoluted interface.
> > When you program host NIC mac address you talk to iLo or BIOS.
> > When you program switch side mac address, you go switch/router/modem.
> >
> > Also programming host params on host side, also doesn't make
> assumption that its connected to eswitch.
> > It also doesn't assume that same connectivity for its life.
> >
> > If you model around how physical devices are configured, it will almost
> never go wrong and still provides same level of flexibility.
> >
> >> We should be able use this to configure port vlan too.
> >>
> >> Also, instead of subport, can we call vport and support different
> >> types of vports - sr-iov, siov, vmdq etc.
> >>
> > At switch level there are just ports.
> > sriov, siov, mdev, vmdq are their couter part (peer) where it is connected.
> >
> >>>
> >>> Also eswitch is flat. There is no need of pf/vf flavour for port.
> >>> It doesn't make sense to define 'mdev' flavour which we are already
> >> working.
> >>> At eswitch level it is just a port, it happen to be connected to vf
> >>> or pf or
> >> other objects, it doesn't matter.
> >>> Port should be flavoured as 'hostport' or 'switchport'.
> >>>
> >>>
> >>>> (using the port ids from above)