RE: [PATCH net-next v2 4/7] devlink: allow subports on devlink PCI ports

[Parav Pandit <parav@mellanox.com>]

> -----Original Message-----
> From: netdev-owner@vger.kernel.org <netdev-owner@vger.kernel.org> On
> Behalf Of Parav Pandit
> Sent: Monday, March 18, 2019 10:57 AM
> To: Jiri Pirko <jiri@resnulli.us>
> Cc: Samudrala, Sridhar <sridhar.samudrala@intel.com>; Jakub Kicinski
> <jakub.kicinski@netronome.com>; davem@davemloft.net;
> netdev@vger.kernel.org; oss-drivers@netronome.com
> Subject: RE: [PATCH net-next v2 4/7] devlink: allow subports on devlink PCI
> ports
> 
> 
> 
> > -----Original Message-----
> > From: Jiri Pirko <jiri@resnulli.us>
> > Sent: Monday, March 18, 2019 7:21 AM
> > To: Parav Pandit <parav@mellanox.com>
> > Cc: Samudrala, Sridhar <sridhar.samudrala@intel.com>; Jakub Kicinski
> > <jakub.kicinski@netronome.com>; davem@davemloft.net;
> > netdev@vger.kernel.org; oss-drivers@netronome.com
> > Subject: Re: [PATCH net-next v2 4/7] devlink: allow subports on
> > devlink PCI ports
> >
> > Fri, Mar 15, 2019 at 10:59:33PM CET, parav@mellanox.com wrote:
> > >
> > >
> > >> -----Original Message-----
> > >> From: Jiri Pirko <jiri@resnulli.us>
> > >> Sent: Friday, March 15, 2019 3:08 PM
> > >> To: Parav Pandit <parav@mellanox.com>
> > >> Cc: Samudrala, Sridhar <sridhar.samudrala@intel.com>; Jakub
> > >> Kicinski <jakub.kicinski@netronome.com>; davem@davemloft.net;
> > >> netdev@vger.kernel.org; oss-drivers@netronome.com
> > >> Subject: Re: [PATCH net-next v2 4/7] devlink: allow subports on
> > >> devlink PCI ports
> > >>
> > >> Fri, Mar 15, 2019 at 04:32:24PM CET, parav@mellanox.com wrote:
> > >> >
> > >> >
> > >> >> -----Original Message-----
> > >> >> From: Samudrala, Sridhar <sridhar.samudrala@intel.com>
> > >> >> Sent: Friday, March 15, 2019 12:58 AM
> > >> >> To: Parav Pandit <parav@mellanox.com>; Jakub Kicinski
> > >> >> <jakub.kicinski@netronome.com>
> > >> >> Cc: Jiri Pirko <jiri@resnulli.us>; davem@davemloft.net;
> > >> >> netdev@vger.kernel.org; oss-drivers@netronome.com
> > >> >> Subject: Re: [PATCH net-next v2 4/7] devlink: allow subports on
> > >> >> devlink PCI ports
> > >> >>
> > >> >>
> > >> >> On 3/14/2019 7:40 PM, Parav Pandit wrote:
> > >> >> >
> > >> >> >
> > >> >> >> -----Original Message-----
> > >> >> >> From: Samudrala, Sridhar <sridhar.samudrala@intel.com>
> > >> >> >> Sent: Thursday, March 14, 2019 9:16 PM
> > >> >> >> To: Parav Pandit <parav@mellanox.com>; Jakub Kicinski
> > >> >> >> <jakub.kicinski@netronome.com>
> > >> >> >> Cc: Jiri Pirko <jiri@resnulli.us>; davem@davemloft.net;
> > >> >> >> netdev@vger.kernel.org; oss-drivers@netronome.com
> > >> >> >> Subject: Re: [PATCH net-next v2 4/7] devlink: allow subports
> > >> >> >> on devlink PCI ports
> > >> >> >>
> > >> >> >>
> > >> >> >>
> > >> >> >> On 3/14/2019 6:28 PM, Parav Pandit wrote:
> > >> >> >>>
> > >> >> >>>
> > >> >> >>>> -----Original Message-----
> > >> >> >>>> From: Jakub Kicinski <jakub.kicinski@netronome.com>
> > >> >> >>>> Sent: Thursday, March 14, 2019 6:39 PM
> > >> >> >>>> To: Parav Pandit <parav@mellanox.com>
> > >> >> >>>> Cc: Jiri Pirko <jiri@resnulli.us>; davem@davemloft.net;
> > >> >> >>>> netdev@vger.kernel.org; oss-drivers@netronome.com
> > >> >> >>>> Subject: Re: [PATCH net-next v2 4/7] devlink: allow
> > >> >> >>>> subports on devlink PCI ports
> > >> >> >>>>
> > >> >> >>>> On Thu, 14 Mar 2019 22:35:36 +0000, Parav Pandit wrote:
> > >> >> >>>>>>> Then instances of flavour pci_vf are going to appear in
> > >> >> >>>>>>> the same devlink instance. Those are the switch ports:
> > >> >> >>>>>>> pci/0000:05:00.0/10002: type eth netdev enp5s0npf0pf0s0
> > >> >> >>>>>>>                           flavour pci_vf pf 0 vf 0
> > >> >> >>>>>>>                           switch_id 00154d130d2f peer
> > >> >> >>>>>>> pci/0000:05:10.1/0
> > >> >> >>>>>>> pci/0000:05:00.0/10003: type eth netdev enp5s0npf0pf0s0
> > >> >> >>>>>>>                           flavour pci_vf pf 0 vf 0 subport 1
> > >> >> >>>>>>>                           switch_id 00154d130d2f peer
> > >> >> >>>>>>> pci/0000:05:10.1/1
> > >> >> >>>>>>>
> > >> >> >>>>>>> With that, peers are going to appear too, and those are
> > >> >> >>>>>>> the actual VF/VF
> > >> >> >>>>>>> subport:
> > >> >> >>>>>>> pci/0000:05:10.1/0: type eth netdev ??? flavour pci_vf_host
> > >> >> >>>>>>>                       peer pci/0000:05:00.0/10002
> > >> >> >>>>>>> pci/0000:05:10.1/1: type eth netdev ??? flavour pci_vf_host
> > >> >> >>>>>>>                       peer pci/0000:05:00.0/10003
> > >> >> >>>>>>>
> > >> >> >>>>>>> Later you can push this VF along with all subports to VM.
> > >> >> >>>>>>> So in VM, you are going to see the VF like this:
> > >> >> >>>>>>> $ devlink dev
> > >> >> >>>>>>> pci/0000:00:08.0
> > >> >> >>>>>>> $ devlink port
> > >> >> >>>>>>> pci/0000:00:08.0/0: type eth netdev ??? flavour
> > >> >> >>>>>>> pci_vf_host
> > >> >> >>>>>>> pci/0000:00:08.0/1: type eth netdev ??? flavour
> > >> >> >>>>>>> pci_vf_host
> > >> >> >>>>>>>
> > >> >> >>>>>>> And back to your question of how are they connected in
> > eswitch.
> > >> >> >>>>>>> That is totally up to the original user John who did the
> > creation.
> > >> >> >>>>>>> He is in charge of the eswitch on baremetal, he would
> > >> >> >>>>>>> configure the forwarding however he likes.
> > >> >> >>>>>>
> > >> >> >>>>>> Ack, so I think you're saying VM has to communicate to
> > >> >> >>>>>> the cloud environment to have this provisioned using some
> > >> >> >>>>>> service API, not a kernel API.  That's what I wanted to
> confirm.
> > >> >> >>>>>>
> > >> >> >>>>>> I don't see any benefit to having the "host ports" under
> > >> >> >>>>>> devlink, as such I think it's a matter of preference.
> > >> >> >>>>>
> > >> >> >>>>> We need 'host ports' to configure parameters of this host
> > >> >> >>>>> port which is not exposed by the rep-netdev.
> > >> >> >>>>> Such as mac address.
> > >> >> >>>>
> > >> >> >>>> Please look at the quote of what Jiri wrote above - the
> > >> >> >>>> host port gets passed to the VM, you can't use it as a
> > >> >> >>>> handle to set the
> > >> MAC.
> > >> >> >>>>
> > >> >> >>>> The way to set the MAC remains:
> > >> >> >>>>
> > >> >> >>>> # devlink port set pci/0000:05:00.0/10002 peer mac_addr
> > >> >> >>>> 00:11:22:33:44:55
> > >> >> >>>>
> > >> >> >>> Even though it can be done, I think this is wrong model to
> > >> >> >>> program
> > >> >> >> hostport mac address using eswitch port.
> > >> >> >>> All devlink objects are control objects, so what is passed
> > >> >> >>> to VM is what is
> > >> >> >> represented by devlink.
> > >> >> >>> VF in the VM will anyway create its devlink object.
> > >> >> >>> What is wrong in programming hostport?
> > >> >> >>> It gives a very clear view to users of topology and objects.
> > >> >> >>
> > >> >> >> The VF or any subport MAC address should be configured by the
> > >> >> >> orchestration layer that is running on the hypervisor and
> > >> >> >> when a VF is assigned to a VF, the host port is not visible
> > >> >> >> to the
> > hypervisor.
> > >> >> > What prevents  creation of hostport due to which is not visible?
> > >> >> > Hostport is control port to program host side of parameters.
> > >> >> > It should be created when user wants to program the parameters.
> > >> >> >
> > >> >> > Model is really straight forward.
> > >> >> > Program host port params using hostport object.
> > >> >> > Program switchport params using rep-netdev.
> > >> >>
> > >> >> IIUC, Jiri/Jakub are proposing creation of 2 devlink objects for
> > >> >> each port - host facing ports and switch facing ports. This is
> > >> >> in addition to the netdevs that are created today.
> > >> >>
> > >> >I am not proposing any different.
> > >> >I am proposing only two changes.
> > >> >1. control hostport params via referring hostport (not via
> > >> >indirect
> > >> >peer)
> > >>
> > >> Not really possible. If you passthrough VF into VM, the hostport
> > >> goes along with it.
> > >>
> > >No.
> > >I am sorry in showing the enumeration which is the source of confusion.
> > >
> > >Below is the right enumeration.
> > >
> > >When VF is enumerated initially in the host, where eswitch devlink
> > >instance
> > is located.
> > >Below enumeration is seen.
> > >
> > >First two entries shows the link between hostport and switchport.
> > >$ devlink port show
> > >pci/0000:05:00.0/10002 eth netdev flavour switchport switch_id
> > >00154d130d2f peer pci/0000:05:00.0/1
> > >
> > >pci/0000:05:00.0/1 eth netdev flavour hostport switch_id 00154d130d2f
> > >peer pci/0000:05:00.0/10002
> >
> > Hostport should not have switch_id.
> >
> > >
> > >pci/0000:05:10.1/0 eth netdev flavour hostport This entry won't be
> > >seen if VF auto probing is disabled. Because than VF is not enumerated.
> > >
> > >As a user, I will be programming the mac address of hostport for a VF.
> > >pci/0000:05:00.0/1 eth netdev flavour hostport switch_id 00154d130d2f
> > >peer pci/0000:05:00.0/10002
> >
> > Hmm, so you are going to have 2 hostports for VF:
> > 1) pci/0000:05:10.1/0
> >    real one, that is going to go to VM - with a separate pci address
> >    and devlink instance.
> 
> Yep. This is the one where Yonatan's port grouping APIs work on.
> 
> > 2) pci/0000:05:00.0/1
> >    dummy one, which is not really a hostport, as there is no netdev
> >    created for it. It only models the other side of cable, which is away
> >    in VM.
> >
> Right. This is the control object which typically hypervisor programs.
> 
> > >
> > >
> > >>
> > >> >2. flavour should not be vf/pf, flavour should be hostport, switchport.
> > >> >Because switch is flat and agnostic of pf/vf/mdev.
> > >>
> > >> Not sure. It's good to have this kind of visibility.
> > >>
> > >port can have label/attribute indicating that this belong to VF-1 or
> > >mdev as
> > long as you are agreeing to have mdev attribute on host port.
> > >(and not ask for abstracting it, because mdev is well defined kernel
> object).
> >
> > Why mdev cannot be another flavour?
> >
> 
> hostport is of type pf/vf/mdev connected to some switchport.
> 
> So proposal is to have,
> port flavour = hostport/switchport
> port type/label = pf/vf/mdev
> 
Instead of having two attributes per port, how about having,
port flavour= physical/cpu/dsa/pf/vf/mdev/switchport.

physical and pf has some overlapping definitions.