Re: [PATCH net-next v2 4/7] devlink: allow subports on devlink PCI ports

[Jiri Pirko <jiri@resnulli.us>]

Wed, Mar 13, 2019 at 05:55:55PM CET, jakub.kicinski@netronome.com wrote:
>On Wed, 13 Mar 2019 17:22:43 +0100, Jiri Pirko wrote:
>> Wed, Mar 13, 2019 at 05:17:31PM CET, jakub.kicinski@netronome.com wrote:
>> >On Wed, 13 Mar 2019 07:07:01 +0100, Jiri Pirko wrote:  
>> >> Tue, Mar 12, 2019 at 09:56:28PM CET, jakub.kicinski@netronome.com wrote:  
>> >> >On Tue, 12 Mar 2019 15:02:39 +0100, Jiri Pirko wrote:    
>> >> >> Tue, Mar 12, 2019 at 03:10:54AM CET, wrote:    
>> >> >> >On Mon, 11 Mar 2019 09:52:04 +0100, Jiri Pirko wrote:      
>> >> >> >> Fri, Mar 08, 2019 at 08:09:43PM CET, wrote:      
>> >> >> >> >If the switchport is in the hypervisor then only the hypervisor can
>> >> >> >> >control switching/forwarding, correct?        
>> >> >> >> 
>> >> >> >> Correct.
>> >> >> >>       
>> >> >> >> >The primary use case for partitioning within a VM (of a VF) would be
>> >> >> >> >containers (and DPDK)?        
>> >> >> >> 
>> >> >> >> Makes sense.
>> >> >> >>       
>> >> >> >> >SR-IOV makes things harder.  Splitting a PF is reasonably easy to grasp.
>> >> >> >> >I'm trying to get a sense of is how would we control an SR-IOV
>> >> >> >> >environment as a whole.        
>> >> >> >> 
>> >> >> >> You mean orchestration?       
>> >> >> >
>> >> >> >Right, orchestration.
>> >> >> >
>> >> >> >To be clear on where I'm going with this - if we want to allow VFs 
>> >> >> >to partition themselves then they have to control what is effectively 
>> >> >> >a "nested" switch.  A per-VF set of rules which would the get      
>> >> >> 
>> >> >> Wait. If you allow to make VF subports (I believe that is what you ment
>> >> >> by VFs partition themselves), that does not mean they will have a
>> >> >> separate nested switch. They would still belong under the same one.    
>> >> >
>> >> >But that existing switch is administered by the hypervisor, how would
>> >> >the VF owners install forwarding rules in a switch they don't control?    
>> >> 
>> >> They won't.  
>> >
>> >Argh.  So how is forwarding configured if there are no rules?  Are you
>> >going to assume its switching on MACs?  We're supposed to offload
>> >software constructs.  If its a software port it needs to be explicitly
>> >switched.  If it's not explicitly switched - we already have macvlan
>> >offload.  
>> 
>> Wait a second. You configure the switch. And for that, you have the
>> switchports (representors). What we are talking about are VF (VF
>> subport) host legs. Am I missing something?
>
>Hm :)  So when VM gets a new port, how is it connected?  Are we
>assuming all ports of a VM are plugged into one big L2 switch?
>The use case for those sub ports is a little murky, sorry about
>the endless confusion :)

Np. When user John (on baremetal, or whenever the devlink instance
with switch port is) creates VF of VF subport by: 
$ devlink dev port add pci/0000:05:00.0 flavour pci_vf pf 0
or by:
$ devlink dev port add pci/0000:05:00.0 flavour pci_vf pf 0 vf 0

Then instances of flavour pci_vf are going to appear in the same devlink
instance. Those are the switch ports:
pci/0000:05:00.0/10002: type eth netdev enp5s0npf0pf0s0
                        flavour pci_vf pf 0 vf 0
                        switch_id 00154d130d2f peer pci/0000:05:10.1/0    
pci/0000:05:00.0/10003: type eth netdev enp5s0npf0pf0s0
                        flavour pci_vf pf 0 vf 0 subport 1
                        switch_id 00154d130d2f peer pci/0000:05:10.1/1

With that, peers are going to appear too, and those are the actual VF/VF
subport:
pci/0000:05:10.1/0: type eth netdev ??? flavour pci_vf_host
                    peer pci/0000:05:00.0/10002
pci/0000:05:10.1/1: type eth netdev ??? flavour pci_vf_host
                    peer pci/0000:05:00.0/10003

Later you can push this VF along with all subports to VM. So in VM, you
are going to see the VF like this:
$ devlink dev
pci/0000:00:08.0
$ devlink port
pci/0000:00:08.0/0: type eth netdev ??? flavour pci_vf_host
pci/0000:00:08.0/1: type eth netdev ??? flavour pci_vf_host

And back to your question of how are they connected in eswitch.
That is totally up to the original user John who did the creation.
He is in charge of the eswitch on baremetal, he would configure
the forwarding however he likes.