All of lore.kernel.org
 help / color / mirror / Atom feed
From: Jiri Pirko <jiri@resnulli.us>
To: Jakub Kicinski <jakub.kicinski@netronome.com>
Cc: davem@davemloft.net, netdev@vger.kernel.org, oss-drivers@netronome.com
Subject: Re: [PATCH net-next v2 4/7] devlink: allow subports on devlink PCI ports
Date: Thu, 7 Mar 2019 10:48:16 +0100	[thread overview]
Message-ID: <20190307094816.GA2190@nanopsycho> (raw)
In-Reply-To: <20190306095638.7c028bdd@cakuba.hsd1.ca.comcast.net>

Wed, Mar 06, 2019 at 06:56:38PM CET, jakub.kicinski@netronome.com wrote:
>On Wed, 6 Mar 2019 13:20:37 +0100, Jiri Pirko wrote:
>> Tue, Mar 05, 2019 at 06:15:34PM CET, jakub.kicinski@netronome.com wrote:
>> >On Tue, 5 Mar 2019 12:06:01 +0100, Jiri Pirko wrote:  
>> >> >> >as ports.  Can we invent a new command (say "partition"?) that'd take
>> >> >> >the bus info where the partition is to be spawned?      
>> >> >> 
>> >> >> Got it. But the question is how different this object would be from the
>> >> >> existing "port" we have today.    
>> >> >
>> >> >They'd be where "the other side of a PCI link" is represented,
>> >> >restricting ports to only ASIC's forwarding plane ports.    
>> >> 
>> >> Basically a "host port", right? It can still be the same port object,
>> >> only with different flavour and attributes. So we would have:
>> >> 
>> >> 1) pci/0000:05:00.0/0: type eth netdev enp5s0np0
>> >>                        flavour physical switch_id 00154d130d2f
>> >> 2) pci/0000:05:00.0/10000: type eth netdev enp5s0npf0s0
>> >>                            flavour pci_pf pf 0 subport 0
>> >>                            switch_id 00154d130d2f
>> >>                            peer pci/0000:05:00.0/1
>> >> 3) pci/0000:05:00.0/10001: type eth netdev enp5s0npf0vf0
>> >>                            flavour pci_vf pf 0 vf 0
>> >>                            switch_id 00154d130d2f
>> >>                            peer pci/0000:05:10.1/0
>> >> 4) pci/0000:05:00.0/10001: type eth netdev enp5s0npf0s1
>> >>                            flavour pci_pf pf 0 subport 1
>> >>                            switch_id 00154d130d2f
>> >>                            peer pci/0000:05:00.0/2
>> >> 5) pci/0000:05:00.0/1: type eth netdev enp5s0f0??
>> >>                        flavour host          <----------------
>> >>                        peer pci/0000:05:00.0/10000
>> >> 6) pci/0000:05:10.1/0: type eth netdev enp5s10f0 
>> >>                        flavour host          <----------------
>> >>                        peer pci/0000:05:00.0/10001
>> >> 7) pci/0000:05:00.0/2: type eth netdev enp5s0f0??
>> >>                        flavour host          <----------------
>> >>                        peer pci/0000:05:00.0/10001
>> >> 
>> >> I think it looks quite clear, it gives complete topology view.  
>> >
>> >Okay, I have some of questions :)
>> >
>> >What do we use for port_index?  
>> 
>> That is just a number totally in control of the driver. Driver can
>> assign it in any way.
>> 
>> >
>> >What are the operations one can perform on "host ports"?  
>> 
>> That is a good question. I would start with *none* and extend it upon
>> needs.
>> 
>> 
>> >
>> >If we have PCI parameters, do they get set on the ASIC side of the port
>> >or the host side of the port?  
>> 
>> Could you give me an example?
>
>Let's take msix_vec_per_pf_min as an example.  
>
>> But I believe that on switch-port side.
>
>Ok.
>
>> >How do those behave when device is passed to VM?  
>> 
>> In case of VF? VF will have separate devlink instance (separate handle,
>> probably "aliased" to the PF handle). So it would disappear from
>> baremetal and appear in VM:
>> $ devlink dev
>> pci/0000:00:10.0
>> $ devlink dev port
>> pci/0000:00:10.1/0: type eth netdev enp5s10f0
>>                     flavour host
>> That's it for the VM.
>> 
>> There's no linkage (peer, alias) between this and the instances on
>> baremetal. 
>
>Ok, I guess this is the main advantage from your perspective?
>The fact that "host ports" are visible inside a VM?

Yep. Also on baremetal.


>Or do you believe that having both ends of a pipe as ports makes the
>topology easier to understand?

That as well.


>
>For creating subdevices, I don't think the handle should ever be port.
>We create new ports on a devlink instance, and configure its forwarding

Okay I agree. Something like:
$ devlink port add pci/0000:00:10.0 .....

It's a bit confusing because "set" accepts port handle (like
pci/0000:00:10.0/1). Probably better would be:
$ devlink dev port add pci/0000:00:10.0 .....


>with offloads of well established Linux SW constructs.  New devices are
>not logically associated with other ports (see how in my patches there
>are 2 "subports" but no main port on that PF - a split not a hierarchy).

Right, basically you have 2 equal objects. Makes sense.


>
>How we want to model forwarding inside a VM (who configures the
>underlying switching) remains unclear.

I don't understand. Could you elaborate a bit?


>
>> >You have a VF devlink instance there - what ports does it show?  
>> 
>> See above.
>> 
>> 
>> >
>> >How do those look when the PF is connected to another host?  Do they
>> >get spawned at all?  
>> 
>> What do you mean by "PF is connected to another host"?
>
>Either "SmartNIC":
>
>http://www.mellanox.com/products/smartnic/?ls=gppc&lsd=SmartNIC-gen-smartnic&gclid=EAIaIQobChMIxIrGmYju4AIVy5yzCh2SFwQJEAAYASAAEgIui_D_BwE
>
>or
>
>Multi-host NIC: http://www.mellanox.com/page/multihost

Right. So in this case, I think that the hostport on specific host
should see devlink instance and the hostport. However, the switchports
should be only on one selected host (I don't see how to do that
differently)


>
>> >Will this not be confusing to DSA folks who have a CPU port?  
>> 
>> Why do you think so?
>
>Host and CPU sound quite similar, it is unclear how they differ, and
>why we have a need for both (from user's perspective).

Hmm, dsa cpu port is something different. It does not have netdev
associated with it. It is just a port which is physically used in order
to send or receive packets on switch ports.

However in our hostport case, it has user facing netdev associated and
user actually uses it to send and receive packets directly (assigns ip
etc).


  parent reply	other threads:[~2019-03-07  9:58 UTC|newest]

Thread overview: 100+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2019-03-01 18:04 [PATCH net-next v2 0/7] devlink: expose PF and VF representors as ports Jakub Kicinski
2019-03-01 18:04 ` [PATCH net-next v2 1/7] nfp: split devlink port init from registration Jakub Kicinski
2019-03-01 18:04 ` [PATCH net-next v2 2/7] devlink: add PF and VF port flavours Jakub Kicinski
2019-03-01 18:04 ` [PATCH net-next v2 3/7] nfp: register devlink ports of all reprs Jakub Kicinski
2019-03-02  8:43   ` Jiri Pirko
2019-03-02 19:07     ` Jakub Kicinski
2019-03-04  7:36       ` Jiri Pirko
2019-03-04 23:32         ` Jakub Kicinski
2019-03-01 18:04 ` [PATCH net-next v2 4/7] devlink: allow subports on devlink PCI ports Jakub Kicinski
2019-03-02  9:41   ` Jiri Pirko
2019-03-02 19:48     ` Jakub Kicinski
2019-03-04  7:56       ` Jiri Pirko
2019-03-05  0:33         ` Jakub Kicinski
2019-03-05 11:06           ` Jiri Pirko
2019-03-05 17:15             ` Jakub Kicinski
2019-03-05 19:59               ` Parav Pandit
2019-03-06 12:20               ` Jiri Pirko
2019-03-06 17:56                 ` Jakub Kicinski
2019-03-07  3:56                   ` Parav Pandit
2019-03-07  9:48                   ` Jiri Pirko [this message]
2019-03-08  2:52                     ` Jakub Kicinski
2019-03-08 14:54                       ` Jiri Pirko
2019-03-08 19:09                         ` Jakub Kicinski
2019-03-11  8:52                           ` Jiri Pirko
2019-03-12  2:10                             ` Jakub Kicinski
2019-03-12 14:02                               ` Jiri Pirko
2019-03-12 20:56                                 ` Jakub Kicinski
2019-03-13  6:07                                   ` Jiri Pirko
2019-03-13 16:17                                     ` Jakub Kicinski
2019-03-13 16:22                                       ` Jiri Pirko
2019-03-13 16:55                                         ` Jakub Kicinski
2019-03-14  7:38                                           ` Jiri Pirko
2019-03-14 22:09                                             ` Jakub Kicinski
2019-03-14 22:35                                               ` Parav Pandit
2019-03-14 23:39                                                 ` Jakub Kicinski
2019-03-15  1:28                                                   ` Parav Pandit
2019-03-15  1:31                                                     ` Parav Pandit
2019-03-15  2:15                                                     ` Samudrala, Sridhar
2019-03-15  2:40                                                       ` Parav Pandit
     [not found]                                                         ` <ae938b4f-5fa9-3c33-8ae6-eab2d3d9f1ec@intel.com>
2019-03-15 15:32                                                           ` Parav Pandit
2019-03-15 20:08                                                             ` Jiri Pirko
2019-03-15 20:44                                                               ` Jakub Kicinski
2019-03-15 22:12                                                                 ` Parav Pandit
2019-03-16  1:16                                                                   ` Jakub Kicinski
2019-03-18 15:43                                                                     ` Parav Pandit
2019-03-18 19:29                                                                       ` Jakub Kicinski
2019-03-18 12:11                                                                 ` Jiri Pirko
2019-03-18 19:16                                                                   ` Jakub Kicinski
2019-03-21  8:45                                                                     ` Jiri Pirko
2019-03-21 15:14                                                                       ` Parav Pandit
2019-03-21 16:14                                                                         ` Jiri Pirko
2019-03-21 16:52                                                                           ` Parav Pandit
2019-03-21 17:20                                                                             ` Jiri Pirko
2019-03-21 17:34                                                                               ` Parav Pandit
2019-03-22 16:27                                                                                 ` Jiri Pirko
2019-03-23  0:37                                                                                   ` Parav Pandit
2019-03-15 21:59                                                               ` Parav Pandit
2019-03-18 12:21                                                                 ` Jiri Pirko
2019-03-18 15:56                                                                   ` Parav Pandit
2019-03-18 16:22                                                                     ` Parav Pandit
2019-03-18 19:36                                                                       ` Jakub Kicinski
2019-03-18 19:44                                                                         ` Parav Pandit
2019-03-18 19:59                                                                           ` Jakub Kicinski
2019-03-18 20:35                                                                             ` Parav Pandit
2019-03-18 21:29                                                                               ` Jakub Kicinski
2019-03-18 22:11                                                                                 ` Parav Pandit
2019-03-20 18:24                                                                                   ` Parav Pandit
2019-03-20 20:22                                                                                     ` Jakub Kicinski
2019-03-20 23:39                                                                                       ` Parav Pandit
2019-03-21  9:08                                                                                       ` Jiri Pirko
2019-03-21 15:03                                                                                         ` Parav Pandit
2019-03-21 16:16                                                                                           ` Jiri Pirko
2019-03-21 16:50                                                                                             ` Parav Pandit
2019-03-21 17:23                                                                                               ` Jiri Pirko
2019-03-21 17:42                                                                                                 ` Parav Pandit
2019-03-22 13:32                                                                                                   ` Jiri Pirko
2019-03-23  0:40                                                                                                     ` Parav Pandit
2019-03-25 20:34                                                                                                       ` Parav Pandit
2019-03-18 19:19                                                                   ` Jakub Kicinski
2019-03-18 19:38                                                                     ` Parav Pandit
2019-03-21  9:09                                                                     ` Jiri Pirko
2019-03-15  7:00                                               ` Jiri Pirko
     [not found]                                 ` <7227d58e-ac58-d549-b921-ca0a0dd3f4b0@intel.com>
2019-03-13  7:37                                   ` Jiri Pirko
2019-03-13 16:03                                     ` Samudrala, Sridhar
2019-03-13 16:24                                       ` Jiri Pirko
2019-03-04 11:19       ` Jiri Pirko
2019-03-05  0:40         ` Jakub Kicinski
2019-03-05 11:07           ` Jiri Pirko
2019-03-04 11:08   ` Jiri Pirko
2019-03-05  0:51     ` Jakub Kicinski
2019-03-05 11:09       ` Jiri Pirko
2019-03-01 18:04 ` [PATCH net-next v2 5/7] nfp: switch to devlink_port_get_phys_port_name() Jakub Kicinski
2019-03-01 18:04 ` [PATCH net-next v2 6/7] devlink: introduce port's peer netdevs Jakub Kicinski
2019-03-01 18:04 ` [PATCH net-next v2 7/7] nfp: expose PF " Jakub Kicinski
2019-03-02 10:13 ` [PATCH net-next v2 0/7] devlink: expose PF and VF representors as ports Jiri Pirko
2019-03-02 19:49   ` [oss-drivers] " Jakub Kicinski
2019-03-04  5:12   ` Parav Pandit
2019-03-04 18:22 ` David Miller
2019-03-20 20:25 ` Jakub Kicinski
2019-03-21  9:11   ` Jiri Pirko

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20190307094816.GA2190@nanopsycho \
    --to=jiri@resnulli.us \
    --cc=davem@davemloft.net \
    --cc=jakub.kicinski@netronome.com \
    --cc=netdev@vger.kernel.org \
    --cc=oss-drivers@netronome.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.