Re: [PATCH net-next v5] devlink: Add devlink port documentation

[Randy Dunlap <rdunlap@infradead.org>]

Hi--

On 12/7/20 2:13 PM, Parav Pandit wrote:
> Added documentation for devlink port and port function related commands.
> 
> Signed-off-by: Parav Pandit <parav@nvidia.com>
> Reviewed-by: Jiri Pirko <jiri@nvidia.com>
> Reviewed-by: Jacob Keller <jacob.e.keller@intel.com>
> ---
> Changelog:
> v4->v5:
>  - described logically ingress and egress point of devlink port
>  - removed networking from devlink port description
>  - rephrased port type description
>  - introdue PCI controller section and description

     introduce

>  - rephrased controller, device, function description
>  - removed confusing eswitch to system wording
>  - rephrased port function description
>  - added example of mac address in port function attribute description

> ---
>  .../networking/devlink/devlink-port.rst       | 116 ++++++++++++++++++
>  Documentation/networking/devlink/index.rst    |   1 +
>  2 files changed, 117 insertions(+)
>  create mode 100644 Documentation/networking/devlink/devlink-port.rst
> 
> diff --git a/Documentation/networking/devlink/devlink-port.rst b/Documentation/networking/devlink/devlink-port.rst
> new file mode 100644
> index 000000000000..dce87d2c07ac
> --- /dev/null
> +++ b/Documentation/networking/devlink/devlink-port.rst
> @@ -0,0 +1,116 @@
> +.. SPDX-License-Identifier: GPL-2.0
> +
> +============
> +Devlink Port
> +============
> +
> +``devlink-port`` is a port that exists on the device. It has a logically
> +separate ingress/egress point of the device. A devlink port can be of one

                                                          port can be any one
   of many flavours.

> +among many flavours. A devlink port flavour along with port attributes
> +describe what a port represents.
> +
> +A device driver that intends to publish a devlink port sets the
> +devlink port attributes and registers the devlink port.
> +
> +Devlink port flavours are described below.
> +
> +.. list-table:: List of devlink port flavours
> +   :widths: 33 90
> +
> +   * - Flavour
> +     - Description
> +   * - ``DEVLINK_PORT_FLAVOUR_PHYSICAL``
> +     - Any kind of physical networking port. This can be an eswitch physical
> +       port or any other physical port on the device.
> +   * - ``DEVLINK_PORT_FLAVOUR_DSA``
> +     - This indicates a DSA interconnect port.
> +   * - ``DEVLINK_PORT_FLAVOUR_CPU``
> +     - This indicates a CPU port applicable only to DSA.
> +   * - ``DEVLINK_PORT_FLAVOUR_PCI_PF``
> +     - This indicates an eswitch port representing a networking port of
> +       PCI physical function (PF).
> +   * - ``DEVLINK_PORT_FLAVOUR_PCI_VF``
> +     - This indicates an eswitch port representing a networking port of
> +       PCI virtual function (VF).
> +   * - ``DEVLINK_PORT_FLAVOUR_VIRTUAL``
> +     - This indicates a virtual port for the virtual PCI device such as PCI VF.
> +
> +Devlink port can have a different type based on the link layer described below.
> +
> +.. list-table:: List of devlink port types
> +   :widths: 23 90
> +
> +   * - Type
> +     - Description
> +   * - ``DEVLINK_PORT_TYPE_ETH``
> +     - Driver should set this port type when a link layer of the port is
> +       Ethernet.
> +   * - ``DEVLINK_PORT_TYPE_IB``
> +     - Driver should set this port type when a link layer of the port is
> +       InfiniBand.
> +   * - ``DEVLINK_PORT_TYPE_AUTO``
> +     - This type is indicated by the user when driver should detect the port
> +       type automatically.
> +
> +PCI controllers
> +---------------
> +In most cases PCI device has only one controller. A controller consists of

either           PCI devices have
or               a PCI device has

> +potentially multiple physical and virtual functions. Such PCI function consists
> +of one or more ports. This port of the function is represented by the devlink
> +eswitch port.
> +
> +A PCI Device connected to multiple CPUs or multiple PCI root complex or

                                                                complexes

> +SmartNIC, however, may have multiple controllers. For a device with multiple
> +controllers, each controller is distinguished by a unique controller number.
> +An eswitch on the PCI device may suppport ports of multiple controllers. 

                                    support

> +
> +An example view of two controller systems::

I think that this is

  An example view of a two-controller system::

instead of 2 controller systems.  ?

> +
> +In this example, external controller (identified by controller number = 1)
> +doesn't have eswitch. Local controller (identified by controller number = 0)
> +has the eswitch. Devlink instance on local controller has eswitch devlink
> +ports representing ports for both the controllers.
> +
> +                 ---------------------------------------------------------
> +                 |                                                       |
> +                 |           --------- ---------         ------- ------- |
> +    -----------  |           | vf(s) | | sf(s) |         |vf(s)| |sf(s)| |
> +    | server  |  | -------   ----/---- ---/----- ------- ---/--- ---/--- |
> +    | pci rc  |=== | pf0 |______/________/       | pf1 |___/_______/     |
> +    | connect |  | -------                       -------                 |
> +    -----------  |     | controller_num=1 (no eswitch)                   |
> +                 ------|--------------------------------------------------
> +                 (internal wire)
> +                       |
> +                 ---------------------------------------------------------
> +                 | devlink eswitch ports and reps                        |
> +                 | ----------------------------------------------------- |
> +                 | |ctrl-0 | ctrl-0 | ctrl-0 | ctrl-0 | ctrl-0 |ctrl-0 | |
> +                 | |pf0    | pf0vfN | pf0sfN | pf1    | pf1vfN |pf1sfN | |
> +                 | ----------------------------------------------------- |
> +                 | |ctrl-1 | ctrl-1 | ctrl-1 | ctrl-1 | ctrl-1 |ctrl-1 | |
> +                 | |pf0    | pf0vfN | pf0sfN | pf1    | pf1vfN |pf1sfN | |
> +                 | ----------------------------------------------------- |
> +                 |                                                       |
> +                 |                                                       |
> +    -----------  |           --------- ---------         ------- ------- |
> +    | smartNIC|  |           | vf(s) | | sf(s) |         |vf(s)| |sf(s)| |
> +    | pci rc  |==| -------   ----/---- ---/----- ------- ---/--- ---/--- |
> +    | connect |  | | pf0 |______/________/       | pf1 |___/_______/     |
> +    -----------  | -------                       -------                 |
> +                 |                                                       |
> +                 |  local controller_num=0 (eswitch)                     |
> +                 ---------------------------------------------------------
> +
> +Port function configuration
> +===========================
> +
> +A user can configure the port function attribute before enumerating the
> +PCI function. Usually it means, user should configure port function attribute
> +before a bus specific device for the function is created. However, when
> +SRIOV is enabled, virtual function devices are created on the PCI bus.
> +Hence, function attribute should be configured before binding virtual
> +function device to the driver.
> +
> +User may set the hardware address of the function represented by the devlink
> +port function. For Ethernet port function this means a MAC address.


-- 
~Randy