From: Parav Pandit <parav@mellanox.com>
To: kuba@kernel.org, davem@davemloft.net, netdev@vger.kernel.org
Cc: Parav Pandit <parav@nvidia.com>, Jiri Pirko <jiri@nvidia.com>
Subject: [PATCH net-next v2 5/6] devlink: Introduce controller number
Date: Tue, 8 Sep 2020 17:42:40 +0300 [thread overview]
Message-ID: <20200908144241.21673-6-parav@mellanox.com> (raw)
In-Reply-To: <20200908144241.21673-1-parav@mellanox.com>
From: Parav Pandit <parav@nvidia.com>
A devlink port may be for a controller consist of PCI device.
A devlink instance holds ports of two types of controllers.
(1) controller discovered on same system where eswitch resides
This is the case where PCI PF/VF of a controller and devlink eswitch
instance both are located on a single system.
(2) controller located on external host system.
This is the case where a controller is located in one system and its
devlink eswitch ports are located in a different system.
When a devlink eswitch instance serves the devlink ports of both
controllers together, PCI PF/VF numbers may overlap.
Due to this a unique phys_port_name cannot be constructed.
For example in below such system controller-A and controller-B, each has
PCI PF pf0 whose eswitch ports are present in controller-A.
These results in phys_port_name as "pf0" for both.
Similar problem exists for VFs and upcoming Sub functions.
An example view of two controller systems:
-----------------------------------------------------
| |
| --------- --------- |
------------- | | vf(s) | | sf(s) | |
| server | | ------- ----/---- ---/----- ------- |
| pci rc |=====| pf0 |______/________/ | pf1 | |
| connection| | ------- ------- |
------------- | | controller-B (no eswitch) (controller num=1)|
------|----------------------------------------------
(internal wire)
|
-----------------------------------------------------
| devlink eswitch ports and reps |
| --------------------------------------------- |
| |ctrl-A | ctrl-B | ctrl-A | ctrl-B | ctrl-B | |
| |pf0 | pf0 | pf0vfN | pf0vfN | pf0sfN | |
| --------------------------------------------- |
| |
| --------- |
| | sf(s) | |
| ------- ---/----- ------- |
| | pf0 |_____/ | pf1 | |
| ------- ------- |
| |
| local controller-A (eswitch) (controller num=0) |
-----------------------------------------------------
An example devlink port for external controller with controller
number = 1 for a VF 1 of PF 0:
$ devlink port show pci/0000:06:00.0/2
pci/0000:06:00.0/2: type eth netdev ens2f0pf0vf1 flavour pcivf controller 1 pfnum 0 vfnum 1 external true splittable false
function:
hw_addr 00:00:00:00:00:00
$ devlink port show pci/0000:06:00.0/2 -jp
{
"port": {
"pci/0000:06:00.0/2": {
"type": "eth",
"netdev": "ens2f0pf0vf1",
"flavour": "pcivf",
"controller": 1,
"pfnum": 0,
"vfnum": 1,
"external": true,
"splittable": false,
"function": {
"hw_addr": "00:00:00:00:00:00"
}
}
}
}
Signed-off-by: Parav Pandit <parav@nvidia.com>
Reviewed-by: Jiri Pirko <jiri@nvidia.com>
---
Changelog:
v1->v2:
- Added text diagram of multiple controllers
- Updated example for a VF
- Addressed comments from Jiri and Jakub
- Moved controller number attribute to PCI port flavours
This enables to better, hirerchical view with controller and its
PF, VF numbers
- Split 'external' and 'controller number' attributes as two
different attributes
- Merged mlx5_core driver to avoid compiliation break
---
.../net/ethernet/mellanox/mlx5/core/en_rep.c | 9 ++++++--
include/net/devlink.h | 9 ++++++--
include/uapi/linux/devlink.h | 1 +
net/core/devlink.c | 23 +++++++++++--------
4 files changed, 29 insertions(+), 13 deletions(-)
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_rep.c b/drivers/net/ethernet/mellanox/mlx5/core/en_rep.c
index 5b3599caa007..135ee26881c9 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/en_rep.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en_rep.c
@@ -1210,11 +1210,13 @@ is_devlink_port_supported(const struct mlx5_core_dev *dev,
static int register_devlink_port(struct mlx5_core_dev *dev,
struct mlx5e_rep_priv *rpriv)
{
+ struct mlx5_esw_offload *offloads = &dev->priv.eswitch->offloads;
struct devlink *devlink = priv_to_devlink(dev);
struct mlx5_eswitch_rep *rep = rpriv->rep;
struct devlink_port_attrs attrs = {};
struct netdev_phys_item_id ppid = {};
unsigned int dl_port_index = 0;
+ u32 controller_num = 0;
bool external;
u16 pfnum;
@@ -1222,6 +1224,8 @@ static int register_devlink_port(struct mlx5_core_dev *dev,
return 0;
external = mlx5_core_is_ecpf_esw_manager(dev);
+ if (external)
+ controller_num = offloads->host_number + 1;
mlx5e_rep_get_port_parent_id(rpriv->netdev, &ppid);
dl_port_index = mlx5_esw_vport_to_devlink_port_index(dev, rep->vport);
pfnum = PCI_FUNC(dev->pdev->devfn);
@@ -1234,11 +1238,12 @@ static int register_devlink_port(struct mlx5_core_dev *dev,
} else if (rep->vport == MLX5_VPORT_PF) {
memcpy(rpriv->dl_port.attrs.switch_id.id, &ppid.id[0], ppid.id_len);
rpriv->dl_port.attrs.switch_id.id_len = ppid.id_len;
- devlink_port_attrs_pci_pf_set(&rpriv->dl_port, pfnum, external);
+ devlink_port_attrs_pci_pf_set(&rpriv->dl_port, controller_num,
+ pfnum, external);
} else if (mlx5_eswitch_is_vf_vport(dev->priv.eswitch, rpriv->rep->vport)) {
memcpy(rpriv->dl_port.attrs.switch_id.id, &ppid.id[0], ppid.id_len);
rpriv->dl_port.attrs.switch_id.id_len = ppid.id_len;
- devlink_port_attrs_pci_vf_set(&rpriv->dl_port,
+ devlink_port_attrs_pci_vf_set(&rpriv->dl_port, controller_num,
pfnum, rep->vport - 1, external);
}
return devlink_port_register(devlink, &rpriv->dl_port, dl_port_index);
diff --git a/include/net/devlink.h b/include/net/devlink.h
index 2dad8c9151f4..eaec0a8cc5ef 100644
--- a/include/net/devlink.h
+++ b/include/net/devlink.h
@@ -59,21 +59,25 @@ struct devlink_port_phys_attrs {
/**
* struct devlink_port_pci_pf_attrs - devlink port's PCI PF attributes
+ * @controller: Associated controller number
* @pf: Associated PCI PF number for this port.
* @external: when set, indicates if a port is for an external controller
*/
struct devlink_port_pci_pf_attrs {
+ u32 controller;
u16 pf;
u8 external:1;
};
/**
* struct devlink_port_pci_vf_attrs - devlink port's PCI VF attributes
+ * @controller: Associated controller number
* @pf: Associated PCI PF number for this port.
* @vf: Associated PCI VF for of the PCI PF for this port.
* @external: when set, indicates if a port is for an external controller
*/
struct devlink_port_pci_vf_attrs {
+ u32 controller;
u16 pf;
u16 vf;
u8 external:1;
@@ -1219,8 +1223,9 @@ void devlink_port_type_ib_set(struct devlink_port *devlink_port,
void devlink_port_type_clear(struct devlink_port *devlink_port);
void devlink_port_attrs_set(struct devlink_port *devlink_port,
struct devlink_port_attrs *devlink_port_attrs);
-void devlink_port_attrs_pci_pf_set(struct devlink_port *devlink_port, u16 pf, bool external);
-void devlink_port_attrs_pci_vf_set(struct devlink_port *devlink_port,
+void devlink_port_attrs_pci_pf_set(struct devlink_port *devlink_port, u32 controller,
+ u16 pf, bool external);
+void devlink_port_attrs_pci_vf_set(struct devlink_port *devlink_port, u32 controller,
u16 pf, u16 vf, bool external);
int devlink_sb_register(struct devlink *devlink, unsigned int sb_index,
u32 size, u16 ingress_pools_count,
diff --git a/include/uapi/linux/devlink.h b/include/uapi/linux/devlink.h
index 40823ed7e05a..40d35145c879 100644
--- a/include/uapi/linux/devlink.h
+++ b/include/uapi/linux/devlink.h
@@ -459,6 +459,7 @@ enum devlink_attr {
DEVLINK_ATTR_PORT_SPLITTABLE, /* u8 */
DEVLINK_ATTR_PORT_EXTERNAL, /* u8 */
+ DEVLINK_ATTR_PORT_CONTROLLER_NUMBER, /* u32 */
/* add new attributes above here, update the policy in devlink.c */
__DEVLINK_ATTR_MAX,
diff --git a/net/core/devlink.c b/net/core/devlink.c
index 6f5f85372721..9cf5b118253b 100644
--- a/net/core/devlink.c
+++ b/net/core/devlink.c
@@ -523,17 +523,18 @@ static int devlink_nl_port_attrs_put(struct sk_buff *msg,
return -EMSGSIZE;
switch (devlink_port->attrs.flavour) {
case DEVLINK_PORT_FLAVOUR_PCI_PF:
- if (nla_put_u16(msg, DEVLINK_ATTR_PORT_PCI_PF_NUMBER,
- attrs->pci_pf.pf))
+ if (nla_put_u32(msg, DEVLINK_ATTR_PORT_CONTROLLER_NUMBER,
+ attrs->pci_pf.controller) ||
+ nla_put_u16(msg, DEVLINK_ATTR_PORT_PCI_PF_NUMBER, attrs->pci_pf.pf))
return -EMSGSIZE;
if (nla_put_u8(msg, DEVLINK_ATTR_PORT_EXTERNAL, attrs->pci_pf.external))
return -EMSGSIZE;
break;
case DEVLINK_PORT_FLAVOUR_PCI_VF:
- if (nla_put_u16(msg, DEVLINK_ATTR_PORT_PCI_PF_NUMBER,
- attrs->pci_vf.pf) ||
- nla_put_u16(msg, DEVLINK_ATTR_PORT_PCI_VF_NUMBER,
- attrs->pci_vf.vf))
+ if (nla_put_u32(msg, DEVLINK_ATTR_PORT_CONTROLLER_NUMBER,
+ attrs->pci_vf.controller) ||
+ nla_put_u16(msg, DEVLINK_ATTR_PORT_PCI_PF_NUMBER, attrs->pci_vf.pf) ||
+ nla_put_u16(msg, DEVLINK_ATTR_PORT_PCI_VF_NUMBER, attrs->pci_vf.vf))
return -EMSGSIZE;
if (nla_put_u8(msg, DEVLINK_ATTR_PORT_EXTERNAL, attrs->pci_vf.external))
return -EMSGSIZE;
@@ -7719,10 +7720,12 @@ EXPORT_SYMBOL_GPL(devlink_port_attrs_set);
* devlink_port_attrs_pci_pf_set - Set PCI PF port attributes
*
* @devlink_port: devlink port
+ * @controller: associated controller number for the devlink port instance
* @pf: associated PF for the devlink port instance
* @external: indicates if the port is for an external controller
*/
-void devlink_port_attrs_pci_pf_set(struct devlink_port *devlink_port, u16 pf, bool external)
+void devlink_port_attrs_pci_pf_set(struct devlink_port *devlink_port, u32 controller,
+ u16 pf, bool external)
{
struct devlink_port_attrs *attrs = &devlink_port->attrs;
int ret;
@@ -7731,7 +7734,7 @@ void devlink_port_attrs_pci_pf_set(struct devlink_port *devlink_port, u16 pf, bo
DEVLINK_PORT_FLAVOUR_PCI_PF);
if (ret)
return;
-
+ attrs->pci_pf.controller = controller;
attrs->pci_pf.pf = pf;
attrs->pci_pf.external = external;
}
@@ -7741,11 +7744,12 @@ EXPORT_SYMBOL_GPL(devlink_port_attrs_pci_pf_set);
* devlink_port_attrs_pci_vf_set - Set PCI VF port attributes
*
* @devlink_port: devlink port
+ * @controller: associated controller number for the devlink port instance
* @pf: associated PF for the devlink port instance
* @vf: associated VF of a PF for the devlink port instance
* @external: indicates if the port is for an external controller
*/
-void devlink_port_attrs_pci_vf_set(struct devlink_port *devlink_port,
+void devlink_port_attrs_pci_vf_set(struct devlink_port *devlink_port, u32 controller,
u16 pf, u16 vf, bool external)
{
struct devlink_port_attrs *attrs = &devlink_port->attrs;
@@ -7755,6 +7759,7 @@ void devlink_port_attrs_pci_vf_set(struct devlink_port *devlink_port,
DEVLINK_PORT_FLAVOUR_PCI_VF);
if (ret)
return;
+ attrs->pci_vf.controller = controller;
attrs->pci_vf.pf = pf;
attrs->pci_vf.vf = vf;
attrs->pci_vf.external = external;
--
2.26.2
next prev parent reply other threads:[~2020-09-08 15:07 UTC|newest]
Thread overview: 51+ messages / expand[flat|nested] mbox.gz Atom feed top
2020-08-25 13:58 [PATCH net-next 0/3] devlink show controller info Parav Pandit
2020-08-25 13:58 ` [PATCH net-next 1/3] devlink: Add comment block for missing port attributes Parav Pandit
2020-08-25 13:58 ` [PATCH net-next 2/3] devlink: Consider other controller while building phys_port_name Parav Pandit
2020-08-26 0:32 ` Jakub Kicinski
2020-08-26 4:27 ` Parav Pandit
2020-08-26 20:07 ` Jakub Kicinski
2020-08-27 4:31 ` Parav Pandit
2020-08-27 18:32 ` Jakub Kicinski
2020-08-27 20:15 ` Parav Pandit
2020-08-27 21:42 ` Jakub Kicinski
2020-08-28 4:27 ` Parav Pandit
2020-08-28 5:08 ` Parav Pandit
2020-08-28 16:43 ` Jakub Kicinski
2020-08-29 3:43 ` Parav Pandit
2020-09-01 8:19 ` Jiri Pirko
2020-09-01 8:53 ` Parav Pandit
2020-09-01 9:17 ` Jiri Pirko
2020-09-01 21:28 ` Jakub Kicinski
2020-09-02 4:26 ` Parav Pandit
2020-09-02 4:44 ` Parav Pandit
2020-09-02 8:00 ` Jiri Pirko
2020-09-02 15:23 ` Jakub Kicinski
2020-09-02 16:18 ` Parav Pandit
2020-09-02 20:10 ` Parav Pandit
2020-09-03 5:54 ` Jiri Pirko
2020-09-03 19:31 ` Jakub Kicinski
2020-09-04 8:43 ` Jiri Pirko
2020-09-06 3:08 ` Parav Pandit
2020-09-06 16:46 ` Jakub Kicinski
2020-09-07 7:21 ` Jiri Pirko
2020-09-07 16:18 ` Jakub Kicinski
2020-08-25 13:58 ` [PATCH net-next 3/3] net/mlx5: E-switch, Set controller attribute for PCI PF and VF ports Parav Pandit
2020-09-08 14:42 ` [PATCH net-next v2 0/6] devlink show controller number Parav Pandit
2020-09-08 14:42 ` [PATCH net-next v2 1/6] net/mlx5: E-switch, Read controller number from device Parav Pandit
2020-09-08 14:42 ` [PATCH net-next v2 2/6] devlink: Add comment block for missing port attributes Parav Pandit
2020-09-08 14:42 ` [PATCH net-next v2 3/6] devlink: Move structure comments outside of structure Parav Pandit
2020-09-08 14:42 ` [PATCH net-next v2 4/6] devlink: Introduce external controller flag Parav Pandit
2020-09-08 14:42 ` Parav Pandit [this message]
2020-09-08 18:50 ` [PATCH net-next v2 5/6] devlink: Introduce controller number Jakub Kicinski
2020-09-09 3:06 ` Parav Pandit
2020-09-08 14:42 ` [PATCH net-next v2 6/6] devlink: Use controller while building phys_port_name Parav Pandit
2020-09-09 4:50 ` [PATCH net-next v3 0/6] devlink show controller number Parav Pandit
2020-09-09 4:50 ` [PATCH net-next v3 1/6] net/mlx5: E-switch, Read controller number from device Parav Pandit
2020-09-09 4:50 ` [PATCH net-next v3 2/6] devlink: Add comment block for missing port attributes Parav Pandit
2020-09-09 4:50 ` [PATCH net-next v3 3/6] devlink: Move structure comments outside of structure Parav Pandit
2020-09-09 4:50 ` [PATCH net-next v3 4/6] devlink: Introduce external controller flag Parav Pandit
2020-09-09 4:50 ` [PATCH net-next v3 5/6] devlink: Introduce controller number Parav Pandit
2020-09-09 4:50 ` [PATCH net-next v3 6/6] devlink: Use controller while building phys_port_name Parav Pandit
2020-09-10 15:02 ` David Ahern
2020-09-09 15:34 ` [PATCH net-next v3 0/6] devlink show controller number Jakub Kicinski
2020-09-09 21:20 ` David Miller
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20200908144241.21673-6-parav@mellanox.com \
--to=parav@mellanox.com \
--cc=davem@davemloft.net \
--cc=jiri@nvidia.com \
--cc=kuba@kernel.org \
--cc=netdev@vger.kernel.org \
--cc=parav@nvidia.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).