* [PATCH net-next v2 0/7] devlink: expose PF and VF representors as ports
@ 2019-03-01 18:04 Jakub Kicinski
2019-03-01 18:04 ` [PATCH net-next v2 1/7] nfp: split devlink port init from registration Jakub Kicinski
` (9 more replies)
0 siblings, 10 replies; 100+ messages in thread
From: Jakub Kicinski @ 2019-03-01 18:04 UTC (permalink / raw)
To: jiri, davem; +Cc: netdev, oss-drivers, Jakub Kicinski
Hi!
This series is a long overdue follow up to Jiri's work on providing
a common .ndo_phys_port_name implementation based on devlink ports.
First devlink port flavours for PF and VF ports are added, and
registered by the NFP. Port numbers and split info are reserved
for physical and DSA ports. For PCIe ports we add pf/vf identifiers.
Note that devices may have more than one PF, including multi host
scenarios where not all pfs are connected to the same host.
The port_index is slightly tricky to figure out, we use a bit of
math to create unique IDs for ports.
Next subports for PCIe ports are introduced. This is in case device
has more than one netdev on a physical function (e.g. multi port
SmartNIC).
With the above features in place we can remove the ndo_phys_port_name
implementation from the NFP and use the standard devlink one for
port netdevs.
Last but not least a concept of peer netdevs is added. In multi-host
scenarios its currently not possible to figure out which PF is
associated with the local host. Peer device is "the other side
of the wire" for PCIe ports. In case of NFP we only link the PF
netdevs, but it should be possible to also link VF peers after
VF driver probes, if users request it.
This is the conceptual image of devlink instances:
HOST A || HOST B
||
PF A | V | V | V | V || PF B | V | V | V
|*F |*F |*F |*F ... || |*F |*F |*F
*port A0 |*port A1 | 0 | 1 | 2 | 3 ||*port B0 |*port B1 | 0 | 1 | 2
||
PCI Express link || PCI Express link
\ \ \ | | | | | / / /
\ \ \ | | | | | / / /
/\ \______\______\'___|___|__________|_______'____/___/___/__ /\
|| |+PF0s0|+PF0s1 |+VF0|+VF1| ...| |+PF1s0|+PF1s1|+VF0|+VF1| ||
i || |------ ------ ----- ---- ----|--- ------ ------ ---- ----| || i
d n H || | <<========== | || d n H
e s O || | ==========>> | || e s O
v t S || | SR-IOV e-switch | || v t S
l a T || | <<========== | || l a T
i n || | ==========>> | || i n
n c A || | ________ _________ ________ | || n c B
k e || | |+Phys 0 |+Phys 1 |+Phys 2 | | || k e
|| \---------------------------------------------------------/ ||
\/ | | | \/
| | |
|| ||
MAC 0 || MAC 1 || MAC 2
|| ||
'+' marks the devlink ports and port (-representor-) netdevs
'*' marks host netdevs (actual VF/PF netdev)
v2 (Jiri):
- update devlink user space output in a commit message;
- split the pci attribute setting helper into pf and vf versions;
- add peer IBDEV_NAME;
- add nest for peer attributes.
Jakub Kicinski (7):
nfp: split devlink port init from registration
devlink: add PF and VF port flavours
nfp: register devlink ports of all reprs
devlink: allow subports on devlink PCI ports
nfp: switch to devlink_port_get_phys_port_name()
devlink: introduce port's peer netdevs
nfp: expose PF peer netdevs
drivers/net/ethernet/netronome/nfp/abm/main.c | 1 +
.../net/ethernet/netronome/nfp/flower/main.c | 1 +
.../net/ethernet/netronome/nfp/nfp_devlink.c | 54 +++++-
.../net/ethernet/netronome/nfp/nfp_net_main.c | 17 +-
.../net/ethernet/netronome/nfp/nfp_net_repr.c | 16 +-
drivers/net/ethernet/netronome/nfp/nfp_port.c | 33 +---
drivers/net/ethernet/netronome/nfp/nfp_port.h | 4 +
include/net/devlink.h | 48 ++++-
include/uapi/linux/devlink.h | 12 ++
net/core/devlink.c | 178 ++++++++++++++++--
10 files changed, 310 insertions(+), 54 deletions(-)
--
2.19.2
^ permalink raw reply [flat|nested] 100+ messages in thread
* [PATCH net-next v2 1/7] nfp: split devlink port init from registration
2019-03-01 18:04 [PATCH net-next v2 0/7] devlink: expose PF and VF representors as ports Jakub Kicinski
@ 2019-03-01 18:04 ` Jakub Kicinski
2019-03-01 18:04 ` [PATCH net-next v2 2/7] devlink: add PF and VF port flavours Jakub Kicinski
` (8 subsequent siblings)
9 siblings, 0 replies; 100+ messages in thread
From: Jakub Kicinski @ 2019-03-01 18:04 UTC (permalink / raw)
To: jiri, davem; +Cc: netdev, oss-drivers, Jakub Kicinski
In the future we want to switch to using the
devlink_port_get_phys_port_name() helper for .ndo_phys_port_name.
This requires that we initialize the devlink ports before the
netdevs are registered. Split the registration from init.
Signed-off-by: Jakub Kicinski <jakub.kicinski@netronome.com>
---
.../net/ethernet/netronome/nfp/nfp_devlink.c | 13 ++++++++++---
.../net/ethernet/netronome/nfp/nfp_net_main.c | 17 ++++++++++++++---
drivers/net/ethernet/netronome/nfp/nfp_port.h | 2 ++
3 files changed, 26 insertions(+), 6 deletions(-)
diff --git a/drivers/net/ethernet/netronome/nfp/nfp_devlink.c b/drivers/net/ethernet/netronome/nfp/nfp_devlink.c
index e9eca99cf493..9af3cb1f2f17 100644
--- a/drivers/net/ethernet/netronome/nfp/nfp_devlink.c
+++ b/drivers/net/ethernet/netronome/nfp/nfp_devlink.c
@@ -350,10 +350,9 @@ const struct devlink_ops nfp_devlink_ops = {
.flash_update = nfp_devlink_flash_update,
};
-int nfp_devlink_port_register(struct nfp_app *app, struct nfp_port *port)
+int nfp_devlink_port_init(struct nfp_app *app, struct nfp_port *port)
{
struct nfp_eth_table_port eth_port;
- struct devlink *devlink;
int ret;
rtnl_lock();
@@ -366,8 +365,16 @@ int nfp_devlink_port_register(struct nfp_app *app, struct nfp_port *port)
devlink_port_attrs_set(&port->dl_port, DEVLINK_PORT_FLAVOUR_PHYSICAL,
eth_port.label_port, eth_port.is_split,
eth_port.label_subport);
+ return 0;
+}
+
+void nfp_devlink_port_clean(struct nfp_port *port)
+{
+}
- devlink = priv_to_devlink(app->pf);
+int nfp_devlink_port_register(struct nfp_app *app, struct nfp_port *port)
+{
+ struct devlink *devlink = priv_to_devlink(app->pf);
return devlink_port_register(devlink, &port->dl_port, port->eth_id);
}
diff --git a/drivers/net/ethernet/netronome/nfp/nfp_net_main.c b/drivers/net/ethernet/netronome/nfp/nfp_net_main.c
index 08f5fdbd8e41..39e87bb73ebf 100644
--- a/drivers/net/ethernet/netronome/nfp/nfp_net_main.c
+++ b/drivers/net/ethernet/netronome/nfp/nfp_net_main.c
@@ -150,9 +150,15 @@ nfp_net_pf_init_vnic(struct nfp_pf *pf, struct nfp_net *nn, unsigned int id)
nn->id = id;
+ if (nn->port) {
+ err = nfp_devlink_port_init(pf->app, nn->port);
+ if (err)
+ return err;
+ }
+
err = nfp_net_init(nn);
if (err)
- return err;
+ goto err_port_clean;
nfp_net_debugfs_vnic_add(nn, pf->ddir);
@@ -167,17 +173,20 @@ nfp_net_pf_init_vnic(struct nfp_pf *pf, struct nfp_net *nn, unsigned int id)
if (nfp_net_is_data_vnic(nn)) {
err = nfp_app_vnic_init(pf->app, nn);
if (err)
- goto err_devlink_port_clean;
+ goto err_port_unreg;
}
return 0;
-err_devlink_port_clean:
+err_port_unreg:
if (nn->port)
nfp_devlink_port_unregister(nn->port);
err_dfs_clean:
nfp_net_debugfs_dir_clean(&nn->debugfs_dir);
nfp_net_clean(nn);
+err_port_clean:
+ if (nn->port)
+ nfp_devlink_port_clean(nn->port);
return err;
}
@@ -224,6 +233,8 @@ static void nfp_net_pf_clean_vnic(struct nfp_pf *pf, struct nfp_net *nn)
nfp_devlink_port_unregister(nn->port);
nfp_net_debugfs_dir_clean(&nn->debugfs_dir);
nfp_net_clean(nn);
+ if (nn->port)
+ nfp_devlink_port_clean(nn->port);
}
static int nfp_net_pf_alloc_irqs(struct nfp_pf *pf)
diff --git a/drivers/net/ethernet/netronome/nfp/nfp_port.h b/drivers/net/ethernet/netronome/nfp/nfp_port.h
index 90ae053f5c07..09c55ca2371a 100644
--- a/drivers/net/ethernet/netronome/nfp/nfp_port.h
+++ b/drivers/net/ethernet/netronome/nfp/nfp_port.h
@@ -129,6 +129,8 @@ int nfp_net_refresh_eth_port(struct nfp_port *port);
void nfp_net_refresh_port_table(struct nfp_port *port);
int nfp_net_refresh_port_table_sync(struct nfp_pf *pf);
+int nfp_devlink_port_init(struct nfp_app *app, struct nfp_port *port);
+void nfp_devlink_port_clean(struct nfp_port *port);
int nfp_devlink_port_register(struct nfp_app *app, struct nfp_port *port);
void nfp_devlink_port_unregister(struct nfp_port *port);
--
2.19.2
^ permalink raw reply related [flat|nested] 100+ messages in thread
* [PATCH net-next v2 2/7] devlink: add PF and VF port flavours
2019-03-01 18:04 [PATCH net-next v2 0/7] devlink: expose PF and VF representors as ports Jakub Kicinski
2019-03-01 18:04 ` [PATCH net-next v2 1/7] nfp: split devlink port init from registration Jakub Kicinski
@ 2019-03-01 18:04 ` Jakub Kicinski
2019-03-01 18:04 ` [PATCH net-next v2 3/7] nfp: register devlink ports of all reprs Jakub Kicinski
` (7 subsequent siblings)
9 siblings, 0 replies; 100+ messages in thread
From: Jakub Kicinski @ 2019-03-01 18:04 UTC (permalink / raw)
To: jiri, davem; +Cc: netdev, oss-drivers, Jakub Kicinski
Current port flavours cover simple switches and DSA. Add PF
and VF flavours to cover "switchdev" SR-IOV NICs.
Example devlink user space output:
$ devlink port
pci/0000:82:00.0/0: type eth netdev p4p1 flavour physical
pci/0000:82:00.0/10000: type eth netdev eth0 flavour pci_pf pf 0
pci/0000:82:00.0/10001: type eth netdev eth1 flavour pci_vf pf 0 vf 0
pci/0000:82:00.0/10002: type eth netdev eth2 flavour pci_vf pf 0 vf 1
$ devlink -jp port
{
"port": {
"pci/0000:82:00.0/0": {
"type": "eth",
"netdev": "p4p1",
"flavour": "physical"
},
"pci/0000:82:00.0/10000": {
"type": "eth",
"netdev": "eth0",
"flavour": "pci_pf",
"pf": 0,
},
"pci/0000:82:00.0/10001": {
"type": "eth",
"netdev": "eth1",
"flavour": "pci_vf",
"pf": 0,
"vf": 0
},
"pci/0000:82:00.0/10002": {
"type": "eth",
"netdev": "eth2",
"flavour": "pci_vf",
"pf": 0,
"vf": 1
}
}
}
v2:
- fix old output in commit message s/pcie_/pci_/ (Jiri);
- split the pci helper into separate ones for PF and VF (Jiri);
- flip the condition in WARN_ON for devlink_port_attrs_set() to
whitelist from blacklist.
Signed-off-by: Jakub Kicinski <jakub.kicinski@netronome.com>
---
include/net/devlink.h | 32 +++++++++++--
include/uapi/linux/devlink.h | 5 ++
net/core/devlink.c | 88 ++++++++++++++++++++++++++++++++----
3 files changed, 113 insertions(+), 12 deletions(-)
diff --git a/include/net/devlink.h b/include/net/devlink.h
index 7f5a0bdca228..00ceff76762c 100644
--- a/include/net/devlink.h
+++ b/include/net/devlink.h
@@ -42,9 +42,19 @@ struct devlink {
struct devlink_port_attrs {
bool set;
enum devlink_port_flavour flavour;
- u32 port_number; /* same value as "split group" */
- bool split;
- u32 split_subport_number;
+ union { /* port identifiers differ per-flavour */
+ /* PHYSICAL, CPU, DSA */
+ struct {
+ bool split;
+ u32 split_subport_number;
+ u32 port_number; /* same value as "split group" */
+ };
+ /* PCI_PF, PCI_VF */
+ struct {
+ u32 pf_number;
+ u32 vf_number;
+ } pci;
+ };
};
struct devlink_port {
@@ -568,6 +578,10 @@ void devlink_port_attrs_set(struct devlink_port *devlink_port,
enum devlink_port_flavour flavour,
u32 port_number, bool split,
u32 split_subport_number);
+void devlink_port_attrs_pci_pf_set(struct devlink_port *devlink_port,
+ u32 pf_number);
+void devlink_port_attrs_pci_vf_set(struct devlink_port *devlink_port,
+ u32 pf_number, u32 vf_number);
int devlink_port_get_phys_port_name(struct devlink_port *devlink_port,
char *name, size_t len);
int devlink_sb_register(struct devlink *devlink, unsigned int sb_index,
@@ -782,6 +796,18 @@ static inline void devlink_port_attrs_set(struct devlink_port *devlink_port,
{
}
+static inline void
+devlink_port_attrs_pci_pf_set(struct devlink_port *devlink_port,
+ u32 pf_number)
+{
+}
+
+static inline void
+devlink_port_attrs_pci_vf_set(struct devlink_port *devlink_port,
+ u32 pf_number, u32 vf_number)
+{
+}
+
static inline int
devlink_port_get_phys_port_name(struct devlink_port *devlink_port,
char *name, size_t len)
diff --git a/include/uapi/linux/devlink.h b/include/uapi/linux/devlink.h
index 5bb4ea67d84f..9ce76d4f640d 100644
--- a/include/uapi/linux/devlink.h
+++ b/include/uapi/linux/devlink.h
@@ -167,6 +167,8 @@ enum devlink_port_flavour {
DEVLINK_PORT_FLAVOUR_DSA, /* Distributed switch architecture
* interconnect port.
*/
+ DEVLINK_PORT_FLAVOUR_PCI_PF, /* PCI Physical function port */
+ DEVLINK_PORT_FLAVOUR_PCI_VF, /* PCI Physical function port */
};
enum devlink_param_cmode {
@@ -332,6 +334,9 @@ enum devlink_attr {
DEVLINK_ATTR_FLASH_UPDATE_FILE_NAME, /* string */
DEVLINK_ATTR_FLASH_UPDATE_COMPONENT, /* string */
+ DEVLINK_ATTR_PORT_PCI_PF_NUMBER, /* u32 */
+ DEVLINK_ATTR_PORT_PCI_VF_NUMBER, /* u32 */
+
/* add new attributes above here, update the policy in devlink.c */
__DEVLINK_ATTR_MAX,
diff --git a/net/core/devlink.c b/net/core/devlink.c
index 6515fbec0dcd..49216b688c5b 100644
--- a/net/core/devlink.c
+++ b/net/core/devlink.c
@@ -516,16 +516,35 @@ static int devlink_nl_port_attrs_put(struct sk_buff *msg,
return 0;
if (nla_put_u16(msg, DEVLINK_ATTR_PORT_FLAVOUR, attrs->flavour))
return -EMSGSIZE;
- if (nla_put_u32(msg, DEVLINK_ATTR_PORT_NUMBER, attrs->port_number))
- return -EMSGSIZE;
- if (!attrs->split)
+
+ switch (attrs->flavour) {
+ case DEVLINK_PORT_FLAVOUR_PHYSICAL:
+ case DEVLINK_PORT_FLAVOUR_CPU:
+ case DEVLINK_PORT_FLAVOUR_DSA:
+ if (nla_put_u32(msg, DEVLINK_ATTR_PORT_NUMBER,
+ attrs->port_number))
+ return -EMSGSIZE;
+
+ if (attrs->split &&
+ (nla_put_u32(msg, DEVLINK_ATTR_PORT_SPLIT_GROUP,
+ attrs->port_number) ||
+ nla_put_u32(msg, DEVLINK_ATTR_PORT_SPLIT_SUBPORT_NUMBER,
+ attrs->split_subport_number)))
+ return -EMSGSIZE;
return 0;
- if (nla_put_u32(msg, DEVLINK_ATTR_PORT_SPLIT_GROUP, attrs->port_number))
- return -EMSGSIZE;
- if (nla_put_u32(msg, DEVLINK_ATTR_PORT_SPLIT_SUBPORT_NUMBER,
- attrs->split_subport_number))
- return -EMSGSIZE;
- return 0;
+ case DEVLINK_PORT_FLAVOUR_PCI_VF:
+ if (nla_put_u32(msg, DEVLINK_ATTR_PORT_PCI_VF_NUMBER,
+ attrs->pci.vf_number))
+ return -EMSGSIZE;
+ /* fall through */
+ case DEVLINK_PORT_FLAVOUR_PCI_PF:
+ if (nla_put_u32(msg, DEVLINK_ATTR_PORT_PCI_PF_NUMBER,
+ attrs->pci.pf_number))
+ return -EMSGSIZE;
+ return 0;
+ default:
+ return -EINVAL;
+ }
}
static int devlink_nl_port_fill(struct sk_buff *msg, struct devlink *devlink,
@@ -5411,6 +5430,10 @@ void devlink_port_attrs_set(struct devlink_port *devlink_port,
{
struct devlink_port_attrs *attrs = &devlink_port->attrs;
+ WARN_ON(flavour != DEVLINK_PORT_FLAVOUR_PHYSICAL &&
+ flavour != DEVLINK_PORT_FLAVOUR_CPU &&
+ flavour != DEVLINK_PORT_FLAVOUR_DSA);
+
attrs->set = true;
attrs->flavour = flavour;
attrs->port_number = port_number;
@@ -5420,6 +5443,46 @@ void devlink_port_attrs_set(struct devlink_port *devlink_port,
}
EXPORT_SYMBOL_GPL(devlink_port_attrs_set);
+/**
+ * devlink_port_attrs_pci_pf_set - Set port attributes for a PCI PF port
+ *
+ * @devlink_port: devlink port
+ * @pf_number: PCI PF number, in multi-host mapping to hosts depends
+ * on the platform
+ */
+void devlink_port_attrs_pci_pf_set(struct devlink_port *devlink_port,
+ u32 pf_number)
+{
+ struct devlink_port_attrs *attrs = &devlink_port->attrs;
+
+ attrs->set = true;
+ attrs->flavour = DEVLINK_PORT_FLAVOUR_PCI_PF;
+ attrs->pci.pf_number = pf_number;
+ devlink_port_notify(devlink_port, DEVLINK_CMD_PORT_NEW);
+}
+EXPORT_SYMBOL_GPL(devlink_port_attrs_pci_pf_set);
+
+/**
+ * devlink_port_attrs_pci_vf_set - Set port attributes for a PCI VF port
+ *
+ * @devlink_port: devlink port
+ * @pf_number: PCI PF number, in multi-host mapping to hosts depends
+ * on the platform
+ * @vf_number: PCI VF number within given PF (ignored for PF itself)
+ */
+void devlink_port_attrs_pci_vf_set(struct devlink_port *devlink_port,
+ u32 pf_number, u32 vf_number)
+{
+ struct devlink_port_attrs *attrs = &devlink_port->attrs;
+
+ attrs->set = true;
+ attrs->flavour = DEVLINK_PORT_FLAVOUR_PCI_VF;
+ attrs->pci.pf_number = pf_number;
+ attrs->pci.vf_number = vf_number;
+ devlink_port_notify(devlink_port, DEVLINK_CMD_PORT_NEW);
+}
+EXPORT_SYMBOL_GPL(devlink_port_attrs_pci_vf_set);
+
int devlink_port_get_phys_port_name(struct devlink_port *devlink_port,
char *name, size_t len)
{
@@ -5444,6 +5507,13 @@ int devlink_port_get_phys_port_name(struct devlink_port *devlink_port,
*/
WARN_ON(1);
return -EINVAL;
+ case DEVLINK_PORT_FLAVOUR_PCI_PF:
+ n = snprintf(name, len, "pf%u", attrs->pci.pf_number);
+ break;
+ case DEVLINK_PORT_FLAVOUR_PCI_VF:
+ n = snprintf(name, len, "pf%uvf%u",
+ attrs->pci.pf_number, attrs->pci.vf_number);
+ break;
}
if (n >= len)
--
2.19.2
^ permalink raw reply related [flat|nested] 100+ messages in thread
* [PATCH net-next v2 3/7] nfp: register devlink ports of all reprs
2019-03-01 18:04 [PATCH net-next v2 0/7] devlink: expose PF and VF representors as ports Jakub Kicinski
2019-03-01 18:04 ` [PATCH net-next v2 1/7] nfp: split devlink port init from registration Jakub Kicinski
2019-03-01 18:04 ` [PATCH net-next v2 2/7] devlink: add PF and VF port flavours Jakub Kicinski
@ 2019-03-01 18:04 ` Jakub Kicinski
2019-03-02 8:43 ` Jiri Pirko
2019-03-01 18:04 ` [PATCH net-next v2 4/7] devlink: allow subports on devlink PCI ports Jakub Kicinski
` (6 subsequent siblings)
9 siblings, 1 reply; 100+ messages in thread
From: Jakub Kicinski @ 2019-03-01 18:04 UTC (permalink / raw)
To: jiri, davem; +Cc: netdev, oss-drivers, Jakub Kicinski
Register all representors as devlink ports.
The port_index is slightly tricky to figure out, we use a bit of
arbitrary math to create unique IDs for PCI ports.
Signed-off-by: Jakub Kicinski <jakub.kicinski@netronome.com>
---
.../net/ethernet/netronome/nfp/nfp_devlink.c | 40 ++++++++++++++++++-
.../net/ethernet/netronome/nfp/nfp_net_repr.c | 16 +++++++-
2 files changed, 53 insertions(+), 3 deletions(-)
diff --git a/drivers/net/ethernet/netronome/nfp/nfp_devlink.c b/drivers/net/ethernet/netronome/nfp/nfp_devlink.c
index 9af3cb1f2f17..bf7fd9614152 100644
--- a/drivers/net/ethernet/netronome/nfp/nfp_devlink.c
+++ b/drivers/net/ethernet/netronome/nfp/nfp_devlink.c
@@ -350,7 +350,8 @@ const struct devlink_ops nfp_devlink_ops = {
.flash_update = nfp_devlink_flash_update,
};
-int nfp_devlink_port_init(struct nfp_app *app, struct nfp_port *port)
+static int
+nfp_devlink_port_init_phys(struct devlink *devlink, struct nfp_port *port)
{
struct nfp_eth_table_port eth_port;
int ret;
@@ -368,6 +369,27 @@ int nfp_devlink_port_init(struct nfp_app *app, struct nfp_port *port)
return 0;
}
+int nfp_devlink_port_init(struct nfp_app *app, struct nfp_port *port)
+{
+ struct devlink *devlink = priv_to_devlink(app->pf);
+
+ switch (port->type) {
+ case NFP_PORT_PHYS_PORT:
+ return nfp_devlink_port_init_phys(devlink, port);
+ case NFP_PORT_PF_PORT:
+ devlink_port_type_eth_set(&port->dl_port, port->netdev);
+ devlink_port_attrs_pci_pf_set(&port->dl_port, port->pf_id);
+ return 0;
+ case NFP_PORT_VF_PORT:
+ devlink_port_type_eth_set(&port->dl_port, port->netdev);
+ devlink_port_attrs_pci_vf_set(&port->dl_port, port->pf_id,
+ port->vf_id);
+ return 0;
+ default:
+ return -EINVAL;
+ }
+}
+
void nfp_devlink_port_clean(struct nfp_port *port)
{
}
@@ -376,7 +398,21 @@ int nfp_devlink_port_register(struct nfp_app *app, struct nfp_port *port)
{
struct devlink *devlink = priv_to_devlink(app->pf);
- return devlink_port_register(devlink, &port->dl_port, port->eth_id);
+ switch (port->type) {
+ case NFP_PORT_PHYS_PORT:
+ return devlink_port_register(devlink, &port->dl_port,
+ port->eth_id);
+ case NFP_PORT_PF_PORT:
+ return devlink_port_register(devlink, &port->dl_port,
+ (port->pf_id + 1) * 10000 +
+ port->pf_split_id * 1000);
+ case NFP_PORT_VF_PORT:
+ return devlink_port_register(devlink, &port->dl_port,
+ (port->pf_id + 1) * 10000 +
+ port->vf_id + 1);
+ default:
+ return -EINVAL;
+ }
}
void nfp_devlink_port_unregister(struct nfp_port *port)
diff --git a/drivers/net/ethernet/netronome/nfp/nfp_net_repr.c b/drivers/net/ethernet/netronome/nfp/nfp_net_repr.c
index d2c803bb4e56..869d22760a6e 100644
--- a/drivers/net/ethernet/netronome/nfp/nfp_net_repr.c
+++ b/drivers/net/ethernet/netronome/nfp/nfp_net_repr.c
@@ -292,7 +292,9 @@ nfp_repr_transfer_features(struct net_device *netdev, struct net_device *lower)
static void nfp_repr_clean(struct nfp_repr *repr)
{
+ nfp_devlink_port_unregister(repr->port);
unregister_netdev(repr->netdev);
+ nfp_devlink_port_clean(repr->port);
nfp_app_repr_clean(repr->app, repr->netdev);
dst_release((struct dst_entry *)repr->dst);
nfp_port_free(repr->port);
@@ -395,12 +397,24 @@ int nfp_repr_init(struct nfp_app *app, struct net_device *netdev,
if (err)
goto err_clean;
- err = register_netdev(netdev);
+ err = nfp_devlink_port_init(app, repr->port);
if (err)
goto err_repr_clean;
+ err = register_netdev(netdev);
+ if (err)
+ goto err_port_clean;
+
+ err = nfp_devlink_port_register(app, repr->port);
+ if (err)
+ goto err_unreg_netdev;
+
return 0;
+err_unreg_netdev:
+ unregister_netdev(repr->netdev);
+err_port_clean:
+ nfp_devlink_port_clean(repr->port);
err_repr_clean:
nfp_app_repr_clean(app, netdev);
err_clean:
--
2.19.2
^ permalink raw reply related [flat|nested] 100+ messages in thread
* [PATCH net-next v2 4/7] devlink: allow subports on devlink PCI ports
2019-03-01 18:04 [PATCH net-next v2 0/7] devlink: expose PF and VF representors as ports Jakub Kicinski
` (2 preceding siblings ...)
2019-03-01 18:04 ` [PATCH net-next v2 3/7] nfp: register devlink ports of all reprs Jakub Kicinski
@ 2019-03-01 18:04 ` Jakub Kicinski
2019-03-02 9:41 ` Jiri Pirko
2019-03-04 11:08 ` Jiri Pirko
2019-03-01 18:04 ` [PATCH net-next v2 5/7] nfp: switch to devlink_port_get_phys_port_name() Jakub Kicinski
` (5 subsequent siblings)
9 siblings, 2 replies; 100+ messages in thread
From: Jakub Kicinski @ 2019-03-01 18:04 UTC (permalink / raw)
To: jiri, davem; +Cc: netdev, oss-drivers, Jakub Kicinski
PCI endpoint corresponds to a PCI device, but such device
can have one more more logical device ports associated with it.
We need a way to distinguish those. Add a PCI subport in the
dumps and print the info in phys_port_name appropriately.
This is not equivalent to port splitting, there is no split
group. It's just a way of representing multiple netdevs on
a single PCI function.
Note that the quality of being multiport pertains only to
the PCI function itself. A PF having multiple netdevs does
not mean that its VFs will also have multiple, or that VFs
are associated with any particular port of a multiport VF.
Example (bus 05 device has subports, bus 82 has only one port per
function):
$ devlink port
pci/0000:05:00.0/0: type eth netdev enp5s0np0 flavour physical
pci/0000:05:00.0/10000: type eth netdev enp5s0npf0s0 flavour pci_pf pf 0 subport 0
pci/0000:05:00.0/4: type eth netdev enp5s0np1 flavour physical
pci/0000:05:00.0/11000: type eth netdev enp5s0npf0s1 flavour pci_pf pf 0 subport 1
pci/0000:82:00.0/0: type eth netdev p4p1 flavour physical
pci/0000:82:00.0/10000: type eth netdev eth0 flavour pci_pf pf 0
$ devlink -jp port
{
"port": {
"pci/0000:05:00.0/0": {
"type": "eth",
"netdev": "enp5s0np0",
"flavour": "physical"
},
"pci/0000:05:00.0/10000": {
"type": "eth",
"netdev": "enp5s0npf0s0",
"flavour": "pci_pf",
"pf": 0,
"subport": 0
},
"pci/0000:05:00.0/4": {
"type": "eth",
"netdev": "enp5s0np1",
"flavour": "physical"
},
"pci/0000:05:00.0/11000": {
"type": "eth",
"netdev": "enp5s0npf0s1",
"flavour": "pci_pf",
"pf": 0,
"subport": 1
},
"pci/0000:82:00.0/0": {
"type": "eth",
"netdev": "p4p1",
"flavour": "physical"
},
"pci/0000:82:00.0/10000": {
"type": "eth",
"netdev": "eth0",
"flavour": "pci_pf",
"pf": 0
}
}
}
Signed-off-by: Jakub Kicinski <jakub.kicinski@netronome.com>
---
.../net/ethernet/netronome/nfp/nfp_devlink.c | 6 ++--
include/net/devlink.h | 13 ++++---
include/uapi/linux/devlink.h | 1 +
net/core/devlink.c | 36 ++++++++++++++++---
4 files changed, 45 insertions(+), 11 deletions(-)
diff --git a/drivers/net/ethernet/netronome/nfp/nfp_devlink.c b/drivers/net/ethernet/netronome/nfp/nfp_devlink.c
index bf7fd9614152..6ad2805f1efc 100644
--- a/drivers/net/ethernet/netronome/nfp/nfp_devlink.c
+++ b/drivers/net/ethernet/netronome/nfp/nfp_devlink.c
@@ -378,12 +378,14 @@ int nfp_devlink_port_init(struct nfp_app *app, struct nfp_port *port)
return nfp_devlink_port_init_phys(devlink, port);
case NFP_PORT_PF_PORT:
devlink_port_type_eth_set(&port->dl_port, port->netdev);
- devlink_port_attrs_pci_pf_set(&port->dl_port, port->pf_id);
+ devlink_port_attrs_pci_pf_set(&port->dl_port, port->pf_id,
+ port->pf_split,
+ port->pf_split_id);
return 0;
case NFP_PORT_VF_PORT:
devlink_port_type_eth_set(&port->dl_port, port->netdev);
devlink_port_attrs_pci_vf_set(&port->dl_port, port->pf_id,
- port->vf_id);
+ port->vf_id, false, 0);
return 0;
default:
return -EINVAL;
diff --git a/include/net/devlink.h b/include/net/devlink.h
index 00ceff76762c..6a29ce80cb38 100644
--- a/include/net/devlink.h
+++ b/include/net/devlink.h
@@ -53,6 +53,8 @@ struct devlink_port_attrs {
struct {
u32 pf_number;
u32 vf_number;
+ bool multiport;
+ u32 subport_number;
} pci;
};
};
@@ -579,9 +581,11 @@ void devlink_port_attrs_set(struct devlink_port *devlink_port,
u32 port_number, bool split,
u32 split_subport_number);
void devlink_port_attrs_pci_pf_set(struct devlink_port *devlink_port,
- u32 pf_number);
+ u32 pf_number, bool multiport,
+ u32 subport_number);
void devlink_port_attrs_pci_vf_set(struct devlink_port *devlink_port,
- u32 pf_number, u32 vf_number);
+ u32 pf_number, u32 vf_number, bool multiport,
+ u32 subport_number);
int devlink_port_get_phys_port_name(struct devlink_port *devlink_port,
char *name, size_t len);
int devlink_sb_register(struct devlink *devlink, unsigned int sb_index,
@@ -798,13 +802,14 @@ static inline void devlink_port_attrs_set(struct devlink_port *devlink_port,
static inline void
devlink_port_attrs_pci_pf_set(struct devlink_port *devlink_port,
- u32 pf_number)
+ u32 pf_number, bool multiport, u32 subport_number)
{
}
static inline void
devlink_port_attrs_pci_vf_set(struct devlink_port *devlink_port,
- u32 pf_number, u32 vf_number)
+ u32 pf_number, u32 vf_number, bool multiport,
+ u32 subport_number)
{
}
diff --git a/include/uapi/linux/devlink.h b/include/uapi/linux/devlink.h
index 9ce76d4f640d..417ae8233cce 100644
--- a/include/uapi/linux/devlink.h
+++ b/include/uapi/linux/devlink.h
@@ -336,6 +336,7 @@ enum devlink_attr {
DEVLINK_ATTR_PORT_PCI_PF_NUMBER, /* u32 */
DEVLINK_ATTR_PORT_PCI_VF_NUMBER, /* u32 */
+ DEVLINK_ATTR_PORT_PCI_SUBPORT, /* u32 */
/* add new attributes above here, update the policy in devlink.c */
diff --git a/net/core/devlink.c b/net/core/devlink.c
index 49216b688c5b..a7dd958be513 100644
--- a/net/core/devlink.c
+++ b/net/core/devlink.c
@@ -541,6 +541,11 @@ static int devlink_nl_port_attrs_put(struct sk_buff *msg,
if (nla_put_u32(msg, DEVLINK_ATTR_PORT_PCI_PF_NUMBER,
attrs->pci.pf_number))
return -EMSGSIZE;
+
+ if (attrs->pci.multiport &&
+ nla_put_u32(msg, DEVLINK_ATTR_PORT_PCI_SUBPORT,
+ attrs->pci.subport_number))
+ return -EMSGSIZE;
return 0;
default:
return -EINVAL;
@@ -5449,15 +5454,20 @@ EXPORT_SYMBOL_GPL(devlink_port_attrs_set);
* @devlink_port: devlink port
* @pf_number: PCI PF number, in multi-host mapping to hosts depends
* on the platform
+ * @multiport: PCI function has more than one logical port
+ * @subport_number: PCI function has more than one logical port
*/
void devlink_port_attrs_pci_pf_set(struct devlink_port *devlink_port,
- u32 pf_number)
+ u32 pf_number, bool multiport,
+ u32 subport_number)
{
struct devlink_port_attrs *attrs = &devlink_port->attrs;
attrs->set = true;
attrs->flavour = DEVLINK_PORT_FLAVOUR_PCI_PF;
attrs->pci.pf_number = pf_number;
+ attrs->pci.multiport = multiport;
+ attrs->pci.subport_number = subport_number;
devlink_port_notify(devlink_port, DEVLINK_CMD_PORT_NEW);
}
EXPORT_SYMBOL_GPL(devlink_port_attrs_pci_pf_set);
@@ -5469,9 +5479,12 @@ EXPORT_SYMBOL_GPL(devlink_port_attrs_pci_pf_set);
* @pf_number: PCI PF number, in multi-host mapping to hosts depends
* on the platform
* @vf_number: PCI VF number within given PF (ignored for PF itself)
+ * @multiport: PCI function has more than one logical port
+ * @subport_number: PCI function has more than one logical port
*/
void devlink_port_attrs_pci_vf_set(struct devlink_port *devlink_port,
- u32 pf_number, u32 vf_number)
+ u32 pf_number, u32 vf_number, bool multiport,
+ u32 subport_number)
{
struct devlink_port_attrs *attrs = &devlink_port->attrs;
@@ -5479,6 +5492,8 @@ void devlink_port_attrs_pci_vf_set(struct devlink_port *devlink_port,
attrs->flavour = DEVLINK_PORT_FLAVOUR_PCI_VF;
attrs->pci.pf_number = pf_number;
attrs->pci.vf_number = vf_number;
+ attrs->pci.multiport = multiport;
+ attrs->pci.subport_number = subport_number;
devlink_port_notify(devlink_port, DEVLINK_CMD_PORT_NEW);
}
EXPORT_SYMBOL_GPL(devlink_port_attrs_pci_vf_set);
@@ -5508,11 +5523,22 @@ int devlink_port_get_phys_port_name(struct devlink_port *devlink_port,
WARN_ON(1);
return -EINVAL;
case DEVLINK_PORT_FLAVOUR_PCI_PF:
- n = snprintf(name, len, "pf%u", attrs->pci.pf_number);
+ if (!attrs->pci.multiport)
+ n = snprintf(name, len, "pf%u", attrs->pci.pf_number);
+ else
+ n = snprintf(name, len, "pf%us%u", attrs->pci.pf_number,
+ attrs->pci.subport_number);
break;
case DEVLINK_PORT_FLAVOUR_PCI_VF:
- n = snprintf(name, len, "pf%uvf%u",
- attrs->pci.pf_number, attrs->pci.vf_number);
+ if (!attrs->pci.multiport)
+ n = snprintf(name, len, "pf%uvf%u",
+ attrs->pci.pf_number,
+ attrs->pci.vf_number);
+ else
+ n = snprintf(name, len, "pf%uvf%us%u",
+ attrs->pci.pf_number,
+ attrs->pci.vf_number,
+ attrs->pci.subport_number);
break;
}
--
2.19.2
^ permalink raw reply related [flat|nested] 100+ messages in thread
* [PATCH net-next v2 5/7] nfp: switch to devlink_port_get_phys_port_name()
2019-03-01 18:04 [PATCH net-next v2 0/7] devlink: expose PF and VF representors as ports Jakub Kicinski
` (3 preceding siblings ...)
2019-03-01 18:04 ` [PATCH net-next v2 4/7] devlink: allow subports on devlink PCI ports Jakub Kicinski
@ 2019-03-01 18:04 ` Jakub Kicinski
2019-03-01 18:04 ` [PATCH net-next v2 6/7] devlink: introduce port's peer netdevs Jakub Kicinski
` (4 subsequent siblings)
9 siblings, 0 replies; 100+ messages in thread
From: Jakub Kicinski @ 2019-03-01 18:04 UTC (permalink / raw)
To: jiri, davem; +Cc: netdev, oss-drivers, Jakub Kicinski
Now that devlink understands all port flavours - switch
to the devlink_port_get_phys_port_name() helper.
Signed-off-by: Jakub Kicinski <jakub.kicinski@netronome.com>
---
drivers/net/ethernet/netronome/nfp/nfp_port.c | 33 +------------------
1 file changed, 1 insertion(+), 32 deletions(-)
diff --git a/drivers/net/ethernet/netronome/nfp/nfp_port.c b/drivers/net/ethernet/netronome/nfp/nfp_port.c
index 93c5bfc0510b..3e2ff8d35e8d 100644
--- a/drivers/net/ethernet/netronome/nfp/nfp_port.c
+++ b/drivers/net/ethernet/netronome/nfp/nfp_port.c
@@ -117,44 +117,13 @@ struct nfp_eth_table_port *nfp_port_get_eth_port(struct nfp_port *port)
int
nfp_port_get_phys_port_name(struct net_device *netdev, char *name, size_t len)
{
- struct nfp_eth_table_port *eth_port;
struct nfp_port *port;
- int n;
port = nfp_port_from_netdev(netdev);
if (!port)
return -EOPNOTSUPP;
- switch (port->type) {
- case NFP_PORT_PHYS_PORT:
- eth_port = __nfp_port_get_eth_port(port);
- if (!eth_port)
- return -EOPNOTSUPP;
-
- if (!eth_port->is_split)
- n = snprintf(name, len, "p%d", eth_port->label_port);
- else
- n = snprintf(name, len, "p%ds%d", eth_port->label_port,
- eth_port->label_subport);
- break;
- case NFP_PORT_PF_PORT:
- if (!port->pf_split)
- n = snprintf(name, len, "pf%d", port->pf_id);
- else
- n = snprintf(name, len, "pf%ds%d", port->pf_id,
- port->pf_split_id);
- break;
- case NFP_PORT_VF_PORT:
- n = snprintf(name, len, "pf%dvf%d", port->pf_id, port->vf_id);
- break;
- default:
- return -EOPNOTSUPP;
- }
-
- if (n >= len)
- return -EINVAL;
-
- return 0;
+ return devlink_port_get_phys_port_name(&port->dl_port, name, len);
}
/**
--
2.19.2
^ permalink raw reply related [flat|nested] 100+ messages in thread
* [PATCH net-next v2 6/7] devlink: introduce port's peer netdevs
2019-03-01 18:04 [PATCH net-next v2 0/7] devlink: expose PF and VF representors as ports Jakub Kicinski
` (4 preceding siblings ...)
2019-03-01 18:04 ` [PATCH net-next v2 5/7] nfp: switch to devlink_port_get_phys_port_name() Jakub Kicinski
@ 2019-03-01 18:04 ` Jakub Kicinski
2019-03-01 18:04 ` [PATCH net-next v2 7/7] nfp: expose PF " Jakub Kicinski
` (3 subsequent siblings)
9 siblings, 0 replies; 100+ messages in thread
From: Jakub Kicinski @ 2019-03-01 18:04 UTC (permalink / raw)
To: jiri, davem; +Cc: netdev, oss-drivers, Jakub Kicinski
Devlink ports represent ports of a switch device (or SR-IOV
NIC which has an embedded switch). In case of SR-IOV when
PCIe PFs are exposed the PFs which are directly connected
to the local machine may also spawn PF netdev (much like
VFs have a port/"repr" and an actual VF netdev).
Allow devlink to expose such linking. There is currently no
way to find out which netdev corresponds to which PF.
Example:
$ devlink port
pci/0000:82:00.0/0: type eth netdev p4p1 flavour physical
pci/0000:82:00.0/10000: type eth netdev eth1 flavour pci_pf pf 0 peer_netdev enp130s0
pci/0000:82:00.0/10001: type eth netdev eth0 flavour pci_vf pf 0 vf 0
pci/0000:82:00.0/10002: type eth netdev eth2 flavour pci_vf pf 0 vf 1
v2: - move the peer info into a nested attr.
Signed-off-by: Jakub Kicinski <jakub.kicinski@netronome.com>
---
include/net/devlink.h | 11 ++++++
include/uapi/linux/devlink.h | 6 ++++
net/core/devlink.c | 68 +++++++++++++++++++++++++++++++++---
3 files changed, 81 insertions(+), 4 deletions(-)
diff --git a/include/net/devlink.h b/include/net/devlink.h
index 6a29ce80cb38..f3ced79a30a8 100644
--- a/include/net/devlink.h
+++ b/include/net/devlink.h
@@ -68,6 +68,7 @@ struct devlink_port {
enum devlink_port_type type;
enum devlink_port_type desired_type;
void *type_dev;
+ void *type_peer;
struct devlink_port_attrs attrs;
};
@@ -573,6 +574,9 @@ int devlink_port_register(struct devlink *devlink,
void devlink_port_unregister(struct devlink_port *devlink_port);
void devlink_port_type_eth_set(struct devlink_port *devlink_port,
struct net_device *netdev);
+void devlink_port_type_eth_set_peer(struct devlink_port *devlink_port,
+ struct net_device *netdev,
+ struct net_device *peer);
void devlink_port_type_ib_set(struct devlink_port *devlink_port,
struct ib_device *ibdev);
void devlink_port_type_clear(struct devlink_port *devlink_port);
@@ -784,6 +788,13 @@ static inline void devlink_port_type_eth_set(struct devlink_port *devlink_port,
{
}
+static inline void
+devlink_port_type_eth_set_peer(struct devlink_port *devlink_port,
+ struct net_device *netdev,
+ struct net_device *peer)
+{
+}
+
static inline void devlink_port_type_ib_set(struct devlink_port *devlink_port,
struct ib_device *ibdev)
{
diff --git a/include/uapi/linux/devlink.h b/include/uapi/linux/devlink.h
index 417ae8233cce..34ed03bee9fc 100644
--- a/include/uapi/linux/devlink.h
+++ b/include/uapi/linux/devlink.h
@@ -338,6 +338,12 @@ enum devlink_attr {
DEVLINK_ATTR_PORT_PCI_VF_NUMBER, /* u32 */
DEVLINK_ATTR_PORT_PCI_SUBPORT, /* u32 */
+ DEVLINK_ATTR_PORT_PEER, /* nested */
+ DEVLINK_ATTR_PORT_PEER_TYPE, /* u16 */
+ DEVLINK_ATTR_PORT_PEER_NETDEV_IFINDEX, /* u32 */
+ DEVLINK_ATTR_PORT_PEER_NETDEV_NAME, /* string */
+ DEVLINK_ATTR_PORT_PEER_IBDEV_NAME, /* string */
+
/* add new attributes above here, update the policy in devlink.c */
__DEVLINK_ATTR_MAX,
diff --git a/net/core/devlink.c b/net/core/devlink.c
index a7dd958be513..75c313b5b616 100644
--- a/net/core/devlink.c
+++ b/net/core/devlink.c
@@ -552,6 +552,47 @@ static int devlink_nl_port_attrs_put(struct sk_buff *msg,
}
}
+static int devlink_nl_port_peer_put(struct sk_buff *msg,
+ struct devlink_port *devlink_port)
+{
+ struct nlattr *peer_attr;
+
+ if (!devlink_port->type_peer)
+ return 0;
+
+ peer_attr = nla_nest_start(msg, DEVLINK_ATTR_PORT_PEER);
+ if (!peer_attr)
+ return -EMSGSIZE;
+
+ /* Peer's type is got to be the same as the port's type */
+ if (nla_put_u16(msg, DEVLINK_ATTR_PORT_PEER_TYPE, devlink_port->type))
+ goto cancel_peer_attr;
+
+ if (devlink_port->type == DEVLINK_PORT_TYPE_ETH) {
+ struct net_device *netdev = devlink_port->type_peer;
+
+ if (nla_put_u32(msg, DEVLINK_ATTR_PORT_PEER_NETDEV_IFINDEX,
+ netdev->ifindex) ||
+ nla_put_string(msg, DEVLINK_ATTR_PORT_PEER_NETDEV_NAME,
+ netdev->name))
+ goto cancel_peer_attr;
+ }
+ if (devlink_port->type == DEVLINK_PORT_TYPE_IB) {
+ struct ib_device *ibdev = devlink_port->type_peer;
+
+ if (ibdev &&
+ nla_put_string(msg, DEVLINK_ATTR_PORT_PEER_IBDEV_NAME,
+ ibdev->name))
+ goto cancel_peer_attr;
+ }
+ nla_nest_end(msg, peer_attr);
+ return 0;
+
+cancel_peer_attr:
+ nla_nest_cancel(msg, peer_attr);
+ return -EMSGSIZE;
+}
+
static int devlink_nl_port_fill(struct sk_buff *msg, struct devlink *devlink,
struct devlink_port *devlink_port,
enum devlink_command cmd, u32 portid,
@@ -593,6 +634,8 @@ static int devlink_nl_port_fill(struct sk_buff *msg, struct devlink *devlink,
}
if (devlink_nl_port_attrs_put(msg, devlink_port))
goto nla_put_failure;
+ if (devlink_nl_port_peer_put(msg, devlink_port))
+ goto nla_put_failure;
genlmsg_end(msg, hdr);
return 0;
@@ -5370,10 +5413,11 @@ EXPORT_SYMBOL_GPL(devlink_port_unregister);
static void __devlink_port_type_set(struct devlink_port *devlink_port,
enum devlink_port_type type,
- void *type_dev)
+ void *type_dev, void *type_peer)
{
devlink_port->type = type;
devlink_port->type_dev = type_dev;
+ devlink_port->type_peer = type_peer;
devlink_port_notify(devlink_port, DEVLINK_CMD_PORT_NEW);
}
@@ -5387,10 +5431,26 @@ void devlink_port_type_eth_set(struct devlink_port *devlink_port,
struct net_device *netdev)
{
return __devlink_port_type_set(devlink_port,
- DEVLINK_PORT_TYPE_ETH, netdev);
+ DEVLINK_PORT_TYPE_ETH, netdev, NULL);
}
EXPORT_SYMBOL_GPL(devlink_port_type_eth_set);
+/**
+ * devlink_port_type_eth_set_peer - Set port type to Ethernet with peer
+ *
+ * @devlink_port: devlink port
+ * @netdev: related netdevice
+ * @peer: for PCIe ports the non-port netdev (actual VF or PF)
+ */
+void devlink_port_type_eth_set_peer(struct devlink_port *devlink_port,
+ struct net_device *netdev,
+ struct net_device *peer)
+{
+ return __devlink_port_type_set(devlink_port,
+ DEVLINK_PORT_TYPE_ETH, netdev, peer);
+}
+EXPORT_SYMBOL_GPL(devlink_port_type_eth_set_peer);
+
/**
* devlink_port_type_ib_set - Set port type to InfiniBand
*
@@ -5401,7 +5461,7 @@ void devlink_port_type_ib_set(struct devlink_port *devlink_port,
struct ib_device *ibdev)
{
return __devlink_port_type_set(devlink_port,
- DEVLINK_PORT_TYPE_IB, ibdev);
+ DEVLINK_PORT_TYPE_IB, ibdev, NULL);
}
EXPORT_SYMBOL_GPL(devlink_port_type_ib_set);
@@ -5413,7 +5473,7 @@ EXPORT_SYMBOL_GPL(devlink_port_type_ib_set);
void devlink_port_type_clear(struct devlink_port *devlink_port)
{
return __devlink_port_type_set(devlink_port,
- DEVLINK_PORT_TYPE_NOTSET, NULL);
+ DEVLINK_PORT_TYPE_NOTSET, NULL, NULL);
}
EXPORT_SYMBOL_GPL(devlink_port_type_clear);
--
2.19.2
^ permalink raw reply related [flat|nested] 100+ messages in thread
* [PATCH net-next v2 7/7] nfp: expose PF peer netdevs
2019-03-01 18:04 [PATCH net-next v2 0/7] devlink: expose PF and VF representors as ports Jakub Kicinski
` (5 preceding siblings ...)
2019-03-01 18:04 ` [PATCH net-next v2 6/7] devlink: introduce port's peer netdevs Jakub Kicinski
@ 2019-03-01 18:04 ` Jakub Kicinski
2019-03-02 10:13 ` [PATCH net-next v2 0/7] devlink: expose PF and VF representors as ports Jiri Pirko
` (2 subsequent siblings)
9 siblings, 0 replies; 100+ messages in thread
From: Jakub Kicinski @ 2019-03-01 18:04 UTC (permalink / raw)
To: jiri, davem; +Cc: netdev, oss-drivers, Jakub Kicinski
Expose PF netdevs as devlink port's peers.
Signed-off-by: Jakub Kicinski <jakub.kicinski@netronome.com>
---
drivers/net/ethernet/netronome/nfp/abm/main.c | 1 +
drivers/net/ethernet/netronome/nfp/flower/main.c | 1 +
drivers/net/ethernet/netronome/nfp/nfp_devlink.c | 3 ++-
drivers/net/ethernet/netronome/nfp/nfp_port.h | 2 ++
4 files changed, 6 insertions(+), 1 deletion(-)
diff --git a/drivers/net/ethernet/netronome/nfp/abm/main.c b/drivers/net/ethernet/netronome/nfp/abm/main.c
index 4d4ff5844c47..8d7ff1200fd4 100644
--- a/drivers/net/ethernet/netronome/nfp/abm/main.c
+++ b/drivers/net/ethernet/netronome/nfp/abm/main.c
@@ -113,6 +113,7 @@ nfp_abm_spawn_repr(struct nfp_app *app, struct nfp_abm_link *alink,
port->pf_id = alink->abm->pf_id;
port->pf_split = app->pf->max_data_vnics > 1;
port->pf_split_id = alink->id;
+ port->peer = alink->vnic->dp.netdev;
port->vnic = alink->vnic->dp.ctrl_bar;
}
diff --git a/drivers/net/ethernet/netronome/nfp/flower/main.c b/drivers/net/ethernet/netronome/nfp/flower/main.c
index 408089133599..13aa21923bf7 100644
--- a/drivers/net/ethernet/netronome/nfp/flower/main.c
+++ b/drivers/net/ethernet/netronome/nfp/flower/main.c
@@ -300,6 +300,7 @@ nfp_flower_spawn_vnic_reprs(struct nfp_app *app,
if (repr_type == NFP_REPR_TYPE_PF) {
port->pf_id = i;
port->vnic = priv->nn->dp.ctrl_bar;
+ port->peer = priv->nn->dp.netdev;
} else {
port->pf_id = 0;
port->vf_id = i;
diff --git a/drivers/net/ethernet/netronome/nfp/nfp_devlink.c b/drivers/net/ethernet/netronome/nfp/nfp_devlink.c
index 6ad2805f1efc..ced21a9951aa 100644
--- a/drivers/net/ethernet/netronome/nfp/nfp_devlink.c
+++ b/drivers/net/ethernet/netronome/nfp/nfp_devlink.c
@@ -377,7 +377,8 @@ int nfp_devlink_port_init(struct nfp_app *app, struct nfp_port *port)
case NFP_PORT_PHYS_PORT:
return nfp_devlink_port_init_phys(devlink, port);
case NFP_PORT_PF_PORT:
- devlink_port_type_eth_set(&port->dl_port, port->netdev);
+ devlink_port_type_eth_set_peer(&port->dl_port, port->netdev,
+ port->peer);
devlink_port_attrs_pci_pf_set(&port->dl_port, port->pf_id,
port->pf_split,
port->pf_split_id);
diff --git a/drivers/net/ethernet/netronome/nfp/nfp_port.h b/drivers/net/ethernet/netronome/nfp/nfp_port.h
index 09c55ca2371a..c75a25cc5cea 100644
--- a/drivers/net/ethernet/netronome/nfp/nfp_port.h
+++ b/drivers/net/ethernet/netronome/nfp/nfp_port.h
@@ -51,6 +51,7 @@ enum nfp_port_flags {
* @eth_forced: for %NFP_PORT_PHYS_PORT port is forced UP or DOWN, don't change
* @eth_port: for %NFP_PORT_PHYS_PORT translated ETH Table port entry
* @eth_stats: for %NFP_PORT_PHYS_PORT MAC stats if available
+ * @peer: for %NFP_PORT_PF_PORT netdev of the actual vNIC, if reachable
* @pf_id: for %NFP_PORT_PF_PORT, %NFP_PORT_VF_PORT ID of the PCI PF (0-3)
* @vf_id: for %NFP_PORT_VF_PORT ID of the PCI VF within @pf_id
* @pf_split: for %NFP_PORT_PF_PORT %true if PCI PF has more than one vNIC
@@ -79,6 +80,7 @@ struct nfp_port {
};
/* NFP_PORT_PF_PORT, NFP_PORT_VF_PORT */
struct {
+ struct net_device *peer;
unsigned int pf_id;
unsigned int vf_id;
bool pf_split;
--
2.19.2
^ permalink raw reply related [flat|nested] 100+ messages in thread
* Re: [PATCH net-next v2 3/7] nfp: register devlink ports of all reprs
2019-03-01 18:04 ` [PATCH net-next v2 3/7] nfp: register devlink ports of all reprs Jakub Kicinski
@ 2019-03-02 8:43 ` Jiri Pirko
2019-03-02 19:07 ` Jakub Kicinski
0 siblings, 1 reply; 100+ messages in thread
From: Jiri Pirko @ 2019-03-02 8:43 UTC (permalink / raw)
To: Jakub Kicinski; +Cc: davem, netdev, oss-drivers
Fri, Mar 01, 2019 at 07:04:49PM CET, jakub.kicinski@netronome.com wrote:
>Register all representors as devlink ports.
>
>The port_index is slightly tricky to figure out, we use a bit of
>arbitrary math to create unique IDs for PCI ports.
>
>Signed-off-by: Jakub Kicinski <jakub.kicinski@netronome.com>
>---
> .../net/ethernet/netronome/nfp/nfp_devlink.c | 40 ++++++++++++++++++-
> .../net/ethernet/netronome/nfp/nfp_net_repr.c | 16 +++++++-
> 2 files changed, 53 insertions(+), 3 deletions(-)
>
>diff --git a/drivers/net/ethernet/netronome/nfp/nfp_devlink.c b/drivers/net/ethernet/netronome/nfp/nfp_devlink.c
>index 9af3cb1f2f17..bf7fd9614152 100644
>--- a/drivers/net/ethernet/netronome/nfp/nfp_devlink.c
>+++ b/drivers/net/ethernet/netronome/nfp/nfp_devlink.c
>@@ -350,7 +350,8 @@ const struct devlink_ops nfp_devlink_ops = {
> .flash_update = nfp_devlink_flash_update,
> };
>
>-int nfp_devlink_port_init(struct nfp_app *app, struct nfp_port *port)
>+static int
>+nfp_devlink_port_init_phys(struct devlink *devlink, struct nfp_port *port)
> {
> struct nfp_eth_table_port eth_port;
> int ret;
>@@ -368,6 +369,27 @@ int nfp_devlink_port_init(struct nfp_app *app, struct nfp_port *port)
> return 0;
> }
>
>+int nfp_devlink_port_init(struct nfp_app *app, struct nfp_port *port)
>+{
>+ struct devlink *devlink = priv_to_devlink(app->pf);
>+
>+ switch (port->type) {
>+ case NFP_PORT_PHYS_PORT:
>+ return nfp_devlink_port_init_phys(devlink, port);
>+ case NFP_PORT_PF_PORT:
>+ devlink_port_type_eth_set(&port->dl_port, port->netdev);
>+ devlink_port_attrs_pci_pf_set(&port->dl_port, port->pf_id);
>+ return 0;
>+ case NFP_PORT_VF_PORT:
>+ devlink_port_type_eth_set(&port->dl_port, port->netdev);
>+ devlink_port_attrs_pci_vf_set(&port->dl_port, port->pf_id,
>+ port->vf_id);
What is the reason to expose vf/pf id for switch port? Isn't it rather
an attribute of a peer?
>+ return 0;
>+ default:
>+ return -EINVAL;
>+ }
>+}
>+
> void nfp_devlink_port_clean(struct nfp_port *port)
> {
> }
>@@ -376,7 +398,21 @@ int nfp_devlink_port_register(struct nfp_app *app, struct nfp_port *port)
> {
> struct devlink *devlink = priv_to_devlink(app->pf);
>
>- return devlink_port_register(devlink, &port->dl_port, port->eth_id);
>+ switch (port->type) {
>+ case NFP_PORT_PHYS_PORT:
>+ return devlink_port_register(devlink, &port->dl_port,
>+ port->eth_id);
>+ case NFP_PORT_PF_PORT:
>+ return devlink_port_register(devlink, &port->dl_port,
>+ (port->pf_id + 1) * 10000 +
>+ port->pf_split_id * 1000);
Wait. What this 10000/1000 magic about?
>+ case NFP_PORT_VF_PORT:
>+ return devlink_port_register(devlink, &port->dl_port,
>+ (port->pf_id + 1) * 10000 +
>+ port->vf_id + 1);
>+ default:
>+ return -EINVAL;
>+ }
> }
>
> void nfp_devlink_port_unregister(struct nfp_port *port)
>diff --git a/drivers/net/ethernet/netronome/nfp/nfp_net_repr.c b/drivers/net/ethernet/netronome/nfp/nfp_net_repr.c
>index d2c803bb4e56..869d22760a6e 100644
>--- a/drivers/net/ethernet/netronome/nfp/nfp_net_repr.c
>+++ b/drivers/net/ethernet/netronome/nfp/nfp_net_repr.c
>@@ -292,7 +292,9 @@ nfp_repr_transfer_features(struct net_device *netdev, struct net_device *lower)
>
> static void nfp_repr_clean(struct nfp_repr *repr)
> {
>+ nfp_devlink_port_unregister(repr->port);
> unregister_netdev(repr->netdev);
>+ nfp_devlink_port_clean(repr->port);
> nfp_app_repr_clean(repr->app, repr->netdev);
> dst_release((struct dst_entry *)repr->dst);
> nfp_port_free(repr->port);
>@@ -395,12 +397,24 @@ int nfp_repr_init(struct nfp_app *app, struct net_device *netdev,
> if (err)
> goto err_clean;
>
>- err = register_netdev(netdev);
>+ err = nfp_devlink_port_init(app, repr->port);
> if (err)
> goto err_repr_clean;
>
>+ err = register_netdev(netdev);
>+ if (err)
>+ goto err_port_clean;
>+
>+ err = nfp_devlink_port_register(app, repr->port);
Don't you want to take my patch ("nfp: register devlink port before
netdev") to change order of register_netdev and devlink_port_register,
include it to this patchset before this patch and change the order in
this patch too? I think it would be clearer to do it from the beginning.
>+ if (err)
>+ goto err_unreg_netdev;
>+
> return 0;
>
>+err_unreg_netdev:
>+ unregister_netdev(repr->netdev);
>+err_port_clean:
>+ nfp_devlink_port_clean(repr->port);
> err_repr_clean:
> nfp_app_repr_clean(app, netdev);
> err_clean:
>--
>2.19.2
>
^ permalink raw reply [flat|nested] 100+ messages in thread
* Re: [PATCH net-next v2 4/7] devlink: allow subports on devlink PCI ports
2019-03-01 18:04 ` [PATCH net-next v2 4/7] devlink: allow subports on devlink PCI ports Jakub Kicinski
@ 2019-03-02 9:41 ` Jiri Pirko
2019-03-02 19:48 ` Jakub Kicinski
2019-03-04 11:08 ` Jiri Pirko
1 sibling, 1 reply; 100+ messages in thread
From: Jiri Pirko @ 2019-03-02 9:41 UTC (permalink / raw)
To: Jakub Kicinski; +Cc: davem, netdev, oss-drivers
Fri, Mar 01, 2019 at 07:04:50PM CET, jakub.kicinski@netronome.com wrote:
>PCI endpoint corresponds to a PCI device, but such device
>can have one more more logical device ports associated with it.
>We need a way to distinguish those. Add a PCI subport in the
>dumps and print the info in phys_port_name appropriately.
>
>This is not equivalent to port splitting, there is no split
>group. It's just a way of representing multiple netdevs on
>a single PCI function.
>
>Note that the quality of being multiport pertains only to
>the PCI function itself. A PF having multiple netdevs does
>not mean that its VFs will also have multiple, or that VFs
>are associated with any particular port of a multiport VF.
>
>Example (bus 05 device has subports, bus 82 has only one port per
>function):
>
>$ devlink port
>pci/0000:05:00.0/0: type eth netdev enp5s0np0 flavour physical
>pci/0000:05:00.0/10000: type eth netdev enp5s0npf0s0 flavour pci_pf pf 0 subport 0
>pci/0000:05:00.0/4: type eth netdev enp5s0np1 flavour physical
>pci/0000:05:00.0/11000: type eth netdev enp5s0npf0s1 flavour pci_pf pf 0 subport 1
So these subport devlink ports are eswitch ports for subports, right?
Please see the following drawing:
+---+ +---+ +---+
pfsub| 5 | vf| 6 | | 7 |pfsub
+-+-+ +-+-+ +-+-+
physical link <---------+ | | |
| | | |
| | | |
| | | |
+-+-+ +-+-+ +-+-+ +-+-+
| 1 | | 2 | | 3 | | 4 |
+--+---+------+---+------+---+------+---+--+
| physical pfsub vf pfsub |
| port port port port |
| |
| eswitch |
| |
| |
+------------------------------------------+
1) pci/0000:05:00.0/0: type eth netdev enp5s0np0 flavour physical switch_id 00154d130d2f
2) pci/0000:05:00.0/10000: type eth netdev enp5s0npf0s0 flavour pci_pf pf 0 subport 0 switch_id 00154d130d2f
3) pci/0000:05:00.0/10001: type eth netdev enp5s0npf0vf0 flavour pci_vf pf 0 vf 0 switch_id 00154d130d2f
4) pci/0000:05:00.0/10001: type eth netdev enp5s0npf0s1 flavour pci_pf pf 0 subport 1 switch_id 00154d130d2f
This is basically what you have and I think we are in sync with that.
But what about 5,6,7? Should they have devlink port instances too?
5) pci/0000:05:00.0/1: type eth netdev enp5s0f0?? flavour ???? pf 0 subport 0
6) pci/0000:05:10.1/0: type eth netdev enp5s10f0 flavour ???? pf 0 vf 0
7) pci/0000:05:00.0/1: type eth netdev enp5s0f0?? flavour ???? pf 0 subport 1
These are the "peers".
I think that there could be flavours "pci_pf" and "pci_vf". Then the
"representors" (switch ports) could have flavours "pci_pf_port" and
"pci_vf_port" or something like that. User can see right away
that is not "PF" of "VF" but rather something "on the other end".
Note there is no "switch_id" for these devlink ports that tells the user
these devlink ports are not part of any switch.
What do you think?
>pci/0000:82:00.0/0: type eth netdev p4p1 flavour physical
>pci/0000:82:00.0/10000: type eth netdev eth0 flavour pci_pf pf 0
>
[...]
^ permalink raw reply [flat|nested] 100+ messages in thread
* Re: [PATCH net-next v2 0/7] devlink: expose PF and VF representors as ports
2019-03-01 18:04 [PATCH net-next v2 0/7] devlink: expose PF and VF representors as ports Jakub Kicinski
` (6 preceding siblings ...)
2019-03-01 18:04 ` [PATCH net-next v2 7/7] nfp: expose PF " Jakub Kicinski
@ 2019-03-02 10:13 ` Jiri Pirko
2019-03-02 19:49 ` [oss-drivers] " Jakub Kicinski
2019-03-04 5:12 ` Parav Pandit
2019-03-04 18:22 ` David Miller
2019-03-20 20:25 ` Jakub Kicinski
9 siblings, 2 replies; 100+ messages in thread
From: Jiri Pirko @ 2019-03-02 10:13 UTC (permalink / raw)
To: Jakub Kicinski; +Cc: davem, netdev, oss-drivers
Fri, Mar 01, 2019 at 07:04:46PM CET, jakub.kicinski@netronome.com wrote:
>Hi!
>
>This series is a long overdue follow up to Jiri's work on providing
>a common .ndo_phys_port_name implementation based on devlink ports.
>
>First devlink port flavours for PF and VF ports are added, and
>registered by the NFP. Port numbers and split info are reserved
>for physical and DSA ports. For PCIe ports we add pf/vf identifiers.
>Note that devices may have more than one PF, including multi host
>scenarios where not all pfs are connected to the same host.
>
>The port_index is slightly tricky to figure out, we use a bit of
>math to create unique IDs for ports.
>
>Next subports for PCIe ports are introduced. This is in case device
>has more than one netdev on a physical function (e.g. multi port
>SmartNIC).
>
>With the above features in place we can remove the ndo_phys_port_name
>implementation from the NFP and use the standard devlink one for
>port netdevs.
>
>Last but not least a concept of peer netdevs is added. In multi-host
>scenarios its currently not possible to figure out which PF is
>associated with the local host. Peer device is "the other side
>of the wire" for PCIe ports. In case of NFP we only link the PF
>netdevs, but it should be possible to also link VF peers after
>VF driver probes, if users request it.
>
>This is the conceptual image of devlink instances:
>
> HOST A || HOST B
> ||
> PF A | V | V | V | V || PF B | V | V | V
> |*F |*F |*F |*F ... || |*F |*F |*F
>*port A0 |*port A1 | 0 | 1 | 2 | 3 ||*port B0 |*port B1 | 0 | 1 | 2
> ||
> PCI Express link || PCI Express link
> \ \ \ | | | | | / / /
> \ \ \ | | | | | / / /
> /\ \______\______\'___|___|__________|_______'____/___/___/__ /\
> || |+PF0s0|+PF0s1 |+VF0|+VF1| ...| |+PF1s0|+PF1s1|+VF0|+VF1| ||
> i || |------ ------ ----- ---- ----|--- ------ ------ ---- ----| || i
>d n H || | <<========== | || d n H
>e s O || | ==========>> | || e s O
>v t S || | SR-IOV e-switch | || v t S
>l a T || | <<========== | || l a T
>i n || | ==========>> | || i n
>n c A || | ________ _________ ________ | || n c B
>k e || | |+Phys 0 |+Phys 1 |+Phys 2 | | || k e
> || \---------------------------------------------------------/ ||
> \/ | | | \/
> | | |
> || ||
> MAC 0 || MAC 1 || MAC 2
> || ||
>
> '+' marks the devlink ports and port (-representor-) netdevs
> '*' marks host netdevs (actual VF/PF netdev)
As I wrote in the reply to patch 4, I think we need to figure out if we
want to model all entities that belong under specific devlink
instance/pci address - which I prefer, or we want to have only eswitch
ports there.
One way or another, I think that it is not good idea to merge this
patchset this late, I would prefer to wait for next net-next opening...
In the meantime we can sync and make this whole thing crystal clear, for
everyone.
Thanks!
^ permalink raw reply [flat|nested] 100+ messages in thread
* Re: [PATCH net-next v2 3/7] nfp: register devlink ports of all reprs
2019-03-02 8:43 ` Jiri Pirko
@ 2019-03-02 19:07 ` Jakub Kicinski
2019-03-04 7:36 ` Jiri Pirko
0 siblings, 1 reply; 100+ messages in thread
From: Jakub Kicinski @ 2019-03-02 19:07 UTC (permalink / raw)
To: Jiri Pirko; +Cc: davem, netdev, oss-drivers
On Sat, 2 Mar 2019 09:43:47 +0100, Jiri Pirko wrote:
> Fri, Mar 01, 2019 at 07:04:49PM CET, jakub.kicinski@netronome.com wrote:
> >Register all representors as devlink ports.
> >
> >The port_index is slightly tricky to figure out, we use a bit of
> >arbitrary math to create unique IDs for PCI ports.
> >
> >Signed-off-by: Jakub Kicinski <jakub.kicinski@netronome.com>
> >---
> > .../net/ethernet/netronome/nfp/nfp_devlink.c | 40 ++++++++++++++++++-
> > .../net/ethernet/netronome/nfp/nfp_net_repr.c | 16 +++++++-
> > 2 files changed, 53 insertions(+), 3 deletions(-)
> >
> >diff --git a/drivers/net/ethernet/netronome/nfp/nfp_devlink.c b/drivers/net/ethernet/netronome/nfp/nfp_devlink.c
> >index 9af3cb1f2f17..bf7fd9614152 100644
> >--- a/drivers/net/ethernet/netronome/nfp/nfp_devlink.c
> >+++ b/drivers/net/ethernet/netronome/nfp/nfp_devlink.c
> >@@ -350,7 +350,8 @@ const struct devlink_ops nfp_devlink_ops = {
> > .flash_update = nfp_devlink_flash_update,
> > };
> >
> >-int nfp_devlink_port_init(struct nfp_app *app, struct nfp_port *port)
> >+static int
> >+nfp_devlink_port_init_phys(struct devlink *devlink, struct nfp_port *port)
> > {
> > struct nfp_eth_table_port eth_port;
> > int ret;
> >@@ -368,6 +369,27 @@ int nfp_devlink_port_init(struct nfp_app *app, struct nfp_port *port)
> > return 0;
> > }
> >
> >+int nfp_devlink_port_init(struct nfp_app *app, struct nfp_port *port)
> >+{
> >+ struct devlink *devlink = priv_to_devlink(app->pf);
> >+
> >+ switch (port->type) {
> >+ case NFP_PORT_PHYS_PORT:
> >+ return nfp_devlink_port_init_phys(devlink, port);
> >+ case NFP_PORT_PF_PORT:
> >+ devlink_port_type_eth_set(&port->dl_port, port->netdev);
> >+ devlink_port_attrs_pci_pf_set(&port->dl_port, port->pf_id);
> >+ return 0;
> >+ case NFP_PORT_VF_PORT:
> >+ devlink_port_type_eth_set(&port->dl_port, port->netdev);
> >+ devlink_port_attrs_pci_vf_set(&port->dl_port, port->pf_id,
> >+ port->vf_id);
>
> What is the reason to expose vf/pf id for switch port? Isn't it rather
> an attribute of a peer?
Naw, its an attribute of the port. I leave the ASIC via PF n or VF m
of PF n. Whatever is on the other side is isolated from the topology
of the ASIC.
Is the physical port ID an attribute of the other end of the cable?
> >+ return 0;
> >+ default:
> >+ return -EINVAL;
> >+ }
> >+}
> >+
> > void nfp_devlink_port_clean(struct nfp_port *port)
> > {
> > }
> >@@ -376,7 +398,21 @@ int nfp_devlink_port_register(struct nfp_app *app, struct nfp_port *port)
> > {
> > struct devlink *devlink = priv_to_devlink(app->pf);
> >
> >- return devlink_port_register(devlink, &port->dl_port, port->eth_id);
> >+ switch (port->type) {
> >+ case NFP_PORT_PHYS_PORT:
> >+ return devlink_port_register(devlink, &port->dl_port,
> >+ port->eth_id);
> >+ case NFP_PORT_PF_PORT:
> >+ return devlink_port_register(devlink, &port->dl_port,
> >+ (port->pf_id + 1) * 10000 +
> >+ port->pf_split_id * 1000);
>
> Wait. What this 10000/1000 magic about?
port_index has to be unique, I need some unique number here, as I
stated both in the commit message and the cover letter, this is
arbitrary.
I can put the datapath port identifier in there but its (a)
meaningless, (b) a bitfield, so it will look like 8972367083. And it
may change depending on the FW load, so its not stable either.
> >+ case NFP_PORT_VF_PORT:
> >+ return devlink_port_register(devlink, &port->dl_port,
> >+ (port->pf_id + 1) * 10000 +
> >+ port->vf_id + 1);
> >+ default:
> >+ return -EINVAL;
> >+ }
> > }
> >
> > void nfp_devlink_port_unregister(struct nfp_port *port)
> >diff --git a/drivers/net/ethernet/netronome/nfp/nfp_net_repr.c b/drivers/net/ethernet/netronome/nfp/nfp_net_repr.c
> >index d2c803bb4e56..869d22760a6e 100644
> >--- a/drivers/net/ethernet/netronome/nfp/nfp_net_repr.c
> >+++ b/drivers/net/ethernet/netronome/nfp/nfp_net_repr.c
> >@@ -292,7 +292,9 @@ nfp_repr_transfer_features(struct net_device *netdev, struct net_device *lower)
> >
> > static void nfp_repr_clean(struct nfp_repr *repr)
> > {
> >+ nfp_devlink_port_unregister(repr->port);
> > unregister_netdev(repr->netdev);
> >+ nfp_devlink_port_clean(repr->port);
> > nfp_app_repr_clean(repr->app, repr->netdev);
> > dst_release((struct dst_entry *)repr->dst);
> > nfp_port_free(repr->port);
> >@@ -395,12 +397,24 @@ int nfp_repr_init(struct nfp_app *app, struct net_device *netdev,
> > if (err)
> > goto err_clean;
> >
> >- err = register_netdev(netdev);
> >+ err = nfp_devlink_port_init(app, repr->port);
> > if (err)
> > goto err_repr_clean;
> >
> >+ err = register_netdev(netdev);
> >+ if (err)
> >+ goto err_port_clean;
> >+
> >+ err = nfp_devlink_port_register(app, repr->port);
>
> Don't you want to take my patch ("nfp: register devlink port before
> netdev") to change order of register_netdev and devlink_port_register,
> include it to this patchset before this patch and change the order in
> this patch too? I think it would be clearer to do it from the beginning.
This way both netdev and devlink_port can get registered fully
initialized. Otherwise we'd get two notifications. Are we trying to
establish some ordering rules to get around the rtnl locking? :)
> >+ if (err)
> >+ goto err_unreg_netdev;
> >+
> > return 0;
> >
> >+err_unreg_netdev:
> >+ unregister_netdev(repr->netdev);
> >+err_port_clean:
> >+ nfp_devlink_port_clean(repr->port);
> > err_repr_clean:
> > nfp_app_repr_clean(app, netdev);
> > err_clean:
^ permalink raw reply [flat|nested] 100+ messages in thread
* Re: [PATCH net-next v2 4/7] devlink: allow subports on devlink PCI ports
2019-03-02 9:41 ` Jiri Pirko
@ 2019-03-02 19:48 ` Jakub Kicinski
2019-03-04 7:56 ` Jiri Pirko
2019-03-04 11:19 ` Jiri Pirko
0 siblings, 2 replies; 100+ messages in thread
From: Jakub Kicinski @ 2019-03-02 19:48 UTC (permalink / raw)
To: Jiri Pirko; +Cc: davem, netdev, oss-drivers
On Sat, 2 Mar 2019 10:41:16 +0100, Jiri Pirko wrote:
> Fri, Mar 01, 2019 at 07:04:50PM CET, jakub.kicinski@netronome.com wrote:
> >PCI endpoint corresponds to a PCI device, but such device
> >can have one more more logical device ports associated with it.
> >We need a way to distinguish those. Add a PCI subport in the
> >dumps and print the info in phys_port_name appropriately.
> >
> >This is not equivalent to port splitting, there is no split
> >group. It's just a way of representing multiple netdevs on
> >a single PCI function.
> >
> >Note that the quality of being multiport pertains only to
> >the PCI function itself. A PF having multiple netdevs does
> >not mean that its VFs will also have multiple, or that VFs
> >are associated with any particular port of a multiport VF.
> >
> >Example (bus 05 device has subports, bus 82 has only one port per
> >function):
> >
> >$ devlink port
> >pci/0000:05:00.0/0: type eth netdev enp5s0np0 flavour physical
> >pci/0000:05:00.0/10000: type eth netdev enp5s0npf0s0 flavour pci_pf pf 0 subport 0
> >pci/0000:05:00.0/4: type eth netdev enp5s0np1 flavour physical
> >pci/0000:05:00.0/11000: type eth netdev enp5s0npf0s1 flavour pci_pf pf 0 subport 1
>
> So these subport devlink ports are eswitch ports for subports, right?
>
> Please see the following drawing:
>
> +---+ +---+ +---+
> pfsub| 5 | vf| 6 | | 7 |pfsub
> +-+-+ +-+-+ +-+-+
> physical link <---------+ | | |
> | | | |
> | | | |
> | | | |
> +-+-+ +-+-+ +-+-+ +-+-+
> | 1 | | 2 | | 3 | | 4 |
> +--+---+------+---+------+---+------+---+--+
> | physical pfsub vf pfsub |
> | port port port port |
> | |
> | eswitch |
> | |
> | |
> +------------------------------------------+
>
> 1) pci/0000:05:00.0/0: type eth netdev enp5s0np0 flavour physical switch_id 00154d130d2f
> 2) pci/0000:05:00.0/10000: type eth netdev enp5s0npf0s0 flavour pci_pf pf 0 subport 0 switch_id 00154d130d2f
> 3) pci/0000:05:00.0/10001: type eth netdev enp5s0npf0vf0 flavour pci_vf pf 0 vf 0 switch_id 00154d130d2f
> 4) pci/0000:05:00.0/10001: type eth netdev enp5s0npf0s1 flavour pci_pf pf 0 subport 1 switch_id 00154d130d2f
>
> This is basically what you have and I think we are in sync with that.
> But what about 5,6,7? Should they have devlink port instances too?
>
> 5) pci/0000:05:00.0/1: type eth netdev enp5s0f0?? flavour ???? pf 0 subport 0
> 6) pci/0000:05:10.1/0: type eth netdev enp5s10f0 flavour ???? pf 0 vf 0
> 7) pci/0000:05:00.0/1: type eth netdev enp5s0f0?? flavour ???? pf 0 subport 1
>
> These are the "peers".
> I think that there could be flavours "pci_pf" and "pci_vf". Then the
> "representors" (switch ports) could have flavours "pci_pf_port" and
> "pci_vf_port" or something like that. User can see right away
> that is not "PF" of "VF" but rather something "on the other end".
> Note there is no "switch_id" for these devlink ports that tells the user
> these devlink ports are not part of any switch.
> What do you think?
Hmmm.. Hm. Hm.
To me its neat if the devlink instance matches an ASIC. I think it's
kind of clear for people to understand what it stands for then. So if
we wanted to do the above we'd have to make the switch_id the first
class identifier for devlink instances, rather than the bus? But then
VF instances don't have a switch ID so that doesn't work...
I need to think about it.
It's also kind of strange that we have to add the noun *port* to the
flavour of... a port... So I would prefer not to have those showing up
as ports. Can we invent a new command (say "partition"?) that'd take
the bus info where the partition is to be spawned?
My next goal is to find a way of grouping multiple bus devices under one
"ASIC" (which is a devlink instance to me) so it can be understood
easily how things are laid out when there is more than one PF connected
to one host.
^ permalink raw reply [flat|nested] 100+ messages in thread
* Re: [oss-drivers] Re: [PATCH net-next v2 0/7] devlink: expose PF and VF representors as ports
2019-03-02 10:13 ` [PATCH net-next v2 0/7] devlink: expose PF and VF representors as ports Jiri Pirko
@ 2019-03-02 19:49 ` Jakub Kicinski
2019-03-04 5:12 ` Parav Pandit
1 sibling, 0 replies; 100+ messages in thread
From: Jakub Kicinski @ 2019-03-02 19:49 UTC (permalink / raw)
To: Jiri Pirko; +Cc: davem, netdev, oss-drivers
On Sat, 2 Mar 2019 11:13:37 +0100, Jiri Pirko wrote:
> >This is the conceptual image of devlink instances:
> >
> > HOST A || HOST B
> > ||
> > PF A | V | V | V | V || PF B | V | V | V
> > |*F |*F |*F |*F ... || |*F |*F |*F
> >*port A0 |*port A1 | 0 | 1 | 2 | 3 ||*port B0 |*port B1 | 0 | 1 | 2
> > ||
> > PCI Express link || PCI Express link
> > \ \ \ | | | | | / / /
> > \ \ \ | | | | | / / /
> > /\ \______\______\'___|___|__________|_______'____/___/___/__ /\
> > || |+PF0s0|+PF0s1 |+VF0|+VF1| ...| |+PF1s0|+PF1s1|+VF0|+VF1| ||
> > i || |------ ------ ----- ---- ----|--- ------ ------ ---- ----| || i
> >d n H || | <<========== | || d n H
> >e s O || | ==========>> | || e s O
> >v t S || | SR-IOV e-switch | || v t S
> >l a T || | <<========== | || l a T
> >i n || | ==========>> | || i n
> >n c A || | ________ _________ ________ | || n c B
> >k e || | |+Phys 0 |+Phys 1 |+Phys 2 | | || k e
> > || \---------------------------------------------------------/ ||
> > \/ | | | \/
> > | | |
> > || ||
> > MAC 0 || MAC 1 || MAC 2
> > || ||
> >
> > '+' marks the devlink ports and port (-representor-) netdevs
> > '*' marks host netdevs (actual VF/PF netdev)
>
> As I wrote in the reply to patch 4, I think we need to figure out if we
> want to model all entities that belong under specific devlink
> instance/pci address - which I prefer, or we want to have only eswitch
> ports there.
>
> One way or another, I think that it is not good idea to merge this
> patchset this late, I would prefer to wait for next net-next opening...
> In the meantime we can sync and make this whole thing crystal clear, for
> everyone.
Makes sense, let's keep talking.
^ permalink raw reply [flat|nested] 100+ messages in thread
* RE: [PATCH net-next v2 0/7] devlink: expose PF and VF representors as ports
2019-03-02 10:13 ` [PATCH net-next v2 0/7] devlink: expose PF and VF representors as ports Jiri Pirko
2019-03-02 19:49 ` [oss-drivers] " Jakub Kicinski
@ 2019-03-04 5:12 ` Parav Pandit
1 sibling, 0 replies; 100+ messages in thread
From: Parav Pandit @ 2019-03-04 5:12 UTC (permalink / raw)
To: Jiri Pirko, Jakub Kicinski; +Cc: davem, netdev, oss-drivers
> -----Original Message-----
> From: netdev-owner@vger.kernel.org <netdev-owner@vger.kernel.org> On
> Behalf Of Jiri Pirko
> Sent: Saturday, March 2, 2019 4:14 AM
> To: Jakub Kicinski <jakub.kicinski@netronome.com>
> Cc: davem@davemloft.net; netdev@vger.kernel.org; oss-
> drivers@netronome.com
> Subject: Re: [PATCH net-next v2 0/7] devlink: expose PF and VF representors
> as ports
>
> Fri, Mar 01, 2019 at 07:04:46PM CET, jakub.kicinski@netronome.com wrote:
> >Hi!
> >
> >This series is a long overdue follow up to Jiri's work on providing a
> >common .ndo_phys_port_name implementation based on devlink ports.
> >
> >First devlink port flavours for PF and VF ports are added, and
> >registered by the NFP. Port numbers and split info are reserved for
> >physical and DSA ports. For PCIe ports we add pf/vf identifiers.
> >Note that devices may have more than one PF, including multi host
> >scenarios where not all pfs are connected to the same host.
> >
> >The port_index is slightly tricky to figure out, we use a bit of math
> >to create unique IDs for ports.
> >
> >Next subports for PCIe ports are introduced. This is in case device has
> >more than one netdev on a physical function (e.g. multi port SmartNIC).
> >
> >With the above features in place we can remove the ndo_phys_port_name
> >implementation from the NFP and use the standard devlink one for port
> >netdevs.
> >
> >Last but not least a concept of peer netdevs is added. In multi-host
> >scenarios its currently not possible to figure out which PF is
> >associated with the local host. Peer device is "the other side of the
> >wire" for PCIe ports. In case of NFP we only link the PF netdevs, but
> >it should be possible to also link VF peers after VF driver probes, if
> >users request it.
> >
> >This is the conceptual image of devlink instances:
> >
> > HOST A || HOST B
> > ||
> > PF A | V | V | V | V || PF B | V | V | V
> > |*F |*F |*F |*F ... || |*F |*F |*F
> >*port A0 |*port A1 | 0 | 1 | 2 | 3 ||*port B0 |*port B1 | 0 | 1 | 2
> > ||
> > PCI Express link || PCI Express link
> > \ \ \ | | | | | / / /
> > \ \ \ | | | | | / / /
> > /\ \______\______\'___|___|__________|_______'____/___/___/__
> /\
> > || |+PF0s0|+PF0s1 |+VF0|+VF1| ...| |+PF1s0|+PF1s1|+VF0|+VF1| ||
> > i || |------ ------ ----- ---- ----|--- ------ ------ ---- ----| || i
> >d n H || | <<========== | || d n H
> >e s O || | ==========>> | || e s O
> >v t S || | SR-IOV e-switch | || v t S
> >l a T || | <<========== | || l a T
> >i n || | ==========>> | || i n
> >n c A || | ________ _________ ________ | || n c B
> >k e || | |+Phys 0 |+Phys 1 |+Phys 2 | | || k e
> > || \---------------------------------------------------------/ ||
> > \/ | | | \/
> > | | |
> > || ||
> > MAC 0 || MAC 1 || MAC 2
> > || ||
> >
> > '+' marks the devlink ports and port (-representor-) netdevs '*' marks
> > host netdevs (actual VF/PF netdev)
>
> As I wrote in the reply to patch 4, I think we need to figure out if we want to
> model all entities that belong under specific devlink instance/pci address -
> which I prefer, or we want to have only eswitch ports there.
>
> One way or another, I think that it is not good idea to merge this patchset
> this late, I would prefer to wait for next net-next opening...
> In the meantime we can sync and make this whole thing crystal clear, for
> everyone.
>
Yes, please.
I replied in other thread, we should not bring peer port concept. It is convoluted.
We just need hostport and switchport representation (likely as port flavours) to configure each separately.
Whether a given devlink device is PF or VF is devlink devie attribute anyway.
^ permalink raw reply [flat|nested] 100+ messages in thread
* Re: [PATCH net-next v2 3/7] nfp: register devlink ports of all reprs
2019-03-02 19:07 ` Jakub Kicinski
@ 2019-03-04 7:36 ` Jiri Pirko
2019-03-04 23:32 ` Jakub Kicinski
0 siblings, 1 reply; 100+ messages in thread
From: Jiri Pirko @ 2019-03-04 7:36 UTC (permalink / raw)
To: Jakub Kicinski; +Cc: davem, netdev, oss-drivers
Sat, Mar 02, 2019 at 08:07:24PM CET, jakub.kicinski@netronome.com wrote:
>On Sat, 2 Mar 2019 09:43:47 +0100, Jiri Pirko wrote:
>> Fri, Mar 01, 2019 at 07:04:49PM CET, jakub.kicinski@netronome.com wrote:
>> >Register all representors as devlink ports.
>> >
>> >The port_index is slightly tricky to figure out, we use a bit of
>> >arbitrary math to create unique IDs for PCI ports.
>> >
>> >Signed-off-by: Jakub Kicinski <jakub.kicinski@netronome.com>
>> >---
>> > .../net/ethernet/netronome/nfp/nfp_devlink.c | 40 ++++++++++++++++++-
>> > .../net/ethernet/netronome/nfp/nfp_net_repr.c | 16 +++++++-
>> > 2 files changed, 53 insertions(+), 3 deletions(-)
>> >
>> >diff --git a/drivers/net/ethernet/netronome/nfp/nfp_devlink.c b/drivers/net/ethernet/netronome/nfp/nfp_devlink.c
>> >index 9af3cb1f2f17..bf7fd9614152 100644
>> >--- a/drivers/net/ethernet/netronome/nfp/nfp_devlink.c
>> >+++ b/drivers/net/ethernet/netronome/nfp/nfp_devlink.c
>> >@@ -350,7 +350,8 @@ const struct devlink_ops nfp_devlink_ops = {
>> > .flash_update = nfp_devlink_flash_update,
>> > };
>> >
>> >-int nfp_devlink_port_init(struct nfp_app *app, struct nfp_port *port)
>> >+static int
>> >+nfp_devlink_port_init_phys(struct devlink *devlink, struct nfp_port *port)
>> > {
>> > struct nfp_eth_table_port eth_port;
>> > int ret;
>> >@@ -368,6 +369,27 @@ int nfp_devlink_port_init(struct nfp_app *app, struct nfp_port *port)
>> > return 0;
>> > }
>> >
>> >+int nfp_devlink_port_init(struct nfp_app *app, struct nfp_port *port)
>> >+{
>> >+ struct devlink *devlink = priv_to_devlink(app->pf);
>> >+
>> >+ switch (port->type) {
>> >+ case NFP_PORT_PHYS_PORT:
>> >+ return nfp_devlink_port_init_phys(devlink, port);
>> >+ case NFP_PORT_PF_PORT:
>> >+ devlink_port_type_eth_set(&port->dl_port, port->netdev);
>> >+ devlink_port_attrs_pci_pf_set(&port->dl_port, port->pf_id);
>> >+ return 0;
>> >+ case NFP_PORT_VF_PORT:
>> >+ devlink_port_type_eth_set(&port->dl_port, port->netdev);
>> >+ devlink_port_attrs_pci_vf_set(&port->dl_port, port->pf_id,
>> >+ port->vf_id);
>>
>> What is the reason to expose vf/pf id for switch port? Isn't it rather
>> an attribute of a peer?
>
>Naw, its an attribute of the port. I leave the ASIC via PF n or VF m
>of PF n. Whatever is on the other side is isolated from the topology
>of the ASIC.
Ok.
>
>Is the physical port ID an attribute of the other end of the cable?
>
>> >+ return 0;
>> >+ default:
>> >+ return -EINVAL;
>> >+ }
>> >+}
>> >+
>> > void nfp_devlink_port_clean(struct nfp_port *port)
>> > {
>> > }
>> >@@ -376,7 +398,21 @@ int nfp_devlink_port_register(struct nfp_app *app, struct nfp_port *port)
>> > {
>> > struct devlink *devlink = priv_to_devlink(app->pf);
>> >
>> >- return devlink_port_register(devlink, &port->dl_port, port->eth_id);
>> >+ switch (port->type) {
>> >+ case NFP_PORT_PHYS_PORT:
>> >+ return devlink_port_register(devlink, &port->dl_port,
>> >+ port->eth_id);
>> >+ case NFP_PORT_PF_PORT:
>> >+ return devlink_port_register(devlink, &port->dl_port,
>> >+ (port->pf_id + 1) * 10000 +
>> >+ port->pf_split_id * 1000);
>>
>> Wait. What this 10000/1000 magic about?
>
>port_index has to be unique, I need some unique number here, as I
>stated both in the commit message and the cover letter, this is
>arbitrary.
You can at least use some defines for that.
>
>I can put the datapath port identifier in there but its (a)
>meaningless, (b) a bitfield, so it will look like 8972367083. And it
>may change depending on the FW load, so its not stable either.
>
>> >+ case NFP_PORT_VF_PORT:
>> >+ return devlink_port_register(devlink, &port->dl_port,
>> >+ (port->pf_id + 1) * 10000 +
>> >+ port->vf_id + 1);
>> >+ default:
>> >+ return -EINVAL;
>> >+ }
>> > }
>> >
>> > void nfp_devlink_port_unregister(struct nfp_port *port)
>> >diff --git a/drivers/net/ethernet/netronome/nfp/nfp_net_repr.c b/drivers/net/ethernet/netronome/nfp/nfp_net_repr.c
>> >index d2c803bb4e56..869d22760a6e 100644
>> >--- a/drivers/net/ethernet/netronome/nfp/nfp_net_repr.c
>> >+++ b/drivers/net/ethernet/netronome/nfp/nfp_net_repr.c
>> >@@ -292,7 +292,9 @@ nfp_repr_transfer_features(struct net_device *netdev, struct net_device *lower)
>> >
>> > static void nfp_repr_clean(struct nfp_repr *repr)
>> > {
>> >+ nfp_devlink_port_unregister(repr->port);
>> > unregister_netdev(repr->netdev);
>> >+ nfp_devlink_port_clean(repr->port);
>> > nfp_app_repr_clean(repr->app, repr->netdev);
>> > dst_release((struct dst_entry *)repr->dst);
>> > nfp_port_free(repr->port);
>> >@@ -395,12 +397,24 @@ int nfp_repr_init(struct nfp_app *app, struct net_device *netdev,
>> > if (err)
>> > goto err_clean;
>> >
>> >- err = register_netdev(netdev);
>> >+ err = nfp_devlink_port_init(app, repr->port);
>> > if (err)
>> > goto err_repr_clean;
>> >
>> >+ err = register_netdev(netdev);
>> >+ if (err)
>> >+ goto err_port_clean;
>> >+
>> >+ err = nfp_devlink_port_register(app, repr->port);
>>
>> Don't you want to take my patch ("nfp: register devlink port before
>> netdev") to change order of register_netdev and devlink_port_register,
>> include it to this patchset before this patch and change the order in
>> this patch too? I think it would be clearer to do it from the beginning.
>
>This way both netdev and devlink_port can get registered fully
>initialized. Otherwise we'd get two notifications. Are we trying to
>establish some ordering rules to get around the rtnl locking? :)
The order of devlink_port_register and register_netdev is given by
layering. For example, for port change, the devlink_port is still there
and registered, only the netdev is unregistered and ib_dev registered
instead of vice versa. It has really no relation to rtnl locking.
>
>> >+ if (err)
>> >+ goto err_unreg_netdev;
>> >+
>> > return 0;
>> >
>> >+err_unreg_netdev:
>> >+ unregister_netdev(repr->netdev);
>> >+err_port_clean:
>> >+ nfp_devlink_port_clean(repr->port);
>> > err_repr_clean:
>> > nfp_app_repr_clean(app, netdev);
>> > err_clean:
^ permalink raw reply [flat|nested] 100+ messages in thread
* Re: [PATCH net-next v2 4/7] devlink: allow subports on devlink PCI ports
2019-03-02 19:48 ` Jakub Kicinski
@ 2019-03-04 7:56 ` Jiri Pirko
2019-03-05 0:33 ` Jakub Kicinski
2019-03-04 11:19 ` Jiri Pirko
1 sibling, 1 reply; 100+ messages in thread
From: Jiri Pirko @ 2019-03-04 7:56 UTC (permalink / raw)
To: Jakub Kicinski; +Cc: davem, netdev, oss-drivers
Sat, Mar 02, 2019 at 08:48:47PM CET, jakub.kicinski@netronome.com wrote:
>On Sat, 2 Mar 2019 10:41:16 +0100, Jiri Pirko wrote:
>> Fri, Mar 01, 2019 at 07:04:50PM CET, jakub.kicinski@netronome.com wrote:
>> >PCI endpoint corresponds to a PCI device, but such device
>> >can have one more more logical device ports associated with it.
>> >We need a way to distinguish those. Add a PCI subport in the
>> >dumps and print the info in phys_port_name appropriately.
>> >
>> >This is not equivalent to port splitting, there is no split
>> >group. It's just a way of representing multiple netdevs on
>> >a single PCI function.
>> >
>> >Note that the quality of being multiport pertains only to
>> >the PCI function itself. A PF having multiple netdevs does
>> >not mean that its VFs will also have multiple, or that VFs
>> >are associated with any particular port of a multiport VF.
>> >
>> >Example (bus 05 device has subports, bus 82 has only one port per
>> >function):
>> >
>> >$ devlink port
>> >pci/0000:05:00.0/0: type eth netdev enp5s0np0 flavour physical
>> >pci/0000:05:00.0/10000: type eth netdev enp5s0npf0s0 flavour pci_pf pf 0 subport 0
>> >pci/0000:05:00.0/4: type eth netdev enp5s0np1 flavour physical
>> >pci/0000:05:00.0/11000: type eth netdev enp5s0npf0s1 flavour pci_pf pf 0 subport 1
>>
>> So these subport devlink ports are eswitch ports for subports, right?
>>
>> Please see the following drawing:
>>
>> +---+ +---+ +---+
>> pfsub| 5 | vf| 6 | | 7 |pfsub
>> +-+-+ +-+-+ +-+-+
>> physical link <---------+ | | |
>> | | | |
>> | | | |
>> | | | |
>> +-+-+ +-+-+ +-+-+ +-+-+
>> | 1 | | 2 | | 3 | | 4 |
>> +--+---+------+---+------+---+------+---+--+
>> | physical pfsub vf pfsub |
>> | port port port port |
>> | |
>> | eswitch |
>> | |
>> | |
>> +------------------------------------------+
>>
>> 1) pci/0000:05:00.0/0: type eth netdev enp5s0np0 flavour physical switch_id 00154d130d2f
>> 2) pci/0000:05:00.0/10000: type eth netdev enp5s0npf0s0 flavour pci_pf pf 0 subport 0 switch_id 00154d130d2f
>> 3) pci/0000:05:00.0/10001: type eth netdev enp5s0npf0vf0 flavour pci_vf pf 0 vf 0 switch_id 00154d130d2f
>> 4) pci/0000:05:00.0/10001: type eth netdev enp5s0npf0s1 flavour pci_pf pf 0 subport 1 switch_id 00154d130d2f
>>
>> This is basically what you have and I think we are in sync with that.
>> But what about 5,6,7? Should they have devlink port instances too?
>>
>> 5) pci/0000:05:00.0/1: type eth netdev enp5s0f0?? flavour ???? pf 0 subport 0
>> 6) pci/0000:05:10.1/0: type eth netdev enp5s10f0 flavour ???? pf 0 vf 0
>> 7) pci/0000:05:00.0/1: type eth netdev enp5s0f0?? flavour ???? pf 0 subport 1
>>
>> These are the "peers".
>> I think that there could be flavours "pci_pf" and "pci_vf". Then the
>> "representors" (switch ports) could have flavours "pci_pf_port" and
>> "pci_vf_port" or something like that. User can see right away
>> that is not "PF" of "VF" but rather something "on the other end".
>> Note there is no "switch_id" for these devlink ports that tells the user
>> these devlink ports are not part of any switch.
>> What do you think?
>
>Hmmm.. Hm. Hm.
>
>To me its neat if the devlink instance matches an ASIC. I think it's
>kind of clear for people to understand what it stands for then. So if
>we wanted to do the above we'd have to make the switch_id the first
>class identifier for devlink instances, rather than the bus? But then
What do you mean by "first class identifier"? Like "a handle"?
>VF instances don't have a switch ID so that doesn't work...
Wait a sec. VF-ports do have. VFs them selves don't. But that is the
same for PF. PF would also not have switch id.
>
>I need to think about it.
>
>It's also kind of strange that we have to add the noun *port* to the
>flavour of... a port... So I would prefer not to have those showing up
Yeah.
>as ports. Can we invent a new command (say "partition"?) that'd take
>the bus info where the partition is to be spawned?
Got it. But the question is how different this object would be from the
existing "port" we have today.
>
>My next goal is to find a way of grouping multiple bus devices under one
>"ASIC" (which is a devlink instance to me) so it can be understood
>easily how things are laid out when there is more than one PF connected
>to one host.
These are the "aliases" you mentioned before right? Makes sense.
^ permalink raw reply [flat|nested] 100+ messages in thread
* Re: [PATCH net-next v2 4/7] devlink: allow subports on devlink PCI ports
2019-03-01 18:04 ` [PATCH net-next v2 4/7] devlink: allow subports on devlink PCI ports Jakub Kicinski
2019-03-02 9:41 ` Jiri Pirko
@ 2019-03-04 11:08 ` Jiri Pirko
2019-03-05 0:51 ` Jakub Kicinski
1 sibling, 1 reply; 100+ messages in thread
From: Jiri Pirko @ 2019-03-04 11:08 UTC (permalink / raw)
To: Jakub Kicinski; +Cc: davem, netdev, oss-drivers
Fri, Mar 01, 2019 at 07:04:50PM CET, jakub.kicinski@netronome.com wrote:
>PCI endpoint corresponds to a PCI device, but such device
>can have one more more logical device ports associated with it.
>We need a way to distinguish those. Add a PCI subport in the
>dumps and print the info in phys_port_name appropriately.
>
>This is not equivalent to port splitting, there is no split
>group. It's just a way of representing multiple netdevs on
>a single PCI function.
>
>Note that the quality of being multiport pertains only to
>the PCI function itself. A PF having multiple netdevs does
>not mean that its VFs will also have multiple, or that VFs
>are associated with any particular port of a multiport VF.
>
>Example (bus 05 device has subports, bus 82 has only one port per
>function):
How do you plan to added/remove these subports?
^ permalink raw reply [flat|nested] 100+ messages in thread
* Re: [PATCH net-next v2 4/7] devlink: allow subports on devlink PCI ports
2019-03-02 19:48 ` Jakub Kicinski
2019-03-04 7:56 ` Jiri Pirko
@ 2019-03-04 11:19 ` Jiri Pirko
2019-03-05 0:40 ` Jakub Kicinski
1 sibling, 1 reply; 100+ messages in thread
From: Jiri Pirko @ 2019-03-04 11:19 UTC (permalink / raw)
To: Jakub Kicinski; +Cc: davem, netdev, oss-drivers
Sat, Mar 02, 2019 at 08:48:47PM CET, jakub.kicinski@netronome.com wrote:
>On Sat, 2 Mar 2019 10:41:16 +0100, Jiri Pirko wrote:
>> Fri, Mar 01, 2019 at 07:04:50PM CET, jakub.kicinski@netronome.com wrote:
>> >PCI endpoint corresponds to a PCI device, but such device
>> >can have one more more logical device ports associated with it.
>> >We need a way to distinguish those. Add a PCI subport in the
>> >dumps and print the info in phys_port_name appropriately.
>> >
>> >This is not equivalent to port splitting, there is no split
>> >group. It's just a way of representing multiple netdevs on
>> >a single PCI function.
>> >
>> >Note that the quality of being multiport pertains only to
>> >the PCI function itself. A PF having multiple netdevs does
>> >not mean that its VFs will also have multiple, or that VFs
>> >are associated with any particular port of a multiport VF.
>> >
>> >Example (bus 05 device has subports, bus 82 has only one port per
>> >function):
>> >
>> >$ devlink port
>> >pci/0000:05:00.0/0: type eth netdev enp5s0np0 flavour physical
>> >pci/0000:05:00.0/10000: type eth netdev enp5s0npf0s0 flavour pci_pf pf 0 subport 0
>> >pci/0000:05:00.0/4: type eth netdev enp5s0np1 flavour physical
>> >pci/0000:05:00.0/11000: type eth netdev enp5s0npf0s1 flavour pci_pf pf 0 subport 1
>>
>> So these subport devlink ports are eswitch ports for subports, right?
>>
>> Please see the following drawing:
>>
>> +---+ +---+ +---+
>> pfsub| 5 | vf| 6 | | 7 |pfsub
>> +-+-+ +-+-+ +-+-+
>> physical link <---------+ | | |
>> | | | |
>> | | | |
>> | | | |
>> +-+-+ +-+-+ +-+-+ +-+-+
>> | 1 | | 2 | | 3 | | 4 |
>> +--+---+------+---+------+---+------+---+--+
>> | physical pfsub vf pfsub |
>> | port port port port |
>> | |
>> | eswitch |
>> | |
>> | |
>> +------------------------------------------+
>>
>> 1) pci/0000:05:00.0/0: type eth netdev enp5s0np0 flavour physical switch_id 00154d130d2f
>> 2) pci/0000:05:00.0/10000: type eth netdev enp5s0npf0s0 flavour pci_pf pf 0 subport 0 switch_id 00154d130d2f
>> 3) pci/0000:05:00.0/10001: type eth netdev enp5s0npf0vf0 flavour pci_vf pf 0 vf 0 switch_id 00154d130d2f
>> 4) pci/0000:05:00.0/10001: type eth netdev enp5s0npf0s1 flavour pci_pf pf 0 subport 1 switch_id 00154d130d2f
>>
>> This is basically what you have and I think we are in sync with that.
>> But what about 5,6,7? Should they have devlink port instances too?
>>
>> 5) pci/0000:05:00.0/1: type eth netdev enp5s0f0?? flavour ???? pf 0 subport 0
>> 6) pci/0000:05:10.1/0: type eth netdev enp5s10f0 flavour ???? pf 0 vf 0
>> 7) pci/0000:05:00.0/1: type eth netdev enp5s0f0?? flavour ???? pf 0 subport 1
>>
>> These are the "peers".
>> I think that there could be flavours "pci_pf" and "pci_vf". Then the
>> "representors" (switch ports) could have flavours "pci_pf_port" and
>> "pci_vf_port" or something like that. User can see right away
>> that is not "PF" of "VF" but rather something "on the other end".
>> Note there is no "switch_id" for these devlink ports that tells the user
>> these devlink ports are not part of any switch.
>> What do you think?
>
>Hmmm.. Hm. Hm.
>
>To me its neat if the devlink instance matches an ASIC. I think it's
>kind of clear for people to understand what it stands for then. So if
>we wanted to do the above we'd have to make the switch_id the first
>class identifier for devlink instances, rather than the bus? But then
>VF instances don't have a switch ID so that doesn't work...
>
>I need to think about it.
>
>It's also kind of strange that we have to add the noun *port* to the
>flavour of... a port... So I would prefer not to have those showing up
>as ports. Can we invent a new command (say "partition"?) that'd take
>the bus info where the partition is to be spawned?
Devlink does not supposed to be only there for switches. From the
beginning the design was to handle cases where the netdev/ib_dev is not
the correct handle. Not only in case you have multiple instances (ports)
for one ASIC, but also in case you have only one. Example use case is
port-type-change (eth->ib,ib->eth).
I chose word "port" as the parent devlink instance is "dev" and if you
partition the ASIC you basically got "ports", each of a different flavour.
And as you said, devlink instance matches one ASIC. Therefore the
devlink instance should contain all bits there are part of that ASIC,
not only switch/eswitch ports. That would be very limitting.
^ permalink raw reply [flat|nested] 100+ messages in thread
* Re: [PATCH net-next v2 0/7] devlink: expose PF and VF representors as ports
2019-03-01 18:04 [PATCH net-next v2 0/7] devlink: expose PF and VF representors as ports Jakub Kicinski
` (7 preceding siblings ...)
2019-03-02 10:13 ` [PATCH net-next v2 0/7] devlink: expose PF and VF representors as ports Jiri Pirko
@ 2019-03-04 18:22 ` David Miller
2019-03-20 20:25 ` Jakub Kicinski
9 siblings, 0 replies; 100+ messages in thread
From: David Miller @ 2019-03-04 18:22 UTC (permalink / raw)
To: jakub.kicinski; +Cc: jiri, netdev, oss-drivers
From: Jakub Kicinski <jakub.kicinski@netronome.com>
Date: Fri, 1 Mar 2019 10:04:46 -0800
> This series is a long overdue follow up to Jiri's work on providing
> a common .ndo_phys_port_name implementation based on devlink ports.
I think this needs enough discussion still that it will have to wait
until the next merge window.
^ permalink raw reply [flat|nested] 100+ messages in thread
* Re: [PATCH net-next v2 3/7] nfp: register devlink ports of all reprs
2019-03-04 7:36 ` Jiri Pirko
@ 2019-03-04 23:32 ` Jakub Kicinski
0 siblings, 0 replies; 100+ messages in thread
From: Jakub Kicinski @ 2019-03-04 23:32 UTC (permalink / raw)
To: Jiri Pirko; +Cc: davem, netdev, oss-drivers
On Mon, 4 Mar 2019 08:36:31 +0100, Jiri Pirko wrote:
> >> >+ case NFP_PORT_PF_PORT:
> >> >+ return devlink_port_register(devlink, &port->dl_port,
> >> >+ (port->pf_id + 1) * 10000 +
> >> >+ port->pf_split_id * 1000);
> >>
> >> Wait. What this 10000/1000 magic about?
> >
> >port_index has to be unique, I need some unique number here, as I
> >stated both in the commit message and the cover letter, this is
> >arbitrary.
>
> You can at least use some defines for that.
Ok.
> >I can put the datapath port identifier in there but its (a)
> >meaningless, (b) a bitfield, so it will look like 8972367083. And it
> >may change depending on the FW load, so its not stable either.
> >> > void nfp_devlink_port_unregister(struct nfp_port *port)
> >> >diff --git a/drivers/net/ethernet/netronome/nfp/nfp_net_repr.c b/drivers/net/ethernet/netronome/nfp/nfp_net_repr.c
> >> >index d2c803bb4e56..869d22760a6e 100644
> >> >--- a/drivers/net/ethernet/netronome/nfp/nfp_net_repr.c
> >> >+++ b/drivers/net/ethernet/netronome/nfp/nfp_net_repr.c
> >> >@@ -395,12 +397,24 @@ int nfp_repr_init(struct nfp_app *app, struct net_device *netdev,
> >> > if (err)
> >> > goto err_clean;
> >> >
> >> >- err = register_netdev(netdev);
> >> >+ err = nfp_devlink_port_init(app, repr->port);
> >> > if (err)
> >> > goto err_repr_clean;
> >> >
> >> >+ err = register_netdev(netdev);
> >> >+ if (err)
> >> >+ goto err_port_clean;
> >> >+
> >> >+ err = nfp_devlink_port_register(app, repr->port);
> >>
> >> Don't you want to take my patch ("nfp: register devlink port before
> >> netdev") to change order of register_netdev and devlink_port_register,
> >> include it to this patchset before this patch and change the order in
> >> this patch too? I think it would be clearer to do it from the beginning.
> >
> >This way both netdev and devlink_port can get registered fully
> >initialized. Otherwise we'd get two notifications. Are we trying to
> >establish some ordering rules to get around the rtnl locking? :)
>
> The order of devlink_port_register and register_netdev is given by
> layering. For example, for port change, the devlink_port is still there
> and registered, only the netdev is unregistered and ib_dev registered
> instead of vice versa. It has really no relation to rtnl locking.
Ok, I shouldn't worry about the notifications too much, I agree the
order you suggests makes sense in principal.
^ permalink raw reply [flat|nested] 100+ messages in thread
* Re: [PATCH net-next v2 4/7] devlink: allow subports on devlink PCI ports
2019-03-04 7:56 ` Jiri Pirko
@ 2019-03-05 0:33 ` Jakub Kicinski
2019-03-05 11:06 ` Jiri Pirko
0 siblings, 1 reply; 100+ messages in thread
From: Jakub Kicinski @ 2019-03-05 0:33 UTC (permalink / raw)
To: Jiri Pirko; +Cc: davem, netdev, oss-drivers
On Mon, 4 Mar 2019 08:56:09 +0100, Jiri Pirko wrote:
> Sat, Mar 02, 2019 at 08:48:47PM CET, jakub.kicinski@netronome.com wrote:
> >On Sat, 2 Mar 2019 10:41:16 +0100, Jiri Pirko wrote:
> >> Fri, Mar 01, 2019 at 07:04:50PM CET, jakub.kicinski@netronome.com wrote:
> >> >PCI endpoint corresponds to a PCI device, but such device
> >> >can have one more more logical device ports associated with it.
> >> >We need a way to distinguish those. Add a PCI subport in the
> >> >dumps and print the info in phys_port_name appropriately.
> >> >
> >> >This is not equivalent to port splitting, there is no split
> >> >group. It's just a way of representing multiple netdevs on
> >> >a single PCI function.
> >> >
> >> >Note that the quality of being multiport pertains only to
> >> >the PCI function itself. A PF having multiple netdevs does
> >> >not mean that its VFs will also have multiple, or that VFs
> >> >are associated with any particular port of a multiport VF.
> >> >
> >> >Example (bus 05 device has subports, bus 82 has only one port per
> >> >function):
> >> >
> >> >$ devlink port
> >> >pci/0000:05:00.0/0: type eth netdev enp5s0np0 flavour physical
> >> >pci/0000:05:00.0/10000: type eth netdev enp5s0npf0s0 flavour pci_pf pf 0 subport 0
> >> >pci/0000:05:00.0/4: type eth netdev enp5s0np1 flavour physical
> >> >pci/0000:05:00.0/11000: type eth netdev enp5s0npf0s1 flavour pci_pf pf 0 subport 1
> >>
> >> So these subport devlink ports are eswitch ports for subports, right?
> >>
> >> Please see the following drawing:
> >>
> >> +---+ +---+ +---+
> >> pfsub| 5 | vf| 6 | | 7 |pfsub
> >> +-+-+ +-+-+ +-+-+
> >> physical link <---------+ | | |
> >> | | | |
> >> | | | |
> >> | | | |
> >> +-+-+ +-+-+ +-+-+ +-+-+
> >> | 1 | | 2 | | 3 | | 4 |
> >> +--+---+------+---+------+---+------+---+--+
> >> | physical pfsub vf pfsub |
> >> | port port port port |
> >> | |
> >> | eswitch |
> >> | |
> >> | |
> >> +------------------------------------------+
> >>
> >> 1) pci/0000:05:00.0/0: type eth netdev enp5s0np0 flavour physical switch_id 00154d130d2f
> >> 2) pci/0000:05:00.0/10000: type eth netdev enp5s0npf0s0 flavour pci_pf pf 0 subport 0 switch_id 00154d130d2f
> >> 3) pci/0000:05:00.0/10001: type eth netdev enp5s0npf0vf0 flavour pci_vf pf 0 vf 0 switch_id 00154d130d2f
> >> 4) pci/0000:05:00.0/10001: type eth netdev enp5s0npf0s1 flavour pci_pf pf 0 subport 1 switch_id 00154d130d2f
> >>
> >> This is basically what you have and I think we are in sync with that.
> >> But what about 5,6,7? Should they have devlink port instances too?
> >>
> >> 5) pci/0000:05:00.0/1: type eth netdev enp5s0f0?? flavour ???? pf 0 subport 0
> >> 6) pci/0000:05:10.1/0: type eth netdev enp5s10f0 flavour ???? pf 0 vf 0
> >> 7) pci/0000:05:00.0/1: type eth netdev enp5s0f0?? flavour ???? pf 0 subport 1
> >>
> >> These are the "peers".
> >> I think that there could be flavours "pci_pf" and "pci_vf". Then the
> >> "representors" (switch ports) could have flavours "pci_pf_port" and
> >> "pci_vf_port" or something like that. User can see right away
> >> that is not "PF" of "VF" but rather something "on the other end".
> >> Note there is no "switch_id" for these devlink ports that tells the user
> >> these devlink ports are not part of any switch.
> >> What do you think?
> >
> >Hmmm.. Hm. Hm.
> >
> >To me its neat if the devlink instance matches an ASIC. I think it's
> >kind of clear for people to understand what it stands for then. So if
> >we wanted to do the above we'd have to make the switch_id the first
> >class identifier for devlink instances, rather than the bus? But then
>
> What do you mean by "first class identifier"? Like "a handle"?
Yes, a handle.
> >VF instances don't have a switch ID so that doesn't work...
>
> Wait a sec. VF-ports do have. VFs them selves don't.
Looking at your example this one:
6) pci/0000:05:10.1/0: type eth netdev enp5s10f0 flavour ???? pf 0 vf 0
that uses VF's DBDF in the devlink instance handle, so I presume this
is a VF's devlink instance that will get passed to the VM together with
the VF device?
> But that is the same for PF. PF would also not have switch id.
Yes :( You'd have to mark what constitutes a devlink instance on your
drawing. The semantics for devlink instances seem to be the focal point
of the discussion.
Right now it seems a little bit that folks on the NIC side see a devlink
instance as a PCI function and on switch side it's the whole ASIC.
> >I need to think about it.
> >
> >It's also kind of strange that we have to add the noun *port* to the
> >flavour of... a port... So I would prefer not to have those showing up
>
> Yeah.
>
> >as ports. Can we invent a new command (say "partition"?) that'd take
> >the bus info where the partition is to be spawned?
>
> Got it. But the question is how different this object would be from the
> existing "port" we have today.
They'd be where "the other side of a PCI link" is represented,
restricting ports to only ASIC's forwarding plane ports.
> >My next goal is to find a way of grouping multiple bus devices under one
> >"ASIC" (which is a devlink instance to me) so it can be understood
> >easily how things are laid out when there is more than one PF connected
> >to one host.
>
> These are the "aliases" you mentioned before right? Makes sense.
Yes.
^ permalink raw reply [flat|nested] 100+ messages in thread
* Re: [PATCH net-next v2 4/7] devlink: allow subports on devlink PCI ports
2019-03-04 11:19 ` Jiri Pirko
@ 2019-03-05 0:40 ` Jakub Kicinski
2019-03-05 11:07 ` Jiri Pirko
0 siblings, 1 reply; 100+ messages in thread
From: Jakub Kicinski @ 2019-03-05 0:40 UTC (permalink / raw)
To: Jiri Pirko; +Cc: davem, netdev, oss-drivers
On Mon, 4 Mar 2019 12:19:02 +0100, Jiri Pirko wrote:
> Sat, Mar 02, 2019 at 08:48:47PM CET, jakub.kicinski@netronome.com wrote:
> >On Sat, 2 Mar 2019 10:41:16 +0100, Jiri Pirko wrote:
> >> Fri, Mar 01, 2019 at 07:04:50PM CET, jakub.kicinski@netronome.com wrote:
> >> >PCI endpoint corresponds to a PCI device, but such device
> >> >can have one more more logical device ports associated with it.
> >> >We need a way to distinguish those. Add a PCI subport in the
> >> >dumps and print the info in phys_port_name appropriately.
> >> >
> >> >This is not equivalent to port splitting, there is no split
> >> >group. It's just a way of representing multiple netdevs on
> >> >a single PCI function.
> >> >
> >> >Note that the quality of being multiport pertains only to
> >> >the PCI function itself. A PF having multiple netdevs does
> >> >not mean that its VFs will also have multiple, or that VFs
> >> >are associated with any particular port of a multiport VF.
> >> >
> >> >Example (bus 05 device has subports, bus 82 has only one port per
> >> >function):
> >> >
> >> >$ devlink port
> >> >pci/0000:05:00.0/0: type eth netdev enp5s0np0 flavour physical
> >> >pci/0000:05:00.0/10000: type eth netdev enp5s0npf0s0 flavour pci_pf pf 0 subport 0
> >> >pci/0000:05:00.0/4: type eth netdev enp5s0np1 flavour physical
> >> >pci/0000:05:00.0/11000: type eth netdev enp5s0npf0s1 flavour pci_pf pf 0 subport 1
> >>
> >> So these subport devlink ports are eswitch ports for subports, right?
> >>
> >> Please see the following drawing:
> >>
> >> +---+ +---+ +---+
> >> pfsub| 5 | vf| 6 | | 7 |pfsub
> >> +-+-+ +-+-+ +-+-+
> >> physical link <---------+ | | |
> >> | | | |
> >> | | | |
> >> | | | |
> >> +-+-+ +-+-+ +-+-+ +-+-+
> >> | 1 | | 2 | | 3 | | 4 |
> >> +--+---+------+---+------+---+------+---+--+
> >> | physical pfsub vf pfsub |
> >> | port port port port |
> >> | |
> >> | eswitch |
> >> | |
> >> | |
> >> +------------------------------------------+
> >>
> >> 1) pci/0000:05:00.0/0: type eth netdev enp5s0np0 flavour physical switch_id 00154d130d2f
> >> 2) pci/0000:05:00.0/10000: type eth netdev enp5s0npf0s0 flavour pci_pf pf 0 subport 0 switch_id 00154d130d2f
> >> 3) pci/0000:05:00.0/10001: type eth netdev enp5s0npf0vf0 flavour pci_vf pf 0 vf 0 switch_id 00154d130d2f
> >> 4) pci/0000:05:00.0/10001: type eth netdev enp5s0npf0s1 flavour pci_pf pf 0 subport 1 switch_id 00154d130d2f
> >>
> >> This is basically what you have and I think we are in sync with that.
> >> But what about 5,6,7? Should they have devlink port instances too?
> >>
> >> 5) pci/0000:05:00.0/1: type eth netdev enp5s0f0?? flavour ???? pf 0 subport 0
> >> 6) pci/0000:05:10.1/0: type eth netdev enp5s10f0 flavour ???? pf 0 vf 0
> >> 7) pci/0000:05:00.0/1: type eth netdev enp5s0f0?? flavour ???? pf 0 subport 1
> >>
> >> These are the "peers".
> >> I think that there could be flavours "pci_pf" and "pci_vf". Then the
> >> "representors" (switch ports) could have flavours "pci_pf_port" and
> >> "pci_vf_port" or something like that. User can see right away
> >> that is not "PF" of "VF" but rather something "on the other end".
> >> Note there is no "switch_id" for these devlink ports that tells the user
> >> these devlink ports are not part of any switch.
> >> What do you think?
> >
> >Hmmm.. Hm. Hm.
> >
> >To me its neat if the devlink instance matches an ASIC. I think it's
> >kind of clear for people to understand what it stands for then. So if
> >we wanted to do the above we'd have to make the switch_id the first
> >class identifier for devlink instances, rather than the bus? But then
> >VF instances don't have a switch ID so that doesn't work...
> >
> >I need to think about it.
> >
> >It's also kind of strange that we have to add the noun *port* to the
> >flavour of... a port... So I would prefer not to have those showing up
> >as ports. Can we invent a new command (say "partition"?) that'd take
> >the bus info where the partition is to be spawned?
>
> Devlink does not supposed to be only there for switches. From the
> beginning the design was to handle cases where the netdev/ib_dev is not
> the correct handle. Not only in case you have multiple instances (ports)
> for one ASIC, but also in case you have only one. Example use case is
> port-type-change (eth->ib,ib->eth).
>
> I chose word "port" as the parent devlink instance is "dev" and if you
> partition the ASIC you basically got "ports", each of a different flavour.
>
> And as you said, devlink instance matches one ASIC. Therefore the
> devlink instance should contain all bits there are part of that ASIC,
> not only switch/eswitch ports. That would be very limitting.
I could read this as us being in full agreement, but I'm not sure..
I think we agree that all objects of an ASIC should be under one
devlink instance, the question remains whether both ends of the pipe
for PCI devices (subdevs or not) should appear under ports or does the
"far end" (from ASICs perspective)/"host end" get its own category.
^ permalink raw reply [flat|nested] 100+ messages in thread
* Re: [PATCH net-next v2 4/7] devlink: allow subports on devlink PCI ports
2019-03-04 11:08 ` Jiri Pirko
@ 2019-03-05 0:51 ` Jakub Kicinski
2019-03-05 11:09 ` Jiri Pirko
0 siblings, 1 reply; 100+ messages in thread
From: Jakub Kicinski @ 2019-03-05 0:51 UTC (permalink / raw)
To: Jiri Pirko; +Cc: davem, netdev, oss-drivers
On Mon, 4 Mar 2019 12:08:57 +0100, Jiri Pirko wrote:
> Fri, Mar 01, 2019 at 07:04:50PM CET, jakub.kicinski@netronome.com wrote:
> >PCI endpoint corresponds to a PCI device, but such device
> >can have one more more logical device ports associated with it.
> >We need a way to distinguish those. Add a PCI subport in the
> >dumps and print the info in phys_port_name appropriately.
> >
> >This is not equivalent to port splitting, there is no split
> >group. It's just a way of representing multiple netdevs on
> >a single PCI function.
> >
> >Note that the quality of being multiport pertains only to
> >the PCI function itself. A PF having multiple netdevs does
> >not mean that its VFs will also have multiple, or that VFs
> >are associated with any particular port of a multiport VF.
> >
> >Example (bus 05 device has subports, bus 82 has only one port per
> >function):
>
> How do you plan to added/remove these subports?
I can't say I got that figured out fully, but I was wondering if we can
have some form of:
$ devlink partition pci/0000:82:00.0 new
pci/0000:82:00.0/1001002
Which would create appropriate sub-port and port (-repr-) netdev.
Plus optionally the ability to work with something like the already
existing mdev infrastructure for passing to a VM. But I haven't even
looked at that, yet.
^ permalink raw reply [flat|nested] 100+ messages in thread
* Re: [PATCH net-next v2 4/7] devlink: allow subports on devlink PCI ports
2019-03-05 0:33 ` Jakub Kicinski
@ 2019-03-05 11:06 ` Jiri Pirko
2019-03-05 17:15 ` Jakub Kicinski
0 siblings, 1 reply; 100+ messages in thread
From: Jiri Pirko @ 2019-03-05 11:06 UTC (permalink / raw)
To: Jakub Kicinski; +Cc: davem, netdev, oss-drivers
Tue, Mar 05, 2019 at 01:33:02AM CET, jakub.kicinski@netronome.com wrote:
>On Mon, 4 Mar 2019 08:56:09 +0100, Jiri Pirko wrote:
>> Sat, Mar 02, 2019 at 08:48:47PM CET, jakub.kicinski@netronome.com wrote:
>> >On Sat, 2 Mar 2019 10:41:16 +0100, Jiri Pirko wrote:
>> >> Fri, Mar 01, 2019 at 07:04:50PM CET, jakub.kicinski@netronome.com wrote:
>> >> >PCI endpoint corresponds to a PCI device, but such device
>> >> >can have one more more logical device ports associated with it.
>> >> >We need a way to distinguish those. Add a PCI subport in the
>> >> >dumps and print the info in phys_port_name appropriately.
>> >> >
>> >> >This is not equivalent to port splitting, there is no split
>> >> >group. It's just a way of representing multiple netdevs on
>> >> >a single PCI function.
>> >> >
>> >> >Note that the quality of being multiport pertains only to
>> >> >the PCI function itself. A PF having multiple netdevs does
>> >> >not mean that its VFs will also have multiple, or that VFs
>> >> >are associated with any particular port of a multiport VF.
>> >> >
>> >> >Example (bus 05 device has subports, bus 82 has only one port per
>> >> >function):
>> >> >
>> >> >$ devlink port
>> >> >pci/0000:05:00.0/0: type eth netdev enp5s0np0 flavour physical
>> >> >pci/0000:05:00.0/10000: type eth netdev enp5s0npf0s0 flavour pci_pf pf 0 subport 0
>> >> >pci/0000:05:00.0/4: type eth netdev enp5s0np1 flavour physical
>> >> >pci/0000:05:00.0/11000: type eth netdev enp5s0npf0s1 flavour pci_pf pf 0 subport 1
>> >>
>> >> So these subport devlink ports are eswitch ports for subports, right?
>> >>
>> >> Please see the following drawing:
>> >>
>> >> +---+ +---+ +---+
>> >> pfsub| 5 | vf| 6 | | 7 |pfsub
>> >> +-+-+ +-+-+ +-+-+
>> >> physical link <---------+ | | |
>> >> | | | |
>> >> | | | |
>> >> | | | |
>> >> +-+-+ +-+-+ +-+-+ +-+-+
>> >> | 1 | | 2 | | 3 | | 4 |
>> >> +--+---+------+---+------+---+------+---+--+
>> >> | physical pfsub vf pfsub |
>> >> | port port port port |
>> >> | |
>> >> | eswitch |
>> >> | |
>> >> | |
>> >> +------------------------------------------+
>> >>
>> >> 1) pci/0000:05:00.0/0: type eth netdev enp5s0np0 flavour physical switch_id 00154d130d2f
>> >> 2) pci/0000:05:00.0/10000: type eth netdev enp5s0npf0s0 flavour pci_pf pf 0 subport 0 switch_id 00154d130d2f
>> >> 3) pci/0000:05:00.0/10001: type eth netdev enp5s0npf0vf0 flavour pci_vf pf 0 vf 0 switch_id 00154d130d2f
>> >> 4) pci/0000:05:00.0/10001: type eth netdev enp5s0npf0s1 flavour pci_pf pf 0 subport 1 switch_id 00154d130d2f
>> >>
>> >> This is basically what you have and I think we are in sync with that.
>> >> But what about 5,6,7? Should they have devlink port instances too?
>> >>
>> >> 5) pci/0000:05:00.0/1: type eth netdev enp5s0f0?? flavour ???? pf 0 subport 0
>> >> 6) pci/0000:05:10.1/0: type eth netdev enp5s10f0 flavour ???? pf 0 vf 0
>> >> 7) pci/0000:05:00.0/1: type eth netdev enp5s0f0?? flavour ???? pf 0 subport 1
>> >>
>> >> These are the "peers".
>> >> I think that there could be flavours "pci_pf" and "pci_vf". Then the
>> >> "representors" (switch ports) could have flavours "pci_pf_port" and
>> >> "pci_vf_port" or something like that. User can see right away
>> >> that is not "PF" of "VF" but rather something "on the other end".
>> >> Note there is no "switch_id" for these devlink ports that tells the user
>> >> these devlink ports are not part of any switch.
>> >> What do you think?
>> >
>> >Hmmm.. Hm. Hm.
>> >
>> >To me its neat if the devlink instance matches an ASIC. I think it's
>> >kind of clear for people to understand what it stands for then. So if
>> >we wanted to do the above we'd have to make the switch_id the first
>> >class identifier for devlink instances, rather than the bus? But then
>>
>> What do you mean by "first class identifier"? Like "a handle"?
>
>Yes, a handle.
Odd.
>
>> >VF instances don't have a switch ID so that doesn't work...
>>
>> Wait a sec. VF-ports do have. VFs them selves don't.
>
>Looking at your example this one:
>
>6) pci/0000:05:10.1/0: type eth netdev enp5s10f0 flavour ???? pf 0 vf 0
>
>that uses VF's DBDF in the devlink instance handle, so I presume this
>is a VF's devlink instance that will get passed to the VM together with
>the VF device?
Yes. Correct. That this does not have switch_id.
>
>> But that is the same for PF. PF would also not have switch id.
>
>Yes :( You'd have to mark what constitutes a devlink instance on your
>drawing. The semantics for devlink instances seem to be the focal point
>of the discussion.
>
>Right now it seems a little bit that folks on the NIC side see a devlink
>instance as a PCI function and on switch side it's the whole ASIC.
I think it should be the whole ASIC for both. I don't see why not. It's
one entity, one set parameter, one flash function, one info report etc.
>
>> >I need to think about it.
>> >
>> >It's also kind of strange that we have to add the noun *port* to the
>> >flavour of... a port... So I would prefer not to have those showing up
>>
>> Yeah.
>>
>> >as ports. Can we invent a new command (say "partition"?) that'd take
>> >the bus info where the partition is to be spawned?
>>
>> Got it. But the question is how different this object would be from the
>> existing "port" we have today.
>
>They'd be where "the other side of a PCI link" is represented,
>restricting ports to only ASIC's forwarding plane ports.
Basically a "host port", right? It can still be the same port object,
only with different flavour and attributes. So we would have:
1) pci/0000:05:00.0/0: type eth netdev enp5s0np0
flavour physical switch_id 00154d130d2f
2) pci/0000:05:00.0/10000: type eth netdev enp5s0npf0s0
flavour pci_pf pf 0 subport 0
switch_id 00154d130d2f
peer pci/0000:05:00.0/1
3) pci/0000:05:00.0/10001: type eth netdev enp5s0npf0vf0
flavour pci_vf pf 0 vf 0
switch_id 00154d130d2f
peer pci/0000:05:10.1/0
4) pci/0000:05:00.0/10001: type eth netdev enp5s0npf0s1
flavour pci_pf pf 0 subport 1
switch_id 00154d130d2f
peer pci/0000:05:00.0/2
5) pci/0000:05:00.0/1: type eth netdev enp5s0f0??
flavour host <----------------
peer pci/0000:05:00.0/10000
6) pci/0000:05:10.1/0: type eth netdev enp5s10f0
flavour host <----------------
peer pci/0000:05:00.0/10001
7) pci/0000:05:00.0/2: type eth netdev enp5s0f0??
flavour host <----------------
peer pci/0000:05:00.0/10001
I think it looks quite clear, it gives complete topology view.
>
>> >My next goal is to find a way of grouping multiple bus devices under one
>> >"ASIC" (which is a devlink instance to me) so it can be understood
>> >easily how things are laid out when there is more than one PF connected
>> >to one host.
>>
>> These are the "aliases" you mentioned before right? Makes sense.
>
>Yes.
^ permalink raw reply [flat|nested] 100+ messages in thread
* Re: [PATCH net-next v2 4/7] devlink: allow subports on devlink PCI ports
2019-03-05 0:40 ` Jakub Kicinski
@ 2019-03-05 11:07 ` Jiri Pirko
0 siblings, 0 replies; 100+ messages in thread
From: Jiri Pirko @ 2019-03-05 11:07 UTC (permalink / raw)
To: Jakub Kicinski; +Cc: davem, netdev, oss-drivers
Tue, Mar 05, 2019 at 01:40:07AM CET, jakub.kicinski@netronome.com wrote:
>On Mon, 4 Mar 2019 12:19:02 +0100, Jiri Pirko wrote:
>> Sat, Mar 02, 2019 at 08:48:47PM CET, jakub.kicinski@netronome.com wrote:
>> >On Sat, 2 Mar 2019 10:41:16 +0100, Jiri Pirko wrote:
>> >> Fri, Mar 01, 2019 at 07:04:50PM CET, jakub.kicinski@netronome.com wrote:
>> >> >PCI endpoint corresponds to a PCI device, but such device
>> >> >can have one more more logical device ports associated with it.
>> >> >We need a way to distinguish those. Add a PCI subport in the
>> >> >dumps and print the info in phys_port_name appropriately.
>> >> >
>> >> >This is not equivalent to port splitting, there is no split
>> >> >group. It's just a way of representing multiple netdevs on
>> >> >a single PCI function.
>> >> >
>> >> >Note that the quality of being multiport pertains only to
>> >> >the PCI function itself. A PF having multiple netdevs does
>> >> >not mean that its VFs will also have multiple, or that VFs
>> >> >are associated with any particular port of a multiport VF.
>> >> >
>> >> >Example (bus 05 device has subports, bus 82 has only one port per
>> >> >function):
>> >> >
>> >> >$ devlink port
>> >> >pci/0000:05:00.0/0: type eth netdev enp5s0np0 flavour physical
>> >> >pci/0000:05:00.0/10000: type eth netdev enp5s0npf0s0 flavour pci_pf pf 0 subport 0
>> >> >pci/0000:05:00.0/4: type eth netdev enp5s0np1 flavour physical
>> >> >pci/0000:05:00.0/11000: type eth netdev enp5s0npf0s1 flavour pci_pf pf 0 subport 1
>> >>
>> >> So these subport devlink ports are eswitch ports for subports, right?
>> >>
>> >> Please see the following drawing:
>> >>
>> >> +---+ +---+ +---+
>> >> pfsub| 5 | vf| 6 | | 7 |pfsub
>> >> +-+-+ +-+-+ +-+-+
>> >> physical link <---------+ | | |
>> >> | | | |
>> >> | | | |
>> >> | | | |
>> >> +-+-+ +-+-+ +-+-+ +-+-+
>> >> | 1 | | 2 | | 3 | | 4 |
>> >> +--+---+------+---+------+---+------+---+--+
>> >> | physical pfsub vf pfsub |
>> >> | port port port port |
>> >> | |
>> >> | eswitch |
>> >> | |
>> >> | |
>> >> +------------------------------------------+
>> >>
>> >> 1) pci/0000:05:00.0/0: type eth netdev enp5s0np0 flavour physical switch_id 00154d130d2f
>> >> 2) pci/0000:05:00.0/10000: type eth netdev enp5s0npf0s0 flavour pci_pf pf 0 subport 0 switch_id 00154d130d2f
>> >> 3) pci/0000:05:00.0/10001: type eth netdev enp5s0npf0vf0 flavour pci_vf pf 0 vf 0 switch_id 00154d130d2f
>> >> 4) pci/0000:05:00.0/10001: type eth netdev enp5s0npf0s1 flavour pci_pf pf 0 subport 1 switch_id 00154d130d2f
>> >>
>> >> This is basically what you have and I think we are in sync with that.
>> >> But what about 5,6,7? Should they have devlink port instances too?
>> >>
>> >> 5) pci/0000:05:00.0/1: type eth netdev enp5s0f0?? flavour ???? pf 0 subport 0
>> >> 6) pci/0000:05:10.1/0: type eth netdev enp5s10f0 flavour ???? pf 0 vf 0
>> >> 7) pci/0000:05:00.0/1: type eth netdev enp5s0f0?? flavour ???? pf 0 subport 1
>> >>
>> >> These are the "peers".
>> >> I think that there could be flavours "pci_pf" and "pci_vf". Then the
>> >> "representors" (switch ports) could have flavours "pci_pf_port" and
>> >> "pci_vf_port" or something like that. User can see right away
>> >> that is not "PF" of "VF" but rather something "on the other end".
>> >> Note there is no "switch_id" for these devlink ports that tells the user
>> >> these devlink ports are not part of any switch.
>> >> What do you think?
>> >
>> >Hmmm.. Hm. Hm.
>> >
>> >To me its neat if the devlink instance matches an ASIC. I think it's
>> >kind of clear for people to understand what it stands for then. So if
>> >we wanted to do the above we'd have to make the switch_id the first
>> >class identifier for devlink instances, rather than the bus? But then
>> >VF instances don't have a switch ID so that doesn't work...
>> >
>> >I need to think about it.
>> >
>> >It's also kind of strange that we have to add the noun *port* to the
>> >flavour of... a port... So I would prefer not to have those showing up
>> >as ports. Can we invent a new command (say "partition"?) that'd take
>> >the bus info where the partition is to be spawned?
>>
>> Devlink does not supposed to be only there for switches. From the
>> beginning the design was to handle cases where the netdev/ib_dev is not
>> the correct handle. Not only in case you have multiple instances (ports)
>> for one ASIC, but also in case you have only one. Example use case is
>> port-type-change (eth->ib,ib->eth).
>>
>> I chose word "port" as the parent devlink instance is "dev" and if you
>> partition the ASIC you basically got "ports", each of a different flavour.
>>
>> And as you said, devlink instance matches one ASIC. Therefore the
>> devlink instance should contain all bits there are part of that ASIC,
>> not only switch/eswitch ports. That would be very limitting.
>
>I could read this as us being in full agreement, but I'm not sure..
>I think we agree that all objects of an ASIC should be under one
>devlink instance, the question remains whether both ends of the pipe
>for PCI devices (subdevs or not) should appear under ports or does the
>"far end" (from ASICs perspective)/"host end" get its own category.
Yep. Please see the suggestion about "flavour host" I did in other reply
in this thread.
^ permalink raw reply [flat|nested] 100+ messages in thread
* Re: [PATCH net-next v2 4/7] devlink: allow subports on devlink PCI ports
2019-03-05 0:51 ` Jakub Kicinski
@ 2019-03-05 11:09 ` Jiri Pirko
0 siblings, 0 replies; 100+ messages in thread
From: Jiri Pirko @ 2019-03-05 11:09 UTC (permalink / raw)
To: Jakub Kicinski; +Cc: davem, netdev, oss-drivers
Tue, Mar 05, 2019 at 01:51:59AM CET, jakub.kicinski@netronome.com wrote:
>On Mon, 4 Mar 2019 12:08:57 +0100, Jiri Pirko wrote:
>> Fri, Mar 01, 2019 at 07:04:50PM CET, jakub.kicinski@netronome.com wrote:
>> >PCI endpoint corresponds to a PCI device, but such device
>> >can have one more more logical device ports associated with it.
>> >We need a way to distinguish those. Add a PCI subport in the
>> >dumps and print the info in phys_port_name appropriately.
>> >
>> >This is not equivalent to port splitting, there is no split
>> >group. It's just a way of representing multiple netdevs on
>> >a single PCI function.
>> >
>> >Note that the quality of being multiport pertains only to
>> >the PCI function itself. A PF having multiple netdevs does
>> >not mean that its VFs will also have multiple, or that VFs
>> >are associated with any particular port of a multiport VF.
>> >
>> >Example (bus 05 device has subports, bus 82 has only one port per
>> >function):
>>
>> How do you plan to added/remove these subports?
>
>I can't say I got that figured out fully, but I was wondering if we can
>have some form of:
>
>$ devlink partition pci/0000:82:00.0 new
>pci/0000:82:00.0/1001002
Parav has something similar in his proposal. Lets figure out the
port/non-port dillema first, then we can design this api.
>
>Which would create appropriate sub-port and port (-repr-) netdev.
>
>Plus optionally the ability to work with something like the already
>existing mdev infrastructure for passing to a VM. But I haven't even
>looked at that, yet.
^ permalink raw reply [flat|nested] 100+ messages in thread
* Re: [PATCH net-next v2 4/7] devlink: allow subports on devlink PCI ports
2019-03-05 11:06 ` Jiri Pirko
@ 2019-03-05 17:15 ` Jakub Kicinski
2019-03-05 19:59 ` Parav Pandit
2019-03-06 12:20 ` Jiri Pirko
0 siblings, 2 replies; 100+ messages in thread
From: Jakub Kicinski @ 2019-03-05 17:15 UTC (permalink / raw)
To: Jiri Pirko; +Cc: davem, netdev, oss-drivers
On Tue, 5 Mar 2019 12:06:01 +0100, Jiri Pirko wrote:
> >> >as ports. Can we invent a new command (say "partition"?) that'd take
> >> >the bus info where the partition is to be spawned?
> >>
> >> Got it. But the question is how different this object would be from the
> >> existing "port" we have today.
> >
> >They'd be where "the other side of a PCI link" is represented,
> >restricting ports to only ASIC's forwarding plane ports.
>
> Basically a "host port", right? It can still be the same port object,
> only with different flavour and attributes. So we would have:
>
> 1) pci/0000:05:00.0/0: type eth netdev enp5s0np0
> flavour physical switch_id 00154d130d2f
> 2) pci/0000:05:00.0/10000: type eth netdev enp5s0npf0s0
> flavour pci_pf pf 0 subport 0
> switch_id 00154d130d2f
> peer pci/0000:05:00.0/1
> 3) pci/0000:05:00.0/10001: type eth netdev enp5s0npf0vf0
> flavour pci_vf pf 0 vf 0
> switch_id 00154d130d2f
> peer pci/0000:05:10.1/0
> 4) pci/0000:05:00.0/10001: type eth netdev enp5s0npf0s1
> flavour pci_pf pf 0 subport 1
> switch_id 00154d130d2f
> peer pci/0000:05:00.0/2
> 5) pci/0000:05:00.0/1: type eth netdev enp5s0f0??
> flavour host <----------------
> peer pci/0000:05:00.0/10000
> 6) pci/0000:05:10.1/0: type eth netdev enp5s10f0
> flavour host <----------------
> peer pci/0000:05:00.0/10001
> 7) pci/0000:05:00.0/2: type eth netdev enp5s0f0??
> flavour host <----------------
> peer pci/0000:05:00.0/10001
>
> I think it looks quite clear, it gives complete topology view.
Okay, I have some of questions :)
What do we use for port_index?
What are the operations one can perform on "host ports"?
If we have PCI parameters, do they get set on the ASIC side of the port
or the host side of the port?
How do those behave when device is passed to VM?
You have a VF devlink instance there - what ports does it show?
How do those look when the PF is connected to another host? Do they
get spawned at all?
Will this not be confusing to DSA folks who have a CPU port?
^ permalink raw reply [flat|nested] 100+ messages in thread
* RE: [PATCH net-next v2 4/7] devlink: allow subports on devlink PCI ports
2019-03-05 17:15 ` Jakub Kicinski
@ 2019-03-05 19:59 ` Parav Pandit
2019-03-06 12:20 ` Jiri Pirko
1 sibling, 0 replies; 100+ messages in thread
From: Parav Pandit @ 2019-03-05 19:59 UTC (permalink / raw)
To: Jakub Kicinski, Jiri Pirko; +Cc: davem, netdev, oss-drivers
> -----Original Message-----
> From: netdev-owner@vger.kernel.org <netdev-owner@vger.kernel.org> On
> Behalf Of Jakub Kicinski
> Sent: Tuesday, March 5, 2019 11:16 AM
> To: Jiri Pirko <jiri@resnulli.us>
> Cc: davem@davemloft.net; netdev@vger.kernel.org; oss-
> drivers@netronome.com
> Subject: Re: [PATCH net-next v2 4/7] devlink: allow subports on devlink PCI
> ports
>
> On Tue, 5 Mar 2019 12:06:01 +0100, Jiri Pirko wrote:
> > >> >as ports. Can we invent a new command (say "partition"?) that'd take
> > >> >the bus info where the partition is to be spawned?
> > >>
> > >> Got it. But the question is how different this object would be from
> > >> the existing "port" we have today.
> > >
> > >They'd be where "the other side of a PCI link" is represented,
> > >restricting ports to only ASIC's forwarding plane ports.
> >
> > Basically a "host port", right? It can still be the same port object,
> > only with different flavour and attributes. So we would have:
> >
> > 1) pci/0000:05:00.0/0: type eth netdev enp5s0np0
> > flavour physical switch_id 00154d130d2f
> > 2) pci/0000:05:00.0/10000: type eth netdev enp5s0npf0s0
> > flavour pci_pf pf 0 subport 0
> > switch_id 00154d130d2f
> > peer pci/0000:05:00.0/1
> > 3) pci/0000:05:00.0/10001: type eth netdev enp5s0npf0vf0
> > flavour pci_vf pf 0 vf 0
> > switch_id 00154d130d2f
> > peer pci/0000:05:10.1/0
> > 4) pci/0000:05:00.0/10001: type eth netdev enp5s0npf0s1
> > flavour pci_pf pf 0 subport 1
> > switch_id 00154d130d2f
> > peer pci/0000:05:00.0/2
> > 5) pci/0000:05:00.0/1: type eth netdev enp5s0f0??
> > flavour host <----------------
> > peer pci/0000:05:00.0/10000
> > 6) pci/0000:05:10.1/0: type eth netdev enp5s10f0
> > flavour host <----------------
> > peer pci/0000:05:00.0/10001
> > 7) pci/0000:05:00.0/2: type eth netdev enp5s0f0??
> > flavour host <----------------
> > peer pci/0000:05:00.0/10001
> >
> > I think it looks quite clear, it gives complete topology view.
>
> Okay, I have some of questions :)
>
> What do we use for port_index?
>
port_index to refer a port for port attribute query/config.
> What are the operations one can perform on "host ports"?
>
Set port parameters.
I see use case for mac address, rdma port guid, port speed that should be emulated.
> If we have PCI parameters, do they get set on the ASIC side of the port or the
> host side of the port?
>
Hostport has host facing parameters.
Eswitch port has switch facing parameters (configured/queried/managed through netdev/ovs).
> How do those behave when device is passed to VM?
>
> You have a VF devlink instance there - what ports does it show?
>
VF devlink port in VM will show the port it got.
It will likely have only read permission for the port attributes as today.
For example, it will not know which switchport it is connected to.
> How do those look when the PF is connected to another host? Do they get
> spawned at all?
>
> Will this not be confusing to DSA folks who have a CPU port?
^ permalink raw reply [flat|nested] 100+ messages in thread
* Re: [PATCH net-next v2 4/7] devlink: allow subports on devlink PCI ports
2019-03-05 17:15 ` Jakub Kicinski
2019-03-05 19:59 ` Parav Pandit
@ 2019-03-06 12:20 ` Jiri Pirko
2019-03-06 17:56 ` Jakub Kicinski
1 sibling, 1 reply; 100+ messages in thread
From: Jiri Pirko @ 2019-03-06 12:20 UTC (permalink / raw)
To: Jakub Kicinski; +Cc: davem, netdev, oss-drivers
Tue, Mar 05, 2019 at 06:15:34PM CET, jakub.kicinski@netronome.com wrote:
>On Tue, 5 Mar 2019 12:06:01 +0100, Jiri Pirko wrote:
>> >> >as ports. Can we invent a new command (say "partition"?) that'd take
>> >> >the bus info where the partition is to be spawned?
>> >>
>> >> Got it. But the question is how different this object would be from the
>> >> existing "port" we have today.
>> >
>> >They'd be where "the other side of a PCI link" is represented,
>> >restricting ports to only ASIC's forwarding plane ports.
>>
>> Basically a "host port", right? It can still be the same port object,
>> only with different flavour and attributes. So we would have:
>>
>> 1) pci/0000:05:00.0/0: type eth netdev enp5s0np0
>> flavour physical switch_id 00154d130d2f
>> 2) pci/0000:05:00.0/10000: type eth netdev enp5s0npf0s0
>> flavour pci_pf pf 0 subport 0
>> switch_id 00154d130d2f
>> peer pci/0000:05:00.0/1
>> 3) pci/0000:05:00.0/10001: type eth netdev enp5s0npf0vf0
>> flavour pci_vf pf 0 vf 0
>> switch_id 00154d130d2f
>> peer pci/0000:05:10.1/0
>> 4) pci/0000:05:00.0/10001: type eth netdev enp5s0npf0s1
>> flavour pci_pf pf 0 subport 1
>> switch_id 00154d130d2f
>> peer pci/0000:05:00.0/2
>> 5) pci/0000:05:00.0/1: type eth netdev enp5s0f0??
>> flavour host <----------------
>> peer pci/0000:05:00.0/10000
>> 6) pci/0000:05:10.1/0: type eth netdev enp5s10f0
>> flavour host <----------------
>> peer pci/0000:05:00.0/10001
>> 7) pci/0000:05:00.0/2: type eth netdev enp5s0f0??
>> flavour host <----------------
>> peer pci/0000:05:00.0/10001
>>
>> I think it looks quite clear, it gives complete topology view.
>
>Okay, I have some of questions :)
>
>What do we use for port_index?
That is just a number totally in control of the driver. Driver can
assign it in any way.
>
>What are the operations one can perform on "host ports"?
That is a good question. I would start with *none* and extend it upon
needs.
>
>If we have PCI parameters, do they get set on the ASIC side of the port
>or the host side of the port?
Could you give me an example? But I believe that on switch-port side.
>
>How do those behave when device is passed to VM?
In case of VF? VF will have separate devlink instance (separate handle,
probably "aliased" to the PF handle). So it would disappear from
baremetal and appear in VM:
$ devlink dev
pci/0000:00:10.0
$ devlink dev port
pci/0000:00:10.1/0: type eth netdev enp5s10f0
flavour host
That's it for the VM.
There's no linkage (peer, alias) between this and the instances on
baremetal.
>
>You have a VF devlink instance there - what ports does it show?
See above.
>
>How do those look when the PF is connected to another host? Do they
>get spawned at all?
What do you mean by "PF is connected to another host"?
>
>Will this not be confusing to DSA folks who have a CPU port?
Why do you think so?
^ permalink raw reply [flat|nested] 100+ messages in thread
* Re: [PATCH net-next v2 4/7] devlink: allow subports on devlink PCI ports
2019-03-06 12:20 ` Jiri Pirko
@ 2019-03-06 17:56 ` Jakub Kicinski
2019-03-07 3:56 ` Parav Pandit
2019-03-07 9:48 ` Jiri Pirko
0 siblings, 2 replies; 100+ messages in thread
From: Jakub Kicinski @ 2019-03-06 17:56 UTC (permalink / raw)
To: Jiri Pirko; +Cc: davem, netdev, oss-drivers
On Wed, 6 Mar 2019 13:20:37 +0100, Jiri Pirko wrote:
> Tue, Mar 05, 2019 at 06:15:34PM CET, jakub.kicinski@netronome.com wrote:
> >On Tue, 5 Mar 2019 12:06:01 +0100, Jiri Pirko wrote:
> >> >> >as ports. Can we invent a new command (say "partition"?) that'd take
> >> >> >the bus info where the partition is to be spawned?
> >> >>
> >> >> Got it. But the question is how different this object would be from the
> >> >> existing "port" we have today.
> >> >
> >> >They'd be where "the other side of a PCI link" is represented,
> >> >restricting ports to only ASIC's forwarding plane ports.
> >>
> >> Basically a "host port", right? It can still be the same port object,
> >> only with different flavour and attributes. So we would have:
> >>
> >> 1) pci/0000:05:00.0/0: type eth netdev enp5s0np0
> >> flavour physical switch_id 00154d130d2f
> >> 2) pci/0000:05:00.0/10000: type eth netdev enp5s0npf0s0
> >> flavour pci_pf pf 0 subport 0
> >> switch_id 00154d130d2f
> >> peer pci/0000:05:00.0/1
> >> 3) pci/0000:05:00.0/10001: type eth netdev enp5s0npf0vf0
> >> flavour pci_vf pf 0 vf 0
> >> switch_id 00154d130d2f
> >> peer pci/0000:05:10.1/0
> >> 4) pci/0000:05:00.0/10001: type eth netdev enp5s0npf0s1
> >> flavour pci_pf pf 0 subport 1
> >> switch_id 00154d130d2f
> >> peer pci/0000:05:00.0/2
> >> 5) pci/0000:05:00.0/1: type eth netdev enp5s0f0??
> >> flavour host <----------------
> >> peer pci/0000:05:00.0/10000
> >> 6) pci/0000:05:10.1/0: type eth netdev enp5s10f0
> >> flavour host <----------------
> >> peer pci/0000:05:00.0/10001
> >> 7) pci/0000:05:00.0/2: type eth netdev enp5s0f0??
> >> flavour host <----------------
> >> peer pci/0000:05:00.0/10001
> >>
> >> I think it looks quite clear, it gives complete topology view.
> >
> >Okay, I have some of questions :)
> >
> >What do we use for port_index?
>
> That is just a number totally in control of the driver. Driver can
> assign it in any way.
>
> >
> >What are the operations one can perform on "host ports"?
>
> That is a good question. I would start with *none* and extend it upon
> needs.
>
>
> >
> >If we have PCI parameters, do they get set on the ASIC side of the port
> >or the host side of the port?
>
> Could you give me an example?
Let's take msix_vec_per_pf_min as an example.
> But I believe that on switch-port side.
Ok.
> >How do those behave when device is passed to VM?
>
> In case of VF? VF will have separate devlink instance (separate handle,
> probably "aliased" to the PF handle). So it would disappear from
> baremetal and appear in VM:
> $ devlink dev
> pci/0000:00:10.0
> $ devlink dev port
> pci/0000:00:10.1/0: type eth netdev enp5s10f0
> flavour host
> That's it for the VM.
>
> There's no linkage (peer, alias) between this and the instances on
> baremetal.
Ok, I guess this is the main advantage from your perspective?
The fact that "host ports" are visible inside a VM?
Or do you believe that having both ends of a pipe as ports makes the
topology easier to understand?
For creating subdevices, I don't think the handle should ever be port.
We create new ports on a devlink instance, and configure its forwarding
with offloads of well established Linux SW constructs. New devices are
not logically associated with other ports (see how in my patches there
are 2 "subports" but no main port on that PF - a split not a hierarchy).
How we want to model forwarding inside a VM (who configures the
underlying switching) remains unclear.
> >You have a VF devlink instance there - what ports does it show?
>
> See above.
>
>
> >
> >How do those look when the PF is connected to another host? Do they
> >get spawned at all?
>
> What do you mean by "PF is connected to another host"?
Either "SmartNIC":
http://www.mellanox.com/products/smartnic/?ls=gppc&lsd=SmartNIC-gen-smartnic&gclid=EAIaIQobChMIxIrGmYju4AIVy5yzCh2SFwQJEAAYASAAEgIui_D_BwE
or
Multi-host NIC: http://www.mellanox.com/page/multihost
> >Will this not be confusing to DSA folks who have a CPU port?
>
> Why do you think so?
Host and CPU sound quite similar, it is unclear how they differ, and
why we have a need for both (from user's perspective).
^ permalink raw reply [flat|nested] 100+ messages in thread
* RE: [PATCH net-next v2 4/7] devlink: allow subports on devlink PCI ports
2019-03-06 17:56 ` Jakub Kicinski
@ 2019-03-07 3:56 ` Parav Pandit
2019-03-07 9:48 ` Jiri Pirko
1 sibling, 0 replies; 100+ messages in thread
From: Parav Pandit @ 2019-03-07 3:56 UTC (permalink / raw)
To: Jakub Kicinski, Jiri Pirko; +Cc: davem, netdev, oss-drivers
Hi Jakub,
> -----Original Message-----
> From: netdev-owner@vger.kernel.org <netdev-owner@vger.kernel.org> On
> Behalf Of Jakub Kicinski
> Sent: Wednesday, March 6, 2019 11:57 AM
> To: Jiri Pirko <jiri@resnulli.us>
> Cc: davem@davemloft.net; netdev@vger.kernel.org; oss-
> drivers@netronome.com
> Subject: Re: [PATCH net-next v2 4/7] devlink: allow subports on devlink PCI
> ports
>
> On Wed, 6 Mar 2019 13:20:37 +0100, Jiri Pirko wrote:
> > Tue, Mar 05, 2019 at 06:15:34PM CET, jakub.kicinski@netronome.com
> wrote:
> > >On Tue, 5 Mar 2019 12:06:01 +0100, Jiri Pirko wrote:
> > >> >> >as ports. Can we invent a new command (say "partition"?) that'd
> take
> > >> >> >the bus info where the partition is to be spawned?
> > >> >>
> > >> >> Got it. But the question is how different this object would be from
> the
> > >> >> existing "port" we have today.
> > >> >
> > >> >They'd be where "the other side of a PCI link" is represented,
> > >> >restricting ports to only ASIC's forwarding plane ports.
> > >>
> > >> Basically a "host port", right? It can still be the same port
> > >> object, only with different flavour and attributes. So we would have:
> > >>
> > >> 1) pci/0000:05:00.0/0: type eth netdev enp5s0np0
> > >> flavour physical switch_id 00154d130d2f
> > >> 2) pci/0000:05:00.0/10000: type eth netdev enp5s0npf0s0
> > >> flavour pci_pf pf 0 subport 0
> > >> switch_id 00154d130d2f
> > >> peer pci/0000:05:00.0/1
> > >> 3) pci/0000:05:00.0/10001: type eth netdev enp5s0npf0vf0
> > >> flavour pci_vf pf 0 vf 0
> > >> switch_id 00154d130d2f
> > >> peer pci/0000:05:10.1/0
> > >> 4) pci/0000:05:00.0/10001: type eth netdev enp5s0npf0s1
> > >> flavour pci_pf pf 0 subport 1
> > >> switch_id 00154d130d2f
> > >> peer pci/0000:05:00.0/2
> > >> 5) pci/0000:05:00.0/1: type eth netdev enp5s0f0??
> > >> flavour host <----------------
> > >> peer pci/0000:05:00.0/10000
> > >> 6) pci/0000:05:10.1/0: type eth netdev enp5s10f0
> > >> flavour host <----------------
> > >> peer pci/0000:05:00.0/10001
> > >> 7) pci/0000:05:00.0/2: type eth netdev enp5s0f0??
> > >> flavour host <----------------
> > >> peer pci/0000:05:00.0/10001
> > >>
> > >> I think it looks quite clear, it gives complete topology view.
> > >
> > >Okay, I have some of questions :)
> > >
> > >What do we use for port_index?
> >
> > That is just a number totally in control of the driver. Driver can
> > assign it in any way.
> >
> > >
> > >What are the operations one can perform on "host ports"?
> >
> > That is a good question. I would start with *none* and extend it upon
> > needs.
> >
> >
> > >
> > >If we have PCI parameters, do they get set on the ASIC side of the
> > >port or the host side of the port?
> >
> > Could you give me an example?
>
> Let's take msix_vec_per_pf_min as an example.
>
> > But I believe that on switch-port side.
>
> Ok.
>
> > >How do those behave when device is passed to VM?
> >
> > In case of VF? VF will have separate devlink instance (separate
> > handle, probably "aliased" to the PF handle). So it would disappear
> > from baremetal and appear in VM:
> > $ devlink dev
> > pci/0000:00:10.0
> > $ devlink dev port
> > pci/0000:00:10.1/0: type eth netdev enp5s10f0
> > flavour host
> > That's it for the VM.
> >
> > There's no linkage (peer, alias) between this and the instances on
> > baremetal.
>
> Ok, I guess this is the main advantage from your perspective?
> The fact that "host ports" are visible inside a VM?
> Or do you believe that having both ends of a pipe as ports makes the
> topology easier to understand?
>
> For creating subdevices, I don't think the handle should ever be port.
I updated the proposal [1], haven't sent updated (reduced) RFC patches yet.
subdevices are created using already existing 'mdev' framework.
You also mentioned in one of the past email discussion.
These subdevices live on 'mdev' bus.
Handle for creating this subdev is its parent PCI device.
Mdev framework exposes bunch of sysfs files for these work.
vendors who wish to get it connected in switchdev mode,
(I guest most of the current switchdev drivers),
will be creating mdev device and associated devlink instance (and port).
This way, created subdev can be provisioned for VM or on host itself using unified scheme.
subdev's parameters, its hostport, its switchport (rep-netdev)
can be controlled similar to VFs.
[1] https://www.mail-archive.com/linux-kernel@vger.kernel.org/msg1948602.html
^ permalink raw reply [flat|nested] 100+ messages in thread
* Re: [PATCH net-next v2 4/7] devlink: allow subports on devlink PCI ports
2019-03-06 17:56 ` Jakub Kicinski
2019-03-07 3:56 ` Parav Pandit
@ 2019-03-07 9:48 ` Jiri Pirko
2019-03-08 2:52 ` Jakub Kicinski
1 sibling, 1 reply; 100+ messages in thread
From: Jiri Pirko @ 2019-03-07 9:48 UTC (permalink / raw)
To: Jakub Kicinski; +Cc: davem, netdev, oss-drivers
Wed, Mar 06, 2019 at 06:56:38PM CET, jakub.kicinski@netronome.com wrote:
>On Wed, 6 Mar 2019 13:20:37 +0100, Jiri Pirko wrote:
>> Tue, Mar 05, 2019 at 06:15:34PM CET, jakub.kicinski@netronome.com wrote:
>> >On Tue, 5 Mar 2019 12:06:01 +0100, Jiri Pirko wrote:
>> >> >> >as ports. Can we invent a new command (say "partition"?) that'd take
>> >> >> >the bus info where the partition is to be spawned?
>> >> >>
>> >> >> Got it. But the question is how different this object would be from the
>> >> >> existing "port" we have today.
>> >> >
>> >> >They'd be where "the other side of a PCI link" is represented,
>> >> >restricting ports to only ASIC's forwarding plane ports.
>> >>
>> >> Basically a "host port", right? It can still be the same port object,
>> >> only with different flavour and attributes. So we would have:
>> >>
>> >> 1) pci/0000:05:00.0/0: type eth netdev enp5s0np0
>> >> flavour physical switch_id 00154d130d2f
>> >> 2) pci/0000:05:00.0/10000: type eth netdev enp5s0npf0s0
>> >> flavour pci_pf pf 0 subport 0
>> >> switch_id 00154d130d2f
>> >> peer pci/0000:05:00.0/1
>> >> 3) pci/0000:05:00.0/10001: type eth netdev enp5s0npf0vf0
>> >> flavour pci_vf pf 0 vf 0
>> >> switch_id 00154d130d2f
>> >> peer pci/0000:05:10.1/0
>> >> 4) pci/0000:05:00.0/10001: type eth netdev enp5s0npf0s1
>> >> flavour pci_pf pf 0 subport 1
>> >> switch_id 00154d130d2f
>> >> peer pci/0000:05:00.0/2
>> >> 5) pci/0000:05:00.0/1: type eth netdev enp5s0f0??
>> >> flavour host <----------------
>> >> peer pci/0000:05:00.0/10000
>> >> 6) pci/0000:05:10.1/0: type eth netdev enp5s10f0
>> >> flavour host <----------------
>> >> peer pci/0000:05:00.0/10001
>> >> 7) pci/0000:05:00.0/2: type eth netdev enp5s0f0??
>> >> flavour host <----------------
>> >> peer pci/0000:05:00.0/10001
>> >>
>> >> I think it looks quite clear, it gives complete topology view.
>> >
>> >Okay, I have some of questions :)
>> >
>> >What do we use for port_index?
>>
>> That is just a number totally in control of the driver. Driver can
>> assign it in any way.
>>
>> >
>> >What are the operations one can perform on "host ports"?
>>
>> That is a good question. I would start with *none* and extend it upon
>> needs.
>>
>>
>> >
>> >If we have PCI parameters, do they get set on the ASIC side of the port
>> >or the host side of the port?
>>
>> Could you give me an example?
>
>Let's take msix_vec_per_pf_min as an example.
>
>> But I believe that on switch-port side.
>
>Ok.
>
>> >How do those behave when device is passed to VM?
>>
>> In case of VF? VF will have separate devlink instance (separate handle,
>> probably "aliased" to the PF handle). So it would disappear from
>> baremetal and appear in VM:
>> $ devlink dev
>> pci/0000:00:10.0
>> $ devlink dev port
>> pci/0000:00:10.1/0: type eth netdev enp5s10f0
>> flavour host
>> That's it for the VM.
>>
>> There's no linkage (peer, alias) between this and the instances on
>> baremetal.
>
>Ok, I guess this is the main advantage from your perspective?
>The fact that "host ports" are visible inside a VM?
Yep. Also on baremetal.
>Or do you believe that having both ends of a pipe as ports makes the
>topology easier to understand?
That as well.
>
>For creating subdevices, I don't think the handle should ever be port.
>We create new ports on a devlink instance, and configure its forwarding
Okay I agree. Something like:
$ devlink port add pci/0000:00:10.0 .....
It's a bit confusing because "set" accepts port handle (like
pci/0000:00:10.0/1). Probably better would be:
$ devlink dev port add pci/0000:00:10.0 .....
>with offloads of well established Linux SW constructs. New devices are
>not logically associated with other ports (see how in my patches there
>are 2 "subports" but no main port on that PF - a split not a hierarchy).
Right, basically you have 2 equal objects. Makes sense.
>
>How we want to model forwarding inside a VM (who configures the
>underlying switching) remains unclear.
I don't understand. Could you elaborate a bit?
>
>> >You have a VF devlink instance there - what ports does it show?
>>
>> See above.
>>
>>
>> >
>> >How do those look when the PF is connected to another host? Do they
>> >get spawned at all?
>>
>> What do you mean by "PF is connected to another host"?
>
>Either "SmartNIC":
>
>http://www.mellanox.com/products/smartnic/?ls=gppc&lsd=SmartNIC-gen-smartnic&gclid=EAIaIQobChMIxIrGmYju4AIVy5yzCh2SFwQJEAAYASAAEgIui_D_BwE
>
>or
>
>Multi-host NIC: http://www.mellanox.com/page/multihost
Right. So in this case, I think that the hostport on specific host
should see devlink instance and the hostport. However, the switchports
should be only on one selected host (I don't see how to do that
differently)
>
>> >Will this not be confusing to DSA folks who have a CPU port?
>>
>> Why do you think so?
>
>Host and CPU sound quite similar, it is unclear how they differ, and
>why we have a need for both (from user's perspective).
Hmm, dsa cpu port is something different. It does not have netdev
associated with it. It is just a port which is physically used in order
to send or receive packets on switch ports.
However in our hostport case, it has user facing netdev associated and
user actually uses it to send and receive packets directly (assigns ip
etc).
^ permalink raw reply [flat|nested] 100+ messages in thread
* Re: [PATCH net-next v2 4/7] devlink: allow subports on devlink PCI ports
2019-03-07 9:48 ` Jiri Pirko
@ 2019-03-08 2:52 ` Jakub Kicinski
2019-03-08 14:54 ` Jiri Pirko
0 siblings, 1 reply; 100+ messages in thread
From: Jakub Kicinski @ 2019-03-08 2:52 UTC (permalink / raw)
To: Jiri Pirko; +Cc: davem, netdev, oss-drivers
On Thu, 7 Mar 2019 10:48:16 +0100, Jiri Pirko wrote:
> Wed, Mar 06, 2019 at 06:56:38PM CET, jakub.kicinski@netronome.com wrote:
> >On Wed, 6 Mar 2019 13:20:37 +0100, Jiri Pirko wrote:
> >For creating subdevices, I don't think the handle should ever be port.
> >We create new ports on a devlink instance, and configure its forwarding
>
> Okay I agree. Something like:
> $ devlink port add pci/0000:00:10.0 .....
>
> It's a bit confusing because "set" accepts port handle (like
> pci/0000:00:10.0/1). Probably better would be:
> $ devlink dev port add pci/0000:00:10.0 .....
>
> >with offloads of well established Linux SW constructs. New devices are
> >not logically associated with other ports (see how in my patches there
> >are 2 "subports" but no main port on that PF - a split not a hierarchy).
>
> Right, basically you have 2 equal objects. Makes sense.
>
> >How we want to model forwarding inside a VM (who configures the
> >underlying switching) remains unclear.
>
> I don't understand. Could you elaborate a bit?
If VF in a VM gets a partitioning request does the new port pop up on
the hypervisor? With a port netdev?
Does the VF also create a port object as well as host port object?
That question is probably independent of host port discussion.
^ permalink raw reply [flat|nested] 100+ messages in thread
* Re: [PATCH net-next v2 4/7] devlink: allow subports on devlink PCI ports
2019-03-08 2:52 ` Jakub Kicinski
@ 2019-03-08 14:54 ` Jiri Pirko
2019-03-08 19:09 ` Jakub Kicinski
0 siblings, 1 reply; 100+ messages in thread
From: Jiri Pirko @ 2019-03-08 14:54 UTC (permalink / raw)
To: Jakub Kicinski; +Cc: davem, netdev, oss-drivers
Fri, Mar 08, 2019 at 03:52:02AM CET, jakub.kicinski@netronome.com wrote:
>On Thu, 7 Mar 2019 10:48:16 +0100, Jiri Pirko wrote:
>> Wed, Mar 06, 2019 at 06:56:38PM CET, jakub.kicinski@netronome.com wrote:
>> >On Wed, 6 Mar 2019 13:20:37 +0100, Jiri Pirko wrote:
>> >For creating subdevices, I don't think the handle should ever be port.
>> >We create new ports on a devlink instance, and configure its forwarding
>>
>> Okay I agree. Something like:
>> $ devlink port add pci/0000:00:10.0 .....
>>
>> It's a bit confusing because "set" accepts port handle (like
>> pci/0000:00:10.0/1). Probably better would be:
>> $ devlink dev port add pci/0000:00:10.0 .....
>>
>> >with offloads of well established Linux SW constructs. New devices are
>> >not logically associated with other ports (see how in my patches there
>> >are 2 "subports" but no main port on that PF - a split not a hierarchy).
>>
>> Right, basically you have 2 equal objects. Makes sense.
>>
>> >How we want to model forwarding inside a VM (who configures the
>> >underlying switching) remains unclear.
>>
>> I don't understand. Could you elaborate a bit?
>
>If VF in a VM gets a partitioning request does the new port pop up on
>the hypervisor? With a port netdev?
Switchport in hypervizor with correct switchid attribute, hostport in
vm. Makes sense?
>
>Does the VF also create a port object as well as host port object?
>
>That question is probably independent of host port discussion.
^ permalink raw reply [flat|nested] 100+ messages in thread
* Re: [PATCH net-next v2 4/7] devlink: allow subports on devlink PCI ports
2019-03-08 14:54 ` Jiri Pirko
@ 2019-03-08 19:09 ` Jakub Kicinski
2019-03-11 8:52 ` Jiri Pirko
0 siblings, 1 reply; 100+ messages in thread
From: Jakub Kicinski @ 2019-03-08 19:09 UTC (permalink / raw)
To: Jiri Pirko; +Cc: davem, netdev, oss-drivers
On Fri, 8 Mar 2019 15:54:21 +0100, Jiri Pirko wrote:
> Fri, Mar 08, 2019 at 03:52:02AM CET, jakub.kicinski@netronome.com wrote:
> >On Thu, 7 Mar 2019 10:48:16 +0100, Jiri Pirko wrote:
> >> Wed, Mar 06, 2019 at 06:56:38PM CET, jakub.kicinski@netronome.com wrote:
> >> >On Wed, 6 Mar 2019 13:20:37 +0100, Jiri Pirko wrote:
> >> >For creating subdevices, I don't think the handle should ever be port.
> >> >We create new ports on a devlink instance, and configure its forwarding
> >>
> >> Okay I agree. Something like:
> >> $ devlink port add pci/0000:00:10.0 .....
> >>
> >> It's a bit confusing because "set" accepts port handle (like
> >> pci/0000:00:10.0/1). Probably better would be:
> >> $ devlink dev port add pci/0000:00:10.0 .....
> >>
> >> >with offloads of well established Linux SW constructs. New devices are
> >> >not logically associated with other ports (see how in my patches there
> >> >are 2 "subports" but no main port on that PF - a split not a hierarchy).
> >>
> >> Right, basically you have 2 equal objects. Makes sense.
> >>
> >> >How we want to model forwarding inside a VM (who configures the
> >> >underlying switching) remains unclear.
> >>
> >> I don't understand. Could you elaborate a bit?
> >
> >If VF in a VM gets a partitioning request does the new port pop up on
> >the hypervisor? With a port netdev?
>
> Switchport in hypervizor with correct switchid attribute, hostport in
> vm. Makes sense?
If the switchport is in the hypervisor then only the hypervisor can
control switching/forwarding, correct?
The primary use case for partitioning within a VM (of a VF) would be
containers (and DPDK)?
SR-IOV makes things harder. Splitting a PF is reasonably easy to grasp.
I'm trying to get a sense of is how would we control an SR-IOV
environment as a whole.
> >Does the VF also create a port object as well as host port object?
> >
> >That question is probably independent of host port discussion.
^ permalink raw reply [flat|nested] 100+ messages in thread
* Re: [PATCH net-next v2 4/7] devlink: allow subports on devlink PCI ports
2019-03-08 19:09 ` Jakub Kicinski
@ 2019-03-11 8:52 ` Jiri Pirko
2019-03-12 2:10 ` Jakub Kicinski
0 siblings, 1 reply; 100+ messages in thread
From: Jiri Pirko @ 2019-03-11 8:52 UTC (permalink / raw)
To: Jakub Kicinski; +Cc: davem, netdev, oss-drivers
Fri, Mar 08, 2019 at 08:09:43PM CET, jakub.kicinski@netronome.com wrote:
>On Fri, 8 Mar 2019 15:54:21 +0100, Jiri Pirko wrote:
>> Fri, Mar 08, 2019 at 03:52:02AM CET, jakub.kicinski@netronome.com wrote:
>> >On Thu, 7 Mar 2019 10:48:16 +0100, Jiri Pirko wrote:
>> >> Wed, Mar 06, 2019 at 06:56:38PM CET, jakub.kicinski@netronome.com wrote:
>> >> >On Wed, 6 Mar 2019 13:20:37 +0100, Jiri Pirko wrote:
>> >> >For creating subdevices, I don't think the handle should ever be port.
>> >> >We create new ports on a devlink instance, and configure its forwarding
>> >>
>> >> Okay I agree. Something like:
>> >> $ devlink port add pci/0000:00:10.0 .....
>> >>
>> >> It's a bit confusing because "set" accepts port handle (like
>> >> pci/0000:00:10.0/1). Probably better would be:
>> >> $ devlink dev port add pci/0000:00:10.0 .....
>> >>
>> >> >with offloads of well established Linux SW constructs. New devices are
>> >> >not logically associated with other ports (see how in my patches there
>> >> >are 2 "subports" but no main port on that PF - a split not a hierarchy).
>> >>
>> >> Right, basically you have 2 equal objects. Makes sense.
>> >>
>> >> >How we want to model forwarding inside a VM (who configures the
>> >> >underlying switching) remains unclear.
>> >>
>> >> I don't understand. Could you elaborate a bit?
>> >
>> >If VF in a VM gets a partitioning request does the new port pop up on
>> >the hypervisor? With a port netdev?
>>
>> Switchport in hypervizor with correct switchid attribute, hostport in
>> vm. Makes sense?
>
>If the switchport is in the hypervisor then only the hypervisor can
>control switching/forwarding, correct?
Correct.
>
>The primary use case for partitioning within a VM (of a VF) would be
>containers (and DPDK)?
Makes sense.
>
>SR-IOV makes things harder. Splitting a PF is reasonably easy to grasp.
>I'm trying to get a sense of is how would we control an SR-IOV
>environment as a whole.
You mean orchestration? I originally planned to implement sriov
orchestration api in devlink too.
>
>> >Does the VF also create a port object as well as host port object?
>> >
>> >That question is probably independent of host port discussion.
>
^ permalink raw reply [flat|nested] 100+ messages in thread
* Re: [PATCH net-next v2 4/7] devlink: allow subports on devlink PCI ports
2019-03-11 8:52 ` Jiri Pirko
@ 2019-03-12 2:10 ` Jakub Kicinski
2019-03-12 14:02 ` Jiri Pirko
0 siblings, 1 reply; 100+ messages in thread
From: Jakub Kicinski @ 2019-03-12 2:10 UTC (permalink / raw)
To: Jiri Pirko; +Cc: davem, netdev, oss-drivers
On Mon, 11 Mar 2019 09:52:04 +0100, Jiri Pirko wrote:
> Fri, Mar 08, 2019 at 08:09:43PM CET, jakub.kicinski@netronome.com wrote:
> >If the switchport is in the hypervisor then only the hypervisor can
> >control switching/forwarding, correct?
>
> Correct.
>
> >The primary use case for partitioning within a VM (of a VF) would be
> >containers (and DPDK)?
>
> Makes sense.
>
> >SR-IOV makes things harder. Splitting a PF is reasonably easy to grasp.
> >I'm trying to get a sense of is how would we control an SR-IOV
> >environment as a whole.
>
> You mean orchestration?
Right, orchestration.
To be clear on where I'm going with this - if we want to allow VFs
to partition themselves then they have to control what is effectively
a "nested" switch. A per-VF set of rules which would the get
"flattened" into the main eswitch rule set. If I was to choose I'd
really rather have this "flattening" be done on the (Linux) hypervisor
and not in the vendor driver and firmware.
I'd much rather have the VM make a "give me another NIC" orchestration
call via some high level REST API than devlink. This makes the
configuration strictly high level to low level:
VM -> cloud net REST API -> cloud agent -> devlink/Linux -> FW -> HW
Without round trips via firmware.
This allows for easy policy enforcement, common code to be maintained
in user space, in high level languages (no 0.5M LoC drivers and 10M LoC
firmware for every driver). It can also be used with software paths
like VirtIO..
Modelling and debugging a nested switch would be a nightmare. What
follows is that we probably shouldn't deal with partitioning of VFs,
but rather only partition via the PF devlink instance, and reassign
the partitions to VMs.
> I originally planned to implement sriov orchestration api in devlink too.
Interesting, would you mind elaborating?
^ permalink raw reply [flat|nested] 100+ messages in thread
* Re: [PATCH net-next v2 4/7] devlink: allow subports on devlink PCI ports
2019-03-12 2:10 ` Jakub Kicinski
@ 2019-03-12 14:02 ` Jiri Pirko
2019-03-12 20:56 ` Jakub Kicinski
[not found] ` <7227d58e-ac58-d549-b921-ca0a0dd3f4b0@intel.com>
0 siblings, 2 replies; 100+ messages in thread
From: Jiri Pirko @ 2019-03-12 14:02 UTC (permalink / raw)
To: Jakub Kicinski; +Cc: davem, netdev, oss-drivers
Tue, Mar 12, 2019 at 03:10:54AM CET, jakub.kicinski@netronome.com wrote:
>On Mon, 11 Mar 2019 09:52:04 +0100, Jiri Pirko wrote:
>> Fri, Mar 08, 2019 at 08:09:43PM CET, jakub.kicinski@netronome.com wrote:
>> >If the switchport is in the hypervisor then only the hypervisor can
>> >control switching/forwarding, correct?
>>
>> Correct.
>>
>> >The primary use case for partitioning within a VM (of a VF) would be
>> >containers (and DPDK)?
>>
>> Makes sense.
>>
>> >SR-IOV makes things harder. Splitting a PF is reasonably easy to grasp.
>> >I'm trying to get a sense of is how would we control an SR-IOV
>> >environment as a whole.
>>
>> You mean orchestration?
>
>Right, orchestration.
>
>To be clear on where I'm going with this - if we want to allow VFs
>to partition themselves then they have to control what is effectively
>a "nested" switch. A per-VF set of rules which would the get
Wait. If you allow to make VF subports (I believe that is what you ment
by VFs partition themselves), that does not mean they will have a
separate nested switch. They would still belong under the same one.
>"flattened" into the main eswitch rule set. If I was to choose I'd
>really rather have this "flattening" be done on the (Linux) hypervisor
>and not in the vendor driver and firmware.
Agreed. Driver should provide one big switch. User should configure it.
>
>I'd much rather have the VM make a "give me another NIC" orchestration
>call via some high level REST API than devlink. This makes the
>configuration strictly high level to low level:
>
> VM -> cloud net REST API -> cloud agent -> devlink/Linux -> FW -> HW
>
>Without round trips via firmware.
Okay. So the "devlink/Linux -> FW" part is going to happen on baremetal.
Makes sense.
>
>This allows for easy policy enforcement, common code to be maintained
>in user space, in high level languages (no 0.5M LoC drivers and 10M LoC
>firmware for every driver). It can also be used with software paths
>like VirtIO..
Agreed.
>
>Modelling and debugging a nested switch would be a nightmare. What
>follows is that we probably shouldn't deal with partitioning of VFs,
>but rather only partition via the PF devlink instance, and reassign
>the partitions to VMs.
Agreed. That must be misunderstanding, I never suggested nested
switches.
>
>> I originally planned to implement sriov orchestration api in devlink too.
>
>Interesting, would you mind elaborating?
I have to think about it. But something like this:
After bootup, you see only physical port, PF switch port and PF host leg.
$ devlink port show
pci/0000:05:00.0/0: type eth netdev enp5s0np0 flavour physical switch_id 00154d130d2
pci/0000:05:00.0/1: type eth netdev ??? flavour pci_pf_host
peer pci/0000:05:00.0/10000
pci/0000:05:00.0/10000: type eth netdev enp5s0npf0pf0s0 flavour pci_pf pf 0 subport 0
switch_id 00154d130d2f peer pci/0000:05:00.0/1
To create new PF subport under PF 0:
$ devlink dev port add pci/0000:05:00.0 flavour pci_pf pf 0
$ devlink port show
pci/0000:05:00.0/0: type eth netdev enp5s0np0 flavour physical switch_id 00154d130d2
pci/0000:05:00.0/1: type eth netdev ??? flavour pci_pf_host
peer pci/0000:05:00.0/10000
pci/0000:05:00.0/10000: type eth netdev enp5s0npf0pf0s0 flavour pci_pf pf 0 subport 0
switch_id 00154d130d2f peer pci/0000:05:00.0/1
pci/0000:05:00.0/2: type eth netdev ??? flavour pci_pf_host <<<<<<<<<<<<<<<<<<
peer pci/0000:05:00.0/10001 <<<<<<<<<<<<<<<<<<
pci/0000:05:00.0/10001: type eth netdev enp5s0npf0pf0s0 flavour pci_pf pf 0 subport 1 <<<<<<<<<<<<<<<<<<
switch_id 00154d130d2f peer pci/0000:05:00.0/2 <<<<<<<<<<<<<<<<<<
To create a new VF under PF 0:
$ devlink dev port add pci/0000:05:00.0 flavour pci_vf pf 0
$ devlink port show
pci/0000:05:00.0/0: type eth netdev enp5s0np0 flavour physical switch_id 00154d130d2
pci/0000:05:00.0/1: type eth netdev ??? flavour pci_pf_host
peer pci/0000:05:00.0/10000
pci/0000:05:00.0/10000: type eth netdev enp5s0npf0pf0s0 flavour pci_pf pf 0 subport 0
switch_id 00154d130d2f peer pci/0000:05:00.0/1
pci/0000:05:00.0/2: type eth netdev ??? flavour pci_pf_host
peer pci/0000:05:00.0/10001
pci/0000:05:00.0/10001: type eth netdev enp5s0npf0pf0s0 flavour pci_pf pf 0 subport 1
switch_id 00154d130d2f peer pci/0000:05:00.0/2
pci/0000:05:10.1/0: type eth netdev ??? flavour pci_vf_host <<<<<<<<<<<<<<<<<<
peer pci/0000:05:00.0/10002 <<<<<<<<<<<<<<<<<<
pci/0000:05:00.0/10002: type eth netdev enp5s0npf0pf0s0 flavour pci_vf pf 0 vf 0 <<<<<<<<<<<<<<<<<<
switch_id 00154d130d2f peer pci/0000:05:10.1/0 <<<<<<<<<<<<<<<<<<
So new VF is created.
To delete, use would need to use the port which is in eswitch:
$ devlink port del pci/0000:05:00.0/2
devlink answers: Operation not permitted
$ devlink port del pci/0000:05:00.0/10001 <<<<<<<<<< this
$ devlink port del pci/0000:05:10.1/0
devlink answers: Operation not permitted
$ devlink port del pci/0000:05:00.0/10002 <<<<<<<<<< this
This actually removes VF.
For VF subports this would work too, we just have to have "subport"
attribute not only for PFs but also for VFs:
To create a new VF subport under PF 0 and VF 0:
$ devlink dev port add pci/0000:05:00.0 flavour pci_vf pf 0 vf 0
$ devlink port show
pci/0000:05:00.0/0: type eth netdev enp5s0np0 flavour physical switch_id 00154d130d2
pci/0000:05:00.0/1: type eth netdev ??? flavour pci_pf_host
peer pci/0000:05:00.0/10000
pci/0000:05:00.0/10000: type eth netdev enp5s0npf0pf0s0 flavour pci_pf pf 0 subport 0
switch_id 00154d130d2f peer pci/0000:05:00.0/1
pci/0000:05:00.0/2: type eth netdev ??? flavour pci_pf_host
peer pci/0000:05:00.0/10001
pci/0000:05:00.0/10001: type eth netdev enp5s0npf0pf0s0 flavour pci_pf pf 0 subport 1
switch_id 00154d130d2f peer pci/0000:05:00.0/2
pci/0000:05:10.1/0: type eth netdev ??? flavour pci_vf_host
peer pci/0000:05:00.0/10002
pci/0000:05:00.0/10002: type eth netdev enp5s0npf0pf0s0 flavour pci_vf pf 0 vf 0 subport 0
switch_id 00154d130d2f peer pci/0000:05:10.1/0
pci/0000:05:10.1/1: type eth netdev ??? flavour pci_vf_host <<<<<<<<<<<<<<<<<<
peer pci/0000:05:00.0/10003 <<<<<<<<<<<<<<<<<<
pci/0000:05:00.0/10003: type eth netdev enp5s0npf0pf0s0 flavour pci_vf pf 0 vf 0 subport 1 <<<<<<<<<<<<<<<<<<
switch_id 00154d130d2f peer pci/0000:05:10.1/1 <<<<<<<<<<<<<<<<<<
^ permalink raw reply [flat|nested] 100+ messages in thread
* Re: [PATCH net-next v2 4/7] devlink: allow subports on devlink PCI ports
2019-03-12 14:02 ` Jiri Pirko
@ 2019-03-12 20:56 ` Jakub Kicinski
2019-03-13 6:07 ` Jiri Pirko
[not found] ` <7227d58e-ac58-d549-b921-ca0a0dd3f4b0@intel.com>
1 sibling, 1 reply; 100+ messages in thread
From: Jakub Kicinski @ 2019-03-12 20:56 UTC (permalink / raw)
To: Jiri Pirko; +Cc: davem, netdev, oss-drivers
On Tue, 12 Mar 2019 15:02:39 +0100, Jiri Pirko wrote:
> Tue, Mar 12, 2019 at 03:10:54AM CET, wrote:
> >On Mon, 11 Mar 2019 09:52:04 +0100, Jiri Pirko wrote:
> >> Fri, Mar 08, 2019 at 08:09:43PM CET, wrote:
> >> >If the switchport is in the hypervisor then only the hypervisor can
> >> >control switching/forwarding, correct?
> >>
> >> Correct.
> >>
> >> >The primary use case for partitioning within a VM (of a VF) would be
> >> >containers (and DPDK)?
> >>
> >> Makes sense.
> >>
> >> >SR-IOV makes things harder. Splitting a PF is reasonably easy to grasp.
> >> >I'm trying to get a sense of is how would we control an SR-IOV
> >> >environment as a whole.
> >>
> >> You mean orchestration?
> >
> >Right, orchestration.
> >
> >To be clear on where I'm going with this - if we want to allow VFs
> >to partition themselves then they have to control what is effectively
> >a "nested" switch. A per-VF set of rules which would the get
>
> Wait. If you allow to make VF subports (I believe that is what you ment
> by VFs partition themselves), that does not mean they will have a
> separate nested switch. They would still belong under the same one.
But that existing switch is administered by the hypervisor, how would
the VF owners install forwarding rules in a switch they don't control?
> >"flattened" into the main eswitch rule set. If I was to choose I'd
> >really rather have this "flattening" be done on the (Linux) hypervisor
> >and not in the vendor driver and firmware.
>
> Agreed. Driver should provide one big switch. User should configure it.
Cool, when you say user - is it the tenant or the provider?
> >I'd much rather have the VM make a "give me another NIC" orchestration
> >call via some high level REST API than devlink. This makes the
> >configuration strictly high level to low level:
> >
> > VM -> cloud net REST API -> cloud agent -> devlink/Linux -> FW -> HW
> >
> >Without round trips via firmware.
>
> Okay. So the "devlink/Linux -> FW" part is going to happen on baremetal.
> Makes sense.
>
> >This allows for easy policy enforcement, common code to be maintained
> >in user space, in high level languages (no 0.5M LoC drivers and 10M LoC
> >firmware for every driver). It can also be used with software paths
> >like VirtIO..
>
> Agreed.
>
> >Modelling and debugging a nested switch would be a nightmare. What
> >follows is that we probably shouldn't deal with partitioning of VFs,
> >but rather only partition via the PF devlink instance, and reassign
> >the partitions to VMs.
>
> Agreed. That must be misunderstanding, I never suggested nested
> switches.
Cool, yes, I was making sure we weren't going in that direction :)
> >> I originally planned to implement sriov orchestration api in devlink too.
> >
> >Interesting, would you mind elaborating?
>
> I have to think about it. But something like this:
> [...]
I see thanks for the examples, they makes things clear!
^ permalink raw reply [flat|nested] 100+ messages in thread
* Re: [PATCH net-next v2 4/7] devlink: allow subports on devlink PCI ports
2019-03-12 20:56 ` Jakub Kicinski
@ 2019-03-13 6:07 ` Jiri Pirko
2019-03-13 16:17 ` Jakub Kicinski
0 siblings, 1 reply; 100+ messages in thread
From: Jiri Pirko @ 2019-03-13 6:07 UTC (permalink / raw)
To: Jakub Kicinski; +Cc: davem, netdev, oss-drivers
Tue, Mar 12, 2019 at 09:56:28PM CET, jakub.kicinski@netronome.com wrote:
>On Tue, 12 Mar 2019 15:02:39 +0100, Jiri Pirko wrote:
>> Tue, Mar 12, 2019 at 03:10:54AM CET, wrote:
>> >On Mon, 11 Mar 2019 09:52:04 +0100, Jiri Pirko wrote:
>> >> Fri, Mar 08, 2019 at 08:09:43PM CET, wrote:
>> >> >If the switchport is in the hypervisor then only the hypervisor can
>> >> >control switching/forwarding, correct?
>> >>
>> >> Correct.
>> >>
>> >> >The primary use case for partitioning within a VM (of a VF) would be
>> >> >containers (and DPDK)?
>> >>
>> >> Makes sense.
>> >>
>> >> >SR-IOV makes things harder. Splitting a PF is reasonably easy to grasp.
>> >> >I'm trying to get a sense of is how would we control an SR-IOV
>> >> >environment as a whole.
>> >>
>> >> You mean orchestration?
>> >
>> >Right, orchestration.
>> >
>> >To be clear on where I'm going with this - if we want to allow VFs
>> >to partition themselves then they have to control what is effectively
>> >a "nested" switch. A per-VF set of rules which would the get
>>
>> Wait. If you allow to make VF subports (I believe that is what you ment
>> by VFs partition themselves), that does not mean they will have a
>> separate nested switch. They would still belong under the same one.
>
>But that existing switch is administered by the hypervisor, how would
>the VF owners install forwarding rules in a switch they don't control?
They won't.
>
>> >"flattened" into the main eswitch rule set. If I was to choose I'd
>> >really rather have this "flattening" be done on the (Linux) hypervisor
>> >and not in the vendor driver and firmware.
>>
>> Agreed. Driver should provide one big switch. User should configure it.
>
>Cool, when you say user - is it the tenant or the provider?
Whoever gets access to the instance.
>
>> >I'd much rather have the VM make a "give me another NIC" orchestration
>> >call via some high level REST API than devlink. This makes the
>> >configuration strictly high level to low level:
>> >
>> > VM -> cloud net REST API -> cloud agent -> devlink/Linux -> FW -> HW
>> >
>> >Without round trips via firmware.
>>
>> Okay. So the "devlink/Linux -> FW" part is going to happen on baremetal.
>> Makes sense.
>>
>> >This allows for easy policy enforcement, common code to be maintained
>> >in user space, in high level languages (no 0.5M LoC drivers and 10M LoC
>> >firmware for every driver). It can also be used with software paths
>> >like VirtIO..
>>
>> Agreed.
>>
>> >Modelling and debugging a nested switch would be a nightmare. What
>> >follows is that we probably shouldn't deal with partitioning of VFs,
>> >but rather only partition via the PF devlink instance, and reassign
>> >the partitions to VMs.
>>
>> Agreed. That must be misunderstanding, I never suggested nested
>> switches.
>
>Cool, yes, I was making sure we weren't going in that direction :)
Okay.
>
>> >> I originally planned to implement sriov orchestration api in devlink too.
>> >
>> >Interesting, would you mind elaborating?
>>
>> I have to think about it. But something like this:
>> [...]
>
>I see thanks for the examples, they makes things clear!
Okay. I will put together some documentation including this. I have some
patches that implement some of the stuff. Your patchset also does some
of that (considering you adjust a thing or two). Lets make this right.
^ permalink raw reply [flat|nested] 100+ messages in thread
* Re: [PATCH net-next v2 4/7] devlink: allow subports on devlink PCI ports
[not found] ` <7227d58e-ac58-d549-b921-ca0a0dd3f4b0@intel.com>
@ 2019-03-13 7:37 ` Jiri Pirko
2019-03-13 16:03 ` Samudrala, Sridhar
0 siblings, 1 reply; 100+ messages in thread
From: Jiri Pirko @ 2019-03-13 7:37 UTC (permalink / raw)
To: Samudrala, Sridhar; +Cc: Jakub Kicinski, davem, netdev, oss-drivers
Wed, Mar 13, 2019 at 07:17:04AM CET, sridhar.samudrala@intel.com wrote:
>
>On 3/12/2019 7:02 AM, Jiri Pirko wrote:
>
>>
>> >
>> > > I originally planned to implement sriov orchestration api in devlink too.
>> >
>> > Interesting, would you mind elaborating?
>>
>> I have to think about it. But something like this:
>>
>> After bootup, you see only physical port, PF switch port and PF host leg.
>
>Is this after changing the eswitch mode to 'switchdev'
I believe so. For new drivers, this should be default and only option.
>
>> $ devlink port show
>> pci/0000:05:00.0/0: type eth netdev enp5s0np0 flavour physical switch_id 00154d130d2
>
>Is this the uplink port representor?
Yes
>
>> pci/0000:05:00.0/1: type eth netdev ??? flavour pci_pf_host
>> peer pci/0000:05:00.0/10000
>
>I guess this is PF netdev
Yes, port
>
>> pci/0000:05:00.0/10000: type eth netdev enp5s0npf0pf0s0 flavour pci_pf pf 0 subport 0
>> switch_id 00154d130d2f peer pci/0000:05:00.0/1
>
>and this one is PF port representor netdev
Yes, port
>
>>
>> To create new PF subport under PF 0:
>> $ devlink dev port add pci/0000:05:00.0 flavour pci_pf pf 0
>
>Can we consider l2-fwd offload macvlan device also as a subport of PF?
What does this have to with with macvlan? Macvlan is a separate soft
driver.
>
>> $ devlink port show
>> pci/0000:05:00.0/0: type eth netdev enp5s0np0 flavour physical switch_id 00154d130d2
>> pci/0000:05:00.0/1: type eth netdev ??? flavour pci_pf_host
>> peer pci/0000:05:00.0/10000
>> pci/0000:05:00.0/10000: type eth netdev enp5s0npf0pf0s0 flavour pci_pf pf 0 subport 0
>> switch_id 00154d130d2f peer pci/0000:05:00.0/1
>> pci/0000:05:00.0/2: type eth netdev ??? flavour pci_pf_host <<<<<<<<<<<<<<<<<<
>> peer pci/0000:05:00.0/10001 <<<<<<<<<<<<<<<<<<
>> pci/0000:05:00.0/10001: type eth netdev enp5s0npf0pf0s0 flavour pci_pf pf 0 subport 1 <<<<<<<<<<<<<<<<<<
>> switch_id 00154d130d2f peer pci/0000:05:00.0/2 <<<<<<<<<<<<<<<<<<
>>
>> To create a new VF under PF 0:
>> $ devlink dev port add pci/0000:05:00.0 flavour pci_vf pf 0
>
>Does this mean that this interface allows creating VFs dynamically one at a
>time?
Yes
>
>> $ devlink port show
>> pci/0000:05:00.0/0: type eth netdev enp5s0np0 flavour physical switch_id 00154d130d2
>> pci/0000:05:00.0/1: type eth netdev ??? flavour pci_pf_host
>> peer pci/0000:05:00.0/10000
>> pci/0000:05:00.0/10000: type eth netdev enp5s0npf0pf0s0 flavour pci_pf pf 0 subport 0
>> switch_id 00154d130d2f peer pci/0000:05:00.0/1
>> pci/0000:05:00.0/2: type eth netdev ??? flavour pci_pf_host
>> peer pci/0000:05:00.0/10001
>> pci/0000:05:00.0/10001: type eth netdev enp5s0npf0pf0s0 flavour pci_pf pf 0 subport 1
>> switch_id 00154d130d2f peer pci/0000:05:00.0/2
>> pci/0000:05:10.1/0: type eth netdev ??? flavour pci_vf_host <<<<<<<<<<<<<<<<<<
>> peer pci/0000:05:00.0/10002 <<<<<<<<<<<<<<<<<<
>> pci/0000:05:00.0/10002: type eth netdev enp5s0npf0pf0s0 flavour pci_vf pf 0 vf 0 <<<<<<<<<<<<<<<<<<
>> switch_id 00154d130d2f peer pci/0000:05:10.1/0 <<<<<<<<<<<<<<<<<<
>>
>> So new VF is created.
>>
>>
>> To delete, use would need to use the port which is in eswitch:
>> $ devlink port del pci/0000:05:00.0/2
>> devlink answers: Operation not permitted
>> $ devlink port del pci/0000:05:00.0/10001 <<<<<<<<<< this
>>
>> $ devlink port del pci/0000:05:10.1/0
>> devlink answers: Operation not permitted
>> $ devlink port del pci/0000:05:00.0/10002 <<<<<<<<<< this
>>
>> This actually removes VF.
>>
>>
>> For VF subports this would work too, we just have to have "subport"
>> attribute not only for PFs but also for VFs:
>>
>> To create a new VF subport under PF 0 and VF 0:
>> $ devlink dev port add pci/0000:05:00.0 flavour pci_vf pf 0 vf 0
>> $ devlink port show
>> pci/0000:05:00.0/0: type eth netdev enp5s0np0 flavour physical switch_id 00154d130d2
>> pci/0000:05:00.0/1: type eth netdev ??? flavour pci_pf_host
>> peer pci/0000:05:00.0/10000
>> pci/0000:05:00.0/10000: type eth netdev enp5s0npf0pf0s0 flavour pci_pf pf 0 subport 0
>> switch_id 00154d130d2f peer pci/0000:05:00.0/1
>> pci/0000:05:00.0/2: type eth netdev ??? flavour pci_pf_host
>> peer pci/0000:05:00.0/10001
>> pci/0000:05:00.0/10001: type eth netdev enp5s0npf0pf0s0 flavour pci_pf pf 0 subport 1
>> switch_id 00154d130d2f peer pci/0000:05:00.0/2
>> pci/0000:05:10.1/0: type eth netdev ??? flavour pci_vf_host
>> peer pci/0000:05:00.0/10002
>> pci/0000:05:00.0/10002: type eth netdev enp5s0npf0pf0s0 flavour pci_vf pf 0 vf 0 subport 0
>> switch_id 00154d130d2f peer pci/0000:05:10.1/0
>> pci/0000:05:10.1/1: type eth netdev ??? flavour pci_vf_host <<<<<<<<<<<<<<<<<<
>> peer pci/0000:05:00.0/10003 <<<<<<<<<<<<<<<<<<
>> pci/0000:05:00.0/10003: type eth netdev enp5s0npf0pf0s0 flavour pci_vf pf 0 vf 0 subport 1 <<<<<<<<<<<<<<<<<<
>> switch_id 00154d130d2f peer pci/0000:05:10.1/1 <<<<<<<<<<<<<<<<<<
>>
>>
^ permalink raw reply [flat|nested] 100+ messages in thread
* Re: [PATCH net-next v2 4/7] devlink: allow subports on devlink PCI ports
2019-03-13 7:37 ` Jiri Pirko
@ 2019-03-13 16:03 ` Samudrala, Sridhar
2019-03-13 16:24 ` Jiri Pirko
0 siblings, 1 reply; 100+ messages in thread
From: Samudrala, Sridhar @ 2019-03-13 16:03 UTC (permalink / raw)
To: Jiri Pirko; +Cc: Jakub Kicinski, davem, netdev, oss-drivers
On 3/13/2019 12:37 AM, Jiri Pirko wrote:
> Wed, Mar 13, 2019 at 07:17:04AM CET, sridhar.samudrala@intel.com wrote:
>>
>> On 3/12/2019 7:02 AM, Jiri Pirko wrote:
>>
>>>
>>>>
>>>>> I originally planned to implement sriov orchestration api in devlink too.
>>>>
>>>> Interesting, would you mind elaborating?
>>>
>>> I have to think about it. But something like this:
>>>
>>> After bootup, you see only physical port, PF switch port and PF host leg.
>>
>> Is this after changing the eswitch mode to 'switchdev'
>
> I believe so. For new drivers, this should be default and only option.
>
>
>>
>>> $ devlink port show
>>> pci/0000:05:00.0/0: type eth netdev enp5s0np0 flavour physical switch_id 00154d130d2
>>
>> Is this the uplink port representor?
>
> Yes
>
>
>>
>>> pci/0000:05:00.0/1: type eth netdev ??? flavour pci_pf_host
>>> peer pci/0000:05:00.0/10000
>>
>> I guess this is PF netdev
>
> Yes, port
>
>
>>
>>> pci/0000:05:00.0/10000: type eth netdev enp5s0npf0pf0s0 flavour pci_pf pf 0 subport 0
>>> switch_id 00154d130d2f peer pci/0000:05:00.0/1
>>
>> and this one is PF port representor netdev
>
> Yes, port
>
>
>>
>>>
>>> To create new PF subport under PF 0:
>>> $ devlink dev port add pci/0000:05:00.0 flavour pci_pf pf 0
>>
>> Can we consider l2-fwd offload macvlan device also as a subport of PF?
>
> What does this have to with with macvlan? Macvlan is a separate soft
> driver.
ethtool -k <pf> l2-fwd-offload on
ip link add link <pf> type macvlan
will create a macvlan netdev but it is backed by a set of separate HW
queues and switching is offloaded to HW. This can be considered as a
subport. In i40e, it is a VMDq VSI.
^ permalink raw reply [flat|nested] 100+ messages in thread
* Re: [PATCH net-next v2 4/7] devlink: allow subports on devlink PCI ports
2019-03-13 6:07 ` Jiri Pirko
@ 2019-03-13 16:17 ` Jakub Kicinski
2019-03-13 16:22 ` Jiri Pirko
0 siblings, 1 reply; 100+ messages in thread
From: Jakub Kicinski @ 2019-03-13 16:17 UTC (permalink / raw)
To: Jiri Pirko; +Cc: davem, netdev, oss-drivers
On Wed, 13 Mar 2019 07:07:01 +0100, Jiri Pirko wrote:
> Tue, Mar 12, 2019 at 09:56:28PM CET, jakub.kicinski@netronome.com wrote:
> >On Tue, 12 Mar 2019 15:02:39 +0100, Jiri Pirko wrote:
> >> Tue, Mar 12, 2019 at 03:10:54AM CET, wrote:
> >> >On Mon, 11 Mar 2019 09:52:04 +0100, Jiri Pirko wrote:
> >> >> Fri, Mar 08, 2019 at 08:09:43PM CET, wrote:
> >> >> >If the switchport is in the hypervisor then only the hypervisor can
> >> >> >control switching/forwarding, correct?
> >> >>
> >> >> Correct.
> >> >>
> >> >> >The primary use case for partitioning within a VM (of a VF) would be
> >> >> >containers (and DPDK)?
> >> >>
> >> >> Makes sense.
> >> >>
> >> >> >SR-IOV makes things harder. Splitting a PF is reasonably easy to grasp.
> >> >> >I'm trying to get a sense of is how would we control an SR-IOV
> >> >> >environment as a whole.
> >> >>
> >> >> You mean orchestration?
> >> >
> >> >Right, orchestration.
> >> >
> >> >To be clear on where I'm going with this - if we want to allow VFs
> >> >to partition themselves then they have to control what is effectively
> >> >a "nested" switch. A per-VF set of rules which would the get
> >>
> >> Wait. If you allow to make VF subports (I believe that is what you ment
> >> by VFs partition themselves), that does not mean they will have a
> >> separate nested switch. They would still belong under the same one.
> >
> >But that existing switch is administered by the hypervisor, how would
> >the VF owners install forwarding rules in a switch they don't control?
>
> They won't.
Argh. So how is forwarding configured if there are no rules? Are you
going to assume its switching on MACs? We're supposed to offload
software constructs. If its a software port it needs to be explicitly
switched. If it's not explicitly switched - we already have macvlan
offload.
> >> >"flattened" into the main eswitch rule set. If I was to choose I'd
> >> >really rather have this "flattening" be done on the (Linux) hypervisor
> >> >and not in the vendor driver and firmware.
> >>
> >> Agreed. Driver should provide one big switch. User should configure it.
> >
> >Cool, when you say user - is it the tenant or the provider?
>
> Whoever gets access to the instance.
>
> >> >I'd much rather have the VM make a "give me another NIC" orchestration
> >> >call via some high level REST API than devlink. This makes the
> >> >configuration strictly high level to low level:
> >> >
> >> > VM -> cloud net REST API -> cloud agent -> devlink/Linux -> FW -> HW
> >> >
> >> >Without round trips via firmware.
> >>
> >> Okay. So the "devlink/Linux -> FW" part is going to happen on baremetal.
> >> Makes sense.
> >>
> >> >This allows for easy policy enforcement, common code to be maintained
> >> >in user space, in high level languages (no 0.5M LoC drivers and 10M LoC
> >> >firmware for every driver). It can also be used with software paths
> >> >like VirtIO..
> >>
> >> Agreed.
> >>
> >> >Modelling and debugging a nested switch would be a nightmare. What
> >> >follows is that we probably shouldn't deal with partitioning of VFs,
> >> >but rather only partition via the PF devlink instance, and reassign
> >> >the partitions to VMs.
> >>
> >> Agreed. That must be misunderstanding, I never suggested nested
> >> switches.
> >
> >Cool, yes, I was making sure we weren't going in that direction :)
>
> Okay.
>
> >> >> I originally planned to implement sriov orchestration api in devlink too.
> >> >
> >> >Interesting, would you mind elaborating?
> >>
> >> I have to think about it. But something like this:
> >> [...]
> >
> >I see thanks for the examples, they makes things clear!
>
> Okay. I will put together some documentation including this. I have some
> patches that implement some of the stuff. Your patchset also does some
> of that (considering you adjust a thing or two). Lets make this right.
Yeah, I feel like I'm again getting further from clarity on what you're
trying to achieve.
^ permalink raw reply [flat|nested] 100+ messages in thread
* Re: [PATCH net-next v2 4/7] devlink: allow subports on devlink PCI ports
2019-03-13 16:17 ` Jakub Kicinski
@ 2019-03-13 16:22 ` Jiri Pirko
2019-03-13 16:55 ` Jakub Kicinski
0 siblings, 1 reply; 100+ messages in thread
From: Jiri Pirko @ 2019-03-13 16:22 UTC (permalink / raw)
To: Jakub Kicinski; +Cc: davem, netdev, oss-drivers
Wed, Mar 13, 2019 at 05:17:31PM CET, jakub.kicinski@netronome.com wrote:
>On Wed, 13 Mar 2019 07:07:01 +0100, Jiri Pirko wrote:
>> Tue, Mar 12, 2019 at 09:56:28PM CET, jakub.kicinski@netronome.com wrote:
>> >On Tue, 12 Mar 2019 15:02:39 +0100, Jiri Pirko wrote:
>> >> Tue, Mar 12, 2019 at 03:10:54AM CET, wrote:
>> >> >On Mon, 11 Mar 2019 09:52:04 +0100, Jiri Pirko wrote:
>> >> >> Fri, Mar 08, 2019 at 08:09:43PM CET, wrote:
>> >> >> >If the switchport is in the hypervisor then only the hypervisor can
>> >> >> >control switching/forwarding, correct?
>> >> >>
>> >> >> Correct.
>> >> >>
>> >> >> >The primary use case for partitioning within a VM (of a VF) would be
>> >> >> >containers (and DPDK)?
>> >> >>
>> >> >> Makes sense.
>> >> >>
>> >> >> >SR-IOV makes things harder. Splitting a PF is reasonably easy to grasp.
>> >> >> >I'm trying to get a sense of is how would we control an SR-IOV
>> >> >> >environment as a whole.
>> >> >>
>> >> >> You mean orchestration?
>> >> >
>> >> >Right, orchestration.
>> >> >
>> >> >To be clear on where I'm going with this - if we want to allow VFs
>> >> >to partition themselves then they have to control what is effectively
>> >> >a "nested" switch. A per-VF set of rules which would the get
>> >>
>> >> Wait. If you allow to make VF subports (I believe that is what you ment
>> >> by VFs partition themselves), that does not mean they will have a
>> >> separate nested switch. They would still belong under the same one.
>> >
>> >But that existing switch is administered by the hypervisor, how would
>> >the VF owners install forwarding rules in a switch they don't control?
>>
>> They won't.
>
>Argh. So how is forwarding configured if there are no rules? Are you
>going to assume its switching on MACs? We're supposed to offload
>software constructs. If its a software port it needs to be explicitly
>switched. If it's not explicitly switched - we already have macvlan
>offload.
Wait a second. You configure the switch. And for that, you have the
switchports (representors). What we are talking about are VF (VF
subport) host legs. Am I missing something?
>
>> >> >"flattened" into the main eswitch rule set. If I was to choose I'd
>> >> >really rather have this "flattening" be done on the (Linux) hypervisor
>> >> >and not in the vendor driver and firmware.
>> >>
>> >> Agreed. Driver should provide one big switch. User should configure it.
>> >
>> >Cool, when you say user - is it the tenant or the provider?
>>
>> Whoever gets access to the instance.
>>
>> >> >I'd much rather have the VM make a "give me another NIC" orchestration
>> >> >call via some high level REST API than devlink. This makes the
>> >> >configuration strictly high level to low level:
>> >> >
>> >> > VM -> cloud net REST API -> cloud agent -> devlink/Linux -> FW -> HW
>> >> >
>> >> >Without round trips via firmware.
>> >>
>> >> Okay. So the "devlink/Linux -> FW" part is going to happen on baremetal.
>> >> Makes sense.
>> >>
>> >> >This allows for easy policy enforcement, common code to be maintained
>> >> >in user space, in high level languages (no 0.5M LoC drivers and 10M LoC
>> >> >firmware for every driver). It can also be used with software paths
>> >> >like VirtIO..
>> >>
>> >> Agreed.
>> >>
>> >> >Modelling and debugging a nested switch would be a nightmare. What
>> >> >follows is that we probably shouldn't deal with partitioning of VFs,
>> >> >but rather only partition via the PF devlink instance, and reassign
>> >> >the partitions to VMs.
>> >>
>> >> Agreed. That must be misunderstanding, I never suggested nested
>> >> switches.
>> >
>> >Cool, yes, I was making sure we weren't going in that direction :)
>>
>> Okay.
>>
>> >> >> I originally planned to implement sriov orchestration api in devlink too.
>> >> >
>> >> >Interesting, would you mind elaborating?
>> >>
>> >> I have to think about it. But something like this:
>> >> [...]
>> >
>> >I see thanks for the examples, they makes things clear!
>>
>> Okay. I will put together some documentation including this. I have some
>> patches that implement some of the stuff. Your patchset also does some
>> of that (considering you adjust a thing or two). Lets make this right.
>
>Yeah, I feel like I'm again getting further from clarity on what you're
>trying to achieve.
:)
^ permalink raw reply [flat|nested] 100+ messages in thread
* Re: [PATCH net-next v2 4/7] devlink: allow subports on devlink PCI ports
2019-03-13 16:03 ` Samudrala, Sridhar
@ 2019-03-13 16:24 ` Jiri Pirko
0 siblings, 0 replies; 100+ messages in thread
From: Jiri Pirko @ 2019-03-13 16:24 UTC (permalink / raw)
To: Samudrala, Sridhar; +Cc: Jakub Kicinski, davem, netdev, oss-drivers
Wed, Mar 13, 2019 at 05:03:17PM CET, sridhar.samudrala@intel.com wrote:
>On 3/13/2019 12:37 AM, Jiri Pirko wrote:
>> Wed, Mar 13, 2019 at 07:17:04AM CET, sridhar.samudrala@intel.com wrote:
>> >
>> > On 3/12/2019 7:02 AM, Jiri Pirko wrote:
>> >
>> > >
>> > > >
>> > > > > I originally planned to implement sriov orchestration api in devlink too.
>> > > >
>> > > > Interesting, would you mind elaborating?
>> > >
>> > > I have to think about it. But something like this:
>> > >
>> > > After bootup, you see only physical port, PF switch port and PF host leg.
>> >
>> > Is this after changing the eswitch mode to 'switchdev'
>>
>> I believe so. For new drivers, this should be default and only option.
>>
>>
>> >
>> > > $ devlink port show
>> > > pci/0000:05:00.0/0: type eth netdev enp5s0np0 flavour physical switch_id 00154d130d2
>> >
>> > Is this the uplink port representor?
>>
>> Yes
>>
>>
>> >
>> > > pci/0000:05:00.0/1: type eth netdev ??? flavour pci_pf_host
>> > > peer pci/0000:05:00.0/10000
>> >
>> > I guess this is PF netdev
>>
>> Yes, port
>>
>>
>> >
>> > > pci/0000:05:00.0/10000: type eth netdev enp5s0npf0pf0s0 flavour pci_pf pf 0 subport 0
>> > > switch_id 00154d130d2f peer pci/0000:05:00.0/1
>> >
>> > and this one is PF port representor netdev
>>
>> Yes, port
>>
>>
>> >
>> > >
>> > > To create new PF subport under PF 0:
>> > > $ devlink dev port add pci/0000:05:00.0 flavour pci_pf pf 0
>> >
>> > Can we consider l2-fwd offload macvlan device also as a subport of PF?
>>
>> What does this have to with with macvlan? Macvlan is a separate soft
>> driver.
>
>ethtool -k <pf> l2-fwd-offload on
>ip link add link <pf> type macvlan
>
>will create a macvlan netdev but it is backed by a set of separate HW queues
>and switching is offloaded to HW. This can be considered as a subport. In
>i40e, it is a VMDq VSI.
Oh, this one. I think that is abuse of macvlan. We should do the
modelling correctly, including visibility of switch ports.
>
>
>
>
^ permalink raw reply [flat|nested] 100+ messages in thread
* Re: [PATCH net-next v2 4/7] devlink: allow subports on devlink PCI ports
2019-03-13 16:22 ` Jiri Pirko
@ 2019-03-13 16:55 ` Jakub Kicinski
2019-03-14 7:38 ` Jiri Pirko
0 siblings, 1 reply; 100+ messages in thread
From: Jakub Kicinski @ 2019-03-13 16:55 UTC (permalink / raw)
To: Jiri Pirko; +Cc: davem, netdev, oss-drivers
On Wed, 13 Mar 2019 17:22:43 +0100, Jiri Pirko wrote:
> Wed, Mar 13, 2019 at 05:17:31PM CET, jakub.kicinski@netronome.com wrote:
> >On Wed, 13 Mar 2019 07:07:01 +0100, Jiri Pirko wrote:
> >> Tue, Mar 12, 2019 at 09:56:28PM CET, jakub.kicinski@netronome.com wrote:
> >> >On Tue, 12 Mar 2019 15:02:39 +0100, Jiri Pirko wrote:
> >> >> Tue, Mar 12, 2019 at 03:10:54AM CET, wrote:
> >> >> >On Mon, 11 Mar 2019 09:52:04 +0100, Jiri Pirko wrote:
> >> >> >> Fri, Mar 08, 2019 at 08:09:43PM CET, wrote:
> >> >> >> >If the switchport is in the hypervisor then only the hypervisor can
> >> >> >> >control switching/forwarding, correct?
> >> >> >>
> >> >> >> Correct.
> >> >> >>
> >> >> >> >The primary use case for partitioning within a VM (of a VF) would be
> >> >> >> >containers (and DPDK)?
> >> >> >>
> >> >> >> Makes sense.
> >> >> >>
> >> >> >> >SR-IOV makes things harder. Splitting a PF is reasonably easy to grasp.
> >> >> >> >I'm trying to get a sense of is how would we control an SR-IOV
> >> >> >> >environment as a whole.
> >> >> >>
> >> >> >> You mean orchestration?
> >> >> >
> >> >> >Right, orchestration.
> >> >> >
> >> >> >To be clear on where I'm going with this - if we want to allow VFs
> >> >> >to partition themselves then they have to control what is effectively
> >> >> >a "nested" switch. A per-VF set of rules which would the get
> >> >>
> >> >> Wait. If you allow to make VF subports (I believe that is what you ment
> >> >> by VFs partition themselves), that does not mean they will have a
> >> >> separate nested switch. They would still belong under the same one.
> >> >
> >> >But that existing switch is administered by the hypervisor, how would
> >> >the VF owners install forwarding rules in a switch they don't control?
> >>
> >> They won't.
> >
> >Argh. So how is forwarding configured if there are no rules? Are you
> >going to assume its switching on MACs? We're supposed to offload
> >software constructs. If its a software port it needs to be explicitly
> >switched. If it's not explicitly switched - we already have macvlan
> >offload.
>
> Wait a second. You configure the switch. And for that, you have the
> switchports (representors). What we are talking about are VF (VF
> subport) host legs. Am I missing something?
Hm :) So when VM gets a new port, how is it connected? Are we
assuming all ports of a VM are plugged into one big L2 switch?
The use case for those sub ports is a little murky, sorry about
the endless confusion :)
^ permalink raw reply [flat|nested] 100+ messages in thread
* Re: [PATCH net-next v2 4/7] devlink: allow subports on devlink PCI ports
2019-03-13 16:55 ` Jakub Kicinski
@ 2019-03-14 7:38 ` Jiri Pirko
2019-03-14 22:09 ` Jakub Kicinski
0 siblings, 1 reply; 100+ messages in thread
From: Jiri Pirko @ 2019-03-14 7:38 UTC (permalink / raw)
To: Jakub Kicinski; +Cc: davem, netdev, oss-drivers
Wed, Mar 13, 2019 at 05:55:55PM CET, jakub.kicinski@netronome.com wrote:
>On Wed, 13 Mar 2019 17:22:43 +0100, Jiri Pirko wrote:
>> Wed, Mar 13, 2019 at 05:17:31PM CET, jakub.kicinski@netronome.com wrote:
>> >On Wed, 13 Mar 2019 07:07:01 +0100, Jiri Pirko wrote:
>> >> Tue, Mar 12, 2019 at 09:56:28PM CET, jakub.kicinski@netronome.com wrote:
>> >> >On Tue, 12 Mar 2019 15:02:39 +0100, Jiri Pirko wrote:
>> >> >> Tue, Mar 12, 2019 at 03:10:54AM CET, wrote:
>> >> >> >On Mon, 11 Mar 2019 09:52:04 +0100, Jiri Pirko wrote:
>> >> >> >> Fri, Mar 08, 2019 at 08:09:43PM CET, wrote:
>> >> >> >> >If the switchport is in the hypervisor then only the hypervisor can
>> >> >> >> >control switching/forwarding, correct?
>> >> >> >>
>> >> >> >> Correct.
>> >> >> >>
>> >> >> >> >The primary use case for partitioning within a VM (of a VF) would be
>> >> >> >> >containers (and DPDK)?
>> >> >> >>
>> >> >> >> Makes sense.
>> >> >> >>
>> >> >> >> >SR-IOV makes things harder. Splitting a PF is reasonably easy to grasp.
>> >> >> >> >I'm trying to get a sense of is how would we control an SR-IOV
>> >> >> >> >environment as a whole.
>> >> >> >>
>> >> >> >> You mean orchestration?
>> >> >> >
>> >> >> >Right, orchestration.
>> >> >> >
>> >> >> >To be clear on where I'm going with this - if we want to allow VFs
>> >> >> >to partition themselves then they have to control what is effectively
>> >> >> >a "nested" switch. A per-VF set of rules which would the get
>> >> >>
>> >> >> Wait. If you allow to make VF subports (I believe that is what you ment
>> >> >> by VFs partition themselves), that does not mean they will have a
>> >> >> separate nested switch. They would still belong under the same one.
>> >> >
>> >> >But that existing switch is administered by the hypervisor, how would
>> >> >the VF owners install forwarding rules in a switch they don't control?
>> >>
>> >> They won't.
>> >
>> >Argh. So how is forwarding configured if there are no rules? Are you
>> >going to assume its switching on MACs? We're supposed to offload
>> >software constructs. If its a software port it needs to be explicitly
>> >switched. If it's not explicitly switched - we already have macvlan
>> >offload.
>>
>> Wait a second. You configure the switch. And for that, you have the
>> switchports (representors). What we are talking about are VF (VF
>> subport) host legs. Am I missing something?
>
>Hm :) So when VM gets a new port, how is it connected? Are we
>assuming all ports of a VM are plugged into one big L2 switch?
>The use case for those sub ports is a little murky, sorry about
>the endless confusion :)
Np. When user John (on baremetal, or whenever the devlink instance
with switch port is) creates VF of VF subport by:
$ devlink dev port add pci/0000:05:00.0 flavour pci_vf pf 0
or by:
$ devlink dev port add pci/0000:05:00.0 flavour pci_vf pf 0 vf 0
Then instances of flavour pci_vf are going to appear in the same devlink
instance. Those are the switch ports:
pci/0000:05:00.0/10002: type eth netdev enp5s0npf0pf0s0
flavour pci_vf pf 0 vf 0
switch_id 00154d130d2f peer pci/0000:05:10.1/0
pci/0000:05:00.0/10003: type eth netdev enp5s0npf0pf0s0
flavour pci_vf pf 0 vf 0 subport 1
switch_id 00154d130d2f peer pci/0000:05:10.1/1
With that, peers are going to appear too, and those are the actual VF/VF
subport:
pci/0000:05:10.1/0: type eth netdev ??? flavour pci_vf_host
peer pci/0000:05:00.0/10002
pci/0000:05:10.1/1: type eth netdev ??? flavour pci_vf_host
peer pci/0000:05:00.0/10003
Later you can push this VF along with all subports to VM. So in VM, you
are going to see the VF like this:
$ devlink dev
pci/0000:00:08.0
$ devlink port
pci/0000:00:08.0/0: type eth netdev ??? flavour pci_vf_host
pci/0000:00:08.0/1: type eth netdev ??? flavour pci_vf_host
And back to your question of how are they connected in eswitch.
That is totally up to the original user John who did the creation.
He is in charge of the eswitch on baremetal, he would configure
the forwarding however he likes.
^ permalink raw reply [flat|nested] 100+ messages in thread
* Re: [PATCH net-next v2 4/7] devlink: allow subports on devlink PCI ports
2019-03-14 7:38 ` Jiri Pirko
@ 2019-03-14 22:09 ` Jakub Kicinski
2019-03-14 22:35 ` Parav Pandit
2019-03-15 7:00 ` Jiri Pirko
0 siblings, 2 replies; 100+ messages in thread
From: Jakub Kicinski @ 2019-03-14 22:09 UTC (permalink / raw)
To: Jiri Pirko; +Cc: davem, netdev, oss-drivers
On Thu, 14 Mar 2019 08:38:40 +0100, Jiri Pirko wrote:
> Wed, Mar 13, 2019 at 05:55:55PM CET, jakub.kicinski@netronome.com wrote:
> >On Wed, 13 Mar 2019 17:22:43 +0100, Jiri Pirko wrote:
> >> Wed, Mar 13, 2019 at 05:17:31PM CET, jakub.kicinski@netronome.com wrote:
> >> >On Wed, 13 Mar 2019 07:07:01 +0100, Jiri Pirko wrote:
> >> >> Tue, Mar 12, 2019 at 09:56:28PM CET, jakub.kicinski@netronome.com wrote:
> >> >> >On Tue, 12 Mar 2019 15:02:39 +0100, Jiri Pirko wrote:
> >> >> >> Tue, Mar 12, 2019 at 03:10:54AM CET, wrote:
> >> >> >> >On Mon, 11 Mar 2019 09:52:04 +0100, Jiri Pirko wrote:
> >> >> >> >> Fri, Mar 08, 2019 at 08:09:43PM CET, wrote:
> >> >> >> >> >If the switchport is in the hypervisor then only the hypervisor can
> >> >> >> >> >control switching/forwarding, correct?
> >> >> >> >>
> >> >> >> >> Correct.
> >> >> >> >>
> >> >> >> >> >The primary use case for partitioning within a VM (of a VF) would be
> >> >> >> >> >containers (and DPDK)?
> >> >> >> >>
> >> >> >> >> Makes sense.
> >> >> >> >>
> >> >> >> >> >SR-IOV makes things harder. Splitting a PF is reasonably easy to grasp.
> >> >> >> >> >I'm trying to get a sense of is how would we control an SR-IOV
> >> >> >> >> >environment as a whole.
> >> >> >> >>
> >> >> >> >> You mean orchestration?
> >> >> >> >
> >> >> >> >Right, orchestration.
> >> >> >> >
> >> >> >> >To be clear on where I'm going with this - if we want to allow VFs
> >> >> >> >to partition themselves then they have to control what is effectively
> >> >> >> >a "nested" switch. A per-VF set of rules which would the get
> >> >> >>
> >> >> >> Wait. If you allow to make VF subports (I believe that is what you ment
> >> >> >> by VFs partition themselves), that does not mean they will have a
> >> >> >> separate nested switch. They would still belong under the same one.
> >> >> >
> >> >> >But that existing switch is administered by the hypervisor, how would
> >> >> >the VF owners install forwarding rules in a switch they don't control?
> >> >>
> >> >> They won't.
> >> >
> >> >Argh. So how is forwarding configured if there are no rules? Are you
> >> >going to assume its switching on MACs? We're supposed to offload
> >> >software constructs. If its a software port it needs to be explicitly
> >> >switched. If it's not explicitly switched - we already have macvlan
> >> >offload.
> >>
> >> Wait a second. You configure the switch. And for that, you have the
> >> switchports (representors). What we are talking about are VF (VF
> >> subport) host legs. Am I missing something?
> >
> >Hm :) So when VM gets a new port, how is it connected? Are we
> >assuming all ports of a VM are plugged into one big L2 switch?
> >The use case for those sub ports is a little murky, sorry about
> >the endless confusion :)
>
> Np. When user John (on baremetal, or whenever the devlink instance
> with switch port is) creates VF of VF subport by:
> $ devlink dev port add pci/0000:05:00.0 flavour pci_vf pf 0
> or by:
> $ devlink dev port add pci/0000:05:00.0 flavour pci_vf pf 0 vf 0
>
> Then instances of flavour pci_vf are going to appear in the same devlink
> instance. Those are the switch ports:
> pci/0000:05:00.0/10002: type eth netdev enp5s0npf0pf0s0
> flavour pci_vf pf 0 vf 0
> switch_id 00154d130d2f peer pci/0000:05:10.1/0
> pci/0000:05:00.0/10003: type eth netdev enp5s0npf0pf0s0
> flavour pci_vf pf 0 vf 0 subport 1
> switch_id 00154d130d2f peer pci/0000:05:10.1/1
>
> With that, peers are going to appear too, and those are the actual VF/VF
> subport:
> pci/0000:05:10.1/0: type eth netdev ??? flavour pci_vf_host
> peer pci/0000:05:00.0/10002
> pci/0000:05:10.1/1: type eth netdev ??? flavour pci_vf_host
> peer pci/0000:05:00.0/10003
>
> Later you can push this VF along with all subports to VM. So in VM, you
> are going to see the VF like this:
> $ devlink dev
> pci/0000:00:08.0
> $ devlink port
> pci/0000:00:08.0/0: type eth netdev ??? flavour pci_vf_host
> pci/0000:00:08.0/1: type eth netdev ??? flavour pci_vf_host
>
> And back to your question of how are they connected in eswitch.
> That is totally up to the original user John who did the creation.
> He is in charge of the eswitch on baremetal, he would configure
> the forwarding however he likes.
Ack, so I think you're saying VM has to communicate to the cloud
environment to have this provisioned using some service API, not
a kernel API. That's what I wanted to confirm.
I don't see any benefit to having the "host ports" under devlink,
as such I think it's a matter of preference. I'll try to describe
the two options to Netronome's FAEs and see which one they find more
intuitive.
Makes sense?
^ permalink raw reply [flat|nested] 100+ messages in thread
* RE: [PATCH net-next v2 4/7] devlink: allow subports on devlink PCI ports
2019-03-14 22:09 ` Jakub Kicinski
@ 2019-03-14 22:35 ` Parav Pandit
2019-03-14 23:39 ` Jakub Kicinski
2019-03-15 7:00 ` Jiri Pirko
1 sibling, 1 reply; 100+ messages in thread
From: Parav Pandit @ 2019-03-14 22:35 UTC (permalink / raw)
To: Jakub Kicinski, Jiri Pirko; +Cc: davem, netdev, oss-drivers
Hi Jakub,
> -----Original Message-----
> From: netdev-owner@vger.kernel.org <netdev-owner@vger.kernel.org> On
> Behalf Of Jakub Kicinski
> Sent: Thursday, March 14, 2019 5:10 PM
> To: Jiri Pirko <jiri@resnulli.us>
> Cc: davem@davemloft.net; netdev@vger.kernel.org; oss-
> drivers@netronome.com
> Subject: Re: [PATCH net-next v2 4/7] devlink: allow subports on devlink PCI
> ports
>
> On Thu, 14 Mar 2019 08:38:40 +0100, Jiri Pirko wrote:
> > Wed, Mar 13, 2019 at 05:55:55PM CET, jakub.kicinski@netronome.com
> wrote:
> > >On Wed, 13 Mar 2019 17:22:43 +0100, Jiri Pirko wrote:
> > >> Wed, Mar 13, 2019 at 05:17:31PM CET, jakub.kicinski@netronome.com
> wrote:
> > >> >On Wed, 13 Mar 2019 07:07:01 +0100, Jiri Pirko wrote:
> > >> >> Tue, Mar 12, 2019 at 09:56:28PM CET,
> jakub.kicinski@netronome.com wrote:
> > >> >> >On Tue, 12 Mar 2019 15:02:39 +0100, Jiri Pirko wrote:
> > >> >> >> Tue, Mar 12, 2019 at 03:10:54AM CET, wrote:
> > >> >> >> >On Mon, 11 Mar 2019 09:52:04 +0100, Jiri Pirko wrote:
> > >> >> >> >> Fri, Mar 08, 2019 at 08:09:43PM CET, wrote:
> > >> >> >> >> >If the switchport is in the hypervisor then only the hypervisor
> can
> > >> >> >> >> >control switching/forwarding, correct?
> > >> >> >> >>
> > >> >> >> >> Correct.
> > >> >> >> >>
> > >> >> >> >> >The primary use case for partitioning within a VM (of a VF)
> would be
> > >> >> >> >> >containers (and DPDK)?
> > >> >> >> >>
> > >> >> >> >> Makes sense.
> > >> >> >> >>
> > >> >> >> >> >SR-IOV makes things harder. Splitting a PF is reasonably easy
> to grasp.
> > >> >> >> >> >I'm trying to get a sense of is how would we control an SR-IOV
> > >> >> >> >> >environment as a whole.
> > >> >> >> >>
> > >> >> >> >> You mean orchestration?
> > >> >> >> >
> > >> >> >> >Right, orchestration.
> > >> >> >> >
> > >> >> >> >To be clear on where I'm going with this - if we want to
> > >> >> >> >allow VFs to partition themselves then they have to control what
> is effectively
> > >> >> >> >a "nested" switch. A per-VF set of rules which would the get
> > >> >> >>
> > >> >> >> Wait. If you allow to make VF subports (I believe that is
> > >> >> >> what you ment by VFs partition themselves), that does not mean
> they will have a
> > >> >> >> separate nested switch. They would still belong under the same
> one.
> > >> >> >
> > >> >> >But that existing switch is administered by the hypervisor, how
> would
> > >> >> >the VF owners install forwarding rules in a switch they don't
> control?
> > >> >>
> > >> >> They won't.
> > >> >
> > >> >Argh. So how is forwarding configured if there are no rules? Are
> > >> >you going to assume its switching on MACs? We're supposed to
> > >> >offload software constructs. If its a software port it needs to
> > >> >be explicitly switched. If it's not explicitly switched - we already have
> macvlan
> > >> >offload.
> > >>
> > >> Wait a second. You configure the switch. And for that, you have the
> > >> switchports (representors). What we are talking about are VF (VF
> > >> subport) host legs. Am I missing something?
> > >
> > >Hm :) So when VM gets a new port, how is it connected? Are we
> > >assuming all ports of a VM are plugged into one big L2 switch?
> > >The use case for those sub ports is a little murky, sorry about the
> > >endless confusion :)
> >
> > Np. When user John (on baremetal, or whenever the devlink instance
> > with switch port is) creates VF of VF subport by:
> > $ devlink dev port add pci/0000:05:00.0 flavour pci_vf pf 0 or by:
> > $ devlink dev port add pci/0000:05:00.0 flavour pci_vf pf 0 vf 0
> >
> > Then instances of flavour pci_vf are going to appear in the same
> > devlink instance. Those are the switch ports:
> > pci/0000:05:00.0/10002: type eth netdev enp5s0npf0pf0s0
> > flavour pci_vf pf 0 vf 0
> > switch_id 00154d130d2f peer pci/0000:05:10.1/0
> > pci/0000:05:00.0/10003: type eth netdev enp5s0npf0pf0s0
> > flavour pci_vf pf 0 vf 0 subport 1
> > switch_id 00154d130d2f peer pci/0000:05:10.1/1
> >
> > With that, peers are going to appear too, and those are the actual
> > VF/VF
> > subport:
> > pci/0000:05:10.1/0: type eth netdev ??? flavour pci_vf_host
> > peer pci/0000:05:00.0/10002
> > pci/0000:05:10.1/1: type eth netdev ??? flavour pci_vf_host
> > peer pci/0000:05:00.0/10003
> >
> > Later you can push this VF along with all subports to VM. So in VM,
> > you are going to see the VF like this:
> > $ devlink dev
> > pci/0000:00:08.0
> > $ devlink port
> > pci/0000:00:08.0/0: type eth netdev ??? flavour pci_vf_host
> > pci/0000:00:08.0/1: type eth netdev ??? flavour pci_vf_host
> >
> > And back to your question of how are they connected in eswitch.
> > That is totally up to the original user John who did the creation.
> > He is in charge of the eswitch on baremetal, he would configure the
> > forwarding however he likes.
>
> Ack, so I think you're saying VM has to communicate to the cloud
> environment to have this provisioned using some service API, not a kernel
> API. That's what I wanted to confirm.
>
> I don't see any benefit to having the "host ports" under devlink, as such I
> think it's a matter of preference.
We need 'host ports' to configure parameters of this
host port which is not exposed by the rep-netdev.
Such as mac address.
> I'll try to describe the two options to
> Netronome's FAEs and see which one they find more intuitive.
>
> Makes sense?
^ permalink raw reply [flat|nested] 100+ messages in thread
* Re: [PATCH net-next v2 4/7] devlink: allow subports on devlink PCI ports
2019-03-14 22:35 ` Parav Pandit
@ 2019-03-14 23:39 ` Jakub Kicinski
2019-03-15 1:28 ` Parav Pandit
0 siblings, 1 reply; 100+ messages in thread
From: Jakub Kicinski @ 2019-03-14 23:39 UTC (permalink / raw)
To: Parav Pandit; +Cc: Jiri Pirko, davem, netdev, oss-drivers
On Thu, 14 Mar 2019 22:35:36 +0000, Parav Pandit wrote:
> > > Then instances of flavour pci_vf are going to appear in the same
> > > devlink instance. Those are the switch ports:
> > > pci/0000:05:00.0/10002: type eth netdev enp5s0npf0pf0s0
> > > flavour pci_vf pf 0 vf 0
> > > switch_id 00154d130d2f peer pci/0000:05:10.1/0
> > > pci/0000:05:00.0/10003: type eth netdev enp5s0npf0pf0s0
> > > flavour pci_vf pf 0 vf 0 subport 1
> > > switch_id 00154d130d2f peer pci/0000:05:10.1/1
> > >
> > > With that, peers are going to appear too, and those are the actual
> > > VF/VF
> > > subport:
> > > pci/0000:05:10.1/0: type eth netdev ??? flavour pci_vf_host
> > > peer pci/0000:05:00.0/10002
> > > pci/0000:05:10.1/1: type eth netdev ??? flavour pci_vf_host
> > > peer pci/0000:05:00.0/10003
> > >
> > > Later you can push this VF along with all subports to VM. So in VM,
> > > you are going to see the VF like this:
> > > $ devlink dev
> > > pci/0000:00:08.0
> > > $ devlink port
> > > pci/0000:00:08.0/0: type eth netdev ??? flavour pci_vf_host
> > > pci/0000:00:08.0/1: type eth netdev ??? flavour pci_vf_host
> > >
> > > And back to your question of how are they connected in eswitch.
> > > That is totally up to the original user John who did the creation.
> > > He is in charge of the eswitch on baremetal, he would configure the
> > > forwarding however he likes.
> >
> > Ack, so I think you're saying VM has to communicate to the cloud
> > environment to have this provisioned using some service API, not a kernel
> > API. That's what I wanted to confirm.
> >
> > I don't see any benefit to having the "host ports" under devlink, as such I
> > think it's a matter of preference.
>
> We need 'host ports' to configure parameters of this
> host port which is not exposed by the rep-netdev.
> Such as mac address.
Please look at the quote of what Jiri wrote above - the host port gets
passed to the VM, you can't use it as a handle to set the MAC.
The way to set the MAC remains:
# devlink port set pci/0000:05:00.0/10002 peer mac_addr 00:11:22:33:44:55
(using the port ids from above)
^ permalink raw reply [flat|nested] 100+ messages in thread
* RE: [PATCH net-next v2 4/7] devlink: allow subports on devlink PCI ports
2019-03-14 23:39 ` Jakub Kicinski
@ 2019-03-15 1:28 ` Parav Pandit
2019-03-15 1:31 ` Parav Pandit
2019-03-15 2:15 ` Samudrala, Sridhar
0 siblings, 2 replies; 100+ messages in thread
From: Parav Pandit @ 2019-03-15 1:28 UTC (permalink / raw)
To: Jakub Kicinski; +Cc: Jiri Pirko, davem, netdev, oss-drivers
> -----Original Message-----
> From: Jakub Kicinski <jakub.kicinski@netronome.com>
> Sent: Thursday, March 14, 2019 6:39 PM
> To: Parav Pandit <parav@mellanox.com>
> Cc: Jiri Pirko <jiri@resnulli.us>; davem@davemloft.net;
> netdev@vger.kernel.org; oss-drivers@netronome.com
> Subject: Re: [PATCH net-next v2 4/7] devlink: allow subports on devlink PCI
> ports
>
> On Thu, 14 Mar 2019 22:35:36 +0000, Parav Pandit wrote:
> > > > Then instances of flavour pci_vf are going to appear in the same
> > > > devlink instance. Those are the switch ports:
> > > > pci/0000:05:00.0/10002: type eth netdev enp5s0npf0pf0s0
> > > > flavour pci_vf pf 0 vf 0
> > > > switch_id 00154d130d2f peer
> > > > pci/0000:05:10.1/0
> > > > pci/0000:05:00.0/10003: type eth netdev enp5s0npf0pf0s0
> > > > flavour pci_vf pf 0 vf 0 subport 1
> > > > switch_id 00154d130d2f peer
> > > > pci/0000:05:10.1/1
> > > >
> > > > With that, peers are going to appear too, and those are the actual
> > > > VF/VF
> > > > subport:
> > > > pci/0000:05:10.1/0: type eth netdev ??? flavour pci_vf_host
> > > > peer pci/0000:05:00.0/10002
> > > > pci/0000:05:10.1/1: type eth netdev ??? flavour pci_vf_host
> > > > peer pci/0000:05:00.0/10003
> > > >
> > > > Later you can push this VF along with all subports to VM. So in
> > > > VM, you are going to see the VF like this:
> > > > $ devlink dev
> > > > pci/0000:00:08.0
> > > > $ devlink port
> > > > pci/0000:00:08.0/0: type eth netdev ??? flavour pci_vf_host
> > > > pci/0000:00:08.0/1: type eth netdev ??? flavour pci_vf_host
> > > >
> > > > And back to your question of how are they connected in eswitch.
> > > > That is totally up to the original user John who did the creation.
> > > > He is in charge of the eswitch on baremetal, he would configure
> > > > the forwarding however he likes.
> > >
> > > Ack, so I think you're saying VM has to communicate to the cloud
> > > environment to have this provisioned using some service API, not a
> > > kernel API. That's what I wanted to confirm.
> > >
> > > I don't see any benefit to having the "host ports" under devlink, as such I
> > > think it's a matter of preference.
> >
> > We need 'host ports' to configure parameters of this host port which
> > is not exposed by the rep-netdev.
> > Such as mac address.
>
> Please look at the quote of what Jiri wrote above - the host port gets passed
> to the VM, you can't use it as a handle to set the MAC.
>
> The way to set the MAC remains:
>
> # devlink port set pci/0000:05:00.0/10002 peer mac_addr 00:11:22:33:44:55
>
Even though it can be done, I think this is wrong model to program hostport mac address using eswitch port.
All devlink objects are control objects, so what is passed to VM is what is represented by devlink.
VF in the VM will anyway create its devlink object.
What is wrong in programming hostport?
It gives a very clear view to users of topology and objects.
Also eswitch is flat. There is no need of pf/vf flavour for port.
It doesn't make sense to define 'mdev' flavour which we are already working.
At eswitch level it is just a port, it happen to be connected to vf or pf or other objects, it doesn't matter.
Port should be flavoured as 'hostport' or 'switchport'.
> (using the port ids from above)
^ permalink raw reply [flat|nested] 100+ messages in thread
* RE: [PATCH net-next v2 4/7] devlink: allow subports on devlink PCI ports
2019-03-15 1:28 ` Parav Pandit
@ 2019-03-15 1:31 ` Parav Pandit
2019-03-15 2:15 ` Samudrala, Sridhar
1 sibling, 0 replies; 100+ messages in thread
From: Parav Pandit @ 2019-03-15 1:31 UTC (permalink / raw)
To: Jakub Kicinski; +Cc: Jiri Pirko, davem, netdev, oss-drivers
> -----Original Message-----
> From: Parav Pandit
> Sent: Thursday, March 14, 2019 8:29 PM
> To: 'Jakub Kicinski' <jakub.kicinski@netronome.com>
> Cc: Jiri Pirko <jiri@resnulli.us>; davem@davemloft.net;
> netdev@vger.kernel.org; oss-drivers@netronome.com
> Subject: RE: [PATCH net-next v2 4/7] devlink: allow subports on devlink PCI
> ports
>
>
>
> > -----Original Message-----
> > From: Jakub Kicinski <jakub.kicinski@netronome.com>
> > Sent: Thursday, March 14, 2019 6:39 PM
> > To: Parav Pandit <parav@mellanox.com>
> > Cc: Jiri Pirko <jiri@resnulli.us>; davem@davemloft.net;
> > netdev@vger.kernel.org; oss-drivers@netronome.com
> > Subject: Re: [PATCH net-next v2 4/7] devlink: allow subports on
> > devlink PCI ports
> >
> > On Thu, 14 Mar 2019 22:35:36 +0000, Parav Pandit wrote:
> > > > > Then instances of flavour pci_vf are going to appear in the same
> > > > > devlink instance. Those are the switch ports:
> > > > > pci/0000:05:00.0/10002: type eth netdev enp5s0npf0pf0s0
> > > > > flavour pci_vf pf 0 vf 0
> > > > > switch_id 00154d130d2f peer
> > > > > pci/0000:05:10.1/0
> > > > > pci/0000:05:00.0/10003: type eth netdev enp5s0npf0pf0s0
> > > > > flavour pci_vf pf 0 vf 0 subport 1
> > > > > switch_id 00154d130d2f peer
> > > > > pci/0000:05:10.1/1
> > > > >
> > > > > With that, peers are going to appear too, and those are the
> > > > > actual VF/VF
> > > > > subport:
> > > > > pci/0000:05:10.1/0: type eth netdev ??? flavour pci_vf_host
> > > > > peer pci/0000:05:00.0/10002
> > > > > pci/0000:05:10.1/1: type eth netdev ??? flavour pci_vf_host
> > > > > peer pci/0000:05:00.0/10003
> > > > >
> > > > > Later you can push this VF along with all subports to VM. So in
> > > > > VM, you are going to see the VF like this:
> > > > > $ devlink dev
> > > > > pci/0000:00:08.0
> > > > > $ devlink port
> > > > > pci/0000:00:08.0/0: type eth netdev ??? flavour pci_vf_host
> > > > > pci/0000:00:08.0/1: type eth netdev ??? flavour pci_vf_host
> > > > >
> > > > > And back to your question of how are they connected in eswitch.
> > > > > That is totally up to the original user John who did the creation.
> > > > > He is in charge of the eswitch on baremetal, he would configure
> > > > > the forwarding however he likes.
> > > >
> > > > Ack, so I think you're saying VM has to communicate to the cloud
> > > > environment to have this provisioned using some service API, not a
> > > > kernel API. That's what I wanted to confirm.
> > > >
> > > > I don't see any benefit to having the "host ports" under devlink,
> > > > as such I think it's a matter of preference.
> > >
> > > We need 'host ports' to configure parameters of this host port which
> > > is not exposed by the rep-netdev.
> > > Such as mac address.
> >
> > Please look at the quote of what Jiri wrote above - the host port gets
> > passed to the VM, you can't use it as a handle to set the MAC.
> >
> > The way to set the MAC remains:
> >
> > # devlink port set pci/0000:05:00.0/10002 peer mac_addr
> > 00:11:22:33:44:55
> >
> Even though it can be done, I think this is wrong model to program hostport
> mac address using eswitch port.
> All devlink objects are control objects, so what is passed to VM is what is
> represented by devlink.
> VF in the VM will anyway create its devlink object.
> What is wrong in programming hostport?
> It gives a very clear view to users of topology and objects.
>
This also make sense for rdma where there are only hostports, and it doesn't have any peer switch ports.
User needs to configure node guid and port_guid.
Programming port_guid for 'hostport' also aligns for this requirement.
We better not program hostport params using this convoluted indirect way.
> Also eswitch is flat. There is no need of pf/vf flavour for port.
> It doesn't make sense to define 'mdev' flavour which we are already working.
> At eswitch level it is just a port, it happen to be connected to vf or pf or
> other objects, it doesn't matter.
> Port should be flavoured as 'hostport' or 'switchport'.
>
>
> > (using the port ids from above)
^ permalink raw reply [flat|nested] 100+ messages in thread
* Re: [PATCH net-next v2 4/7] devlink: allow subports on devlink PCI ports
2019-03-15 1:28 ` Parav Pandit
2019-03-15 1:31 ` Parav Pandit
@ 2019-03-15 2:15 ` Samudrala, Sridhar
2019-03-15 2:40 ` Parav Pandit
1 sibling, 1 reply; 100+ messages in thread
From: Samudrala, Sridhar @ 2019-03-15 2:15 UTC (permalink / raw)
To: Parav Pandit, Jakub Kicinski; +Cc: Jiri Pirko, davem, netdev, oss-drivers
On 3/14/2019 6:28 PM, Parav Pandit wrote:
>
>
>> -----Original Message-----
>> From: Jakub Kicinski <jakub.kicinski@netronome.com>
>> Sent: Thursday, March 14, 2019 6:39 PM
>> To: Parav Pandit <parav@mellanox.com>
>> Cc: Jiri Pirko <jiri@resnulli.us>; davem@davemloft.net;
>> netdev@vger.kernel.org; oss-drivers@netronome.com
>> Subject: Re: [PATCH net-next v2 4/7] devlink: allow subports on devlink PCI
>> ports
>>
>> On Thu, 14 Mar 2019 22:35:36 +0000, Parav Pandit wrote:
>>>>> Then instances of flavour pci_vf are going to appear in the same
>>>>> devlink instance. Those are the switch ports:
>>>>> pci/0000:05:00.0/10002: type eth netdev enp5s0npf0pf0s0
>>>>> flavour pci_vf pf 0 vf 0
>>>>> switch_id 00154d130d2f peer
>>>>> pci/0000:05:10.1/0
>>>>> pci/0000:05:00.0/10003: type eth netdev enp5s0npf0pf0s0
>>>>> flavour pci_vf pf 0 vf 0 subport 1
>>>>> switch_id 00154d130d2f peer
>>>>> pci/0000:05:10.1/1
>>>>>
>>>>> With that, peers are going to appear too, and those are the actual
>>>>> VF/VF
>>>>> subport:
>>>>> pci/0000:05:10.1/0: type eth netdev ??? flavour pci_vf_host
>>>>> peer pci/0000:05:00.0/10002
>>>>> pci/0000:05:10.1/1: type eth netdev ??? flavour pci_vf_host
>>>>> peer pci/0000:05:00.0/10003
>>>>>
>>>>> Later you can push this VF along with all subports to VM. So in
>>>>> VM, you are going to see the VF like this:
>>>>> $ devlink dev
>>>>> pci/0000:00:08.0
>>>>> $ devlink port
>>>>> pci/0000:00:08.0/0: type eth netdev ??? flavour pci_vf_host
>>>>> pci/0000:00:08.0/1: type eth netdev ??? flavour pci_vf_host
>>>>>
>>>>> And back to your question of how are they connected in eswitch.
>>>>> That is totally up to the original user John who did the creation.
>>>>> He is in charge of the eswitch on baremetal, he would configure
>>>>> the forwarding however he likes.
>>>>
>>>> Ack, so I think you're saying VM has to communicate to the cloud
>>>> environment to have this provisioned using some service API, not a
>>>> kernel API. That's what I wanted to confirm.
>>>>
>>>> I don't see any benefit to having the "host ports" under devlink, as such I
>>>> think it's a matter of preference.
>>>
>>> We need 'host ports' to configure parameters of this host port which
>>> is not exposed by the rep-netdev.
>>> Such as mac address.
>>
>> Please look at the quote of what Jiri wrote above - the host port gets passed
>> to the VM, you can't use it as a handle to set the MAC.
>>
>> The way to set the MAC remains:
>>
>> # devlink port set pci/0000:05:00.0/10002 peer mac_addr 00:11:22:33:44:55
>>
> Even though it can be done, I think this is wrong model to program hostport mac address using eswitch port.
> All devlink objects are control objects, so what is passed to VM is what is represented by devlink.
> VF in the VM will anyway create its devlink object.
> What is wrong in programming hostport?
> It gives a very clear view to users of topology and objects.
The VF or any subport MAC address should be configured by the
orchestration layer that is running on the hypervisor and when a VF is
assigned to a VF, the host port is not visible to the hypervisor.
Currently we have ndo_set_vf_mac_addr api that works with PF netdev, but
i think we are trying to move away from that API and do all the
configuration via the port representor netdevs. As the mac address
cannot be configured using this netdev, i think Jakub is suggesting
creating a devlink opject for each port representor and use that
interface to set peer mac address. We should be able use this to
configure port vlan too.
Also, instead of subport, can we call vport and support different types
of vports - sr-iov, siov, vmdq etc.
>
> Also eswitch is flat. There is no need of pf/vf flavour for port.
> It doesn't make sense to define 'mdev' flavour which we are already working.
> At eswitch level it is just a port, it happen to be connected to vf or pf or other objects, it doesn't matter.
> Port should be flavoured as 'hostport' or 'switchport'.
>
>
>> (using the port ids from above)
^ permalink raw reply [flat|nested] 100+ messages in thread
* RE: [PATCH net-next v2 4/7] devlink: allow subports on devlink PCI ports
2019-03-15 2:15 ` Samudrala, Sridhar
@ 2019-03-15 2:40 ` Parav Pandit
[not found] ` <ae938b4f-5fa9-3c33-8ae6-eab2d3d9f1ec@intel.com>
0 siblings, 1 reply; 100+ messages in thread
From: Parav Pandit @ 2019-03-15 2:40 UTC (permalink / raw)
To: Samudrala, Sridhar, Jakub Kicinski; +Cc: Jiri Pirko, davem, netdev, oss-drivers
> -----Original Message-----
> From: Samudrala, Sridhar <sridhar.samudrala@intel.com>
> Sent: Thursday, March 14, 2019 9:16 PM
> To: Parav Pandit <parav@mellanox.com>; Jakub Kicinski
> <jakub.kicinski@netronome.com>
> Cc: Jiri Pirko <jiri@resnulli.us>; davem@davemloft.net;
> netdev@vger.kernel.org; oss-drivers@netronome.com
> Subject: Re: [PATCH net-next v2 4/7] devlink: allow subports on devlink PCI
> ports
>
>
>
> On 3/14/2019 6:28 PM, Parav Pandit wrote:
> >
> >
> >> -----Original Message-----
> >> From: Jakub Kicinski <jakub.kicinski@netronome.com>
> >> Sent: Thursday, March 14, 2019 6:39 PM
> >> To: Parav Pandit <parav@mellanox.com>
> >> Cc: Jiri Pirko <jiri@resnulli.us>; davem@davemloft.net;
> >> netdev@vger.kernel.org; oss-drivers@netronome.com
> >> Subject: Re: [PATCH net-next v2 4/7] devlink: allow subports on
> >> devlink PCI ports
> >>
> >> On Thu, 14 Mar 2019 22:35:36 +0000, Parav Pandit wrote:
> >>>>> Then instances of flavour pci_vf are going to appear in the same
> >>>>> devlink instance. Those are the switch ports:
> >>>>> pci/0000:05:00.0/10002: type eth netdev enp5s0npf0pf0s0
> >>>>> flavour pci_vf pf 0 vf 0
> >>>>> switch_id 00154d130d2f peer
> >>>>> pci/0000:05:10.1/0
> >>>>> pci/0000:05:00.0/10003: type eth netdev enp5s0npf0pf0s0
> >>>>> flavour pci_vf pf 0 vf 0 subport 1
> >>>>> switch_id 00154d130d2f peer
> >>>>> pci/0000:05:10.1/1
> >>>>>
> >>>>> With that, peers are going to appear too, and those are the actual
> >>>>> VF/VF
> >>>>> subport:
> >>>>> pci/0000:05:10.1/0: type eth netdev ??? flavour pci_vf_host
> >>>>> peer pci/0000:05:00.0/10002
> >>>>> pci/0000:05:10.1/1: type eth netdev ??? flavour pci_vf_host
> >>>>> peer pci/0000:05:00.0/10003
> >>>>>
> >>>>> Later you can push this VF along with all subports to VM. So in
> >>>>> VM, you are going to see the VF like this:
> >>>>> $ devlink dev
> >>>>> pci/0000:00:08.0
> >>>>> $ devlink port
> >>>>> pci/0000:00:08.0/0: type eth netdev ??? flavour pci_vf_host
> >>>>> pci/0000:00:08.0/1: type eth netdev ??? flavour pci_vf_host
> >>>>>
> >>>>> And back to your question of how are they connected in eswitch.
> >>>>> That is totally up to the original user John who did the creation.
> >>>>> He is in charge of the eswitch on baremetal, he would configure
> >>>>> the forwarding however he likes.
> >>>>
> >>>> Ack, so I think you're saying VM has to communicate to the cloud
> >>>> environment to have this provisioned using some service API, not a
> >>>> kernel API. That's what I wanted to confirm.
> >>>>
> >>>> I don't see any benefit to having the "host ports" under devlink,
> >>>> as such I think it's a matter of preference.
> >>>
> >>> We need 'host ports' to configure parameters of this host port which
> >>> is not exposed by the rep-netdev.
> >>> Such as mac address.
> >>
> >> Please look at the quote of what Jiri wrote above - the host port
> >> gets passed to the VM, you can't use it as a handle to set the MAC.
> >>
> >> The way to set the MAC remains:
> >>
> >> # devlink port set pci/0000:05:00.0/10002 peer mac_addr
> >> 00:11:22:33:44:55
> >>
> > Even though it can be done, I think this is wrong model to program
> hostport mac address using eswitch port.
> > All devlink objects are control objects, so what is passed to VM is what is
> represented by devlink.
> > VF in the VM will anyway create its devlink object.
> > What is wrong in programming hostport?
> > It gives a very clear view to users of topology and objects.
>
> The VF or any subport MAC address should be configured by the
> orchestration layer that is running on the hypervisor and when a VF is
> assigned to a VF, the host port is not visible to the hypervisor.
What prevents creation of hostport due to which is not visible?
Hostport is control port to program host side of parameters.
It should be created when user wants to program the parameters.
Model is really straight forward.
Program host port params using hostport object.
Program switchport params using rep-netdev.
> Currently we have ndo_set_vf_mac_addr api that works with PF netdev, but i
> think we are trying to move away from that API and do all the configuration
> via the port representor netdevs.
This is fine rep-netdev represents eswitch port.
You normally don't go to switch to program host port params.
> As the mac address cannot be configured
> using this netdev, i think Jakub is suggesting creating a devlink opject for
> each port representor and use that interface to set peer mac address.
I understand but is convoluted interface.
When you program host NIC mac address you talk to iLo or BIOS.
When you program switch side mac address, you go switch/router/modem.
Also programming host params on host side, also doesn't make assumption that its connected to eswitch.
It also doesn't assume that same connectivity for its life.
If you model around how physical devices are configured, it will almost never go wrong and still provides same level of flexibility.
> We should be able use this to configure port vlan too.
>
> Also, instead of subport, can we call vport and support different types of
> vports - sr-iov, siov, vmdq etc.
>
At switch level there are just ports.
sriov, siov, mdev, vmdq are their couter part (peer) where it is connected.
> >
> > Also eswitch is flat. There is no need of pf/vf flavour for port.
> > It doesn't make sense to define 'mdev' flavour which we are already
> working.
> > At eswitch level it is just a port, it happen to be connected to vf or pf or
> other objects, it doesn't matter.
> > Port should be flavoured as 'hostport' or 'switchport'.
> >
> >
> >> (using the port ids from above)
^ permalink raw reply [flat|nested] 100+ messages in thread
* Re: [PATCH net-next v2 4/7] devlink: allow subports on devlink PCI ports
2019-03-14 22:09 ` Jakub Kicinski
2019-03-14 22:35 ` Parav Pandit
@ 2019-03-15 7:00 ` Jiri Pirko
1 sibling, 0 replies; 100+ messages in thread
From: Jiri Pirko @ 2019-03-15 7:00 UTC (permalink / raw)
To: Jakub Kicinski; +Cc: davem, netdev, oss-drivers
Thu, Mar 14, 2019 at 11:09:45PM CET, jakub.kicinski@netronome.com wrote:
>On Thu, 14 Mar 2019 08:38:40 +0100, Jiri Pirko wrote:
>> Wed, Mar 13, 2019 at 05:55:55PM CET, jakub.kicinski@netronome.com wrote:
>> >On Wed, 13 Mar 2019 17:22:43 +0100, Jiri Pirko wrote:
>> >> Wed, Mar 13, 2019 at 05:17:31PM CET, jakub.kicinski@netronome.com wrote:
>> >> >On Wed, 13 Mar 2019 07:07:01 +0100, Jiri Pirko wrote:
>> >> >> Tue, Mar 12, 2019 at 09:56:28PM CET, jakub.kicinski@netronome.com wrote:
>> >> >> >On Tue, 12 Mar 2019 15:02:39 +0100, Jiri Pirko wrote:
>> >> >> >> Tue, Mar 12, 2019 at 03:10:54AM CET, wrote:
>> >> >> >> >On Mon, 11 Mar 2019 09:52:04 +0100, Jiri Pirko wrote:
>> >> >> >> >> Fri, Mar 08, 2019 at 08:09:43PM CET, wrote:
>> >> >> >> >> >If the switchport is in the hypervisor then only the hypervisor can
>> >> >> >> >> >control switching/forwarding, correct?
>> >> >> >> >>
>> >> >> >> >> Correct.
>> >> >> >> >>
>> >> >> >> >> >The primary use case for partitioning within a VM (of a VF) would be
>> >> >> >> >> >containers (and DPDK)?
>> >> >> >> >>
>> >> >> >> >> Makes sense.
>> >> >> >> >>
>> >> >> >> >> >SR-IOV makes things harder. Splitting a PF is reasonably easy to grasp.
>> >> >> >> >> >I'm trying to get a sense of is how would we control an SR-IOV
>> >> >> >> >> >environment as a whole.
>> >> >> >> >>
>> >> >> >> >> You mean orchestration?
>> >> >> >> >
>> >> >> >> >Right, orchestration.
>> >> >> >> >
>> >> >> >> >To be clear on where I'm going with this - if we want to allow VFs
>> >> >> >> >to partition themselves then they have to control what is effectively
>> >> >> >> >a "nested" switch. A per-VF set of rules which would the get
>> >> >> >>
>> >> >> >> Wait. If you allow to make VF subports (I believe that is what you ment
>> >> >> >> by VFs partition themselves), that does not mean they will have a
>> >> >> >> separate nested switch. They would still belong under the same one.
>> >> >> >
>> >> >> >But that existing switch is administered by the hypervisor, how would
>> >> >> >the VF owners install forwarding rules in a switch they don't control?
>> >> >>
>> >> >> They won't.
>> >> >
>> >> >Argh. So how is forwarding configured if there are no rules? Are you
>> >> >going to assume its switching on MACs? We're supposed to offload
>> >> >software constructs. If its a software port it needs to be explicitly
>> >> >switched. If it's not explicitly switched - we already have macvlan
>> >> >offload.
>> >>
>> >> Wait a second. You configure the switch. And for that, you have the
>> >> switchports (representors). What we are talking about are VF (VF
>> >> subport) host legs. Am I missing something?
>> >
>> >Hm :) So when VM gets a new port, how is it connected? Are we
>> >assuming all ports of a VM are plugged into one big L2 switch?
>> >The use case for those sub ports is a little murky, sorry about
>> >the endless confusion :)
>>
>> Np. When user John (on baremetal, or whenever the devlink instance
>> with switch port is) creates VF of VF subport by:
>> $ devlink dev port add pci/0000:05:00.0 flavour pci_vf pf 0
>> or by:
>> $ devlink dev port add pci/0000:05:00.0 flavour pci_vf pf 0 vf 0
>>
>> Then instances of flavour pci_vf are going to appear in the same devlink
>> instance. Those are the switch ports:
>> pci/0000:05:00.0/10002: type eth netdev enp5s0npf0pf0s0
>> flavour pci_vf pf 0 vf 0
>> switch_id 00154d130d2f peer pci/0000:05:10.1/0
>> pci/0000:05:00.0/10003: type eth netdev enp5s0npf0pf0s0
>> flavour pci_vf pf 0 vf 0 subport 1
>> switch_id 00154d130d2f peer pci/0000:05:10.1/1
>>
>> With that, peers are going to appear too, and those are the actual VF/VF
>> subport:
>> pci/0000:05:10.1/0: type eth netdev ??? flavour pci_vf_host
>> peer pci/0000:05:00.0/10002
>> pci/0000:05:10.1/1: type eth netdev ??? flavour pci_vf_host
>> peer pci/0000:05:00.0/10003
>>
>> Later you can push this VF along with all subports to VM. So in VM, you
>> are going to see the VF like this:
>> $ devlink dev
>> pci/0000:00:08.0
>> $ devlink port
>> pci/0000:00:08.0/0: type eth netdev ??? flavour pci_vf_host
>> pci/0000:00:08.0/1: type eth netdev ??? flavour pci_vf_host
>>
>> And back to your question of how are they connected in eswitch.
>> That is totally up to the original user John who did the creation.
>> He is in charge of the eswitch on baremetal, he would configure
>> the forwarding however he likes.
>
>Ack, so I think you're saying VM has to communicate to the cloud
>environment to have this provisioned using some service API, not
>a kernel API. That's what I wanted to confirm.
Okay.
>
>I don't see any benefit to having the "host ports" under devlink,
>as such I think it's a matter of preference. I'll try to describe
>the two options to Netronome's FAEs and see which one they find more
>intuitive.
Yeah, the "host ports" are probably not a must. I just like to have them
for visibility purposes. No big deal to implement them.
>
>Makes sense?
Okay. Thanks!
^ permalink raw reply [flat|nested] 100+ messages in thread
* RE: [PATCH net-next v2 4/7] devlink: allow subports on devlink PCI ports
[not found] ` <ae938b4f-5fa9-3c33-8ae6-eab2d3d9f1ec@intel.com>
@ 2019-03-15 15:32 ` Parav Pandit
2019-03-15 20:08 ` Jiri Pirko
0 siblings, 1 reply; 100+ messages in thread
From: Parav Pandit @ 2019-03-15 15:32 UTC (permalink / raw)
To: Samudrala, Sridhar, Jakub Kicinski; +Cc: Jiri Pirko, davem, netdev, oss-drivers
> -----Original Message-----
> From: Samudrala, Sridhar <sridhar.samudrala@intel.com>
> Sent: Friday, March 15, 2019 12:58 AM
> To: Parav Pandit <parav@mellanox.com>; Jakub Kicinski
> <jakub.kicinski@netronome.com>
> Cc: Jiri Pirko <jiri@resnulli.us>; davem@davemloft.net;
> netdev@vger.kernel.org; oss-drivers@netronome.com
> Subject: Re: [PATCH net-next v2 4/7] devlink: allow subports on devlink PCI
> ports
>
>
> On 3/14/2019 7:40 PM, Parav Pandit wrote:
> >
> >
> >> -----Original Message-----
> >> From: Samudrala, Sridhar <sridhar.samudrala@intel.com>
> >> Sent: Thursday, March 14, 2019 9:16 PM
> >> To: Parav Pandit <parav@mellanox.com>; Jakub Kicinski
> >> <jakub.kicinski@netronome.com>
> >> Cc: Jiri Pirko <jiri@resnulli.us>; davem@davemloft.net;
> >> netdev@vger.kernel.org; oss-drivers@netronome.com
> >> Subject: Re: [PATCH net-next v2 4/7] devlink: allow subports on
> >> devlink PCI ports
> >>
> >>
> >>
> >> On 3/14/2019 6:28 PM, Parav Pandit wrote:
> >>>
> >>>
> >>>> -----Original Message-----
> >>>> From: Jakub Kicinski <jakub.kicinski@netronome.com>
> >>>> Sent: Thursday, March 14, 2019 6:39 PM
> >>>> To: Parav Pandit <parav@mellanox.com>
> >>>> Cc: Jiri Pirko <jiri@resnulli.us>; davem@davemloft.net;
> >>>> netdev@vger.kernel.org; oss-drivers@netronome.com
> >>>> Subject: Re: [PATCH net-next v2 4/7] devlink: allow subports on
> >>>> devlink PCI ports
> >>>>
> >>>> On Thu, 14 Mar 2019 22:35:36 +0000, Parav Pandit wrote:
> >>>>>>> Then instances of flavour pci_vf are going to appear in the same
> >>>>>>> devlink instance. Those are the switch ports:
> >>>>>>> pci/0000:05:00.0/10002: type eth netdev enp5s0npf0pf0s0
> >>>>>>> flavour pci_vf pf 0 vf 0
> >>>>>>> switch_id 00154d130d2f peer
> >>>>>>> pci/0000:05:10.1/0
> >>>>>>> pci/0000:05:00.0/10003: type eth netdev enp5s0npf0pf0s0
> >>>>>>> flavour pci_vf pf 0 vf 0 subport 1
> >>>>>>> switch_id 00154d130d2f peer
> >>>>>>> pci/0000:05:10.1/1
> >>>>>>>
> >>>>>>> With that, peers are going to appear too, and those are the
> >>>>>>> actual VF/VF
> >>>>>>> subport:
> >>>>>>> pci/0000:05:10.1/0: type eth netdev ??? flavour pci_vf_host
> >>>>>>> peer pci/0000:05:00.0/10002
> >>>>>>> pci/0000:05:10.1/1: type eth netdev ??? flavour pci_vf_host
> >>>>>>> peer pci/0000:05:00.0/10003
> >>>>>>>
> >>>>>>> Later you can push this VF along with all subports to VM. So in
> >>>>>>> VM, you are going to see the VF like this:
> >>>>>>> $ devlink dev
> >>>>>>> pci/0000:00:08.0
> >>>>>>> $ devlink port
> >>>>>>> pci/0000:00:08.0/0: type eth netdev ??? flavour pci_vf_host
> >>>>>>> pci/0000:00:08.0/1: type eth netdev ??? flavour pci_vf_host
> >>>>>>>
> >>>>>>> And back to your question of how are they connected in eswitch.
> >>>>>>> That is totally up to the original user John who did the creation.
> >>>>>>> He is in charge of the eswitch on baremetal, he would configure
> >>>>>>> the forwarding however he likes.
> >>>>>>
> >>>>>> Ack, so I think you're saying VM has to communicate to the cloud
> >>>>>> environment to have this provisioned using some service API, not
> >>>>>> a kernel API. That's what I wanted to confirm.
> >>>>>>
> >>>>>> I don't see any benefit to having the "host ports" under devlink,
> >>>>>> as such I think it's a matter of preference.
> >>>>>
> >>>>> We need 'host ports' to configure parameters of this host port
> >>>>> which is not exposed by the rep-netdev.
> >>>>> Such as mac address.
> >>>>
> >>>> Please look at the quote of what Jiri wrote above - the host port
> >>>> gets passed to the VM, you can't use it as a handle to set the MAC.
> >>>>
> >>>> The way to set the MAC remains:
> >>>>
> >>>> # devlink port set pci/0000:05:00.0/10002 peer mac_addr
> >>>> 00:11:22:33:44:55
> >>>>
> >>> Even though it can be done, I think this is wrong model to program
> >> hostport mac address using eswitch port.
> >>> All devlink objects are control objects, so what is passed to VM is
> >>> what is
> >> represented by devlink.
> >>> VF in the VM will anyway create its devlink object.
> >>> What is wrong in programming hostport?
> >>> It gives a very clear view to users of topology and objects.
> >>
> >> The VF or any subport MAC address should be configured by the
> >> orchestration layer that is running on the hypervisor and when a VF
> >> is assigned to a VF, the host port is not visible to the hypervisor.
> > What prevents creation of hostport due to which is not visible?
> > Hostport is control port to program host side of parameters.
> > It should be created when user wants to program the parameters.
> >
> > Model is really straight forward.
> > Program host port params using hostport object.
> > Program switchport params using rep-netdev.
>
> IIUC, Jiri/Jakub are proposing creation of 2 devlink objects for each port -
> host facing ports and switch facing ports. This is in addition to the netdevs
> that are created today.
>
I am not proposing any different.
I am proposing only two changes.
1. control hostport params via referring hostport (not via indirect peer)
2. flavour should not be vf/pf, flavour should be hostport, switchport.
Because switch is flat and agnostic of pf/vf/mdev.
> Are you suggesting that all the devlink objects should be visible only at the
> hypervisor layer?
>
Of course not.
Ports and params controlled by hypervisor should be exposed at hypervisor/eswitch wherever its parent devlink instance exist.
Ports which should be visible inside a VM should be exposed inside a VM.
So for a given VF,
If eswitch is at hypervisor level,
$ devlink port show
pci/0000:05:00.0/10002 eth netdev flavour switchport switch_id 00154d130d2f peer pci/0000:05:10.1/0
pci/0000:05:10.1/0 eth netdev flavour hostport switch_id 00154d130d2f peer pci/0000:05:00.0/10002
where VF is enumerated,
$ devlink port show
pci/0000:05:10.1/0 eth netdev flavour hostport
This is because unprivileged VF doesn't have visibility to eswitch and its links.
> I think the terminology need to be defined clearly so that we are all on the
> same page.
>
> >
> >> Currently we have ndo_set_vf_mac_addr api that works with PF netdev,
> >> but i think we are trying to move away from that API and do all the
> >> configuration via the port representor netdevs.
> > This is fine rep-netdev represents eswitch port.
> > You normally don't go to switch to program host port params.
> >
> >> As the mac address cannot be configured using this netdev, i think
> >> Jakub is suggesting creating a devlink opject for each port
> >> representor and use that interface to set peer mac address.
> >
> > I understand but is convoluted interface.
> > When you program host NIC mac address you talk to iLo or BIOS.
> > When you program switch side mac address, you go switch/router/modem.
> >
> > Also programming host params on host side, also doesn't make
> assumption that its connected to eswitch.
> > It also doesn't assume that same connectivity for its life.
> >
> > If you model around how physical devices are configured, it will almost
> never go wrong and still provides same level of flexibility.
> >
> >> We should be able use this to configure port vlan too.
> >>
> >> Also, instead of subport, can we call vport and support different
> >> types of vports - sr-iov, siov, vmdq etc.
> >>
> > At switch level there are just ports.
> > sriov, siov, mdev, vmdq are their couter part (peer) where it is connected.
> >
> >>>
> >>> Also eswitch is flat. There is no need of pf/vf flavour for port.
> >>> It doesn't make sense to define 'mdev' flavour which we are already
> >> working.
> >>> At eswitch level it is just a port, it happen to be connected to vf
> >>> or pf or
> >> other objects, it doesn't matter.
> >>> Port should be flavoured as 'hostport' or 'switchport'.
> >>>
> >>>
> >>>> (using the port ids from above)
^ permalink raw reply [flat|nested] 100+ messages in thread
* Re: [PATCH net-next v2 4/7] devlink: allow subports on devlink PCI ports
2019-03-15 15:32 ` Parav Pandit
@ 2019-03-15 20:08 ` Jiri Pirko
2019-03-15 20:44 ` Jakub Kicinski
2019-03-15 21:59 ` Parav Pandit
0 siblings, 2 replies; 100+ messages in thread
From: Jiri Pirko @ 2019-03-15 20:08 UTC (permalink / raw)
To: Parav Pandit
Cc: Samudrala, Sridhar, Jakub Kicinski, davem, netdev, oss-drivers
Fri, Mar 15, 2019 at 04:32:24PM CET, parav@mellanox.com wrote:
>
>
>> -----Original Message-----
>> From: Samudrala, Sridhar <sridhar.samudrala@intel.com>
>> Sent: Friday, March 15, 2019 12:58 AM
>> To: Parav Pandit <parav@mellanox.com>; Jakub Kicinski
>> <jakub.kicinski@netronome.com>
>> Cc: Jiri Pirko <jiri@resnulli.us>; davem@davemloft.net;
>> netdev@vger.kernel.org; oss-drivers@netronome.com
>> Subject: Re: [PATCH net-next v2 4/7] devlink: allow subports on devlink PCI
>> ports
>>
>>
>> On 3/14/2019 7:40 PM, Parav Pandit wrote:
>> >
>> >
>> >> -----Original Message-----
>> >> From: Samudrala, Sridhar <sridhar.samudrala@intel.com>
>> >> Sent: Thursday, March 14, 2019 9:16 PM
>> >> To: Parav Pandit <parav@mellanox.com>; Jakub Kicinski
>> >> <jakub.kicinski@netronome.com>
>> >> Cc: Jiri Pirko <jiri@resnulli.us>; davem@davemloft.net;
>> >> netdev@vger.kernel.org; oss-drivers@netronome.com
>> >> Subject: Re: [PATCH net-next v2 4/7] devlink: allow subports on
>> >> devlink PCI ports
>> >>
>> >>
>> >>
>> >> On 3/14/2019 6:28 PM, Parav Pandit wrote:
>> >>>
>> >>>
>> >>>> -----Original Message-----
>> >>>> From: Jakub Kicinski <jakub.kicinski@netronome.com>
>> >>>> Sent: Thursday, March 14, 2019 6:39 PM
>> >>>> To: Parav Pandit <parav@mellanox.com>
>> >>>> Cc: Jiri Pirko <jiri@resnulli.us>; davem@davemloft.net;
>> >>>> netdev@vger.kernel.org; oss-drivers@netronome.com
>> >>>> Subject: Re: [PATCH net-next v2 4/7] devlink: allow subports on
>> >>>> devlink PCI ports
>> >>>>
>> >>>> On Thu, 14 Mar 2019 22:35:36 +0000, Parav Pandit wrote:
>> >>>>>>> Then instances of flavour pci_vf are going to appear in the same
>> >>>>>>> devlink instance. Those are the switch ports:
>> >>>>>>> pci/0000:05:00.0/10002: type eth netdev enp5s0npf0pf0s0
>> >>>>>>> flavour pci_vf pf 0 vf 0
>> >>>>>>> switch_id 00154d130d2f peer
>> >>>>>>> pci/0000:05:10.1/0
>> >>>>>>> pci/0000:05:00.0/10003: type eth netdev enp5s0npf0pf0s0
>> >>>>>>> flavour pci_vf pf 0 vf 0 subport 1
>> >>>>>>> switch_id 00154d130d2f peer
>> >>>>>>> pci/0000:05:10.1/1
>> >>>>>>>
>> >>>>>>> With that, peers are going to appear too, and those are the
>> >>>>>>> actual VF/VF
>> >>>>>>> subport:
>> >>>>>>> pci/0000:05:10.1/0: type eth netdev ??? flavour pci_vf_host
>> >>>>>>> peer pci/0000:05:00.0/10002
>> >>>>>>> pci/0000:05:10.1/1: type eth netdev ??? flavour pci_vf_host
>> >>>>>>> peer pci/0000:05:00.0/10003
>> >>>>>>>
>> >>>>>>> Later you can push this VF along with all subports to VM. So in
>> >>>>>>> VM, you are going to see the VF like this:
>> >>>>>>> $ devlink dev
>> >>>>>>> pci/0000:00:08.0
>> >>>>>>> $ devlink port
>> >>>>>>> pci/0000:00:08.0/0: type eth netdev ??? flavour pci_vf_host
>> >>>>>>> pci/0000:00:08.0/1: type eth netdev ??? flavour pci_vf_host
>> >>>>>>>
>> >>>>>>> And back to your question of how are they connected in eswitch.
>> >>>>>>> That is totally up to the original user John who did the creation.
>> >>>>>>> He is in charge of the eswitch on baremetal, he would configure
>> >>>>>>> the forwarding however he likes.
>> >>>>>>
>> >>>>>> Ack, so I think you're saying VM has to communicate to the cloud
>> >>>>>> environment to have this provisioned using some service API, not
>> >>>>>> a kernel API. That's what I wanted to confirm.
>> >>>>>>
>> >>>>>> I don't see any benefit to having the "host ports" under devlink,
>> >>>>>> as such I think it's a matter of preference.
>> >>>>>
>> >>>>> We need 'host ports' to configure parameters of this host port
>> >>>>> which is not exposed by the rep-netdev.
>> >>>>> Such as mac address.
>> >>>>
>> >>>> Please look at the quote of what Jiri wrote above - the host port
>> >>>> gets passed to the VM, you can't use it as a handle to set the MAC.
>> >>>>
>> >>>> The way to set the MAC remains:
>> >>>>
>> >>>> # devlink port set pci/0000:05:00.0/10002 peer mac_addr
>> >>>> 00:11:22:33:44:55
>> >>>>
>> >>> Even though it can be done, I think this is wrong model to program
>> >> hostport mac address using eswitch port.
>> >>> All devlink objects are control objects, so what is passed to VM is
>> >>> what is
>> >> represented by devlink.
>> >>> VF in the VM will anyway create its devlink object.
>> >>> What is wrong in programming hostport?
>> >>> It gives a very clear view to users of topology and objects.
>> >>
>> >> The VF or any subport MAC address should be configured by the
>> >> orchestration layer that is running on the hypervisor and when a VF
>> >> is assigned to a VF, the host port is not visible to the hypervisor.
>> > What prevents creation of hostport due to which is not visible?
>> > Hostport is control port to program host side of parameters.
>> > It should be created when user wants to program the parameters.
>> >
>> > Model is really straight forward.
>> > Program host port params using hostport object.
>> > Program switchport params using rep-netdev.
>>
>> IIUC, Jiri/Jakub are proposing creation of 2 devlink objects for each port -
>> host facing ports and switch facing ports. This is in addition to the netdevs
>> that are created today.
>>
>I am not proposing any different.
>I am proposing only two changes.
>1. control hostport params via referring hostport (not via indirect peer)
Not really possible. If you passthrough VF into VM, the hostport goes
along with it.
>2. flavour should not be vf/pf, flavour should be hostport, switchport.
>Because switch is flat and agnostic of pf/vf/mdev.
Not sure. It's good to have this kind of visibility.
>
>> Are you suggesting that all the devlink objects should be visible only at the
>> hypervisor layer?
>>
>Of course not.
>
>Ports and params controlled by hypervisor should be exposed at hypervisor/eswitch wherever its parent devlink instance exist.
>Ports which should be visible inside a VM should be exposed inside a VM.
>So for a given VF,
>
>If eswitch is at hypervisor level,
>$ devlink port show
>pci/0000:05:00.0/10002 eth netdev flavour switchport switch_id 00154d130d2f peer pci/0000:05:10.1/0
>pci/0000:05:10.1/0 eth netdev flavour hostport switch_id 00154d130d2f peer pci/0000:05:00.0/10002
>
>where VF is enumerated,
>$ devlink port show
>pci/0000:05:10.1/0 eth netdev flavour hostport
So this is how it looks like in VM, right?
>This is because unprivileged VF doesn't have visibility to eswitch and its links.
>
>> I think the terminology need to be defined clearly so that we are all on the
>> same page.
>>
>> >
>> >> Currently we have ndo_set_vf_mac_addr api that works with PF netdev,
>> >> but i think we are trying to move away from that API and do all the
>> >> configuration via the port representor netdevs.
>> > This is fine rep-netdev represents eswitch port.
>> > You normally don't go to switch to program host port params.
>> >
>> >> As the mac address cannot be configured using this netdev, i think
>> >> Jakub is suggesting creating a devlink opject for each port
>> >> representor and use that interface to set peer mac address.
>> >
>> > I understand but is convoluted interface.
>> > When you program host NIC mac address you talk to iLo or BIOS.
>> > When you program switch side mac address, you go switch/router/modem.
>> >
>> > Also programming host params on host side, also doesn't make
>> assumption that its connected to eswitch.
>> > It also doesn't assume that same connectivity for its life.
>> >
>> > If you model around how physical devices are configured, it will almost
>> never go wrong and still provides same level of flexibility.
>> >
>> >> We should be able use this to configure port vlan too.
>> >>
>> >> Also, instead of subport, can we call vport and support different
>> >> types of vports - sr-iov, siov, vmdq etc.
>> >>
>> > At switch level there are just ports.
>> > sriov, siov, mdev, vmdq are their couter part (peer) where it is connected.
>> >
>> >>>
>> >>> Also eswitch is flat. There is no need of pf/vf flavour for port.
>> >>> It doesn't make sense to define 'mdev' flavour which we are already
>> >> working.
>> >>> At eswitch level it is just a port, it happen to be connected to vf
>> >>> or pf or
>> >> other objects, it doesn't matter.
>> >>> Port should be flavoured as 'hostport' or 'switchport'.
>> >>>
>> >>>
>> >>>> (using the port ids from above)
^ permalink raw reply [flat|nested] 100+ messages in thread
* Re: [PATCH net-next v2 4/7] devlink: allow subports on devlink PCI ports
2019-03-15 20:08 ` Jiri Pirko
@ 2019-03-15 20:44 ` Jakub Kicinski
2019-03-15 22:12 ` Parav Pandit
2019-03-18 12:11 ` Jiri Pirko
2019-03-15 21:59 ` Parav Pandit
1 sibling, 2 replies; 100+ messages in thread
From: Jakub Kicinski @ 2019-03-15 20:44 UTC (permalink / raw)
To: Jiri Pirko; +Cc: Parav Pandit, Samudrala, Sridhar, davem, netdev, oss-drivers
On Fri, 15 Mar 2019 21:08:14 +0100, Jiri Pirko wrote:
> >> IIUC, Jiri/Jakub are proposing creation of 2 devlink objects for each port -
> >> host facing ports and switch facing ports. This is in addition to the netdevs
> >> that are created today.
To be clear I'm not in favour of the dual-object proposal.
> >I am not proposing any different.
> >I am proposing only two changes.
> >1. control hostport params via referring hostport (not via indirect peer)
>
> Not really possible. If you passthrough VF into VM, the hostport goes
> along with it.
>
> >2. flavour should not be vf/pf, flavour should be hostport, switchport.
> >Because switch is flat and agnostic of pf/vf/mdev.
>
> Not sure. It's good to have this kind of visibility.
Yes, this subthread honestly makes me go from 60% sure to 95% sure we
shouldn't do the dual object thing :( Seems like Parav is already
confused by it and suggests host port can exist without switch port :(
> >> Are you suggesting that all the devlink objects should be visible only at the
> >> hypervisor layer?
> >>
> >Of course not.
> >
> >Ports and params controlled by hypervisor should be exposed at hypervisor/eswitch wherever its parent devlink instance exist.
> >Ports which should be visible inside a VM should be exposed inside a VM.
> >So for a given VF,
> >
> >If eswitch is at hypervisor level,
> >$ devlink port show
> >pci/0000:05:00.0/10002 eth netdev flavour switchport switch_id 00154d130d2f peer pci/0000:05:10.1/0
> >pci/0000:05:10.1/0 eth netdev flavour hostport switch_id 00154d130d2f peer pci/0000:05:00.0/10002
> >
> >where VF is enumerated,
> >$ devlink port show
> >pci/0000:05:10.1/0 eth netdev flavour hostport
>
> So this is how it looks like in VM, right?
>
> >This is because unprivileged VF doesn't have visibility to eswitch and its links.
^ permalink raw reply [flat|nested] 100+ messages in thread
* RE: [PATCH net-next v2 4/7] devlink: allow subports on devlink PCI ports
2019-03-15 20:08 ` Jiri Pirko
2019-03-15 20:44 ` Jakub Kicinski
@ 2019-03-15 21:59 ` Parav Pandit
2019-03-18 12:21 ` Jiri Pirko
1 sibling, 1 reply; 100+ messages in thread
From: Parav Pandit @ 2019-03-15 21:59 UTC (permalink / raw)
To: Jiri Pirko; +Cc: Samudrala, Sridhar, Jakub Kicinski, davem, netdev, oss-drivers
> -----Original Message-----
> From: Jiri Pirko <jiri@resnulli.us>
> Sent: Friday, March 15, 2019 3:08 PM
> To: Parav Pandit <parav@mellanox.com>
> Cc: Samudrala, Sridhar <sridhar.samudrala@intel.com>; Jakub Kicinski
> <jakub.kicinski@netronome.com>; davem@davemloft.net;
> netdev@vger.kernel.org; oss-drivers@netronome.com
> Subject: Re: [PATCH net-next v2 4/7] devlink: allow subports on devlink PCI
> ports
>
> Fri, Mar 15, 2019 at 04:32:24PM CET, parav@mellanox.com wrote:
> >
> >
> >> -----Original Message-----
> >> From: Samudrala, Sridhar <sridhar.samudrala@intel.com>
> >> Sent: Friday, March 15, 2019 12:58 AM
> >> To: Parav Pandit <parav@mellanox.com>; Jakub Kicinski
> >> <jakub.kicinski@netronome.com>
> >> Cc: Jiri Pirko <jiri@resnulli.us>; davem@davemloft.net;
> >> netdev@vger.kernel.org; oss-drivers@netronome.com
> >> Subject: Re: [PATCH net-next v2 4/7] devlink: allow subports on
> >> devlink PCI ports
> >>
> >>
> >> On 3/14/2019 7:40 PM, Parav Pandit wrote:
> >> >
> >> >
> >> >> -----Original Message-----
> >> >> From: Samudrala, Sridhar <sridhar.samudrala@intel.com>
> >> >> Sent: Thursday, March 14, 2019 9:16 PM
> >> >> To: Parav Pandit <parav@mellanox.com>; Jakub Kicinski
> >> >> <jakub.kicinski@netronome.com>
> >> >> Cc: Jiri Pirko <jiri@resnulli.us>; davem@davemloft.net;
> >> >> netdev@vger.kernel.org; oss-drivers@netronome.com
> >> >> Subject: Re: [PATCH net-next v2 4/7] devlink: allow subports on
> >> >> devlink PCI ports
> >> >>
> >> >>
> >> >>
> >> >> On 3/14/2019 6:28 PM, Parav Pandit wrote:
> >> >>>
> >> >>>
> >> >>>> -----Original Message-----
> >> >>>> From: Jakub Kicinski <jakub.kicinski@netronome.com>
> >> >>>> Sent: Thursday, March 14, 2019 6:39 PM
> >> >>>> To: Parav Pandit <parav@mellanox.com>
> >> >>>> Cc: Jiri Pirko <jiri@resnulli.us>; davem@davemloft.net;
> >> >>>> netdev@vger.kernel.org; oss-drivers@netronome.com
> >> >>>> Subject: Re: [PATCH net-next v2 4/7] devlink: allow subports on
> >> >>>> devlink PCI ports
> >> >>>>
> >> >>>> On Thu, 14 Mar 2019 22:35:36 +0000, Parav Pandit wrote:
> >> >>>>>>> Then instances of flavour pci_vf are going to appear in the
> >> >>>>>>> same devlink instance. Those are the switch ports:
> >> >>>>>>> pci/0000:05:00.0/10002: type eth netdev enp5s0npf0pf0s0
> >> >>>>>>> flavour pci_vf pf 0 vf 0
> >> >>>>>>> switch_id 00154d130d2f peer
> >> >>>>>>> pci/0000:05:10.1/0
> >> >>>>>>> pci/0000:05:00.0/10003: type eth netdev enp5s0npf0pf0s0
> >> >>>>>>> flavour pci_vf pf 0 vf 0 subport 1
> >> >>>>>>> switch_id 00154d130d2f peer
> >> >>>>>>> pci/0000:05:10.1/1
> >> >>>>>>>
> >> >>>>>>> With that, peers are going to appear too, and those are the
> >> >>>>>>> actual VF/VF
> >> >>>>>>> subport:
> >> >>>>>>> pci/0000:05:10.1/0: type eth netdev ??? flavour pci_vf_host
> >> >>>>>>> peer pci/0000:05:00.0/10002
> >> >>>>>>> pci/0000:05:10.1/1: type eth netdev ??? flavour pci_vf_host
> >> >>>>>>> peer pci/0000:05:00.0/10003
> >> >>>>>>>
> >> >>>>>>> Later you can push this VF along with all subports to VM. So
> >> >>>>>>> in VM, you are going to see the VF like this:
> >> >>>>>>> $ devlink dev
> >> >>>>>>> pci/0000:00:08.0
> >> >>>>>>> $ devlink port
> >> >>>>>>> pci/0000:00:08.0/0: type eth netdev ??? flavour pci_vf_host
> >> >>>>>>> pci/0000:00:08.0/1: type eth netdev ??? flavour pci_vf_host
> >> >>>>>>>
> >> >>>>>>> And back to your question of how are they connected in eswitch.
> >> >>>>>>> That is totally up to the original user John who did the creation.
> >> >>>>>>> He is in charge of the eswitch on baremetal, he would
> >> >>>>>>> configure the forwarding however he likes.
> >> >>>>>>
> >> >>>>>> Ack, so I think you're saying VM has to communicate to the
> >> >>>>>> cloud environment to have this provisioned using some service
> >> >>>>>> API, not a kernel API. That's what I wanted to confirm.
> >> >>>>>>
> >> >>>>>> I don't see any benefit to having the "host ports" under
> >> >>>>>> devlink, as such I think it's a matter of preference.
> >> >>>>>
> >> >>>>> We need 'host ports' to configure parameters of this host port
> >> >>>>> which is not exposed by the rep-netdev.
> >> >>>>> Such as mac address.
> >> >>>>
> >> >>>> Please look at the quote of what Jiri wrote above - the host
> >> >>>> port gets passed to the VM, you can't use it as a handle to set the
> MAC.
> >> >>>>
> >> >>>> The way to set the MAC remains:
> >> >>>>
> >> >>>> # devlink port set pci/0000:05:00.0/10002 peer mac_addr
> >> >>>> 00:11:22:33:44:55
> >> >>>>
> >> >>> Even though it can be done, I think this is wrong model to
> >> >>> program
> >> >> hostport mac address using eswitch port.
> >> >>> All devlink objects are control objects, so what is passed to VM
> >> >>> is what is
> >> >> represented by devlink.
> >> >>> VF in the VM will anyway create its devlink object.
> >> >>> What is wrong in programming hostport?
> >> >>> It gives a very clear view to users of topology and objects.
> >> >>
> >> >> The VF or any subport MAC address should be configured by the
> >> >> orchestration layer that is running on the hypervisor and when a
> >> >> VF is assigned to a VF, the host port is not visible to the hypervisor.
> >> > What prevents creation of hostport due to which is not visible?
> >> > Hostport is control port to program host side of parameters.
> >> > It should be created when user wants to program the parameters.
> >> >
> >> > Model is really straight forward.
> >> > Program host port params using hostport object.
> >> > Program switchport params using rep-netdev.
> >>
> >> IIUC, Jiri/Jakub are proposing creation of 2 devlink objects for each
> >> port - host facing ports and switch facing ports. This is in addition
> >> to the netdevs that are created today.
> >>
> >I am not proposing any different.
> >I am proposing only two changes.
> >1. control hostport params via referring hostport (not via indirect
> >peer)
>
> Not really possible. If you passthrough VF into VM, the hostport goes along
> with it.
>
No.
I am sorry in showing the enumeration which is the source of confusion.
Below is the right enumeration.
When VF is enumerated initially in the host, where eswitch devlink instance is located.
Below enumeration is seen.
First two entries shows the link between hostport and switchport.
$ devlink port show
pci/0000:05:00.0/10002 eth netdev flavour switchport switch_id 00154d130d2f peer pci/0000:05:00.0/1
pci/0000:05:00.0/1 eth netdev flavour hostport switch_id 00154d130d2f peer pci/0000:05:00.0/10002
pci/0000:05:10.1/0 eth netdev flavour hostport
This entry won't be seen if VF auto probing is disabled. Because than VF is not enumerated.
As a user, I will be programming the mac address of hostport for a VF.
pci/0000:05:00.0/1 eth netdev flavour hostport switch_id 00154d130d2f peer pci/0000:05:00.0/10002
>
> >2. flavour should not be vf/pf, flavour should be hostport, switchport.
> >Because switch is flat and agnostic of pf/vf/mdev.
>
> Not sure. It's good to have this kind of visibility.
>
port can have label/attribute indicating that this belong to VF-1 or mdev as long as you are agreeing to have mdev attribute on host port.
(and not ask for abstracting it, because mdev is well defined kernel object).
>
> >
> >> Are you suggesting that all the devlink objects should be visible
> >> only at the hypervisor layer?
> >>
> >Of course not.
> >
> >Ports and params controlled by hypervisor should be exposed at
> hypervisor/eswitch wherever its parent devlink instance exist.
> >Ports which should be visible inside a VM should be exposed inside a VM.
> >So for a given VF,
> >
> >If eswitch is at hypervisor level,
> >$ devlink port show
> >pci/0000:05:00.0/10002 eth netdev flavour switchport switch_id
> >00154d130d2f peer pci/0000:05:10.1/0
> >pci/0000:05:10.1/0 eth netdev flavour hostport switch_id 00154d130d2f
> >peer pci/0000:05:00.0/10002
> >
> >where VF is enumerated,
> >$ devlink port show
> >pci/0000:05:10.1/0 eth netdev flavour hostport
>
> So this is how it looks like in VM, right?
>
Yep.
Once VF is mapped to VM only two entries are seen and hostport can be still controlled.
$ devlink port show
pci/0000:05:00.0/10002 eth netdev flavour switchport switch_id 00154d130d2f peer pci/0000:05:00.0/1
pci/0000:05:00.0/1 eth netdev flavour hostport switch_id 00154d130d2f peer pci/0000:05:00.0/10002
This addresses the case for Infiniband where there is no eswitch, but hostports exists and should be managed.
We shouldn't be inventing new devlink APIs or create a fake sw eswitch object which doesn't exist in hw.
>
> >This is because unprivileged VF doesn't have visibility to eswitch and its
> links.
> >
> >> I think the terminology need to be defined clearly so that we are all
> >> on the same page.
> >>
> >> >
> >> >> Currently we have ndo_set_vf_mac_addr api that works with PF
> >> >> netdev, but i think we are trying to move away from that API and
> >> >> do all the configuration via the port representor netdevs.
> >> > This is fine rep-netdev represents eswitch port.
> >> > You normally don't go to switch to program host port params.
> >> >
> >> >> As the mac address cannot be configured using this netdev, i think
> >> >> Jakub is suggesting creating a devlink opject for each port
> >> >> representor and use that interface to set peer mac address.
> >> >
> >> > I understand but is convoluted interface.
> >> > When you program host NIC mac address you talk to iLo or BIOS.
> >> > When you program switch side mac address, you go
> switch/router/modem.
> >> >
> >> > Also programming host params on host side, also doesn't make
> >> assumption that its connected to eswitch.
> >> > It also doesn't assume that same connectivity for its life.
> >> >
> >> > If you model around how physical devices are configured, it will
> >> > almost
> >> never go wrong and still provides same level of flexibility.
> >> >
> >> >> We should be able use this to configure port vlan too.
> >> >>
> >> >> Also, instead of subport, can we call vport and support different
> >> >> types of vports - sr-iov, siov, vmdq etc.
> >> >>
> >> > At switch level there are just ports.
> >> > sriov, siov, mdev, vmdq are their couter part (peer) where it is
> connected.
> >> >
> >> >>>
> >> >>> Also eswitch is flat. There is no need of pf/vf flavour for port.
> >> >>> It doesn't make sense to define 'mdev' flavour which we are
> >> >>> already
> >> >> working.
> >> >>> At eswitch level it is just a port, it happen to be connected to
> >> >>> vf or pf or
> >> >> other objects, it doesn't matter.
> >> >>> Port should be flavoured as 'hostport' or 'switchport'.
> >> >>>
> >> >>>
> >> >>>> (using the port ids from above)
^ permalink raw reply [flat|nested] 100+ messages in thread
* RE: [PATCH net-next v2 4/7] devlink: allow subports on devlink PCI ports
2019-03-15 20:44 ` Jakub Kicinski
@ 2019-03-15 22:12 ` Parav Pandit
2019-03-16 1:16 ` Jakub Kicinski
2019-03-18 12:11 ` Jiri Pirko
1 sibling, 1 reply; 100+ messages in thread
From: Parav Pandit @ 2019-03-15 22:12 UTC (permalink / raw)
To: Jakub Kicinski, Jiri Pirko; +Cc: Samudrala, Sridhar, davem, netdev, oss-drivers
> -----Original Message-----
> From: Jakub Kicinski <jakub.kicinski@netronome.com>
> Sent: Friday, March 15, 2019 3:45 PM
> To: Jiri Pirko <jiri@resnulli.us>
> Cc: Parav Pandit <parav@mellanox.com>; Samudrala, Sridhar
> <sridhar.samudrala@intel.com>; davem@davemloft.net;
> netdev@vger.kernel.org; oss-drivers@netronome.com
> Subject: Re: [PATCH net-next v2 4/7] devlink: allow subports on devlink PCI
> ports
>
> On Fri, 15 Mar 2019 21:08:14 +0100, Jiri Pirko wrote:
> > >> IIUC, Jiri/Jakub are proposing creation of 2 devlink objects for
> > >> each port - host facing ports and switch facing ports. This is in
> > >> addition to the netdevs that are created today.
>
> To be clear I'm not in favour of the dual-object proposal.
>
> > >I am not proposing any different.
> > >I am proposing only two changes.
> > >1. control hostport params via referring hostport (not via indirect
> > >peer)
> >
> > Not really possible. If you passthrough VF into VM, the hostport goes
> > along with it.
> >
> > >2. flavour should not be vf/pf, flavour should be hostport, switchport.
> > >Because switch is flat and agnostic of pf/vf/mdev.
> >
> > Not sure. It's good to have this kind of visibility.
>
> Yes, this subthread honestly makes me go from 60% sure to 95% sure we
> shouldn't do the dual object thing :( Seems like Parav is already confused by
> it and suggests host port can exist without switch port :(
>
I am almost sure that I am not confused.
I am clear that hostports should be configured by devlink
instance which has the capability to program it.
When hostport is in VF, that VF usually won't have privilege
to program it and won't have visibility to eswitch either.
Why would you like to start with restrictive model of peer view only?
Hostports exist for infiniband HCA without switchport.
We should be able to manage hostport objects without creating fake eswitch sw object.
Jakub,
Can you please point to some example other than veth-pair where you configure host param (such as mac address) through a switch?
An existing example will help me to map it to devlink eswitch proposal.
If we go peer programming route, what are your thoughts on
how should we program infiniband hostports which doesn't have peer ports?
> > >> Are you suggesting that all the devlink objects should be visible
> > >> only at the hypervisor layer?
> > >>
> > >Of course not.
> > >
> > >Ports and params controlled by hypervisor should be exposed at
> hypervisor/eswitch wherever its parent devlink instance exist.
> > >Ports which should be visible inside a VM should be exposed inside a VM.
> > >So for a given VF,
> > >
> > >If eswitch is at hypervisor level,
> > >$ devlink port show
> > >pci/0000:05:00.0/10002 eth netdev flavour switchport switch_id
> > >00154d130d2f peer pci/0000:05:10.1/0
> > >pci/0000:05:10.1/0 eth netdev flavour hostport switch_id 00154d130d2f
> > >peer pci/0000:05:00.0/10002
> > >
> > >where VF is enumerated,
> > >$ devlink port show
> > >pci/0000:05:10.1/0 eth netdev flavour hostport
> >
> > So this is how it looks like in VM, right?
> >
> > >This is because unprivileged VF doesn't have visibility to eswitch and its
> links.
^ permalink raw reply [flat|nested] 100+ messages in thread
* Re: [PATCH net-next v2 4/7] devlink: allow subports on devlink PCI ports
2019-03-15 22:12 ` Parav Pandit
@ 2019-03-16 1:16 ` Jakub Kicinski
2019-03-18 15:43 ` Parav Pandit
0 siblings, 1 reply; 100+ messages in thread
From: Jakub Kicinski @ 2019-03-16 1:16 UTC (permalink / raw)
To: Parav Pandit; +Cc: Jiri Pirko, Samudrala, Sridhar, davem, netdev, oss-drivers
On Fri, 15 Mar 2019 22:12:13 +0000, Parav Pandit wrote:
> > On Fri, 15 Mar 2019 21:08:14 +0100, Jiri Pirko wrote:
> > > >> IIUC, Jiri/Jakub are proposing creation of 2 devlink objects for
> > > >> each port - host facing ports and switch facing ports. This is in
> > > >> addition to the netdevs that are created today.
> >
> > To be clear I'm not in favour of the dual-object proposal.
> >
> > > >I am not proposing any different.
> > > >I am proposing only two changes.
> > > >1. control hostport params via referring hostport (not via indirect
> > > >peer)
> > >
> > > Not really possible. If you passthrough VF into VM, the hostport goes
> > > along with it.
> > >
> > > >2. flavour should not be vf/pf, flavour should be hostport, switchport.
> > > >Because switch is flat and agnostic of pf/vf/mdev.
> > >
> > > Not sure. It's good to have this kind of visibility.
> >
> > Yes, this subthread honestly makes me go from 60% sure to 95% sure we
> > shouldn't do the dual object thing :( Seems like Parav is already confused by
> > it and suggests host port can exist without switch port :(
> >
> I am almost sure that I am not confused.
> I am clear that hostports should be configured by devlink
> instance which has the capability to program it.
Right now a devlink port is something that the datapath of an ASIC can
address. All flavours we have presently are basically various MACs -
physical (front panel ports), DSA - for ASIC interconnects on a
multi-ASIC board, CPU - for connecting to a MAC of a NIC.
Jiri's flavour proposal was strictly extending the same logic to
SR-IOV. Each object addressable within the datapath gets a port.
The datapath's ID can be used as port_index.
I just reimplemented his patches here and added the subports which I
think he wasn't aware of as they are a quirk of old NFP ASICs.
Having 3 objects for the same datapath ID is a significant departure
from the existing devlink port semantics.
> When hostport is in VF, that VF usually won't have privilege
> to program it and won't have visibility to eswitch either.
If VM has no visibility into the eswitch and no permission to configure
things, what use does the object serve?
> Why would you like to start with restrictive model of peer view only?
"Restrictive model" is one way of putting it. I'd rather say that we
are not adding objects which:
(a) do not adhere to current semantics;
(b) have no distinct function.
We can make the "add MAC address" command not use the word peer:
devlink port addr_pool add pci/0000:05:00.0/10003 type eth 00:11:22:33:44:55
devlink port addr_pool del pci/0000:05:00.0/10003 type eth 00:11:22:33:44:55
if the "peer" doesn't sit right.
> Hostports exist for infiniband HCA without switchport.
> We should be able to manage hostport objects without creating fake eswitch sw object.
It sounds like the RDMA subsystem is lacking a model to represent all
its objects, but that's RDMA's problem to solve..
In netdev world we have netdevs for ports which a used for bulk of the
configuration, most importantly - forwarding.
> Jakub,
> Can you please point to some example other than veth-pair where you
> configure host param (such as mac address) through a switch?
Existing "legacy" SR-IOV NDOs.
> An existing example will help me to map it to devlink eswitch proposal.
> If we go peer programming route, what are your thoughts on
> how should we program infiniband hostports which doesn't have peer ports?
Again, you may be trying to fix RDMA's lack of control objects, which
may be better fixed elsewhere..
^ permalink raw reply [flat|nested] 100+ messages in thread
* Re: [PATCH net-next v2 4/7] devlink: allow subports on devlink PCI ports
2019-03-15 20:44 ` Jakub Kicinski
2019-03-15 22:12 ` Parav Pandit
@ 2019-03-18 12:11 ` Jiri Pirko
2019-03-18 19:16 ` Jakub Kicinski
1 sibling, 1 reply; 100+ messages in thread
From: Jiri Pirko @ 2019-03-18 12:11 UTC (permalink / raw)
To: Jakub Kicinski
Cc: Parav Pandit, Samudrala, Sridhar, davem, netdev, oss-drivers
Fri, Mar 15, 2019 at 09:44:54PM CET, jakub.kicinski@netronome.com wrote:
>On Fri, 15 Mar 2019 21:08:14 +0100, Jiri Pirko wrote:
>> >> IIUC, Jiri/Jakub are proposing creation of 2 devlink objects for each port -
>> >> host facing ports and switch facing ports. This is in addition to the netdevs
>> >> that are created today.
>
>To be clear I'm not in favour of the dual-object proposal.
>
>> >I am not proposing any different.
>> >I am proposing only two changes.
>> >1. control hostport params via referring hostport (not via indirect peer)
>>
>> Not really possible. If you passthrough VF into VM, the hostport goes
>> along with it.
>>
>> >2. flavour should not be vf/pf, flavour should be hostport, switchport.
>> >Because switch is flat and agnostic of pf/vf/mdev.
>>
>> Not sure. It's good to have this kind of visibility.
>
>Yes, this subthread honestly makes me go from 60% sure to 95% sure we
>shouldn't do the dual object thing :( Seems like Parav is already
>confused by it and suggests host port can exist without switch port :(
Although I understand your hesitation, the host ports are also
associated with the asic and should be under the devlink instance.
It is just a matter of proper documentation and clear code to avoid
confusions.
>
>> >> Are you suggesting that all the devlink objects should be visible only at the
>> >> hypervisor layer?
>> >>
>> >Of course not.
>> >
>> >Ports and params controlled by hypervisor should be exposed at hypervisor/eswitch wherever its parent devlink instance exist.
>> >Ports which should be visible inside a VM should be exposed inside a VM.
>> >So for a given VF,
>> >
>> >If eswitch is at hypervisor level,
>> >$ devlink port show
>> >pci/0000:05:00.0/10002 eth netdev flavour switchport switch_id 00154d130d2f peer pci/0000:05:10.1/0
>> >pci/0000:05:10.1/0 eth netdev flavour hostport switch_id 00154d130d2f peer pci/0000:05:00.0/10002
>> >
>> >where VF is enumerated,
>> >$ devlink port show
>> >pci/0000:05:10.1/0 eth netdev flavour hostport
>>
>> So this is how it looks like in VM, right?
>>
>> >This is because unprivileged VF doesn't have visibility to eswitch and its links.
>
^ permalink raw reply [flat|nested] 100+ messages in thread
* Re: [PATCH net-next v2 4/7] devlink: allow subports on devlink PCI ports
2019-03-15 21:59 ` Parav Pandit
@ 2019-03-18 12:21 ` Jiri Pirko
2019-03-18 15:56 ` Parav Pandit
2019-03-18 19:19 ` Jakub Kicinski
0 siblings, 2 replies; 100+ messages in thread
From: Jiri Pirko @ 2019-03-18 12:21 UTC (permalink / raw)
To: Parav Pandit
Cc: Samudrala, Sridhar, Jakub Kicinski, davem, netdev, oss-drivers
Fri, Mar 15, 2019 at 10:59:33PM CET, parav@mellanox.com wrote:
>
>
>> -----Original Message-----
>> From: Jiri Pirko <jiri@resnulli.us>
>> Sent: Friday, March 15, 2019 3:08 PM
>> To: Parav Pandit <parav@mellanox.com>
>> Cc: Samudrala, Sridhar <sridhar.samudrala@intel.com>; Jakub Kicinski
>> <jakub.kicinski@netronome.com>; davem@davemloft.net;
>> netdev@vger.kernel.org; oss-drivers@netronome.com
>> Subject: Re: [PATCH net-next v2 4/7] devlink: allow subports on devlink PCI
>> ports
>>
>> Fri, Mar 15, 2019 at 04:32:24PM CET, parav@mellanox.com wrote:
>> >
>> >
>> >> -----Original Message-----
>> >> From: Samudrala, Sridhar <sridhar.samudrala@intel.com>
>> >> Sent: Friday, March 15, 2019 12:58 AM
>> >> To: Parav Pandit <parav@mellanox.com>; Jakub Kicinski
>> >> <jakub.kicinski@netronome.com>
>> >> Cc: Jiri Pirko <jiri@resnulli.us>; davem@davemloft.net;
>> >> netdev@vger.kernel.org; oss-drivers@netronome.com
>> >> Subject: Re: [PATCH net-next v2 4/7] devlink: allow subports on
>> >> devlink PCI ports
>> >>
>> >>
>> >> On 3/14/2019 7:40 PM, Parav Pandit wrote:
>> >> >
>> >> >
>> >> >> -----Original Message-----
>> >> >> From: Samudrala, Sridhar <sridhar.samudrala@intel.com>
>> >> >> Sent: Thursday, March 14, 2019 9:16 PM
>> >> >> To: Parav Pandit <parav@mellanox.com>; Jakub Kicinski
>> >> >> <jakub.kicinski@netronome.com>
>> >> >> Cc: Jiri Pirko <jiri@resnulli.us>; davem@davemloft.net;
>> >> >> netdev@vger.kernel.org; oss-drivers@netronome.com
>> >> >> Subject: Re: [PATCH net-next v2 4/7] devlink: allow subports on
>> >> >> devlink PCI ports
>> >> >>
>> >> >>
>> >> >>
>> >> >> On 3/14/2019 6:28 PM, Parav Pandit wrote:
>> >> >>>
>> >> >>>
>> >> >>>> -----Original Message-----
>> >> >>>> From: Jakub Kicinski <jakub.kicinski@netronome.com>
>> >> >>>> Sent: Thursday, March 14, 2019 6:39 PM
>> >> >>>> To: Parav Pandit <parav@mellanox.com>
>> >> >>>> Cc: Jiri Pirko <jiri@resnulli.us>; davem@davemloft.net;
>> >> >>>> netdev@vger.kernel.org; oss-drivers@netronome.com
>> >> >>>> Subject: Re: [PATCH net-next v2 4/7] devlink: allow subports on
>> >> >>>> devlink PCI ports
>> >> >>>>
>> >> >>>> On Thu, 14 Mar 2019 22:35:36 +0000, Parav Pandit wrote:
>> >> >>>>>>> Then instances of flavour pci_vf are going to appear in the
>> >> >>>>>>> same devlink instance. Those are the switch ports:
>> >> >>>>>>> pci/0000:05:00.0/10002: type eth netdev enp5s0npf0pf0s0
>> >> >>>>>>> flavour pci_vf pf 0 vf 0
>> >> >>>>>>> switch_id 00154d130d2f peer
>> >> >>>>>>> pci/0000:05:10.1/0
>> >> >>>>>>> pci/0000:05:00.0/10003: type eth netdev enp5s0npf0pf0s0
>> >> >>>>>>> flavour pci_vf pf 0 vf 0 subport 1
>> >> >>>>>>> switch_id 00154d130d2f peer
>> >> >>>>>>> pci/0000:05:10.1/1
>> >> >>>>>>>
>> >> >>>>>>> With that, peers are going to appear too, and those are the
>> >> >>>>>>> actual VF/VF
>> >> >>>>>>> subport:
>> >> >>>>>>> pci/0000:05:10.1/0: type eth netdev ??? flavour pci_vf_host
>> >> >>>>>>> peer pci/0000:05:00.0/10002
>> >> >>>>>>> pci/0000:05:10.1/1: type eth netdev ??? flavour pci_vf_host
>> >> >>>>>>> peer pci/0000:05:00.0/10003
>> >> >>>>>>>
>> >> >>>>>>> Later you can push this VF along with all subports to VM. So
>> >> >>>>>>> in VM, you are going to see the VF like this:
>> >> >>>>>>> $ devlink dev
>> >> >>>>>>> pci/0000:00:08.0
>> >> >>>>>>> $ devlink port
>> >> >>>>>>> pci/0000:00:08.0/0: type eth netdev ??? flavour pci_vf_host
>> >> >>>>>>> pci/0000:00:08.0/1: type eth netdev ??? flavour pci_vf_host
>> >> >>>>>>>
>> >> >>>>>>> And back to your question of how are they connected in eswitch.
>> >> >>>>>>> That is totally up to the original user John who did the creation.
>> >> >>>>>>> He is in charge of the eswitch on baremetal, he would
>> >> >>>>>>> configure the forwarding however he likes.
>> >> >>>>>>
>> >> >>>>>> Ack, so I think you're saying VM has to communicate to the
>> >> >>>>>> cloud environment to have this provisioned using some service
>> >> >>>>>> API, not a kernel API. That's what I wanted to confirm.
>> >> >>>>>>
>> >> >>>>>> I don't see any benefit to having the "host ports" under
>> >> >>>>>> devlink, as such I think it's a matter of preference.
>> >> >>>>>
>> >> >>>>> We need 'host ports' to configure parameters of this host port
>> >> >>>>> which is not exposed by the rep-netdev.
>> >> >>>>> Such as mac address.
>> >> >>>>
>> >> >>>> Please look at the quote of what Jiri wrote above - the host
>> >> >>>> port gets passed to the VM, you can't use it as a handle to set the
>> MAC.
>> >> >>>>
>> >> >>>> The way to set the MAC remains:
>> >> >>>>
>> >> >>>> # devlink port set pci/0000:05:00.0/10002 peer mac_addr
>> >> >>>> 00:11:22:33:44:55
>> >> >>>>
>> >> >>> Even though it can be done, I think this is wrong model to
>> >> >>> program
>> >> >> hostport mac address using eswitch port.
>> >> >>> All devlink objects are control objects, so what is passed to VM
>> >> >>> is what is
>> >> >> represented by devlink.
>> >> >>> VF in the VM will anyway create its devlink object.
>> >> >>> What is wrong in programming hostport?
>> >> >>> It gives a very clear view to users of topology and objects.
>> >> >>
>> >> >> The VF or any subport MAC address should be configured by the
>> >> >> orchestration layer that is running on the hypervisor and when a
>> >> >> VF is assigned to a VF, the host port is not visible to the hypervisor.
>> >> > What prevents creation of hostport due to which is not visible?
>> >> > Hostport is control port to program host side of parameters.
>> >> > It should be created when user wants to program the parameters.
>> >> >
>> >> > Model is really straight forward.
>> >> > Program host port params using hostport object.
>> >> > Program switchport params using rep-netdev.
>> >>
>> >> IIUC, Jiri/Jakub are proposing creation of 2 devlink objects for each
>> >> port - host facing ports and switch facing ports. This is in addition
>> >> to the netdevs that are created today.
>> >>
>> >I am not proposing any different.
>> >I am proposing only two changes.
>> >1. control hostport params via referring hostport (not via indirect
>> >peer)
>>
>> Not really possible. If you passthrough VF into VM, the hostport goes along
>> with it.
>>
>No.
>I am sorry in showing the enumeration which is the source of confusion.
>
>Below is the right enumeration.
>
>When VF is enumerated initially in the host, where eswitch devlink instance is located.
>Below enumeration is seen.
>
>First two entries shows the link between hostport and switchport.
>$ devlink port show
>pci/0000:05:00.0/10002 eth netdev flavour switchport switch_id 00154d130d2f peer pci/0000:05:00.0/1
>
>pci/0000:05:00.0/1 eth netdev flavour hostport switch_id 00154d130d2f peer pci/0000:05:00.0/10002
Hostport should not have switch_id.
>
>pci/0000:05:10.1/0 eth netdev flavour hostport
>This entry won't be seen if VF auto probing is disabled. Because than VF is not enumerated.
>
>As a user, I will be programming the mac address of hostport for a VF.
>pci/0000:05:00.0/1 eth netdev flavour hostport switch_id 00154d130d2f peer pci/0000:05:00.0/10002
Hmm, so you are going to have 2 hostports for VF:
1) pci/0000:05:10.1/0
real one, that is going to go to VM - with a separate pci address
and devlink instance.
2) pci/0000:05:00.0/1
dummy one, which is not really a hostport, as there is no netdev
created for it. It only models the other side of cable, which is away
in VM.
>
>
>>
>> >2. flavour should not be vf/pf, flavour should be hostport, switchport.
>> >Because switch is flat and agnostic of pf/vf/mdev.
>>
>> Not sure. It's good to have this kind of visibility.
>>
>port can have label/attribute indicating that this belong to VF-1 or mdev as long as you are agreeing to have mdev attribute on host port.
>(and not ask for abstracting it, because mdev is well defined kernel object).
Why mdev cannot be another flavour?
>
>>
>> >
>> >> Are you suggesting that all the devlink objects should be visible
>> >> only at the hypervisor layer?
>> >>
>> >Of course not.
>> >
>> >Ports and params controlled by hypervisor should be exposed at
>> hypervisor/eswitch wherever its parent devlink instance exist.
>> >Ports which should be visible inside a VM should be exposed inside a VM.
>> >So for a given VF,
>> >
>> >If eswitch is at hypervisor level,
>> >$ devlink port show
>> >pci/0000:05:00.0/10002 eth netdev flavour switchport switch_id
>> >00154d130d2f peer pci/0000:05:10.1/0
>> >pci/0000:05:10.1/0 eth netdev flavour hostport switch_id 00154d130d2f
>> >peer pci/0000:05:00.0/10002
>> >
>> >where VF is enumerated,
>> >$ devlink port show
>> >pci/0000:05:10.1/0 eth netdev flavour hostport
>>
>> So this is how it looks like in VM, right?
>>
>Yep.
>Once VF is mapped to VM only two entries are seen and hostport can be still controlled.
>
>$ devlink port show
>pci/0000:05:00.0/10002 eth netdev flavour switchport switch_id 00154d130d2f peer pci/0000:05:00.0/1
>
>pci/0000:05:00.0/1 eth netdev flavour hostport switch_id 00154d130d2f peer pci/0000:05:00.0/10002
>
>This addresses the case for Infiniband where there is no eswitch, but hostports exists and should be managed.
>We shouldn't be inventing new devlink APIs or create a fake sw eswitch object which doesn't exist in hw.
>
>>
>> >This is because unprivileged VF doesn't have visibility to eswitch and its
>> links.
>> >
>> >> I think the terminology need to be defined clearly so that we are all
>> >> on the same page.
>> >>
>> >> >
>> >> >> Currently we have ndo_set_vf_mac_addr api that works with PF
>> >> >> netdev, but i think we are trying to move away from that API and
>> >> >> do all the configuration via the port representor netdevs.
>> >> > This is fine rep-netdev represents eswitch port.
>> >> > You normally don't go to switch to program host port params.
>> >> >
>> >> >> As the mac address cannot be configured using this netdev, i think
>> >> >> Jakub is suggesting creating a devlink opject for each port
>> >> >> representor and use that interface to set peer mac address.
>> >> >
>> >> > I understand but is convoluted interface.
>> >> > When you program host NIC mac address you talk to iLo or BIOS.
>> >> > When you program switch side mac address, you go
>> switch/router/modem.
>> >> >
>> >> > Also programming host params on host side, also doesn't make
>> >> assumption that its connected to eswitch.
>> >> > It also doesn't assume that same connectivity for its life.
>> >> >
>> >> > If you model around how physical devices are configured, it will
>> >> > almost
>> >> never go wrong and still provides same level of flexibility.
>> >> >
>> >> >> We should be able use this to configure port vlan too.
>> >> >>
>> >> >> Also, instead of subport, can we call vport and support different
>> >> >> types of vports - sr-iov, siov, vmdq etc.
>> >> >>
>> >> > At switch level there are just ports.
>> >> > sriov, siov, mdev, vmdq are their couter part (peer) where it is
>> connected.
>> >> >
>> >> >>>
>> >> >>> Also eswitch is flat. There is no need of pf/vf flavour for port.
>> >> >>> It doesn't make sense to define 'mdev' flavour which we are
>> >> >>> already
>> >> >> working.
>> >> >>> At eswitch level it is just a port, it happen to be connected to
>> >> >>> vf or pf or
>> >> >> other objects, it doesn't matter.
>> >> >>> Port should be flavoured as 'hostport' or 'switchport'.
>> >> >>>
>> >> >>>
>> >> >>>> (using the port ids from above)
^ permalink raw reply [flat|nested] 100+ messages in thread
* RE: [PATCH net-next v2 4/7] devlink: allow subports on devlink PCI ports
2019-03-16 1:16 ` Jakub Kicinski
@ 2019-03-18 15:43 ` Parav Pandit
2019-03-18 19:29 ` Jakub Kicinski
0 siblings, 1 reply; 100+ messages in thread
From: Parav Pandit @ 2019-03-18 15:43 UTC (permalink / raw)
To: Jakub Kicinski; +Cc: Jiri Pirko, Samudrala, Sridhar, davem, netdev, oss-drivers
> -----Original Message-----
> From: Jakub Kicinski <jakub.kicinski@netronome.com>
> Sent: Friday, March 15, 2019 8:16 PM
> To: Parav Pandit <parav@mellanox.com>
> Cc: Jiri Pirko <jiri@resnulli.us>; Samudrala, Sridhar
> <sridhar.samudrala@intel.com>; davem@davemloft.net;
> netdev@vger.kernel.org; oss-drivers@netronome.com
> Subject: Re: [PATCH net-next v2 4/7] devlink: allow subports on devlink PCI
> ports
>
> On Fri, 15 Mar 2019 22:12:13 +0000, Parav Pandit wrote:
> > > On Fri, 15 Mar 2019 21:08:14 +0100, Jiri Pirko wrote:
> > > > >> IIUC, Jiri/Jakub are proposing creation of 2 devlink objects
> > > > >> for each port - host facing ports and switch facing ports. This
> > > > >> is in addition to the netdevs that are created today.
> > >
> > > To be clear I'm not in favour of the dual-object proposal.
> > >
> > > > >I am not proposing any different.
> > > > >I am proposing only two changes.
> > > > >1. control hostport params via referring hostport (not via
> > > > >indirect
> > > > >peer)
> > > >
> > > > Not really possible. If you passthrough VF into VM, the hostport
> > > > goes along with it.
> > > >
> > > > >2. flavour should not be vf/pf, flavour should be hostport, switchport.
> > > > >Because switch is flat and agnostic of pf/vf/mdev.
> > > >
> > > > Not sure. It's good to have this kind of visibility.
> > >
> > > Yes, this subthread honestly makes me go from 60% sure to 95% sure
> > > we shouldn't do the dual object thing :( Seems like Parav is
> > > already confused by it and suggests host port can exist without
> > > switch port :(
> > >
> > I am almost sure that I am not confused.
> > I am clear that hostports should be configured by devlink instance
> > which has the capability to program it.
>
> Right now a devlink port is something that the datapath of an ASIC can
> address. All flavours we have presently are basically various MACs - physical
> (front panel ports), DSA - for ASIC interconnects on a multi-ASIC board, CPU -
> for connecting to a MAC of a NIC.
>
Devlink port implementation in commit doesn't say that it is for ASIC datapath or limited to ASIC datapath id.
It is not right to say that 'whole datapath' object should be represented with just single object 'port'.
Datapath involves various stages in ASIC each does different processing.
These datapath objects are interconnected, i.e. hostport is connected to switchport.
Commit [1] says devlink port is physical port. However we already have 3 flavours of port.
> Jiri's flavour proposal was strictly extending the same logic to SR-IOV. Each
> object addressable within the datapath gets a port.
> The datapath's ID can be used as port_index.
>
And as I said, it is already restrictive.
Port is a port, it can be labeled for vf/pf, but flavour is not really vf/pf.
Also label applies more on the hostport side vs switchport.
> I just reimplemented his patches here and added the subports which I think
> he wasn't aware of as they are a quirk of old NFP ASICs.
>
> Having 3 objects for the same datapath ID is a significant departure from the
> existing devlink port semantics.
>
It is really not same datapath ID.
Because if that is the case, we should be programming mac address on the rep-netdev itself.
But we are not doing that because rep-netdev represents only 'eswitch port'.
> > When hostport is in VF, that VF usually won't have privilege to
> > program it and won't have visibility to eswitch either.
>
> If VM has no visibility into the eswitch and no permission to configure
> things, what use does the object serve?
>
To view device properties, health, RO registers, more importantly its port details.
Yonatan is working on grouping these devlink ports and those are control through devlink APIs.
Jiri is actively internally reviewing those patches since last 3+ weeks, not finished yet.
So this visibility is needed anyway.
> > Why would you like to start with restrictive model of peer view only?
>
> "Restrictive model" is one way of putting it. I'd rather say that we are not
> adding objects which:
> (a) do not adhere to current semantics;
> (b) have no distinct function.
>
hostport certainly has distinct function than switchport.
i.e. to program host side parameters. (eth.mac, rdma.port_guid and more in future).
> We can make the "add MAC address" command not use the word peer:
>
> devlink port addr_pool add pci/0000:05:00.0/10003 type eth
> 00:11:22:33:44:55 devlink port addr_pool del pci/0000:05:00.0/10003 type
> eth 00:11:22:33:44:55
>
> if the "peer" doesn't sit right.
>
> > Hostports exist for infiniband HCA without switchport.
> > We should be able to manage hostport objects without creating fake
> eswitch sw object.
>
> It sounds like the RDMA subsystem is lacking a model to represent all its
> objects, but that's RDMA's problem to solve..
>
devlink framework is not limited to Ethernet, it operates on bus/device notion.
So for Ethernet vendors program mac address.
For rdma vendor programs port_guid (which is equivalent of mac address).
devlink also publishes rdma device info today.
net/core/devlink.c has very well established IB device info exposed via devlink_nl_port_fill() for more than 3 years now in commit [2].
It is not fair to say create, solve it somewhere else.
> In netdev world we have netdevs for ports which a used for bulk of the
> configuration, most importantly - forwarding.
>
> > Jakub,
> > Can you please point to some example other than veth-pair where you
> > configure host param (such as mac address) through a switch?
>
> Existing "legacy" SR-IOV NDOs.
>
That is perfect example of programming hostport parameters, without a eswitch..
At high level, I was looking where you open switch GUI/cli or something equivalent that program's host's mac address..
So far we don't have such equivalent good example yet..
> > An existing example will help me to map it to devlink eswitch proposal.
> > If we go peer programming route, what are your thoughts on how should
> > we program infiniband hostports which doesn't have peer ports?
>
> Again, you may be trying to fix RDMA's lack of control objects, which may be
> better fixed elsewhere..
devlink port is link agnostic control object.
[1] bfcd3a46617209454cfc0947ab093e37fd1e84ef
[2] commit id bfcd3a466
^ permalink raw reply [flat|nested] 100+ messages in thread
* RE: [PATCH net-next v2 4/7] devlink: allow subports on devlink PCI ports
2019-03-18 12:21 ` Jiri Pirko
@ 2019-03-18 15:56 ` Parav Pandit
2019-03-18 16:22 ` Parav Pandit
2019-03-18 19:19 ` Jakub Kicinski
1 sibling, 1 reply; 100+ messages in thread
From: Parav Pandit @ 2019-03-18 15:56 UTC (permalink / raw)
To: Jiri Pirko; +Cc: Samudrala, Sridhar, Jakub Kicinski, davem, netdev, oss-drivers
> -----Original Message-----
> From: Jiri Pirko <jiri@resnulli.us>
> Sent: Monday, March 18, 2019 7:21 AM
> To: Parav Pandit <parav@mellanox.com>
> Cc: Samudrala, Sridhar <sridhar.samudrala@intel.com>; Jakub Kicinski
> <jakub.kicinski@netronome.com>; davem@davemloft.net;
> netdev@vger.kernel.org; oss-drivers@netronome.com
> Subject: Re: [PATCH net-next v2 4/7] devlink: allow subports on devlink PCI
> ports
>
> Fri, Mar 15, 2019 at 10:59:33PM CET, parav@mellanox.com wrote:
> >
> >
> >> -----Original Message-----
> >> From: Jiri Pirko <jiri@resnulli.us>
> >> Sent: Friday, March 15, 2019 3:08 PM
> >> To: Parav Pandit <parav@mellanox.com>
> >> Cc: Samudrala, Sridhar <sridhar.samudrala@intel.com>; Jakub Kicinski
> >> <jakub.kicinski@netronome.com>; davem@davemloft.net;
> >> netdev@vger.kernel.org; oss-drivers@netronome.com
> >> Subject: Re: [PATCH net-next v2 4/7] devlink: allow subports on
> >> devlink PCI ports
> >>
> >> Fri, Mar 15, 2019 at 04:32:24PM CET, parav@mellanox.com wrote:
> >> >
> >> >
> >> >> -----Original Message-----
> >> >> From: Samudrala, Sridhar <sridhar.samudrala@intel.com>
> >> >> Sent: Friday, March 15, 2019 12:58 AM
> >> >> To: Parav Pandit <parav@mellanox.com>; Jakub Kicinski
> >> >> <jakub.kicinski@netronome.com>
> >> >> Cc: Jiri Pirko <jiri@resnulli.us>; davem@davemloft.net;
> >> >> netdev@vger.kernel.org; oss-drivers@netronome.com
> >> >> Subject: Re: [PATCH net-next v2 4/7] devlink: allow subports on
> >> >> devlink PCI ports
> >> >>
> >> >>
> >> >> On 3/14/2019 7:40 PM, Parav Pandit wrote:
> >> >> >
> >> >> >
> >> >> >> -----Original Message-----
> >> >> >> From: Samudrala, Sridhar <sridhar.samudrala@intel.com>
> >> >> >> Sent: Thursday, March 14, 2019 9:16 PM
> >> >> >> To: Parav Pandit <parav@mellanox.com>; Jakub Kicinski
> >> >> >> <jakub.kicinski@netronome.com>
> >> >> >> Cc: Jiri Pirko <jiri@resnulli.us>; davem@davemloft.net;
> >> >> >> netdev@vger.kernel.org; oss-drivers@netronome.com
> >> >> >> Subject: Re: [PATCH net-next v2 4/7] devlink: allow subports on
> >> >> >> devlink PCI ports
> >> >> >>
> >> >> >>
> >> >> >>
> >> >> >> On 3/14/2019 6:28 PM, Parav Pandit wrote:
> >> >> >>>
> >> >> >>>
> >> >> >>>> -----Original Message-----
> >> >> >>>> From: Jakub Kicinski <jakub.kicinski@netronome.com>
> >> >> >>>> Sent: Thursday, March 14, 2019 6:39 PM
> >> >> >>>> To: Parav Pandit <parav@mellanox.com>
> >> >> >>>> Cc: Jiri Pirko <jiri@resnulli.us>; davem@davemloft.net;
> >> >> >>>> netdev@vger.kernel.org; oss-drivers@netronome.com
> >> >> >>>> Subject: Re: [PATCH net-next v2 4/7] devlink: allow subports
> >> >> >>>> on devlink PCI ports
> >> >> >>>>
> >> >> >>>> On Thu, 14 Mar 2019 22:35:36 +0000, Parav Pandit wrote:
> >> >> >>>>>>> Then instances of flavour pci_vf are going to appear in
> >> >> >>>>>>> the same devlink instance. Those are the switch ports:
> >> >> >>>>>>> pci/0000:05:00.0/10002: type eth netdev enp5s0npf0pf0s0
> >> >> >>>>>>> flavour pci_vf pf 0 vf 0
> >> >> >>>>>>> switch_id 00154d130d2f peer
> >> >> >>>>>>> pci/0000:05:10.1/0
> >> >> >>>>>>> pci/0000:05:00.0/10003: type eth netdev enp5s0npf0pf0s0
> >> >> >>>>>>> flavour pci_vf pf 0 vf 0 subport 1
> >> >> >>>>>>> switch_id 00154d130d2f peer
> >> >> >>>>>>> pci/0000:05:10.1/1
> >> >> >>>>>>>
> >> >> >>>>>>> With that, peers are going to appear too, and those are
> >> >> >>>>>>> the actual VF/VF
> >> >> >>>>>>> subport:
> >> >> >>>>>>> pci/0000:05:10.1/0: type eth netdev ??? flavour pci_vf_host
> >> >> >>>>>>> peer pci/0000:05:00.0/10002
> >> >> >>>>>>> pci/0000:05:10.1/1: type eth netdev ??? flavour pci_vf_host
> >> >> >>>>>>> peer pci/0000:05:00.0/10003
> >> >> >>>>>>>
> >> >> >>>>>>> Later you can push this VF along with all subports to VM.
> >> >> >>>>>>> So in VM, you are going to see the VF like this:
> >> >> >>>>>>> $ devlink dev
> >> >> >>>>>>> pci/0000:00:08.0
> >> >> >>>>>>> $ devlink port
> >> >> >>>>>>> pci/0000:00:08.0/0: type eth netdev ??? flavour
> >> >> >>>>>>> pci_vf_host
> >> >> >>>>>>> pci/0000:00:08.0/1: type eth netdev ??? flavour
> >> >> >>>>>>> pci_vf_host
> >> >> >>>>>>>
> >> >> >>>>>>> And back to your question of how are they connected in
> eswitch.
> >> >> >>>>>>> That is totally up to the original user John who did the
> creation.
> >> >> >>>>>>> He is in charge of the eswitch on baremetal, he would
> >> >> >>>>>>> configure the forwarding however he likes.
> >> >> >>>>>>
> >> >> >>>>>> Ack, so I think you're saying VM has to communicate to the
> >> >> >>>>>> cloud environment to have this provisioned using some
> >> >> >>>>>> service API, not a kernel API. That's what I wanted to confirm.
> >> >> >>>>>>
> >> >> >>>>>> I don't see any benefit to having the "host ports" under
> >> >> >>>>>> devlink, as such I think it's a matter of preference.
> >> >> >>>>>
> >> >> >>>>> We need 'host ports' to configure parameters of this host
> >> >> >>>>> port which is not exposed by the rep-netdev.
> >> >> >>>>> Such as mac address.
> >> >> >>>>
> >> >> >>>> Please look at the quote of what Jiri wrote above - the host
> >> >> >>>> port gets passed to the VM, you can't use it as a handle to
> >> >> >>>> set the
> >> MAC.
> >> >> >>>>
> >> >> >>>> The way to set the MAC remains:
> >> >> >>>>
> >> >> >>>> # devlink port set pci/0000:05:00.0/10002 peer mac_addr
> >> >> >>>> 00:11:22:33:44:55
> >> >> >>>>
> >> >> >>> Even though it can be done, I think this is wrong model to
> >> >> >>> program
> >> >> >> hostport mac address using eswitch port.
> >> >> >>> All devlink objects are control objects, so what is passed to
> >> >> >>> VM is what is
> >> >> >> represented by devlink.
> >> >> >>> VF in the VM will anyway create its devlink object.
> >> >> >>> What is wrong in programming hostport?
> >> >> >>> It gives a very clear view to users of topology and objects.
> >> >> >>
> >> >> >> The VF or any subport MAC address should be configured by the
> >> >> >> orchestration layer that is running on the hypervisor and when
> >> >> >> a VF is assigned to a VF, the host port is not visible to the
> hypervisor.
> >> >> > What prevents creation of hostport due to which is not visible?
> >> >> > Hostport is control port to program host side of parameters.
> >> >> > It should be created when user wants to program the parameters.
> >> >> >
> >> >> > Model is really straight forward.
> >> >> > Program host port params using hostport object.
> >> >> > Program switchport params using rep-netdev.
> >> >>
> >> >> IIUC, Jiri/Jakub are proposing creation of 2 devlink objects for
> >> >> each port - host facing ports and switch facing ports. This is in
> >> >> addition to the netdevs that are created today.
> >> >>
> >> >I am not proposing any different.
> >> >I am proposing only two changes.
> >> >1. control hostport params via referring hostport (not via indirect
> >> >peer)
> >>
> >> Not really possible. If you passthrough VF into VM, the hostport goes
> >> along with it.
> >>
> >No.
> >I am sorry in showing the enumeration which is the source of confusion.
> >
> >Below is the right enumeration.
> >
> >When VF is enumerated initially in the host, where eswitch devlink instance
> is located.
> >Below enumeration is seen.
> >
> >First two entries shows the link between hostport and switchport.
> >$ devlink port show
> >pci/0000:05:00.0/10002 eth netdev flavour switchport switch_id
> >00154d130d2f peer pci/0000:05:00.0/1
> >
> >pci/0000:05:00.0/1 eth netdev flavour hostport switch_id 00154d130d2f
> >peer pci/0000:05:00.0/10002
>
> Hostport should not have switch_id.
>
> >
> >pci/0000:05:10.1/0 eth netdev flavour hostport This entry won't be seen
> >if VF auto probing is disabled. Because than VF is not enumerated.
> >
> >As a user, I will be programming the mac address of hostport for a VF.
> >pci/0000:05:00.0/1 eth netdev flavour hostport switch_id 00154d130d2f
> >peer pci/0000:05:00.0/10002
>
> Hmm, so you are going to have 2 hostports for VF:
> 1) pci/0000:05:10.1/0
> real one, that is going to go to VM - with a separate pci address
> and devlink instance.
Yep. This is the one where Yonatan's port grouping APIs work on.
> 2) pci/0000:05:00.0/1
> dummy one, which is not really a hostport, as there is no netdev
> created for it. It only models the other side of cable, which is away
> in VM.
>
Right. This is the control object which typically hypervisor programs.
> >
> >
> >>
> >> >2. flavour should not be vf/pf, flavour should be hostport, switchport.
> >> >Because switch is flat and agnostic of pf/vf/mdev.
> >>
> >> Not sure. It's good to have this kind of visibility.
> >>
> >port can have label/attribute indicating that this belong to VF-1 or mdev as
> long as you are agreeing to have mdev attribute on host port.
> >(and not ask for abstracting it, because mdev is well defined kernel object).
>
> Why mdev cannot be another flavour?
>
hostport is of type pf/vf/mdev connected to some switchport.
So proposal is to have,
port flavour = hostport/switchport
port type/label = pf/vf/mdev
> >
> >>
> >> >
> >> >> Are you suggesting that all the devlink objects should be visible
> >> >> only at the hypervisor layer?
> >> >>
> >> >Of course not.
> >> >
> >> >Ports and params controlled by hypervisor should be exposed at
> >> hypervisor/eswitch wherever its parent devlink instance exist.
> >> >Ports which should be visible inside a VM should be exposed inside a
> VM.
> >> >So for a given VF,
> >> >
> >> >If eswitch is at hypervisor level,
> >> >$ devlink port show
> >> >pci/0000:05:00.0/10002 eth netdev flavour switchport switch_id
> >> >00154d130d2f peer pci/0000:05:10.1/0
> >> >pci/0000:05:10.1/0 eth netdev flavour hostport switch_id
> >> >00154d130d2f peer pci/0000:05:00.0/10002
> >> >
> >> >where VF is enumerated,
> >> >$ devlink port show
> >> >pci/0000:05:10.1/0 eth netdev flavour hostport
> >>
> >> So this is how it looks like in VM, right?
> >>
> >Yep.
> >Once VF is mapped to VM only two entries are seen and hostport can be
> still controlled.
> >
> >$ devlink port show
> >pci/0000:05:00.0/10002 eth netdev flavour switchport switch_id
> >00154d130d2f peer pci/0000:05:00.0/1
> >
> >pci/0000:05:00.0/1 eth netdev flavour hostport switch_id 00154d130d2f
> >peer pci/0000:05:00.0/10002
> >
> >This addresses the case for Infiniband where there is no eswitch, but
> hostports exists and should be managed.
> >We shouldn't be inventing new devlink APIs or create a fake sw eswitch
> object which doesn't exist in hw.
> >
> >>
> >> >This is because unprivileged VF doesn't have visibility to eswitch
> >> >and its
> >> links.
> >> >
> >> >> I think the terminology need to be defined clearly so that we are
> >> >> all on the same page.
> >> >>
> >> >> >
> >> >> >> Currently we have ndo_set_vf_mac_addr api that works with PF
> >> >> >> netdev, but i think we are trying to move away from that API
> >> >> >> and do all the configuration via the port representor netdevs.
> >> >> > This is fine rep-netdev represents eswitch port.
> >> >> > You normally don't go to switch to program host port params.
> >> >> >
> >> >> >> As the mac address cannot be configured using this netdev, i
> >> >> >> think Jakub is suggesting creating a devlink opject for each
> >> >> >> port representor and use that interface to set peer mac address.
> >> >> >
> >> >> > I understand but is convoluted interface.
> >> >> > When you program host NIC mac address you talk to iLo or BIOS.
> >> >> > When you program switch side mac address, you go
> >> switch/router/modem.
> >> >> >
> >> >> > Also programming host params on host side, also doesn't make
> >> >> assumption that its connected to eswitch.
> >> >> > It also doesn't assume that same connectivity for its life.
> >> >> >
> >> >> > If you model around how physical devices are configured, it will
> >> >> > almost
> >> >> never go wrong and still provides same level of flexibility.
> >> >> >
> >> >> >> We should be able use this to configure port vlan too.
> >> >> >>
> >> >> >> Also, instead of subport, can we call vport and support
> >> >> >> different types of vports - sr-iov, siov, vmdq etc.
> >> >> >>
> >> >> > At switch level there are just ports.
> >> >> > sriov, siov, mdev, vmdq are their couter part (peer) where it is
> >> connected.
> >> >> >
> >> >> >>>
> >> >> >>> Also eswitch is flat. There is no need of pf/vf flavour for port.
> >> >> >>> It doesn't make sense to define 'mdev' flavour which we are
> >> >> >>> already
> >> >> >> working.
> >> >> >>> At eswitch level it is just a port, it happen to be connected
> >> >> >>> to vf or pf or
> >> >> >> other objects, it doesn't matter.
> >> >> >>> Port should be flavoured as 'hostport' or 'switchport'.
> >> >> >>>
> >> >> >>>
> >> >> >>>> (using the port ids from above)
^ permalink raw reply [flat|nested] 100+ messages in thread
* RE: [PATCH net-next v2 4/7] devlink: allow subports on devlink PCI ports
2019-03-18 15:56 ` Parav Pandit
@ 2019-03-18 16:22 ` Parav Pandit
2019-03-18 19:36 ` Jakub Kicinski
0 siblings, 1 reply; 100+ messages in thread
From: Parav Pandit @ 2019-03-18 16:22 UTC (permalink / raw)
To: Parav Pandit, Jiri Pirko
Cc: Samudrala, Sridhar, Jakub Kicinski, davem, netdev, oss-drivers
> -----Original Message-----
> From: netdev-owner@vger.kernel.org <netdev-owner@vger.kernel.org> On
> Behalf Of Parav Pandit
> Sent: Monday, March 18, 2019 10:57 AM
> To: Jiri Pirko <jiri@resnulli.us>
> Cc: Samudrala, Sridhar <sridhar.samudrala@intel.com>; Jakub Kicinski
> <jakub.kicinski@netronome.com>; davem@davemloft.net;
> netdev@vger.kernel.org; oss-drivers@netronome.com
> Subject: RE: [PATCH net-next v2 4/7] devlink: allow subports on devlink PCI
> ports
>
>
>
> > -----Original Message-----
> > From: Jiri Pirko <jiri@resnulli.us>
> > Sent: Monday, March 18, 2019 7:21 AM
> > To: Parav Pandit <parav@mellanox.com>
> > Cc: Samudrala, Sridhar <sridhar.samudrala@intel.com>; Jakub Kicinski
> > <jakub.kicinski@netronome.com>; davem@davemloft.net;
> > netdev@vger.kernel.org; oss-drivers@netronome.com
> > Subject: Re: [PATCH net-next v2 4/7] devlink: allow subports on
> > devlink PCI ports
> >
> > Fri, Mar 15, 2019 at 10:59:33PM CET, parav@mellanox.com wrote:
> > >
> > >
> > >> -----Original Message-----
> > >> From: Jiri Pirko <jiri@resnulli.us>
> > >> Sent: Friday, March 15, 2019 3:08 PM
> > >> To: Parav Pandit <parav@mellanox.com>
> > >> Cc: Samudrala, Sridhar <sridhar.samudrala@intel.com>; Jakub
> > >> Kicinski <jakub.kicinski@netronome.com>; davem@davemloft.net;
> > >> netdev@vger.kernel.org; oss-drivers@netronome.com
> > >> Subject: Re: [PATCH net-next v2 4/7] devlink: allow subports on
> > >> devlink PCI ports
> > >>
> > >> Fri, Mar 15, 2019 at 04:32:24PM CET, parav@mellanox.com wrote:
> > >> >
> > >> >
> > >> >> -----Original Message-----
> > >> >> From: Samudrala, Sridhar <sridhar.samudrala@intel.com>
> > >> >> Sent: Friday, March 15, 2019 12:58 AM
> > >> >> To: Parav Pandit <parav@mellanox.com>; Jakub Kicinski
> > >> >> <jakub.kicinski@netronome.com>
> > >> >> Cc: Jiri Pirko <jiri@resnulli.us>; davem@davemloft.net;
> > >> >> netdev@vger.kernel.org; oss-drivers@netronome.com
> > >> >> Subject: Re: [PATCH net-next v2 4/7] devlink: allow subports on
> > >> >> devlink PCI ports
> > >> >>
> > >> >>
> > >> >> On 3/14/2019 7:40 PM, Parav Pandit wrote:
> > >> >> >
> > >> >> >
> > >> >> >> -----Original Message-----
> > >> >> >> From: Samudrala, Sridhar <sridhar.samudrala@intel.com>
> > >> >> >> Sent: Thursday, March 14, 2019 9:16 PM
> > >> >> >> To: Parav Pandit <parav@mellanox.com>; Jakub Kicinski
> > >> >> >> <jakub.kicinski@netronome.com>
> > >> >> >> Cc: Jiri Pirko <jiri@resnulli.us>; davem@davemloft.net;
> > >> >> >> netdev@vger.kernel.org; oss-drivers@netronome.com
> > >> >> >> Subject: Re: [PATCH net-next v2 4/7] devlink: allow subports
> > >> >> >> on devlink PCI ports
> > >> >> >>
> > >> >> >>
> > >> >> >>
> > >> >> >> On 3/14/2019 6:28 PM, Parav Pandit wrote:
> > >> >> >>>
> > >> >> >>>
> > >> >> >>>> -----Original Message-----
> > >> >> >>>> From: Jakub Kicinski <jakub.kicinski@netronome.com>
> > >> >> >>>> Sent: Thursday, March 14, 2019 6:39 PM
> > >> >> >>>> To: Parav Pandit <parav@mellanox.com>
> > >> >> >>>> Cc: Jiri Pirko <jiri@resnulli.us>; davem@davemloft.net;
> > >> >> >>>> netdev@vger.kernel.org; oss-drivers@netronome.com
> > >> >> >>>> Subject: Re: [PATCH net-next v2 4/7] devlink: allow
> > >> >> >>>> subports on devlink PCI ports
> > >> >> >>>>
> > >> >> >>>> On Thu, 14 Mar 2019 22:35:36 +0000, Parav Pandit wrote:
> > >> >> >>>>>>> Then instances of flavour pci_vf are going to appear in
> > >> >> >>>>>>> the same devlink instance. Those are the switch ports:
> > >> >> >>>>>>> pci/0000:05:00.0/10002: type eth netdev enp5s0npf0pf0s0
> > >> >> >>>>>>> flavour pci_vf pf 0 vf 0
> > >> >> >>>>>>> switch_id 00154d130d2f peer
> > >> >> >>>>>>> pci/0000:05:10.1/0
> > >> >> >>>>>>> pci/0000:05:00.0/10003: type eth netdev enp5s0npf0pf0s0
> > >> >> >>>>>>> flavour pci_vf pf 0 vf 0 subport 1
> > >> >> >>>>>>> switch_id 00154d130d2f peer
> > >> >> >>>>>>> pci/0000:05:10.1/1
> > >> >> >>>>>>>
> > >> >> >>>>>>> With that, peers are going to appear too, and those are
> > >> >> >>>>>>> the actual VF/VF
> > >> >> >>>>>>> subport:
> > >> >> >>>>>>> pci/0000:05:10.1/0: type eth netdev ??? flavour pci_vf_host
> > >> >> >>>>>>> peer pci/0000:05:00.0/10002
> > >> >> >>>>>>> pci/0000:05:10.1/1: type eth netdev ??? flavour pci_vf_host
> > >> >> >>>>>>> peer pci/0000:05:00.0/10003
> > >> >> >>>>>>>
> > >> >> >>>>>>> Later you can push this VF along with all subports to VM.
> > >> >> >>>>>>> So in VM, you are going to see the VF like this:
> > >> >> >>>>>>> $ devlink dev
> > >> >> >>>>>>> pci/0000:00:08.0
> > >> >> >>>>>>> $ devlink port
> > >> >> >>>>>>> pci/0000:00:08.0/0: type eth netdev ??? flavour
> > >> >> >>>>>>> pci_vf_host
> > >> >> >>>>>>> pci/0000:00:08.0/1: type eth netdev ??? flavour
> > >> >> >>>>>>> pci_vf_host
> > >> >> >>>>>>>
> > >> >> >>>>>>> And back to your question of how are they connected in
> > eswitch.
> > >> >> >>>>>>> That is totally up to the original user John who did the
> > creation.
> > >> >> >>>>>>> He is in charge of the eswitch on baremetal, he would
> > >> >> >>>>>>> configure the forwarding however he likes.
> > >> >> >>>>>>
> > >> >> >>>>>> Ack, so I think you're saying VM has to communicate to
> > >> >> >>>>>> the cloud environment to have this provisioned using some
> > >> >> >>>>>> service API, not a kernel API. That's what I wanted to
> confirm.
> > >> >> >>>>>>
> > >> >> >>>>>> I don't see any benefit to having the "host ports" under
> > >> >> >>>>>> devlink, as such I think it's a matter of preference.
> > >> >> >>>>>
> > >> >> >>>>> We need 'host ports' to configure parameters of this host
> > >> >> >>>>> port which is not exposed by the rep-netdev.
> > >> >> >>>>> Such as mac address.
> > >> >> >>>>
> > >> >> >>>> Please look at the quote of what Jiri wrote above - the
> > >> >> >>>> host port gets passed to the VM, you can't use it as a
> > >> >> >>>> handle to set the
> > >> MAC.
> > >> >> >>>>
> > >> >> >>>> The way to set the MAC remains:
> > >> >> >>>>
> > >> >> >>>> # devlink port set pci/0000:05:00.0/10002 peer mac_addr
> > >> >> >>>> 00:11:22:33:44:55
> > >> >> >>>>
> > >> >> >>> Even though it can be done, I think this is wrong model to
> > >> >> >>> program
> > >> >> >> hostport mac address using eswitch port.
> > >> >> >>> All devlink objects are control objects, so what is passed
> > >> >> >>> to VM is what is
> > >> >> >> represented by devlink.
> > >> >> >>> VF in the VM will anyway create its devlink object.
> > >> >> >>> What is wrong in programming hostport?
> > >> >> >>> It gives a very clear view to users of topology and objects.
> > >> >> >>
> > >> >> >> The VF or any subport MAC address should be configured by the
> > >> >> >> orchestration layer that is running on the hypervisor and
> > >> >> >> when a VF is assigned to a VF, the host port is not visible
> > >> >> >> to the
> > hypervisor.
> > >> >> > What prevents creation of hostport due to which is not visible?
> > >> >> > Hostport is control port to program host side of parameters.
> > >> >> > It should be created when user wants to program the parameters.
> > >> >> >
> > >> >> > Model is really straight forward.
> > >> >> > Program host port params using hostport object.
> > >> >> > Program switchport params using rep-netdev.
> > >> >>
> > >> >> IIUC, Jiri/Jakub are proposing creation of 2 devlink objects for
> > >> >> each port - host facing ports and switch facing ports. This is
> > >> >> in addition to the netdevs that are created today.
> > >> >>
> > >> >I am not proposing any different.
> > >> >I am proposing only two changes.
> > >> >1. control hostport params via referring hostport (not via
> > >> >indirect
> > >> >peer)
> > >>
> > >> Not really possible. If you passthrough VF into VM, the hostport
> > >> goes along with it.
> > >>
> > >No.
> > >I am sorry in showing the enumeration which is the source of confusion.
> > >
> > >Below is the right enumeration.
> > >
> > >When VF is enumerated initially in the host, where eswitch devlink
> > >instance
> > is located.
> > >Below enumeration is seen.
> > >
> > >First two entries shows the link between hostport and switchport.
> > >$ devlink port show
> > >pci/0000:05:00.0/10002 eth netdev flavour switchport switch_id
> > >00154d130d2f peer pci/0000:05:00.0/1
> > >
> > >pci/0000:05:00.0/1 eth netdev flavour hostport switch_id 00154d130d2f
> > >peer pci/0000:05:00.0/10002
> >
> > Hostport should not have switch_id.
> >
> > >
> > >pci/0000:05:10.1/0 eth netdev flavour hostport This entry won't be
> > >seen if VF auto probing is disabled. Because than VF is not enumerated.
> > >
> > >As a user, I will be programming the mac address of hostport for a VF.
> > >pci/0000:05:00.0/1 eth netdev flavour hostport switch_id 00154d130d2f
> > >peer pci/0000:05:00.0/10002
> >
> > Hmm, so you are going to have 2 hostports for VF:
> > 1) pci/0000:05:10.1/0
> > real one, that is going to go to VM - with a separate pci address
> > and devlink instance.
>
> Yep. This is the one where Yonatan's port grouping APIs work on.
>
> > 2) pci/0000:05:00.0/1
> > dummy one, which is not really a hostport, as there is no netdev
> > created for it. It only models the other side of cable, which is away
> > in VM.
> >
> Right. This is the control object which typically hypervisor programs.
>
> > >
> > >
> > >>
> > >> >2. flavour should not be vf/pf, flavour should be hostport, switchport.
> > >> >Because switch is flat and agnostic of pf/vf/mdev.
> > >>
> > >> Not sure. It's good to have this kind of visibility.
> > >>
> > >port can have label/attribute indicating that this belong to VF-1 or
> > >mdev as
> > long as you are agreeing to have mdev attribute on host port.
> > >(and not ask for abstracting it, because mdev is well defined kernel
> object).
> >
> > Why mdev cannot be another flavour?
> >
>
> hostport is of type pf/vf/mdev connected to some switchport.
>
> So proposal is to have,
> port flavour = hostport/switchport
> port type/label = pf/vf/mdev
>
Instead of having two attributes per port, how about having,
port flavour= physical/cpu/dsa/pf/vf/mdev/switchport.
physical and pf has some overlapping definitions.
^ permalink raw reply [flat|nested] 100+ messages in thread
* Re: [PATCH net-next v2 4/7] devlink: allow subports on devlink PCI ports
2019-03-18 12:11 ` Jiri Pirko
@ 2019-03-18 19:16 ` Jakub Kicinski
2019-03-21 8:45 ` Jiri Pirko
0 siblings, 1 reply; 100+ messages in thread
From: Jakub Kicinski @ 2019-03-18 19:16 UTC (permalink / raw)
To: Jiri Pirko; +Cc: Parav Pandit, Samudrala, Sridhar, davem, netdev, oss-drivers
On Mon, 18 Mar 2019 13:11:54 +0100, Jiri Pirko wrote:
> >> >2. flavour should not be vf/pf, flavour should be hostport, switchport.
> >> >Because switch is flat and agnostic of pf/vf/mdev.
> >>
> >> Not sure. It's good to have this kind of visibility.
> >
> >Yes, this subthread honestly makes me go from 60% sure to 95% sure we
> >shouldn't do the dual object thing :( Seems like Parav is already
> >confused by it and suggests host port can exist without switch port :(
>
> Although I understand your hesitation, the host ports are also
> associated with the asic and should be under the devlink instance.
> It is just a matter of proper documentation and clear code to avoid
> confusions.
They are certainly a part and belong to the ASIC, the question in my
mind is more along the lines of do we want "one pipe/one port" or is
it okay to have multiple software objects of the same kind for those
objects.
To put it differently - do want a port object for each port of the ASIC
or do we want a port object for each netdev..
^ permalink raw reply [flat|nested] 100+ messages in thread
* Re: [PATCH net-next v2 4/7] devlink: allow subports on devlink PCI ports
2019-03-18 12:21 ` Jiri Pirko
2019-03-18 15:56 ` Parav Pandit
@ 2019-03-18 19:19 ` Jakub Kicinski
2019-03-18 19:38 ` Parav Pandit
2019-03-21 9:09 ` Jiri Pirko
1 sibling, 2 replies; 100+ messages in thread
From: Jakub Kicinski @ 2019-03-18 19:19 UTC (permalink / raw)
To: Jiri Pirko; +Cc: Parav Pandit, Samudrala, Sridhar, davem, netdev, oss-drivers
On Mon, 18 Mar 2019 13:21:05 +0100, Jiri Pirko wrote:
> >First two entries shows the link between hostport and switchport.
> >$ devlink port show
> >pci/0000:05:00.0/10002 eth netdev flavour switchport switch_id 00154d130d2f peer pci/0000:05:00.0/1
> >
> >pci/0000:05:00.0/1 eth netdev flavour hostport switch_id 00154d130d2f peer pci/0000:05:00.0/10002
>
> Hostport should not have switch_id.
Isn't a concept of a port of a switch without a switch ID a red flag?
^ permalink raw reply [flat|nested] 100+ messages in thread
* Re: [PATCH net-next v2 4/7] devlink: allow subports on devlink PCI ports
2019-03-18 15:43 ` Parav Pandit
@ 2019-03-18 19:29 ` Jakub Kicinski
0 siblings, 0 replies; 100+ messages in thread
From: Jakub Kicinski @ 2019-03-18 19:29 UTC (permalink / raw)
To: Parav Pandit; +Cc: Jiri Pirko, Samudrala, Sridhar, davem, netdev, oss-drivers
On Mon, 18 Mar 2019 15:43:20 +0000, Parav Pandit wrote:
> > -----Original Message-----
> > From: Jakub Kicinski <jakub.kicinski@netronome.com>
> > Sent: Friday, March 15, 2019 8:16 PM
> > To: Parav Pandit <parav@mellanox.com>
> > Cc: Jiri Pirko <jiri@resnulli.us>; Samudrala, Sridhar
> > <sridhar.samudrala@intel.com>; davem@davemloft.net;
> > netdev@vger.kernel.org; oss-drivers@netronome.com
> > Subject: Re: [PATCH net-next v2 4/7] devlink: allow subports on devlink PCI
> > ports
> >
> > On Fri, 15 Mar 2019 22:12:13 +0000, Parav Pandit wrote:
> > > > On Fri, 15 Mar 2019 21:08:14 +0100, Jiri Pirko wrote:
> > > > > >> IIUC, Jiri/Jakub are proposing creation of 2 devlink objects
> > > > > >> for each port - host facing ports and switch facing ports. This
> > > > > >> is in addition to the netdevs that are created today.
> > > >
> > > > To be clear I'm not in favour of the dual-object proposal.
> > > >
> > > > > >I am not proposing any different.
> > > > > >I am proposing only two changes.
> > > > > >1. control hostport params via referring hostport (not via
> > > > > >indirect
> > > > > >peer)
> > > > >
> > > > > Not really possible. If you passthrough VF into VM, the hostport
> > > > > goes along with it.
> > > > >
> > > > > >2. flavour should not be vf/pf, flavour should be hostport, switchport.
> > > > > >Because switch is flat and agnostic of pf/vf/mdev.
> > > > >
> > > > > Not sure. It's good to have this kind of visibility.
> > > >
> > > > Yes, this subthread honestly makes me go from 60% sure to 95% sure
> > > > we shouldn't do the dual object thing :( Seems like Parav is
> > > > already confused by it and suggests host port can exist without
> > > > switch port :(
> > > >
> > > I am almost sure that I am not confused.
> > > I am clear that hostports should be configured by devlink instance
> > > which has the capability to program it.
> >
> > Right now a devlink port is something that the datapath of an ASIC can
> > address. All flavours we have presently are basically various MACs - physical
> > (front panel ports), DSA - for ASIC interconnects on a multi-ASIC board, CPU -
> > for connecting to a MAC of a NIC.
> >
> Devlink port implementation in commit doesn't say that it is for ASIC datapath or limited to ASIC datapath id.
> It is not right to say that 'whole datapath' object should be represented with just single object 'port'.
> Datapath involves various stages in ASIC each does different processing.
> These datapath objects are interconnected, i.e. hostport is connected to switchport.
> Commit [1] says devlink port is physical port. However we already have 3 flavours of port.
>
> > Jiri's flavour proposal was strictly extending the same logic to SR-IOV. Each
> > object addressable within the datapath gets a port.
> > The datapath's ID can be used as port_index.
> >
> And as I said, it is already restrictive.
> Port is a port, it can be labeled for vf/pf, but flavour is not really vf/pf.
> Also label applies more on the hostport side vs switchport.
>
> > I just reimplemented his patches here and added the subports which I think
> > he wasn't aware of as they are a quirk of old NFP ASICs.
> >
> > Having 3 objects for the same datapath ID is a significant departure from the
> > existing devlink port semantics.
> >
> It is really not same datapath ID.
> Because if that is the case, we should be programming mac address on the rep-netdev itself.
> But we are not doing that because rep-netdev represents only 'eswitch port'.
Okay, I explained the history to you here, you can write your own if
you want.
> > > When hostport is in VF, that VF usually won't have privilege to
> > > program it and won't have visibility to eswitch either.
> >
> > If VM has no visibility into the eswitch and no permission to configure
> > things, what use does the object serve?
> >
> To view device properties, health, RO registers, more importantly its port details.
Device != port.
> Yonatan is working on grouping these devlink ports and those are control through devlink APIs.
> Jiri is actively internally reviewing those patches since last 3+ weeks, not finished yet.
> So this visibility is needed anyway.
No idea what "grouping devlink ports" may refer to, but I'd be
surprised if it's relevant to VMs.
> > > Why would you like to start with restrictive model of peer view only?
> >
> > "Restrictive model" is one way of putting it. I'd rather say that we are not
> > adding objects which:
> > (a) do not adhere to current semantics;
> > (b) have no distinct function.
> >
> hostport certainly has distinct function than switchport.
> i.e. to program host side parameters. (eth.mac, rdma.port_guid and more in future).
Yeah, Ethernet or IB address, and so many other things (we just can't
happen to think about any right now)...
> > We can make the "add MAC address" command not use the word peer:
> >
> > devlink port addr_pool add pci/0000:05:00.0/10003 type eth
> > 00:11:22:33:44:55 devlink port addr_pool del pci/0000:05:00.0/10003 type
> > eth 00:11:22:33:44:55
> >
> > if the "peer" doesn't sit right.
> >
> > > Hostports exist for infiniband HCA without switchport.
> > > We should be able to manage hostport objects without creating fake
> > eswitch sw object.
> >
> > It sounds like the RDMA subsystem is lacking a model to represent all its
> > objects, but that's RDMA's problem to solve..
> >
> devlink framework is not limited to Ethernet, it operates on bus/device notion.
> So for Ethernet vendors program mac address.
> For rdma vendor programs port_guid (which is equivalent of mac address).
>
> devlink also publishes rdma device info today.
> net/core/devlink.c has very well established IB device info exposed via devlink_nl_port_fill() for more than 3 years now in commit [2].
> It is not fair to say create, solve it somewhere else.
>
> > In netdev world we have netdevs for ports which a used for bulk of the
> > configuration, most importantly - forwarding.
> >
> > > Jakub,
> > > Can you please point to some example other than veth-pair where you
> > > configure host param (such as mac address) through a switch?
> >
> > Existing "legacy" SR-IOV NDOs.
> >
> That is perfect example of programming hostport parameters, without a eswitch..
> At high level, I was looking where you open switch GUI/cli or something equivalent that program's host's mac address..
> So far we don't have such equivalent good example yet..
>
> > > An existing example will help me to map it to devlink eswitch proposal.
> > > If we go peer programming route, what are your thoughts on how should
> > > we program infiniband hostports which doesn't have peer ports?
> >
> > Again, you may be trying to fix RDMA's lack of control objects, which may be
> > better fixed elsewhere..
>
> devlink port is link agnostic control object.
>
> [1] bfcd3a46617209454cfc0947ab093e37fd1e84ef
> [2] commit id bfcd3a466
^ permalink raw reply [flat|nested] 100+ messages in thread
* Re: [PATCH net-next v2 4/7] devlink: allow subports on devlink PCI ports
2019-03-18 16:22 ` Parav Pandit
@ 2019-03-18 19:36 ` Jakub Kicinski
2019-03-18 19:44 ` Parav Pandit
0 siblings, 1 reply; 100+ messages in thread
From: Jakub Kicinski @ 2019-03-18 19:36 UTC (permalink / raw)
To: Parav Pandit; +Cc: Jiri Pirko, Samudrala, Sridhar, davem, netdev, oss-drivers
On Mon, 18 Mar 2019 16:22:33 +0000, Parav Pandit wrote:
>>>>>>2. flavour should not be vf/pf, flavour should be hostport, switchport.
>>> >Because switch is flat and agnostic of pf/vf/mdev.
>>>>>
>>>>> Not sure. It's good to have this kind of visibility.
>>>>>
>>>> port can have label/attribute indicating that this belong to VF-1 or
>>>> mdev as long as you are agreeing to have mdev attribute on host port.
>>>> (and not ask for abstracting it, because mdev is well defined kernel object).
>>>
>>> Why mdev cannot be another flavour?
>>>
>>
>> hostport is of type pf/vf/mdev connected to some switchport.
>>
>> So proposal is to have,
>> port flavour = hostport/switchport
>> port type/label = pf/vf/mdev
>>
> Instead of having two attributes per port, how about having,
> port flavour= physical/cpu/dsa/pf/vf/mdev/switchport.
>
> physical and pf has some overlapping definitions.
What "overlapping definitions" do physical and PF have?
Sounds like you're referring to limitations of Mellanox HW.
^ permalink raw reply [flat|nested] 100+ messages in thread
* RE: [PATCH net-next v2 4/7] devlink: allow subports on devlink PCI ports
2019-03-18 19:19 ` Jakub Kicinski
@ 2019-03-18 19:38 ` Parav Pandit
2019-03-21 9:09 ` Jiri Pirko
1 sibling, 0 replies; 100+ messages in thread
From: Parav Pandit @ 2019-03-18 19:38 UTC (permalink / raw)
To: Jakub Kicinski, Jiri Pirko; +Cc: Samudrala, Sridhar, davem, netdev, oss-drivers
> -----Original Message-----
> From: Jakub Kicinski <jakub.kicinski@netronome.com>
> Sent: Monday, March 18, 2019 2:19 PM
> To: Jiri Pirko <jiri@resnulli.us>
> Cc: Parav Pandit <parav@mellanox.com>; Samudrala, Sridhar
> <sridhar.samudrala@intel.com>; davem@davemloft.net;
> netdev@vger.kernel.org; oss-drivers@netronome.com
> Subject: Re: [PATCH net-next v2 4/7] devlink: allow subports on devlink PCI
> ports
>
> On Mon, 18 Mar 2019 13:21:05 +0100, Jiri Pirko wrote:
> > >First two entries shows the link between hostport and switchport.
> > >$ devlink port show
> > >pci/0000:05:00.0/10002 eth netdev flavour switchport switch_id
> 00154d130d2f peer pci/0000:05:00.0/1
> > >
> > >pci/0000:05:00.0/1 eth netdev flavour hostport switch_id 00154d130d2f
> peer pci/0000:05:00.0/10002
> >
> > Hostport should not have switch_id.
>
> Isn't a concept of a port of a switch without a switch ID a red flag?
It shows that a given hostport is connected to a given switchid at given switchport.
But I believe this info can be dropped, because this is visible at the switchport side.
So when showing hostport, it can be just only,
pci/0000:05:00.0/1 eth netdev flavour hostport
^ permalink raw reply [flat|nested] 100+ messages in thread
* RE: [PATCH net-next v2 4/7] devlink: allow subports on devlink PCI ports
2019-03-18 19:36 ` Jakub Kicinski
@ 2019-03-18 19:44 ` Parav Pandit
2019-03-18 19:59 ` Jakub Kicinski
0 siblings, 1 reply; 100+ messages in thread
From: Parav Pandit @ 2019-03-18 19:44 UTC (permalink / raw)
To: Jakub Kicinski; +Cc: Jiri Pirko, Samudrala, Sridhar, davem, netdev, oss-drivers
> -----Original Message-----
> From: Jakub Kicinski <jakub.kicinski@netronome.com>
> Sent: Monday, March 18, 2019 2:37 PM
> To: Parav Pandit <parav@mellanox.com>
> Cc: Jiri Pirko <jiri@resnulli.us>; Samudrala, Sridhar
> <sridhar.samudrala@intel.com>; davem@davemloft.net;
> netdev@vger.kernel.org; oss-drivers@netronome.com
> Subject: Re: [PATCH net-next v2 4/7] devlink: allow subports on devlink PCI
> ports
>
> On Mon, 18 Mar 2019 16:22:33 +0000, Parav Pandit wrote:
> >>>>>>2. flavour should not be vf/pf, flavour should be hostport, switchport.
> >>> >Because switch is flat and agnostic of pf/vf/mdev.
> >>>>>
> >>>>> Not sure. It's good to have this kind of visibility.
> >>>>>
> >>>> port can have label/attribute indicating that this belong to VF-1
> >>>> or mdev as long as you are agreeing to have mdev attribute on host
> port.
> >>>> (and not ask for abstracting it, because mdev is well defined kernel
> object).
> >>>
> >>> Why mdev cannot be another flavour?
> >>>
> >>
> >> hostport is of type pf/vf/mdev connected to some switchport.
> >>
> >> So proposal is to have,
> >> port flavour = hostport/switchport
> >> port type/label = pf/vf/mdev
> >>
> > Instead of having two attributes per port, how about having, port
> > flavour= physical/cpu/dsa/pf/vf/mdev/switchport.
> >
> > physical and pf has some overlapping definitions.
>
> What "overlapping definitions" do physical and PF have?
PF has physically user facing port.
And physical port in include/uapi/linux/devlink.h also describe that.
^ permalink raw reply [flat|nested] 100+ messages in thread
* RE: [PATCH net-next v2 4/7] devlink: allow subports on devlink PCI ports
2019-03-18 19:44 ` Parav Pandit
@ 2019-03-18 19:59 ` Jakub Kicinski
2019-03-18 20:35 ` Parav Pandit
0 siblings, 1 reply; 100+ messages in thread
From: Jakub Kicinski @ 2019-03-18 19:59 UTC (permalink / raw)
To: Parav Pandit; +Cc: Jiri Pirko, Samudrala, Sridhar, davem, netdev, oss-drivers
On Mon, 18 Mar 2019 19:44:21 +0000, Parav Pandit wrote:
> > -----Original Message-----
> > From: Jakub Kicinski <jakub.kicinski@netronome.com>
> > Sent: Monday, March 18, 2019 2:37 PM
> > To: Parav Pandit <parav@mellanox.com>
> > Cc: Jiri Pirko <jiri@resnulli.us>; Samudrala, Sridhar
> > <sridhar.samudrala@intel.com>; davem@davemloft.net;
> > netdev@vger.kernel.org; oss-drivers@netronome.com
> > Subject: Re: [PATCH net-next v2 4/7] devlink: allow subports on devlink PCI
> > ports
> >
> > On Mon, 18 Mar 2019 16:22:33 +0000, Parav Pandit wrote:
> > >>>>>>2. flavour should not be vf/pf, flavour should be hostport, switchport.
> > >>> >Because switch is flat and agnostic of pf/vf/mdev.
> > >>>>>
> > >>>>> Not sure. It's good to have this kind of visibility.
> > >>>>>
> > >>>> port can have label/attribute indicating that this belong to VF-1
> > >>>> or mdev as long as you are agreeing to have mdev attribute on host port.
> > >>>> (and not ask for abstracting it, because mdev is well defined kernel object).
> > >>>
> > >>> Why mdev cannot be another flavour?
> > >>>
> > >>
> > >> hostport is of type pf/vf/mdev connected to some switchport.
> > >>
> > >> So proposal is to have,
> > >> port flavour = hostport/switchport
> > >> port type/label = pf/vf/mdev
> > >>
> > > Instead of having two attributes per port, how about having, port
> > > flavour= physical/cpu/dsa/pf/vf/mdev/switchport.
> > >
> > > physical and pf has some overlapping definitions.
> >
> > What "overlapping definitions" do physical and PF have?
> PF has physically user facing port.
PF doesn't "have a user facing port" in switchdev mode. It's a
limitation of Mellanox HW that you have some strong association
there.
> And physical port in include/uapi/linux/devlink.h also describe that.
By "that" you must mean that the physical is a user facing port.
^ permalink raw reply [flat|nested] 100+ messages in thread
* RE: [PATCH net-next v2 4/7] devlink: allow subports on devlink PCI ports
2019-03-18 19:59 ` Jakub Kicinski
@ 2019-03-18 20:35 ` Parav Pandit
2019-03-18 21:29 ` Jakub Kicinski
0 siblings, 1 reply; 100+ messages in thread
From: Parav Pandit @ 2019-03-18 20:35 UTC (permalink / raw)
To: Jakub Kicinski; +Cc: Jiri Pirko, Samudrala, Sridhar, davem, netdev, oss-drivers
> -----Original Message-----
> From: Jakub Kicinski <jakub.kicinski@netronome.com>
> Sent: Monday, March 18, 2019 3:00 PM
> To: Parav Pandit <parav@mellanox.com>
> Cc: Jiri Pirko <jiri@resnulli.us>; Samudrala, Sridhar
> <sridhar.samudrala@intel.com>; davem@davemloft.net;
> netdev@vger.kernel.org; oss-drivers@netronome.com
> Subject: RE: [PATCH net-next v2 4/7] devlink: allow subports on devlink PCI
> ports
>
> On Mon, 18 Mar 2019 19:44:21 +0000, Parav Pandit wrote:
> > > -----Original Message-----
> > > From: Jakub Kicinski <jakub.kicinski@netronome.com>
> > > Sent: Monday, March 18, 2019 2:37 PM
> > > To: Parav Pandit <parav@mellanox.com>
> > > Cc: Jiri Pirko <jiri@resnulli.us>; Samudrala, Sridhar
> > > <sridhar.samudrala@intel.com>; davem@davemloft.net;
> > > netdev@vger.kernel.org; oss-drivers@netronome.com
> > > Subject: Re: [PATCH net-next v2 4/7] devlink: allow subports on
> > > devlink PCI ports
> > >
> > > On Mon, 18 Mar 2019 16:22:33 +0000, Parav Pandit wrote:
> > > >>>>>>2. flavour should not be vf/pf, flavour should be hostport,
> switchport.
> > > >>> >Because switch is flat and agnostic of pf/vf/mdev.
> > > >>>>>
> > > >>>>> Not sure. It's good to have this kind of visibility.
> > > >>>>>
> > > >>>> port can have label/attribute indicating that this belong to
> > > >>>> VF-1 or mdev as long as you are agreeing to have mdev attribute on
> host port.
> > > >>>> (and not ask for abstracting it, because mdev is well defined kernel
> object).
> > > >>>
> > > >>> Why mdev cannot be another flavour?
> > > >>>
> > > >>
> > > >> hostport is of type pf/vf/mdev connected to some switchport.
> > > >>
> > > >> So proposal is to have,
> > > >> port flavour = hostport/switchport port type/label = pf/vf/mdev
> > > >>
> > > > Instead of having two attributes per port, how about having, port
> > > > flavour= physical/cpu/dsa/pf/vf/mdev/switchport.
> > > >
> > > > physical and pf has some overlapping definitions.
> > >
> > > What "overlapping definitions" do physical and PF have?
> > PF has physically user facing port.
>
> PF doesn't "have a user facing port" in switchdev mode.
Physical port described in include/uapi/linux/devlink.h as DEVLINK_PORT_FLAVOUR_PHYSICAL is not related to switchdev or legacy mode.
As the comment block describe it is 'any kind of port physical facing user'.
Current mlx5 driver doesn't expose ports as physical regardless of switchdev/legacy mode.
> It's a limitation of Mellanox HW that you have some strong association there.
>
Not sure why you keep saying that. Any code reference that I should look at?
Or maybe you can explain what is that limitation, because I am not aware of any.
> > And physical port in include/uapi/linux/devlink.h also describe that.
>
> By "that" you must mean that the physical is a user facing port.
Can you please describe the difference between 'PF port' and 'physical port of include/uapi/linux/devlink.h'?
I must have missed this crisp definition in discussion between you and Jiri.
I am in meantime checking the thread.
^ permalink raw reply [flat|nested] 100+ messages in thread
* Re: [PATCH net-next v2 4/7] devlink: allow subports on devlink PCI ports
2019-03-18 20:35 ` Parav Pandit
@ 2019-03-18 21:29 ` Jakub Kicinski
2019-03-18 22:11 ` Parav Pandit
0 siblings, 1 reply; 100+ messages in thread
From: Jakub Kicinski @ 2019-03-18 21:29 UTC (permalink / raw)
To: Parav Pandit; +Cc: Jiri Pirko, Samudrala, Sridhar, davem, netdev, oss-drivers
On Mon, 18 Mar 2019 20:35:02 +0000, Parav Pandit wrote:
> > > > > physical and pf has some overlapping definitions.
> > > >
> > > > What "overlapping definitions" do physical and PF have?
> > > PF has physically user facing port.
> >
> > PF doesn't "have a user facing port" in switchdev mode.
>
> Physical port described in include/uapi/linux/devlink.h as
> DEVLINK_PORT_FLAVOUR_PHYSICAL is not related to switchdev or legacy
> mode.
I said "PF doesn't ...", you're now talking about physical?
> As the comment block describe it is 'any kind of port physical
> facing user'.
Are you saying PCI function is physical? Just because PF stands for
Physical Function?
Physical port in devlink means a port in the front panel where
networking cable goes.
> Current mlx5 driver doesn't expose ports as physical regardless of
> switchdev/legacy mode.
Today mlx5 doesn't expose devlink ports at all.
> > It's a limitation of Mellanox HW that you have some strong
> > association there.
> Not sure why you keep saying that. Any code reference that I should
> look at? Or maybe you can explain what is that limitation, because I
> am not aware of any.
NIC designs originating from traditional NICs were build as pipelines
from PCI to wire or from wire to PCI. Reportedly it makes it hard to
completely divorce the PCI PF from the wire port (physical port).
Which is why you may think that "PF has physically user facing port".
> > > And physical port in include/uapi/linux/devlink.h also describe
> > > that.
> >
> > By "that" you must mean that the physical is a user facing port.
>
> Can you please describe the difference between 'PF port' and
> 'physical port of include/uapi/linux/devlink.h'? I must have missed
> this crisp definition in discussion between you and Jiri. I am in
> meantime checking the thread.
Perhaps start with the cover letter which includes an ASCII drawing?
Using Mellanox nomenclature - PF port is a "representor" for the PF
which may be on another Host (SmartNIC or multihost). It's pretty
much the same thing as a VF port/"representor".
Physical port is the hole on the panel of the adapter where cable goes.
^ permalink raw reply [flat|nested] 100+ messages in thread
* RE: [PATCH net-next v2 4/7] devlink: allow subports on devlink PCI ports
2019-03-18 21:29 ` Jakub Kicinski
@ 2019-03-18 22:11 ` Parav Pandit
2019-03-20 18:24 ` Parav Pandit
0 siblings, 1 reply; 100+ messages in thread
From: Parav Pandit @ 2019-03-18 22:11 UTC (permalink / raw)
To: Jakub Kicinski; +Cc: Jiri Pirko, Samudrala, Sridhar, davem, netdev, oss-drivers
> -----Original Message-----
> From: Jakub Kicinski <jakub.kicinski@netronome.com>
> Sent: Monday, March 18, 2019 4:30 PM
> To: Parav Pandit <parav@mellanox.com>
> Cc: Jiri Pirko <jiri@resnulli.us>; Samudrala, Sridhar
> <sridhar.samudrala@intel.com>; davem@davemloft.net;
> netdev@vger.kernel.org; oss-drivers@netronome.com
> Subject: Re: [PATCH net-next v2 4/7] devlink: allow subports on devlink PCI
> ports
>
> On Mon, 18 Mar 2019 20:35:02 +0000, Parav Pandit wrote:
> > > > > > physical and pf has some overlapping definitions.
> > > > >
> > > > > What "overlapping definitions" do physical and PF have?
> > > > PF has physically user facing port.
> > >
> > > PF doesn't "have a user facing port" in switchdev mode.
> >
> > Physical port described in include/uapi/linux/devlink.h as
> > DEVLINK_PORT_FLAVOUR_PHYSICAL is not related to switchdev or legacy
> > mode.
>
> I said "PF doesn't ...", you're now talking about physical?
>
> > As the comment block describe it is 'any kind of port physical facing
> > user'.
>
> Are you saying PCI function is physical? Just because PF stands for Physical
> Function?
>
> Physical port in devlink means a port in the front panel where networking
> cable goes.
>
> > Current mlx5 driver doesn't expose ports as physical regardless of
> > switchdev/legacy mode.
>
> Today mlx5 doesn't expose devlink ports at all.
>
> > > It's a limitation of Mellanox HW that you have some strong
> > > association there.
> > Not sure why you keep saying that. Any code reference that I should
> > look at? Or maybe you can explain what is that limitation, because I
> > am not aware of any.
>
> NIC designs originating from traditional NICs were build as pipelines from PCI
> to wire or from wire to PCI. Reportedly it makes it hard to completely
> divorce the PCI PF from the wire port (physical port).
>
> Which is why you may think that "PF has physically user facing port".
>
> > > > And physical port in include/uapi/linux/devlink.h also describe
> > > > that.
> > >
> > > By "that" you must mean that the physical is a user facing port.
> >
> > Can you please describe the difference between 'PF port' and 'physical
> > port of include/uapi/linux/devlink.h'? I must have missed this crisp
> > definition in discussion between you and Jiri. I am in meantime
> > checking the thread.
>
> Perhaps start with the cover letter which includes an ASCII drawing?
>
> Using Mellanox nomenclature - PF port is a "representor" for the PF which
> may be on another Host (SmartNIC or multihost). It's pretty much the same
> thing as a VF port/"representor".
>
Yes. We are aligned here. :-)
I see your point, where in multi-host scenario, a physical port may be 1, but PF ports are 4, because of 4 PFs for 4 hosts.
(just an example of 4 hosts with their own mac address sharing 1 physical port).
When there is no multihost and one to one mapping between a PF and physical links,
there is some overlap between PF port and physical port attributes.
I believe, such overlap is fine as long as we have unique indices for the ports.
So I am ok to have flavours as physical/cpu/dsa/pf/vf/mdev/switchport.
(last 4 as new port flavours).
> Physical port is the hole on the panel of the adapter where cable goes.
^ permalink raw reply [flat|nested] 100+ messages in thread
* RE: [PATCH net-next v2 4/7] devlink: allow subports on devlink PCI ports
2019-03-18 22:11 ` Parav Pandit
@ 2019-03-20 18:24 ` Parav Pandit
2019-03-20 20:22 ` Jakub Kicinski
0 siblings, 1 reply; 100+ messages in thread
From: Parav Pandit @ 2019-03-20 18:24 UTC (permalink / raw)
To: Parav Pandit, Jakub Kicinski
Cc: Jiri Pirko, Samudrala, Sridhar, davem, netdev, oss-drivers
Hi Jiri, Jakub, Samudrala Sridhar,
> -----Original Message-----
> From: netdev-owner@vger.kernel.org <netdev-owner@vger.kernel.org> On
> Behalf Of Parav Pandit
> Sent: Monday, March 18, 2019 5:12 PM
> To: Jakub Kicinski <jakub.kicinski@netronome.com>
> Cc: Jiri Pirko <jiri@resnulli.us>; Samudrala, Sridhar
> <sridhar.samudrala@intel.com>; davem@davemloft.net;
> netdev@vger.kernel.org; oss-drivers@netronome.com
> Subject: RE: [PATCH net-next v2 4/7] devlink: allow subports on devlink PCI
> ports
>
>
>
> > -----Original Message-----
> > From: Jakub Kicinski <jakub.kicinski@netronome.com>
> > Sent: Monday, March 18, 2019 4:30 PM
> > To: Parav Pandit <parav@mellanox.com>
> > Cc: Jiri Pirko <jiri@resnulli.us>; Samudrala, Sridhar
> > <sridhar.samudrala@intel.com>; davem@davemloft.net;
> > netdev@vger.kernel.org; oss-drivers@netronome.com
> > Subject: Re: [PATCH net-next v2 4/7] devlink: allow subports on
> > devlink PCI ports
> >
> > On Mon, 18 Mar 2019 20:35:02 +0000, Parav Pandit wrote:
> > > > > > > physical and pf has some overlapping definitions.
> > > > > >
> > > > > > What "overlapping definitions" do physical and PF have?
> > > > > PF has physically user facing port.
> > > >
> > > > PF doesn't "have a user facing port" in switchdev mode.
> > >
> > > Physical port described in include/uapi/linux/devlink.h as
> > > DEVLINK_PORT_FLAVOUR_PHYSICAL is not related to switchdev or legacy
> > > mode.
> >
> > I said "PF doesn't ...", you're now talking about physical?
> >
> > > As the comment block describe it is 'any kind of port physical
> > > facing user'.
> >
> > Are you saying PCI function is physical? Just because PF stands for
> > Physical Function?
> >
> > Physical port in devlink means a port in the front panel where
> > networking cable goes.
> >
> > > Current mlx5 driver doesn't expose ports as physical regardless of
> > > switchdev/legacy mode.
> >
> > Today mlx5 doesn't expose devlink ports at all.
> >
> > > > It's a limitation of Mellanox HW that you have some strong
> > > > association there.
> > > Not sure why you keep saying that. Any code reference that I should
> > > look at? Or maybe you can explain what is that limitation, because I
> > > am not aware of any.
> >
> > NIC designs originating from traditional NICs were build as pipelines
> > from PCI to wire or from wire to PCI. Reportedly it makes it hard to
> > completely divorce the PCI PF from the wire port (physical port).
> >
> > Which is why you may think that "PF has physically user facing port".
> >
> > > > > And physical port in include/uapi/linux/devlink.h also describe
> > > > > that.
> > > >
> > > > By "that" you must mean that the physical is a user facing port.
> > >
> > > Can you please describe the difference between 'PF port' and
> > > 'physical port of include/uapi/linux/devlink.h'? I must have missed
> > > this crisp definition in discussion between you and Jiri. I am in
> > > meantime checking the thread.
> >
> > Perhaps start with the cover letter which includes an ASCII drawing?
> >
> > Using Mellanox nomenclature - PF port is a "representor" for the PF
> > which may be on another Host (SmartNIC or multihost). It's pretty
> > much the same thing as a VF port/"representor".
> >
> Yes. We are aligned here. :-)
> I see your point, where in multi-host scenario, a physical port may be 1, but
> PF ports are 4, because of 4 PFs for 4 hosts.
> (just an example of 4 hosts with their own mac address sharing 1 physical
> port).
>
> When there is no multihost and one to one mapping between a PF and
> physical links, there is some overlap between PF port and physical port
> attributes.
> I believe, such overlap is fine as long as we have unique indices for the ports.
>
> So I am ok to have flavours as physical/cpu/dsa/pf/vf/mdev/switchport.
> (last 4 as new port flavours).
>
> > Physical port is the hole on the panel of the adapter where cable goes.
So my take away from above discussion are:
1. Following new port flavours should be added pci_pf/pci_vf/mdev/switchport.
a. Switchport indicates port on the eswitch. Normally this port has rep-netdev attached to it.
b. host side port flavours are pci_pf/pci_vf/mdev which may be connected to switchport
2. host side port flavours are not limited to Ethernet, as it is for devlink's port instance.
3. Each port is continue to be accessed using unique port index.
4. host side ports and switchport are control objects.
a. switch side ports reside where current eswitch object of devlink instance reside
b. for a given VF/PF/mdev such host side ports may be in hypervisor or VM or both
depending on the privilege
5. eth.mac_address, rdma.port_guid can be programmed at
host port flavours by extending as $ devlink port param set...
(similar to devlink dev param set)
6. more host port params can be added in future when user need arise
7. rep-netdev continue to be eswitch (switchport) representor at the switch side.
a. Hence rep-netdev cannot be used for programming host port's parameters.
8. eswitch devlink instance knows when VF/PF/mdev's switchport are created/removed.
Hence, those will be created/deleted by eswitch.
Similarly for host port flavours too.
Does it look fine? Did I miss something?
We would like to progress on incremental patches for item-4 and
any prep work needed to reach to item-4.
^ permalink raw reply [flat|nested] 100+ messages in thread
* Re: [PATCH net-next v2 4/7] devlink: allow subports on devlink PCI ports
2019-03-20 18:24 ` Parav Pandit
@ 2019-03-20 20:22 ` Jakub Kicinski
2019-03-20 23:39 ` Parav Pandit
2019-03-21 9:08 ` Jiri Pirko
0 siblings, 2 replies; 100+ messages in thread
From: Jakub Kicinski @ 2019-03-20 20:22 UTC (permalink / raw)
To: Parav Pandit; +Cc: Jiri Pirko, Samudrala, Sridhar, davem, netdev, oss-drivers
On Wed, 20 Mar 2019 18:24:15 +0000, Parav Pandit wrote:
> Hi Jiri, Jakub, Samudrala Sridhar,
> > > > > > And physical port in include/uapi/linux/devlink.h also describe
> > > > > > that.
> > > > >
> > > > > By "that" you must mean that the physical is a user facing port.
> > > >
> > > > Can you please describe the difference between 'PF port' and
> > > > 'physical port of include/uapi/linux/devlink.h'? I must have missed
> > > > this crisp definition in discussion between you and Jiri. I am in
> > > > meantime checking the thread.
> > >
> > > Perhaps start with the cover letter which includes an ASCII drawing?
> > >
> > > Using Mellanox nomenclature - PF port is a "representor" for the PF
> > > which may be on another Host (SmartNIC or multihost). It's pretty
> > > much the same thing as a VF port/"representor".
> > >
> > Yes. We are aligned here. :-)
> > I see your point, where in multi-host scenario, a physical port may be 1, but
> > PF ports are 4, because of 4 PFs for 4 hosts.
> > (just an example of 4 hosts with their own mac address sharing 1 physical
> > port).
> >
> > When there is no multihost and one to one mapping between a PF and
> > physical links, there is some overlap between PF port and physical port
> > attributes.
> > I believe, such overlap is fine as long as we have unique indices for the ports.
> >
> > So I am ok to have flavours as physical/cpu/dsa/pf/vf/mdev/switchport.
> > (last 4 as new port flavours).
> >
> > > Physical port is the hole on the panel of the adapter where cable goes.
>
> So my take away from above discussion are:
> 1. Following new port flavours should be added pci_pf/pci_vf/mdev/switchport.
> a. Switchport indicates port on the eswitch. Normally this port has rep-netdev attached to it.
I don't understand the "switchport". Surely physical ports are also
attached to the eswitch? And one of the main purpose of adding the
pci_pf/pci_vf flavours was to generate phys_port_name for the port
netdevs.
Please don't use the term representor if possible. Representor for
most developers describes the way the netdev is implemented in the
driver, so for Mellanox and Netronome different ports will be
representors and non-representors. That's why I prefer port netdev
(attached to eswitch, has switch_id) and host netdev (PF/VF netdev,
vNIC, VSI, etc).
> b. host side port flavours are pci_pf/pci_vf/mdev which may be connected to switchport
See above, pci_pf/pci_vf are needed for phys_port_name generation.
> 2. host side port flavours are not limited to Ethernet, as it is for devlink's port instance.
>
> 3. Each port is continue to be accessed using unique port index.
>
> 4. host side ports and switchport are control objects.
> a. switch side ports reside where current eswitch object of devlink instance reside
> b. for a given VF/PF/mdev such host side ports may be in hypervisor or VM or both
> depending on the privilege
>
> 5. eth.mac_address, rdma.port_guid can be programmed at
> host port flavours by extending as $ devlink port param set...
> (similar to devlink dev param set)
You can keep restating that's your position, but I have *not* conceded
to that.
> 6. more host port params can be added in future when user need arise
>
> 7. rep-netdev continue to be eswitch (switchport) representor at the switch side.
> a. Hence rep-netdev cannot be used for programming host port's parameters.
>
> 8. eswitch devlink instance knows when VF/PF/mdev's switchport are created/removed.
> Hence, those will be created/deleted by eswitch.
> Similarly for host port flavours too.
>
> Does it look fine? Did I miss something?
> We would like to progress on incremental patches for item-4 and
> any prep work needed to reach to item-4.
^ permalink raw reply [flat|nested] 100+ messages in thread
* Re: [PATCH net-next v2 0/7] devlink: expose PF and VF representors as ports
2019-03-01 18:04 [PATCH net-next v2 0/7] devlink: expose PF and VF representors as ports Jakub Kicinski
` (8 preceding siblings ...)
2019-03-04 18:22 ` David Miller
@ 2019-03-20 20:25 ` Jakub Kicinski
2019-03-21 9:11 ` Jiri Pirko
9 siblings, 1 reply; 100+ messages in thread
From: Jakub Kicinski @ 2019-03-20 20:25 UTC (permalink / raw)
To: jiri; +Cc: davem, netdev, oss-drivers
On Fri, 1 Mar 2019 10:04:46 -0800, Jakub Kicinski wrote:
> Hi!
>
> This series is a long overdue follow up to Jiri's work on providing
> a common .ndo_phys_port_name implementation based on devlink ports.
Hi Jiri,
unfortunately I need to focus on some urgent work, so I won't have time
to work on this.
^ permalink raw reply [flat|nested] 100+ messages in thread
* RE: [PATCH net-next v2 4/7] devlink: allow subports on devlink PCI ports
2019-03-20 20:22 ` Jakub Kicinski
@ 2019-03-20 23:39 ` Parav Pandit
2019-03-21 9:08 ` Jiri Pirko
1 sibling, 0 replies; 100+ messages in thread
From: Parav Pandit @ 2019-03-20 23:39 UTC (permalink / raw)
To: Jakub Kicinski; +Cc: Jiri Pirko, Samudrala, Sridhar, davem, netdev, oss-drivers
> -----Original Message-----
> From: Jakub Kicinski <jakub.kicinski@netronome.com>
> Sent: Wednesday, March 20, 2019 3:23 PM
> To: Parav Pandit <parav@mellanox.com>
> Cc: Jiri Pirko <jiri@resnulli.us>; Samudrala, Sridhar
> <sridhar.samudrala@intel.com>; davem@davemloft.net;
> netdev@vger.kernel.org; oss-drivers@netronome.com
> Subject: Re: [PATCH net-next v2 4/7] devlink: allow subports on devlink PCI
> ports
>
> On Wed, 20 Mar 2019 18:24:15 +0000, Parav Pandit wrote:
> > Hi Jiri, Jakub, Samudrala Sridhar,
> > > > > > > And physical port in include/uapi/linux/devlink.h also
> > > > > > > describe that.
> > > > > >
> > > > > > By "that" you must mean that the physical is a user facing port.
> > > > >
> > > > > Can you please describe the difference between 'PF port' and
> > > > > 'physical port of include/uapi/linux/devlink.h'? I must have
> > > > > missed this crisp definition in discussion between you and Jiri.
> > > > > I am in meantime checking the thread.
> > > >
> > > > Perhaps start with the cover letter which includes an ASCII drawing?
> > > >
> > > > Using Mellanox nomenclature - PF port is a "representor" for the
> > > > PF which may be on another Host (SmartNIC or multihost). It's
> > > > pretty much the same thing as a VF port/"representor".
> > > >
> > > Yes. We are aligned here. :-)
> > > I see your point, where in multi-host scenario, a physical port may
> > > be 1, but PF ports are 4, because of 4 PFs for 4 hosts.
> > > (just an example of 4 hosts with their own mac address sharing 1
> > > physical port).
> > >
> > > When there is no multihost and one to one mapping between a PF and
> > > physical links, there is some overlap between PF port and physical
> > > port attributes.
> > > I believe, such overlap is fine as long as we have unique indices for the
> ports.
> > >
> > > So I am ok to have flavours as physical/cpu/dsa/pf/vf/mdev/switchport.
> > > (last 4 as new port flavours).
> > >
> > > > Physical port is the hole on the panel of the adapter where cable goes.
> >
> > So my take away from above discussion are:
> > 1. Following new port flavours should be added
> pci_pf/pci_vf/mdev/switchport.
> > a. Switchport indicates port on the eswitch. Normally this port has rep-
> netdev attached to it.
>
> I don't understand the "switchport". Surely physical ports are also attached
> to the eswitch?
Yes, physical port is attached to eswitch too.
switchport is the one which is not directly visible to the user on the physical panel.
Jiri captured that in the diagram visible here.
[1] https://www.mail-archive.com/netdev@vger.kernel.org/msg289477.html
> And one of the main purpose of adding the pci_pf/pci_vf
> flavours was to generate phys_port_name for the port netdevs.
>
Yes, this is surely needed due to current direction of placement of pf/vf info in rep-netdev phys_port_name.
So this information is available from the past example in thread.
I copied here and extended some parts of it.
Below example is for one PCI function that has two physical ports, one VF, and one mdev.
It enumerates 6 devlink ports.
VF 1 is physical port 1 of PF 0.
mdev uuidX's port 0 is connected to PF 0, port 0.
$ devlink port show
pci/0000:05:00.0/0 eth netdev repndev_pf0_p0 flavour physical switch_id 00154d130d2f
pci/0000:05:00.0/1 eth netdev repndev_pf0_p1 flavour physical switch_id 00154d130d2f
pci/0000:05:00.0/10001 eth netdev repndev_pf0_vf_1 flavour switchport switch_id 00154d130d2f peer pci/0000:05:00.0/1
pci/0000:05:00.0/10002 eth netdev repndev_pf0_p0_mdev_8000 flavour switchport switch_id 00154d130d2f peer mdev/uuidX/0
pci/0000:05:00.0/1 eth netdev flavour vf_ctrl vf 1
mdev/uuidX/0 eth netdev flavour mdev_ctrl
Here eswitch side creates phys_port_name from the peer port's information.
Something like,
struct devlink_peer_info {
struct devlink *dev;
struct devlink_port *port;
};
struct devlink_port {
[...] existing fields;
struct devlink_peer_info peer_info;
};
Depending on peer port flavour, it can construct appropriate phys_port_name ndo_ops return value.
I changed port flavour name to vf_ctrl and mdev_ctrl to make it more explicit for clarity.
Let's discuss what is missing in this model.. or part that's incorrect...
> Please don't use the term representor if possible. Representor for most
> developers describes the way the netdev is implemented in the driver, so for
> Mellanox and Netronome different ports will be representors and non-
> representors. That's why I prefer port netdev (attached to eswitch, has
> switch_id) and host netdev (PF/VF netdev, vNIC, VSI, etc).
>
Frankly I don't see much difference in both the proposals.
No matter which way we go, peer_info is needed anyway for vf/pf/mdev.
Only exception is introduction of explicit host_port_side object (instead of indirect peer way).
This gives clear visibility to users and addresses rdma non_sriov case too.
> > b. host side port flavours are pci_pf/pci_vf/mdev which may be
> > connected to switchport
>
> See above, pci_pf/pci_vf are needed for phys_port_name generation.
>
> > 2. host side port flavours are not limited to Ethernet, as it is for devlink's
> port instance.
> >
> > 3. Each port is continue to be accessed using unique port index.
> >
> > 4. host side ports and switchport are control objects.
> > a. switch side ports reside where current eswitch object of devlink
> > instance reside b. for a given VF/PF/mdev such host side ports may be
> > in hypervisor or VM or both depending on the privilege
> >
> > 5. eth.mac_address, rdma.port_guid can be programmed at host port
> > flavours by extending as $ devlink port param set...
> > (similar to devlink dev param set)
>
> You can keep restating that's your position, but I have *not* conceded to
> that.
>
I understand. Lets discuss any short comings of it (if any) due to which switch side flavours to be defined via indirect peer programming.
> > 6. more host port params can be added in future when user need arise
> >
> > 7. rep-netdev continue to be eswitch (switchport) representor at the switch
> side.
> > a. Hence rep-netdev cannot be used for programming host port's
> parameters.
> >
> > 8. eswitch devlink instance knows when VF/PF/mdev's switchport are
> created/removed.
> > Hence, those will be created/deleted by eswitch.
> > Similarly for host port flavours too.
> >
> > Does it look fine? Did I miss something?
> > We would like to progress on incremental patches for item-4 and any
> > prep work needed to reach to item-4.
^ permalink raw reply [flat|nested] 100+ messages in thread
* Re: [PATCH net-next v2 4/7] devlink: allow subports on devlink PCI ports
2019-03-18 19:16 ` Jakub Kicinski
@ 2019-03-21 8:45 ` Jiri Pirko
2019-03-21 15:14 ` Parav Pandit
0 siblings, 1 reply; 100+ messages in thread
From: Jiri Pirko @ 2019-03-21 8:45 UTC (permalink / raw)
To: Jakub Kicinski
Cc: Parav Pandit, Samudrala, Sridhar, davem, netdev, oss-drivers
Mon, Mar 18, 2019 at 08:16:42PM CET, jakub.kicinski@netronome.com wrote:
>On Mon, 18 Mar 2019 13:11:54 +0100, Jiri Pirko wrote:
>> >> >2. flavour should not be vf/pf, flavour should be hostport, switchport.
>> >> >Because switch is flat and agnostic of pf/vf/mdev.
>> >>
>> >> Not sure. It's good to have this kind of visibility.
>> >
>> >Yes, this subthread honestly makes me go from 60% sure to 95% sure we
>> >shouldn't do the dual object thing :( Seems like Parav is already
>> >confused by it and suggests host port can exist without switch port :(
>>
>> Although I understand your hesitation, the host ports are also
>> associated with the asic and should be under the devlink instance.
>> It is just a matter of proper documentation and clear code to avoid
>> confusions.
>
>They are certainly a part and belong to the ASIC, the question in my
>mind is more along the lines of do we want "one pipe/one port" or is
>it okay to have multiple software objects of the same kind for those
>objects.
>
>To put it differently - do want a port object for each port of the ASIC
>or do we want a port object for each netdev..
Perhaps "port" name of the object is misleading. From the beginning, I
ment to have it for both switch ports and host ports. I admit that "host
port" is a bit misleading, as it is not really a port of eswitch, but
the counter part. But if we introduce another object for that purpose in
devlink (like "partititon"), it would be a lot of duplication I think.
Question is, do we need the "host port"? Can't we just put a relation to
host netdev in the eswitch port.
So as you suggest, we would have
devlink_port -+-- switch netdev/ibdev
|
+-- host netdev/ibdev
So the "weights" of both switch/host netdev/ibdev to devlink_port
relations would be equivalent.
Then, the devlink_port would represent the whole "pipe" with both ends.
More I think about it, the more it makes sense to me...
^ permalink raw reply [flat|nested] 100+ messages in thread
* Re: [PATCH net-next v2 4/7] devlink: allow subports on devlink PCI ports
2019-03-20 20:22 ` Jakub Kicinski
2019-03-20 23:39 ` Parav Pandit
@ 2019-03-21 9:08 ` Jiri Pirko
2019-03-21 15:03 ` Parav Pandit
1 sibling, 1 reply; 100+ messages in thread
From: Jiri Pirko @ 2019-03-21 9:08 UTC (permalink / raw)
To: Jakub Kicinski
Cc: Parav Pandit, Samudrala, Sridhar, davem, netdev, oss-drivers
Wed, Mar 20, 2019 at 09:22:57PM CET, jakub.kicinski@netronome.com wrote:
>On Wed, 20 Mar 2019 18:24:15 +0000, Parav Pandit wrote:
>> Hi Jiri, Jakub, Samudrala Sridhar,
>> > > > > > And physical port in include/uapi/linux/devlink.h also describe
>> > > > > > that.
>> > > > >
>> > > > > By "that" you must mean that the physical is a user facing port.
>> > > >
>> > > > Can you please describe the difference between 'PF port' and
>> > > > 'physical port of include/uapi/linux/devlink.h'? I must have missed
>> > > > this crisp definition in discussion between you and Jiri. I am in
>> > > > meantime checking the thread.
>> > >
>> > > Perhaps start with the cover letter which includes an ASCII drawing?
>> > >
>> > > Using Mellanox nomenclature - PF port is a "representor" for the PF
>> > > which may be on another Host (SmartNIC or multihost). It's pretty
>> > > much the same thing as a VF port/"representor".
>> > >
>> > Yes. We are aligned here. :-)
>> > I see your point, where in multi-host scenario, a physical port may be 1, but
>> > PF ports are 4, because of 4 PFs for 4 hosts.
>> > (just an example of 4 hosts with their own mac address sharing 1 physical
>> > port).
>> >
>> > When there is no multihost and one to one mapping between a PF and
>> > physical links, there is some overlap between PF port and physical port
>> > attributes.
>> > I believe, such overlap is fine as long as we have unique indices for the ports.
>> >
>> > So I am ok to have flavours as physical/cpu/dsa/pf/vf/mdev/switchport.
>> > (last 4 as new port flavours).
>> >
>> > > Physical port is the hole on the panel of the adapter where cable goes.
>>
>> So my take away from above discussion are:
>> 1. Following new port flavours should be added pci_pf/pci_vf/mdev/switchport.
>> a. Switchport indicates port on the eswitch. Normally this port has rep-netdev attached to it.
>
>I don't understand the "switchport". Surely physical ports are also
>attached to the eswitch? And one of the main purpose of adding the
>pci_pf/pci_vf flavours was to generate phys_port_name for the port
>netdevs.
>
>Please don't use the term representor if possible. Representor for
>most developers describes the way the netdev is implemented in the
>driver, so for Mellanox and Netronome different ports will be
>representors and non-representors. That's why I prefer port netdev
>(attached to eswitch, has switch_id) and host netdev (PF/VF netdev,
>vNIC, VSI, etc).
>
>> b. host side port flavours are pci_pf/pci_vf/mdev which may be connected to switchport
>
>See above, pci_pf/pci_vf are needed for phys_port_name generation.
Yep, that makes sense.
>
>> 2. host side port flavours are not limited to Ethernet, as it is for devlink's port instance.
>>
>> 3. Each port is continue to be accessed using unique port index.
>>
>> 4. host side ports and switchport are control objects.
>> a. switch side ports reside where current eswitch object of devlink instance reside
>> b. for a given VF/PF/mdev such host side ports may be in hypervisor or VM or both
>> depending on the privilege
>>
>> 5. eth.mac_address, rdma.port_guid can be programmed at
>> host port flavours by extending as $ devlink port param set...
>> (similar to devlink dev param set)
>
>You can keep restating that's your position, but I have *not* conceded
>to that.
I'm also not convinced that host dummy ports are good idea to hold
these.
>
>> 6. more host port params can be added in future when user need arise
>>
>> 7. rep-netdev continue to be eswitch (switchport) representor at the switch side.
>> a. Hence rep-netdev cannot be used for programming host port's parameters.
>>
>> 8. eswitch devlink instance knows when VF/PF/mdev's switchport are created/removed.
>> Hence, those will be created/deleted by eswitch.
>> Similarly for host port flavours too.
>>
>> Does it look fine? Did I miss something?
>> We would like to progress on incremental patches for item-4 and
>> any prep work needed to reach to item-4.
^ permalink raw reply [flat|nested] 100+ messages in thread
* Re: [PATCH net-next v2 4/7] devlink: allow subports on devlink PCI ports
2019-03-18 19:19 ` Jakub Kicinski
2019-03-18 19:38 ` Parav Pandit
@ 2019-03-21 9:09 ` Jiri Pirko
1 sibling, 0 replies; 100+ messages in thread
From: Jiri Pirko @ 2019-03-21 9:09 UTC (permalink / raw)
To: Jakub Kicinski
Cc: Parav Pandit, Samudrala, Sridhar, davem, netdev, oss-drivers
Mon, Mar 18, 2019 at 08:19:00PM CET, jakub.kicinski@netronome.com wrote:
>On Mon, 18 Mar 2019 13:21:05 +0100, Jiri Pirko wrote:
>> >First two entries shows the link between hostport and switchport.
>> >$ devlink port show
>> >pci/0000:05:00.0/10002 eth netdev flavour switchport switch_id 00154d130d2f peer pci/0000:05:00.0/1
>> >
>> >pci/0000:05:00.0/1 eth netdev flavour hostport switch_id 00154d130d2f peer pci/0000:05:00.0/10002
>>
>> Hostport should not have switch_id.
>
>Isn't a concept of a port of a switch without a switch ID a red flag?
If we want to have hostports as devlink ports, they should not have
switch id. They are not switch ports but rather counterparts of the
switch ports.
^ permalink raw reply [flat|nested] 100+ messages in thread
* Re: [PATCH net-next v2 0/7] devlink: expose PF and VF representors as ports
2019-03-20 20:25 ` Jakub Kicinski
@ 2019-03-21 9:11 ` Jiri Pirko
0 siblings, 0 replies; 100+ messages in thread
From: Jiri Pirko @ 2019-03-21 9:11 UTC (permalink / raw)
To: Jakub Kicinski; +Cc: davem, netdev, oss-drivers
Wed, Mar 20, 2019 at 09:25:53PM CET, jakub.kicinski@netronome.com wrote:
>On Fri, 1 Mar 2019 10:04:46 -0800, Jakub Kicinski wrote:
>> Hi!
>>
>> This series is a long overdue follow up to Jiri's work on providing
>> a common .ndo_phys_port_name implementation based on devlink ports.
>
>Hi Jiri,
>
>unfortunately I need to focus on some urgent work, so I won't have time
>to work on this.
Okay. I'll start pushing my devlink patches I have in qeueu and we'll
get back to this later, no worries.
^ permalink raw reply [flat|nested] 100+ messages in thread
* RE: [PATCH net-next v2 4/7] devlink: allow subports on devlink PCI ports
2019-03-21 9:08 ` Jiri Pirko
@ 2019-03-21 15:03 ` Parav Pandit
2019-03-21 16:16 ` Jiri Pirko
0 siblings, 1 reply; 100+ messages in thread
From: Parav Pandit @ 2019-03-21 15:03 UTC (permalink / raw)
To: Jiri Pirko, Jakub Kicinski; +Cc: Samudrala, Sridhar, davem, netdev, oss-drivers
Hi Jiri,
> -----Original Message-----
> From: Jiri Pirko <jiri@resnulli.us>
> Sent: Thursday, March 21, 2019 4:08 AM
> To: Jakub Kicinski <jakub.kicinski@netronome.com>
> Cc: Parav Pandit <parav@mellanox.com>; Samudrala, Sridhar
> <sridhar.samudrala@intel.com>; davem@davemloft.net;
> netdev@vger.kernel.org; oss-drivers@netronome.com
> Subject: Re: [PATCH net-next v2 4/7] devlink: allow subports on devlink PCI
> ports
>
> Wed, Mar 20, 2019 at 09:22:57PM CET, jakub.kicinski@netronome.com
> wrote:
> >On Wed, 20 Mar 2019 18:24:15 +0000, Parav Pandit wrote:
> >> Hi Jiri, Jakub, Samudrala Sridhar,
> >> > > > > > And physical port in include/uapi/linux/devlink.h also
> >> > > > > > describe that.
> >> > > > >
> >> > > > > By "that" you must mean that the physical is a user facing port.
> >> > > >
> >> > > > Can you please describe the difference between 'PF port' and
> >> > > > 'physical port of include/uapi/linux/devlink.h'? I must have
> >> > > > missed this crisp definition in discussion between you and
> >> > > > Jiri. I am in meantime checking the thread.
> >> > >
> >> > > Perhaps start with the cover letter which includes an ASCII drawing?
> >> > >
> >> > > Using Mellanox nomenclature - PF port is a "representor" for the
> >> > > PF which may be on another Host (SmartNIC or multihost). It's
> >> > > pretty much the same thing as a VF port/"representor".
> >> > >
> >> > Yes. We are aligned here. :-)
> >> > I see your point, where in multi-host scenario, a physical port may
> >> > be 1, but PF ports are 4, because of 4 PFs for 4 hosts.
> >> > (just an example of 4 hosts with their own mac address sharing 1
> >> > physical port).
> >> >
> >> > When there is no multihost and one to one mapping between a PF and
> >> > physical links, there is some overlap between PF port and physical
> >> > port attributes.
> >> > I believe, such overlap is fine as long as we have unique indices for the
> ports.
> >> >
> >> > So I am ok to have flavours as physical/cpu/dsa/pf/vf/mdev/switchport.
> >> > (last 4 as new port flavours).
> >> >
> >> > > Physical port is the hole on the panel of the adapter where cable
> goes.
> >>
> >> So my take away from above discussion are:
> >> 1. Following new port flavours should be added
> pci_pf/pci_vf/mdev/switchport.
> >> a. Switchport indicates port on the eswitch. Normally this port has rep-
> netdev attached to it.
> >
> >I don't understand the "switchport". Surely physical ports are also
> >attached to the eswitch? And one of the main purpose of adding the
> >pci_pf/pci_vf flavours was to generate phys_port_name for the port
> >netdevs.
> >
> >Please don't use the term representor if possible. Representor for
> >most developers describes the way the netdev is implemented in the
> >driver, so for Mellanox and Netronome different ports will be
> >representors and non-representors. That's why I prefer port netdev
> >(attached to eswitch, has switch_id) and host netdev (PF/VF netdev,
> >vNIC, VSI, etc).
> >
> >> b. host side port flavours are pci_pf/pci_vf/mdev which may be
> >> connected to switchport
> >
> >See above, pci_pf/pci_vf are needed for phys_port_name generation.
>
> Yep, that makes sense.
>
>
> >
> >> 2. host side port flavours are not limited to Ethernet, as it is for devlink's
> port instance.
> >>
> >> 3. Each port is continue to be accessed using unique port index.
> >>
> >> 4. host side ports and switchport are control objects.
> >> a. switch side ports reside where current eswitch object of devlink
> >> instance reside b. for a given VF/PF/mdev such host side ports may be
> >> in hypervisor or VM or both depending on the privilege
> >>
> >> 5. eth.mac_address, rdma.port_guid can be programmed at host port
> >> flavours by extending as $ devlink port param set...
> >> (similar to devlink dev param set)
> >
> >You can keep restating that's your position, but I have *not* conceded
> >to that.
>
> I'm also not convinced that host dummy ports are good idea to hold these.
>
>
I didn't understand what do you mean my dummy port.
Can you explain what is wrong in programming host port params using host_port object?
Few questions are unanswered in my past 2 or 3 emails.
Can you please go through them?
Can you point to some example switch API where you program host params at switch?
> >
> >> 6. more host port params can be added in future when user need arise
> >>
> >> 7. rep-netdev continue to be eswitch (switchport) representor at the
> switch side.
> >> a. Hence rep-netdev cannot be used for programming host port's
> parameters.
> >>
> >> 8. eswitch devlink instance knows when VF/PF/mdev's switchport are
> created/removed.
> >> Hence, those will be created/deleted by eswitch.
> >> Similarly for host port flavours too.
> >>
> >> Does it look fine? Did I miss something?
> >> We would like to progress on incremental patches for item-4 and any
> >> prep work needed to reach to item-4.
^ permalink raw reply [flat|nested] 100+ messages in thread
* RE: [PATCH net-next v2 4/7] devlink: allow subports on devlink PCI ports
2019-03-21 8:45 ` Jiri Pirko
@ 2019-03-21 15:14 ` Parav Pandit
2019-03-21 16:14 ` Jiri Pirko
0 siblings, 1 reply; 100+ messages in thread
From: Parav Pandit @ 2019-03-21 15:14 UTC (permalink / raw)
To: Jiri Pirko, Jakub Kicinski; +Cc: Samudrala, Sridhar, davem, netdev, oss-drivers
> -----Original Message-----
> From: Jiri Pirko <jiri@resnulli.us>
> Sent: Thursday, March 21, 2019 3:45 AM
> To: Jakub Kicinski <jakub.kicinski@netronome.com>
> Cc: Parav Pandit <parav@mellanox.com>; Samudrala, Sridhar
> <sridhar.samudrala@intel.com>; davem@davemloft.net;
> netdev@vger.kernel.org; oss-drivers@netronome.com
> Subject: Re: [PATCH net-next v2 4/7] devlink: allow subports on devlink PCI
> ports
>
> Mon, Mar 18, 2019 at 08:16:42PM CET, jakub.kicinski@netronome.com
> wrote:
> >On Mon, 18 Mar 2019 13:11:54 +0100, Jiri Pirko wrote:
> >> >> >2. flavour should not be vf/pf, flavour should be hostport, switchport.
> >> >> >Because switch is flat and agnostic of pf/vf/mdev.
> >> >>
> >> >> Not sure. It's good to have this kind of visibility.
> >> >
> >> >Yes, this subthread honestly makes me go from 60% sure to 95% sure
> >> >we shouldn't do the dual object thing :( Seems like Parav is
> >> >already confused by it and suggests host port can exist without
> >> >switch port :(
> >>
> >> Although I understand your hesitation, the host ports are also
> >> associated with the asic and should be under the devlink instance.
> >> It is just a matter of proper documentation and clear code to avoid
> >> confusions.
> >
> >They are certainly a part and belong to the ASIC, the question in my
> >mind is more along the lines of do we want "one pipe/one port" or is it
> >okay to have multiple software objects of the same kind for those
> >objects.
> >
> >To put it differently - do want a port object for each port of the ASIC
> >or do we want a port object for each netdev..
>
> Perhaps "port" name of the object is misleading. From the beginning, I ment
> to have it for both switch ports and host ports. I admit that "host port" is a
> bit misleading, as it is not really a port of eswitch, but the counter part. But
> if we introduce another object for that purpose in devlink (like "partititon"),
> it would be a lot of duplication I think.
>
> Question is, do we need the "host port"? Can't we just put a relation to host
> netdev in the eswitch port.
>
Can you please explain how does it work for rdma for non sriov use case?
Do we have to create a fake eswitch object?
> So as you suggest, we would have
> devlink_port -+-- switch netdev/ibdev
> |
> +-- host netdev/ibdev
>
How does this work for rdma to program single node_guid for dual port ibdev?
Did you actually read the recent example I showed in [1]?
[1] https://marc.info/?l=linux-netdev&m=155312521817191&w=2
And why it doesn't address all the use cases of pf/vf/mdev, ibdev, netdev?
> So the "weights" of both switch/host netdev/ibdev to devlink_port relations
> would be equivalent.
>
> Then, the devlink_port would represent the whole "pipe" with both ends.
>
> More I think about it, the more it makes sense to me...
^ permalink raw reply [flat|nested] 100+ messages in thread
* Re: [PATCH net-next v2 4/7] devlink: allow subports on devlink PCI ports
2019-03-21 15:14 ` Parav Pandit
@ 2019-03-21 16:14 ` Jiri Pirko
2019-03-21 16:52 ` Parav Pandit
0 siblings, 1 reply; 100+ messages in thread
From: Jiri Pirko @ 2019-03-21 16:14 UTC (permalink / raw)
To: Parav Pandit
Cc: Jakub Kicinski, Samudrala, Sridhar, davem, netdev, oss-drivers
Thu, Mar 21, 2019 at 04:14:53PM CET, parav@mellanox.com wrote:
>
>
>> -----Original Message-----
>> From: Jiri Pirko <jiri@resnulli.us>
>> Sent: Thursday, March 21, 2019 3:45 AM
>> To: Jakub Kicinski <jakub.kicinski@netronome.com>
>> Cc: Parav Pandit <parav@mellanox.com>; Samudrala, Sridhar
>> <sridhar.samudrala@intel.com>; davem@davemloft.net;
>> netdev@vger.kernel.org; oss-drivers@netronome.com
>> Subject: Re: [PATCH net-next v2 4/7] devlink: allow subports on devlink PCI
>> ports
>>
>> Mon, Mar 18, 2019 at 08:16:42PM CET, jakub.kicinski@netronome.com
>> wrote:
>> >On Mon, 18 Mar 2019 13:11:54 +0100, Jiri Pirko wrote:
>> >> >> >2. flavour should not be vf/pf, flavour should be hostport, switchport.
>> >> >> >Because switch is flat and agnostic of pf/vf/mdev.
>> >> >>
>> >> >> Not sure. It's good to have this kind of visibility.
>> >> >
>> >> >Yes, this subthread honestly makes me go from 60% sure to 95% sure
>> >> >we shouldn't do the dual object thing :( Seems like Parav is
>> >> >already confused by it and suggests host port can exist without
>> >> >switch port :(
>> >>
>> >> Although I understand your hesitation, the host ports are also
>> >> associated with the asic and should be under the devlink instance.
>> >> It is just a matter of proper documentation and clear code to avoid
>> >> confusions.
>> >
>> >They are certainly a part and belong to the ASIC, the question in my
>> >mind is more along the lines of do we want "one pipe/one port" or is it
>> >okay to have multiple software objects of the same kind for those
>> >objects.
>> >
>> >To put it differently - do want a port object for each port of the ASIC
>> >or do we want a port object for each netdev..
>>
>> Perhaps "port" name of the object is misleading. From the beginning, I ment
>> to have it for both switch ports and host ports. I admit that "host port" is a
>> bit misleading, as it is not really a port of eswitch, but the counter part. But
>> if we introduce another object for that purpose in devlink (like "partititon"),
>> it would be a lot of duplication I think.
>>
>> Question is, do we need the "host port"? Can't we just put a relation to host
>> netdev in the eswitch port.
>>
>Can you please explain how does it work for rdma for non sriov use case?
>Do we have to create a fake eswitch object?
Could you please provide details on "rdma for non sriov use case"?
>
>> So as you suggest, we would have
>> devlink_port -+-- switch netdev/ibdev
>> |
>> +-- host netdev/ibdev
>>
>
>How does this work for rdma to program single node_guid for dual port ibdev?
>Did you actually read the recent example I showed in [1]?
>[1] https://marc.info/?l=linux-netdev&m=155312521817191&w=2
>And why it doesn't address all the use cases of pf/vf/mdev, ibdev, netdev?
>
>> So the "weights" of both switch/host netdev/ibdev to devlink_port relations
>> would be equivalent.
>>
>> Then, the devlink_port would represent the whole "pipe" with both ends.
>>
>> More I think about it, the more it makes sense to me...
^ permalink raw reply [flat|nested] 100+ messages in thread
* Re: [PATCH net-next v2 4/7] devlink: allow subports on devlink PCI ports
2019-03-21 15:03 ` Parav Pandit
@ 2019-03-21 16:16 ` Jiri Pirko
2019-03-21 16:50 ` Parav Pandit
0 siblings, 1 reply; 100+ messages in thread
From: Jiri Pirko @ 2019-03-21 16:16 UTC (permalink / raw)
To: Parav Pandit
Cc: Jakub Kicinski, Samudrala, Sridhar, davem, netdev, oss-drivers
Thu, Mar 21, 2019 at 04:03:58PM CET, parav@mellanox.com wrote:
>Hi Jiri,
>
>> -----Original Message-----
>> From: Jiri Pirko <jiri@resnulli.us>
>> Sent: Thursday, March 21, 2019 4:08 AM
>> To: Jakub Kicinski <jakub.kicinski@netronome.com>
>> Cc: Parav Pandit <parav@mellanox.com>; Samudrala, Sridhar
>> <sridhar.samudrala@intel.com>; davem@davemloft.net;
>> netdev@vger.kernel.org; oss-drivers@netronome.com
>> Subject: Re: [PATCH net-next v2 4/7] devlink: allow subports on devlink PCI
>> ports
>>
>> Wed, Mar 20, 2019 at 09:22:57PM CET, jakub.kicinski@netronome.com
>> wrote:
>> >On Wed, 20 Mar 2019 18:24:15 +0000, Parav Pandit wrote:
>> >> Hi Jiri, Jakub, Samudrala Sridhar,
>> >> > > > > > And physical port in include/uapi/linux/devlink.h also
>> >> > > > > > describe that.
>> >> > > > >
>> >> > > > > By "that" you must mean that the physical is a user facing port.
>> >> > > >
>> >> > > > Can you please describe the difference between 'PF port' and
>> >> > > > 'physical port of include/uapi/linux/devlink.h'? I must have
>> >> > > > missed this crisp definition in discussion between you and
>> >> > > > Jiri. I am in meantime checking the thread.
>> >> > >
>> >> > > Perhaps start with the cover letter which includes an ASCII drawing?
>> >> > >
>> >> > > Using Mellanox nomenclature - PF port is a "representor" for the
>> >> > > PF which may be on another Host (SmartNIC or multihost). It's
>> >> > > pretty much the same thing as a VF port/"representor".
>> >> > >
>> >> > Yes. We are aligned here. :-)
>> >> > I see your point, where in multi-host scenario, a physical port may
>> >> > be 1, but PF ports are 4, because of 4 PFs for 4 hosts.
>> >> > (just an example of 4 hosts with their own mac address sharing 1
>> >> > physical port).
>> >> >
>> >> > When there is no multihost and one to one mapping between a PF and
>> >> > physical links, there is some overlap between PF port and physical
>> >> > port attributes.
>> >> > I believe, such overlap is fine as long as we have unique indices for the
>> ports.
>> >> >
>> >> > So I am ok to have flavours as physical/cpu/dsa/pf/vf/mdev/switchport.
>> >> > (last 4 as new port flavours).
>> >> >
>> >> > > Physical port is the hole on the panel of the adapter where cable
>> goes.
>> >>
>> >> So my take away from above discussion are:
>> >> 1. Following new port flavours should be added
>> pci_pf/pci_vf/mdev/switchport.
>> >> a. Switchport indicates port on the eswitch. Normally this port has rep-
>> netdev attached to it.
>> >
>> >I don't understand the "switchport". Surely physical ports are also
>> >attached to the eswitch? And one of the main purpose of adding the
>> >pci_pf/pci_vf flavours was to generate phys_port_name for the port
>> >netdevs.
>> >
>> >Please don't use the term representor if possible. Representor for
>> >most developers describes the way the netdev is implemented in the
>> >driver, so for Mellanox and Netronome different ports will be
>> >representors and non-representors. That's why I prefer port netdev
>> >(attached to eswitch, has switch_id) and host netdev (PF/VF netdev,
>> >vNIC, VSI, etc).
>> >
>> >> b. host side port flavours are pci_pf/pci_vf/mdev which may be
>> >> connected to switchport
>> >
>> >See above, pci_pf/pci_vf are needed for phys_port_name generation.
>>
>> Yep, that makes sense.
>>
>>
>> >
>> >> 2. host side port flavours are not limited to Ethernet, as it is for devlink's
>> port instance.
>> >>
>> >> 3. Each port is continue to be accessed using unique port index.
>> >>
>> >> 4. host side ports and switchport are control objects.
>> >> a. switch side ports reside where current eswitch object of devlink
>> >> instance reside b. for a given VF/PF/mdev such host side ports may be
>> >> in hypervisor or VM or both depending on the privilege
>> >>
>> >> 5. eth.mac_address, rdma.port_guid can be programmed at host port
>> >> flavours by extending as $ devlink port param set...
>> >> (similar to devlink dev param set)
>> >
>> >You can keep restating that's your position, but I have *not* conceded
>> >to that.
>>
>> I'm also not convinced that host dummy ports are good idea to hold these.
>>
>>
>I didn't understand what do you mean my dummy port.
It's a port for a VF host port which is not actually in the host but in
the vm. Very confusing.
>Can you explain what is wrong in programming host port params using host_port object?
>Few questions are unanswered in my past 2 or 3 emails.
>Can you please go through them?
>Can you point to some example switch API where you program host params at switch?
>
>> >
>> >> 6. more host port params can be added in future when user need arise
>> >>
>> >> 7. rep-netdev continue to be eswitch (switchport) representor at the
>> switch side.
>> >> a. Hence rep-netdev cannot be used for programming host port's
>> parameters.
>> >>
>> >> 8. eswitch devlink instance knows when VF/PF/mdev's switchport are
>> created/removed.
>> >> Hence, those will be created/deleted by eswitch.
>> >> Similarly for host port flavours too.
>> >>
>> >> Does it look fine? Did I miss something?
>> >> We would like to progress on incremental patches for item-4 and any
>> >> prep work needed to reach to item-4.
^ permalink raw reply [flat|nested] 100+ messages in thread
* RE: [PATCH net-next v2 4/7] devlink: allow subports on devlink PCI ports
2019-03-21 16:16 ` Jiri Pirko
@ 2019-03-21 16:50 ` Parav Pandit
2019-03-21 17:23 ` Jiri Pirko
0 siblings, 1 reply; 100+ messages in thread
From: Parav Pandit @ 2019-03-21 16:50 UTC (permalink / raw)
To: Jiri Pirko; +Cc: Jakub Kicinski, Samudrala, Sridhar, davem, netdev, oss-drivers
> -----Original Message-----
> From: Jiri Pirko <jiri@resnulli.us>
> Sent: Thursday, March 21, 2019 11:16 AM
> To: Parav Pandit <parav@mellanox.com>
> Cc: Jakub Kicinski <jakub.kicinski@netronome.com>; Samudrala, Sridhar
> <sridhar.samudrala@intel.com>; davem@davemloft.net;
> netdev@vger.kernel.org; oss-drivers@netronome.com
> Subject: Re: [PATCH net-next v2 4/7] devlink: allow subports on devlink PCI
> ports
>
> Thu, Mar 21, 2019 at 04:03:58PM CET, parav@mellanox.com wrote:
> >Hi Jiri,
> >
> >> -----Original Message-----
> >> From: Jiri Pirko <jiri@resnulli.us>
> >> Sent: Thursday, March 21, 2019 4:08 AM
> >> To: Jakub Kicinski <jakub.kicinski@netronome.com>
> >> Cc: Parav Pandit <parav@mellanox.com>; Samudrala, Sridhar
> >> <sridhar.samudrala@intel.com>; davem@davemloft.net;
> >> netdev@vger.kernel.org; oss-drivers@netronome.com
> >> Subject: Re: [PATCH net-next v2 4/7] devlink: allow subports on
> >> devlink PCI ports
> >>
> >> Wed, Mar 20, 2019 at 09:22:57PM CET, jakub.kicinski@netronome.com
> >> wrote:
> >> >On Wed, 20 Mar 2019 18:24:15 +0000, Parav Pandit wrote:
> >> >> Hi Jiri, Jakub, Samudrala Sridhar,
> >> >> > > > > > And physical port in include/uapi/linux/devlink.h also
> >> >> > > > > > describe that.
> >> >> > > > >
> >> >> > > > > By "that" you must mean that the physical is a user facing port.
> >> >> > > >
> >> >> > > > Can you please describe the difference between 'PF port' and
> >> >> > > > 'physical port of include/uapi/linux/devlink.h'? I must have
> >> >> > > > missed this crisp definition in discussion between you and
> >> >> > > > Jiri. I am in meantime checking the thread.
> >> >> > >
> >> >> > > Perhaps start with the cover letter which includes an ASCII
> drawing?
> >> >> > >
> >> >> > > Using Mellanox nomenclature - PF port is a "representor" for
> >> >> > > the PF which may be on another Host (SmartNIC or multihost).
> >> >> > > It's pretty much the same thing as a VF port/"representor".
> >> >> > >
> >> >> > Yes. We are aligned here. :-)
> >> >> > I see your point, where in multi-host scenario, a physical port
> >> >> > may be 1, but PF ports are 4, because of 4 PFs for 4 hosts.
> >> >> > (just an example of 4 hosts with their own mac address sharing 1
> >> >> > physical port).
> >> >> >
> >> >> > When there is no multihost and one to one mapping between a PF
> >> >> > and physical links, there is some overlap between PF port and
> >> >> > physical port attributes.
> >> >> > I believe, such overlap is fine as long as we have unique
> >> >> > indices for the
> >> ports.
> >> >> >
> >> >> > So I am ok to have flavours as
> physical/cpu/dsa/pf/vf/mdev/switchport.
> >> >> > (last 4 as new port flavours).
> >> >> >
> >> >> > > Physical port is the hole on the panel of the adapter where
> >> >> > > cable
> >> goes.
> >> >>
> >> >> So my take away from above discussion are:
> >> >> 1. Following new port flavours should be added
> >> pci_pf/pci_vf/mdev/switchport.
> >> >> a. Switchport indicates port on the eswitch. Normally this port
> >> >> has rep-
> >> netdev attached to it.
> >> >
> >> >I don't understand the "switchport". Surely physical ports are also
> >> >attached to the eswitch? And one of the main purpose of adding the
> >> >pci_pf/pci_vf flavours was to generate phys_port_name for the port
> >> >netdevs.
> >> >
> >> >Please don't use the term representor if possible. Representor for
> >> >most developers describes the way the netdev is implemented in the
> >> >driver, so for Mellanox and Netronome different ports will be
> >> >representors and non-representors. That's why I prefer port netdev
> >> >(attached to eswitch, has switch_id) and host netdev (PF/VF netdev,
> >> >vNIC, VSI, etc).
> >> >
> >> >> b. host side port flavours are pci_pf/pci_vf/mdev which may be
> >> >> connected to switchport
> >> >
> >> >See above, pci_pf/pci_vf are needed for phys_port_name generation.
> >>
> >> Yep, that makes sense.
> >>
> >>
> >> >
> >> >> 2. host side port flavours are not limited to Ethernet, as it is
> >> >> for devlink's
> >> port instance.
> >> >>
> >> >> 3. Each port is continue to be accessed using unique port index.
> >> >>
> >> >> 4. host side ports and switchport are control objects.
> >> >> a. switch side ports reside where current eswitch object of
> >> >> devlink instance reside b. for a given VF/PF/mdev such host side
> >> >> ports may be in hypervisor or VM or both depending on the
> >> >> privilege
> >> >>
> >> >> 5. eth.mac_address, rdma.port_guid can be programmed at host port
> >> >> flavours by extending as $ devlink port param set...
> >> >> (similar to devlink dev param set)
> >> >
> >> >You can keep restating that's your position, but I have *not*
> >> >conceded to that.
> >>
> >> I'm also not convinced that host dummy ports are good idea to hold
> these.
> >>
> >>
> >I didn't understand what do you mean my dummy port.
>
> It's a port for a VF host port which is not actually in the host but in the vm.
> Very confusing.
>
It is the vf_ctrl flavour. I don't see it any different than rep-netdev.
rep-netdev is not that confusing to us that represent eswitch vport.
Why vf_ctrl flavour port that represents otherside of the pipe as you have shown in example?
Why it that confusing?
> >Can you explain what is wrong in programming host port params using
> host_port object?
> >Few questions are unanswered in my past 2 or 3 emails.
> >Can you please go through them?
> >Can you point to some example switch API where you program host params
> at switch?
> >
> >> >
> >> >> 6. more host port params can be added in future when user need
> >> >> arise
> >> >>
> >> >> 7. rep-netdev continue to be eswitch (switchport) representor at
> >> >> the
> >> switch side.
> >> >> a. Hence rep-netdev cannot be used for programming host port's
> >> parameters.
> >> >>
> >> >> 8. eswitch devlink instance knows when VF/PF/mdev's switchport are
> >> created/removed.
> >> >> Hence, those will be created/deleted by eswitch.
> >> >> Similarly for host port flavours too.
> >> >>
> >> >> Does it look fine? Did I miss something?
> >> >> We would like to progress on incremental patches for item-4 and
> >> >> any prep work needed to reach to item-4.
^ permalink raw reply [flat|nested] 100+ messages in thread
* RE: [PATCH net-next v2 4/7] devlink: allow subports on devlink PCI ports
2019-03-21 16:14 ` Jiri Pirko
@ 2019-03-21 16:52 ` Parav Pandit
2019-03-21 17:20 ` Jiri Pirko
0 siblings, 1 reply; 100+ messages in thread
From: Parav Pandit @ 2019-03-21 16:52 UTC (permalink / raw)
To: Jiri Pirko; +Cc: Jakub Kicinski, Samudrala, Sridhar, davem, netdev, oss-drivers
> -----Original Message-----
> From: Jiri Pirko <jiri@resnulli.us>
> Sent: Thursday, March 21, 2019 11:14 AM
> To: Parav Pandit <parav@mellanox.com>
> Cc: Jakub Kicinski <jakub.kicinski@netronome.com>; Samudrala, Sridhar
> <sridhar.samudrala@intel.com>; davem@davemloft.net;
> netdev@vger.kernel.org; oss-drivers@netronome.com
> Subject: Re: [PATCH net-next v2 4/7] devlink: allow subports on devlink PCI
> ports
>
> Thu, Mar 21, 2019 at 04:14:53PM CET, parav@mellanox.com wrote:
> >
> >
> >> -----Original Message-----
> >> From: Jiri Pirko <jiri@resnulli.us>
> >> Sent: Thursday, March 21, 2019 3:45 AM
> >> To: Jakub Kicinski <jakub.kicinski@netronome.com>
> >> Cc: Parav Pandit <parav@mellanox.com>; Samudrala, Sridhar
> >> <sridhar.samudrala@intel.com>; davem@davemloft.net;
> >> netdev@vger.kernel.org; oss-drivers@netronome.com
> >> Subject: Re: [PATCH net-next v2 4/7] devlink: allow subports on
> >> devlink PCI ports
> >>
> >> Mon, Mar 18, 2019 at 08:16:42PM CET, jakub.kicinski@netronome.com
> >> wrote:
> >> >On Mon, 18 Mar 2019 13:11:54 +0100, Jiri Pirko wrote:
> >> >> >> >2. flavour should not be vf/pf, flavour should be hostport,
> switchport.
> >> >> >> >Because switch is flat and agnostic of pf/vf/mdev.
> >> >> >>
> >> >> >> Not sure. It's good to have this kind of visibility.
> >> >> >
> >> >> >Yes, this subthread honestly makes me go from 60% sure to 95%
> >> >> >sure we shouldn't do the dual object thing :( Seems like Parav
> >> >> >is already confused by it and suggests host port can exist
> >> >> >without switch port :(
> >> >>
> >> >> Although I understand your hesitation, the host ports are also
> >> >> associated with the asic and should be under the devlink instance.
> >> >> It is just a matter of proper documentation and clear code to
> >> >> avoid confusions.
> >> >
> >> >They are certainly a part and belong to the ASIC, the question in my
> >> >mind is more along the lines of do we want "one pipe/one port" or is
> >> >it okay to have multiple software objects of the same kind for those
> >> >objects.
> >> >
> >> >To put it differently - do want a port object for each port of the
> >> >ASIC or do we want a port object for each netdev..
> >>
> >> Perhaps "port" name of the object is misleading. From the beginning,
> >> I ment to have it for both switch ports and host ports. I admit that
> >> "host port" is a bit misleading, as it is not really a port of
> >> eswitch, but the counter part. But if we introduce another object for
> >> that purpose in devlink (like "partititon"), it would be a lot of duplication
> I think.
> >>
> >> Question is, do we need the "host port"? Can't we just put a relation
> >> to host netdev in the eswitch port.
> >>
> >Can you please explain how does it work for rdma for non sriov use case?
> >Do we have to create a fake eswitch object?
>
> Could you please provide details on "rdma for non sriov use case"?
>
There are multiple mdevs on PFs that happen to have link layer as IB and those devlink instances have port that deserved to be configured same way as that of Eth.
^ permalink raw reply [flat|nested] 100+ messages in thread
* Re: [PATCH net-next v2 4/7] devlink: allow subports on devlink PCI ports
2019-03-21 16:52 ` Parav Pandit
@ 2019-03-21 17:20 ` Jiri Pirko
2019-03-21 17:34 ` Parav Pandit
0 siblings, 1 reply; 100+ messages in thread
From: Jiri Pirko @ 2019-03-21 17:20 UTC (permalink / raw)
To: Parav Pandit
Cc: Jakub Kicinski, Samudrala, Sridhar, davem, netdev, oss-drivers
Thu, Mar 21, 2019 at 05:52:09PM CET, parav@mellanox.com wrote:
>
>
>> -----Original Message-----
>> From: Jiri Pirko <jiri@resnulli.us>
>> Sent: Thursday, March 21, 2019 11:14 AM
>> To: Parav Pandit <parav@mellanox.com>
>> Cc: Jakub Kicinski <jakub.kicinski@netronome.com>; Samudrala, Sridhar
>> <sridhar.samudrala@intel.com>; davem@davemloft.net;
>> netdev@vger.kernel.org; oss-drivers@netronome.com
>> Subject: Re: [PATCH net-next v2 4/7] devlink: allow subports on devlink PCI
>> ports
>>
>> Thu, Mar 21, 2019 at 04:14:53PM CET, parav@mellanox.com wrote:
>> >
>> >
>> >> -----Original Message-----
>> >> From: Jiri Pirko <jiri@resnulli.us>
>> >> Sent: Thursday, March 21, 2019 3:45 AM
>> >> To: Jakub Kicinski <jakub.kicinski@netronome.com>
>> >> Cc: Parav Pandit <parav@mellanox.com>; Samudrala, Sridhar
>> >> <sridhar.samudrala@intel.com>; davem@davemloft.net;
>> >> netdev@vger.kernel.org; oss-drivers@netronome.com
>> >> Subject: Re: [PATCH net-next v2 4/7] devlink: allow subports on
>> >> devlink PCI ports
>> >>
>> >> Mon, Mar 18, 2019 at 08:16:42PM CET, jakub.kicinski@netronome.com
>> >> wrote:
>> >> >On Mon, 18 Mar 2019 13:11:54 +0100, Jiri Pirko wrote:
>> >> >> >> >2. flavour should not be vf/pf, flavour should be hostport,
>> switchport.
>> >> >> >> >Because switch is flat and agnostic of pf/vf/mdev.
>> >> >> >>
>> >> >> >> Not sure. It's good to have this kind of visibility.
>> >> >> >
>> >> >> >Yes, this subthread honestly makes me go from 60% sure to 95%
>> >> >> >sure we shouldn't do the dual object thing :( Seems like Parav
>> >> >> >is already confused by it and suggests host port can exist
>> >> >> >without switch port :(
>> >> >>
>> >> >> Although I understand your hesitation, the host ports are also
>> >> >> associated with the asic and should be under the devlink instance.
>> >> >> It is just a matter of proper documentation and clear code to
>> >> >> avoid confusions.
>> >> >
>> >> >They are certainly a part and belong to the ASIC, the question in my
>> >> >mind is more along the lines of do we want "one pipe/one port" or is
>> >> >it okay to have multiple software objects of the same kind for those
>> >> >objects.
>> >> >
>> >> >To put it differently - do want a port object for each port of the
>> >> >ASIC or do we want a port object for each netdev..
>> >>
>> >> Perhaps "port" name of the object is misleading. From the beginning,
>> >> I ment to have it for both switch ports and host ports. I admit that
>> >> "host port" is a bit misleading, as it is not really a port of
>> >> eswitch, but the counter part. But if we introduce another object for
>> >> that purpose in devlink (like "partititon"), it would be a lot of duplication
>> I think.
>> >>
>> >> Question is, do we need the "host port"? Can't we just put a relation
>> >> to host netdev in the eswitch port.
>> >>
>> >Can you please explain how does it work for rdma for non sriov use case?
>> >Do we have to create a fake eswitch object?
>>
>> Could you please provide details on "rdma for non sriov use case"?
>>
>There are multiple mdevs on PFs that happen to have link layer as IB and those devlink instances have port that deserved to be configured same way as that of Eth.
Could you please describe it a bit more. There is still an eswitch
through which the traffic is going, isn't it?
^ permalink raw reply [flat|nested] 100+ messages in thread
* Re: [PATCH net-next v2 4/7] devlink: allow subports on devlink PCI ports
2019-03-21 16:50 ` Parav Pandit
@ 2019-03-21 17:23 ` Jiri Pirko
2019-03-21 17:42 ` Parav Pandit
0 siblings, 1 reply; 100+ messages in thread
From: Jiri Pirko @ 2019-03-21 17:23 UTC (permalink / raw)
To: Parav Pandit
Cc: Jakub Kicinski, Samudrala, Sridhar, davem, netdev, oss-drivers
Thu, Mar 21, 2019 at 05:50:37PM CET, parav@mellanox.com wrote:
>
>
>> -----Original Message-----
>> From: Jiri Pirko <jiri@resnulli.us>
>> Sent: Thursday, March 21, 2019 11:16 AM
>> To: Parav Pandit <parav@mellanox.com>
>> Cc: Jakub Kicinski <jakub.kicinski@netronome.com>; Samudrala, Sridhar
>> <sridhar.samudrala@intel.com>; davem@davemloft.net;
>> netdev@vger.kernel.org; oss-drivers@netronome.com
>> Subject: Re: [PATCH net-next v2 4/7] devlink: allow subports on devlink PCI
>> ports
>>
>> Thu, Mar 21, 2019 at 04:03:58PM CET, parav@mellanox.com wrote:
>> >Hi Jiri,
>> >
>> >> -----Original Message-----
>> >> From: Jiri Pirko <jiri@resnulli.us>
>> >> Sent: Thursday, March 21, 2019 4:08 AM
>> >> To: Jakub Kicinski <jakub.kicinski@netronome.com>
>> >> Cc: Parav Pandit <parav@mellanox.com>; Samudrala, Sridhar
>> >> <sridhar.samudrala@intel.com>; davem@davemloft.net;
>> >> netdev@vger.kernel.org; oss-drivers@netronome.com
>> >> Subject: Re: [PATCH net-next v2 4/7] devlink: allow subports on
>> >> devlink PCI ports
>> >>
>> >> Wed, Mar 20, 2019 at 09:22:57PM CET, jakub.kicinski@netronome.com
>> >> wrote:
>> >> >On Wed, 20 Mar 2019 18:24:15 +0000, Parav Pandit wrote:
>> >> >> Hi Jiri, Jakub, Samudrala Sridhar,
>> >> >> > > > > > And physical port in include/uapi/linux/devlink.h also
>> >> >> > > > > > describe that.
>> >> >> > > > >
>> >> >> > > > > By "that" you must mean that the physical is a user facing port.
>> >> >> > > >
>> >> >> > > > Can you please describe the difference between 'PF port' and
>> >> >> > > > 'physical port of include/uapi/linux/devlink.h'? I must have
>> >> >> > > > missed this crisp definition in discussion between you and
>> >> >> > > > Jiri. I am in meantime checking the thread.
>> >> >> > >
>> >> >> > > Perhaps start with the cover letter which includes an ASCII
>> drawing?
>> >> >> > >
>> >> >> > > Using Mellanox nomenclature - PF port is a "representor" for
>> >> >> > > the PF which may be on another Host (SmartNIC or multihost).
>> >> >> > > It's pretty much the same thing as a VF port/"representor".
>> >> >> > >
>> >> >> > Yes. We are aligned here. :-)
>> >> >> > I see your point, where in multi-host scenario, a physical port
>> >> >> > may be 1, but PF ports are 4, because of 4 PFs for 4 hosts.
>> >> >> > (just an example of 4 hosts with their own mac address sharing 1
>> >> >> > physical port).
>> >> >> >
>> >> >> > When there is no multihost and one to one mapping between a PF
>> >> >> > and physical links, there is some overlap between PF port and
>> >> >> > physical port attributes.
>> >> >> > I believe, such overlap is fine as long as we have unique
>> >> >> > indices for the
>> >> ports.
>> >> >> >
>> >> >> > So I am ok to have flavours as
>> physical/cpu/dsa/pf/vf/mdev/switchport.
>> >> >> > (last 4 as new port flavours).
>> >> >> >
>> >> >> > > Physical port is the hole on the panel of the adapter where
>> >> >> > > cable
>> >> goes.
>> >> >>
>> >> >> So my take away from above discussion are:
>> >> >> 1. Following new port flavours should be added
>> >> pci_pf/pci_vf/mdev/switchport.
>> >> >> a. Switchport indicates port on the eswitch. Normally this port
>> >> >> has rep-
>> >> netdev attached to it.
>> >> >
>> >> >I don't understand the "switchport". Surely physical ports are also
>> >> >attached to the eswitch? And one of the main purpose of adding the
>> >> >pci_pf/pci_vf flavours was to generate phys_port_name for the port
>> >> >netdevs.
>> >> >
>> >> >Please don't use the term representor if possible. Representor for
>> >> >most developers describes the way the netdev is implemented in the
>> >> >driver, so for Mellanox and Netronome different ports will be
>> >> >representors and non-representors. That's why I prefer port netdev
>> >> >(attached to eswitch, has switch_id) and host netdev (PF/VF netdev,
>> >> >vNIC, VSI, etc).
>> >> >
>> >> >> b. host side port flavours are pci_pf/pci_vf/mdev which may be
>> >> >> connected to switchport
>> >> >
>> >> >See above, pci_pf/pci_vf are needed for phys_port_name generation.
>> >>
>> >> Yep, that makes sense.
>> >>
>> >>
>> >> >
>> >> >> 2. host side port flavours are not limited to Ethernet, as it is
>> >> >> for devlink's
>> >> port instance.
>> >> >>
>> >> >> 3. Each port is continue to be accessed using unique port index.
>> >> >>
>> >> >> 4. host side ports and switchport are control objects.
>> >> >> a. switch side ports reside where current eswitch object of
>> >> >> devlink instance reside b. for a given VF/PF/mdev such host side
>> >> >> ports may be in hypervisor or VM or both depending on the
>> >> >> privilege
>> >> >>
>> >> >> 5. eth.mac_address, rdma.port_guid can be programmed at host port
>> >> >> flavours by extending as $ devlink port param set...
>> >> >> (similar to devlink dev param set)
>> >> >
>> >> >You can keep restating that's your position, but I have *not*
>> >> >conceded to that.
>> >>
>> >> I'm also not convinced that host dummy ports are good idea to hold
>> these.
>> >>
>> >>
>> >I didn't understand what do you mean my dummy port.
>>
>> It's a port for a VF host port which is not actually in the host but in the vm.
>> Very confusing.
>>
>It is the vf_ctrl flavour. I don't see it any different than rep-netdev.
>rep-netdev is not that confusing to us that represent eswitch vport.
>Why vf_ctrl flavour port that represents otherside of the pipe as you have shown in example?
>Why it that confusing?
Because sometimes it is there only once (PF), sometimes twice (VF) - and
one of these is kind-of zombie.
>
>
>> >Can you explain what is wrong in programming host port params using
>> host_port object?
>> >Few questions are unanswered in my past 2 or 3 emails.
>> >Can you please go through them?
>> >Can you point to some example switch API where you program host params
>> at switch?
>> >
>> >> >
>> >> >> 6. more host port params can be added in future when user need
>> >> >> arise
>> >> >>
>> >> >> 7. rep-netdev continue to be eswitch (switchport) representor at
>> >> >> the
>> >> switch side.
>> >> >> a. Hence rep-netdev cannot be used for programming host port's
>> >> parameters.
>> >> >>
>> >> >> 8. eswitch devlink instance knows when VF/PF/mdev's switchport are
>> >> created/removed.
>> >> >> Hence, those will be created/deleted by eswitch.
>> >> >> Similarly for host port flavours too.
>> >> >>
>> >> >> Does it look fine? Did I miss something?
>> >> >> We would like to progress on incremental patches for item-4 and
>> >> >> any prep work needed to reach to item-4.
^ permalink raw reply [flat|nested] 100+ messages in thread
* RE: [PATCH net-next v2 4/7] devlink: allow subports on devlink PCI ports
2019-03-21 17:20 ` Jiri Pirko
@ 2019-03-21 17:34 ` Parav Pandit
2019-03-22 16:27 ` Jiri Pirko
0 siblings, 1 reply; 100+ messages in thread
From: Parav Pandit @ 2019-03-21 17:34 UTC (permalink / raw)
To: Jiri Pirko; +Cc: Jakub Kicinski, Samudrala, Sridhar, davem, netdev, oss-drivers
> -----Original Message-----
> From: Jiri Pirko <jiri@resnulli.us>
> Sent: Thursday, March 21, 2019 12:21 PM
> To: Parav Pandit <parav@mellanox.com>
> Cc: Jakub Kicinski <jakub.kicinski@netronome.com>; Samudrala, Sridhar
> <sridhar.samudrala@intel.com>; davem@davemloft.net;
> netdev@vger.kernel.org; oss-drivers@netronome.com
> Subject: Re: [PATCH net-next v2 4/7] devlink: allow subports on devlink PCI
> ports
>
> Thu, Mar 21, 2019 at 05:52:09PM CET, parav@mellanox.com wrote:
> >
> >
> >> -----Original Message-----
> >> From: Jiri Pirko <jiri@resnulli.us>
> >> Sent: Thursday, March 21, 2019 11:14 AM
> >> To: Parav Pandit <parav@mellanox.com>
> >> Cc: Jakub Kicinski <jakub.kicinski@netronome.com>; Samudrala, Sridhar
> >> <sridhar.samudrala@intel.com>; davem@davemloft.net;
> >> netdev@vger.kernel.org; oss-drivers@netronome.com
> >> Subject: Re: [PATCH net-next v2 4/7] devlink: allow subports on
> >> devlink PCI ports
> >>
> >> Thu, Mar 21, 2019 at 04:14:53PM CET, parav@mellanox.com wrote:
> >> >
> >> >
> >> >> -----Original Message-----
> >> >> From: Jiri Pirko <jiri@resnulli.us>
> >> >> Sent: Thursday, March 21, 2019 3:45 AM
> >> >> To: Jakub Kicinski <jakub.kicinski@netronome.com>
> >> >> Cc: Parav Pandit <parav@mellanox.com>; Samudrala, Sridhar
> >> >> <sridhar.samudrala@intel.com>; davem@davemloft.net;
> >> >> netdev@vger.kernel.org; oss-drivers@netronome.com
> >> >> Subject: Re: [PATCH net-next v2 4/7] devlink: allow subports on
> >> >> devlink PCI ports
> >> >>
> >> >> Mon, Mar 18, 2019 at 08:16:42PM CET, jakub.kicinski@netronome.com
> >> >> wrote:
> >> >> >On Mon, 18 Mar 2019 13:11:54 +0100, Jiri Pirko wrote:
> >> >> >> >> >2. flavour should not be vf/pf, flavour should be hostport,
> >> switchport.
> >> >> >> >> >Because switch is flat and agnostic of pf/vf/mdev.
> >> >> >> >>
> >> >> >> >> Not sure. It's good to have this kind of visibility.
> >> >> >> >
> >> >> >> >Yes, this subthread honestly makes me go from 60% sure to 95%
> >> >> >> >sure we shouldn't do the dual object thing :( Seems like
> >> >> >> >Parav is already confused by it and suggests host port can
> >> >> >> >exist without switch port :(
> >> >> >>
> >> >> >> Although I understand your hesitation, the host ports are also
> >> >> >> associated with the asic and should be under the devlink instance.
> >> >> >> It is just a matter of proper documentation and clear code to
> >> >> >> avoid confusions.
> >> >> >
> >> >> >They are certainly a part and belong to the ASIC, the question in
> >> >> >my mind is more along the lines of do we want "one pipe/one port"
> >> >> >or is it okay to have multiple software objects of the same kind
> >> >> >for those objects.
> >> >> >
> >> >> >To put it differently - do want a port object for each port of
> >> >> >the ASIC or do we want a port object for each netdev..
> >> >>
> >> >> Perhaps "port" name of the object is misleading. From the
> >> >> beginning, I ment to have it for both switch ports and host ports.
> >> >> I admit that "host port" is a bit misleading, as it is not really
> >> >> a port of eswitch, but the counter part. But if we introduce
> >> >> another object for that purpose in devlink (like "partititon"), it
> >> >> would be a lot of duplication
> >> I think.
> >> >>
> >> >> Question is, do we need the "host port"? Can't we just put a
> >> >> relation to host netdev in the eswitch port.
> >> >>
> >> >Can you please explain how does it work for rdma for non sriov use
> case?
> >> >Do we have to create a fake eswitch object?
> >>
> >> Could you please provide details on "rdma for non sriov use case"?
> >>
> >There are multiple mdevs on PFs that happen to have link layer as IB and
> those devlink instances have port that deserved to be configured same way
> as that of Eth.
>
> Could you please describe it a bit more. There is still an eswitch through
> which the traffic is going, isn't it?
Yes, there is an eswitch but it doesn't have switch side of vports.
It is equivalent to legacy mode.
I hope you are not thinking to create fake eswitch vports. :-)
^ permalink raw reply [flat|nested] 100+ messages in thread
* RE: [PATCH net-next v2 4/7] devlink: allow subports on devlink PCI ports
2019-03-21 17:23 ` Jiri Pirko
@ 2019-03-21 17:42 ` Parav Pandit
2019-03-22 13:32 ` Jiri Pirko
0 siblings, 1 reply; 100+ messages in thread
From: Parav Pandit @ 2019-03-21 17:42 UTC (permalink / raw)
To: Jiri Pirko; +Cc: Jakub Kicinski, Samudrala, Sridhar, davem, netdev, oss-drivers
> -----Original Message-----
> From: Jiri Pirko <jiri@resnulli.us>
> Sent: Thursday, March 21, 2019 12:24 PM
> To: Parav Pandit <parav@mellanox.com>
> Cc: Jakub Kicinski <jakub.kicinski@netronome.com>; Samudrala, Sridhar
> <sridhar.samudrala@intel.com>; davem@davemloft.net;
> netdev@vger.kernel.org; oss-drivers@netronome.com
> Subject: Re: [PATCH net-next v2 4/7] devlink: allow subports on devlink PCI
> ports
>
> Thu, Mar 21, 2019 at 05:50:37PM CET, parav@mellanox.com wrote:
> >
> >
> >> -----Original Message-----
> >> From: Jiri Pirko <jiri@resnulli.us>
> >> Sent: Thursday, March 21, 2019 11:16 AM
> >> To: Parav Pandit <parav@mellanox.com>
> >> Cc: Jakub Kicinski <jakub.kicinski@netronome.com>; Samudrala, Sridhar
> >> <sridhar.samudrala@intel.com>; davem@davemloft.net;
> >> netdev@vger.kernel.org; oss-drivers@netronome.com
> >> Subject: Re: [PATCH net-next v2 4/7] devlink: allow subports on
> >> devlink PCI ports
> >>
> >> Thu, Mar 21, 2019 at 04:03:58PM CET, parav@mellanox.com wrote:
> >> >Hi Jiri,
> >> >
> >> >> -----Original Message-----
> >> >> From: Jiri Pirko <jiri@resnulli.us>
> >> >> Sent: Thursday, March 21, 2019 4:08 AM
> >> >> To: Jakub Kicinski <jakub.kicinski@netronome.com>
> >> >> Cc: Parav Pandit <parav@mellanox.com>; Samudrala, Sridhar
> >> >> <sridhar.samudrala@intel.com>; davem@davemloft.net;
> >> >> netdev@vger.kernel.org; oss-drivers@netronome.com
> >> >> Subject: Re: [PATCH net-next v2 4/7] devlink: allow subports on
> >> >> devlink PCI ports
> >> >>
> >> >> Wed, Mar 20, 2019 at 09:22:57PM CET, jakub.kicinski@netronome.com
> >> >> wrote:
> >> >> >On Wed, 20 Mar 2019 18:24:15 +0000, Parav Pandit wrote:
> >> >> >> Hi Jiri, Jakub, Samudrala Sridhar,
> >> >> >> > > > > > And physical port in include/uapi/linux/devlink.h
> >> >> >> > > > > > also describe that.
> >> >> >> > > > >
> >> >> >> > > > > By "that" you must mean that the physical is a user facing
> port.
> >> >> >> > > >
> >> >> >> > > > Can you please describe the difference between 'PF port'
> >> >> >> > > > and 'physical port of include/uapi/linux/devlink.h'? I
> >> >> >> > > > must have missed this crisp definition in discussion
> >> >> >> > > > between you and Jiri. I am in meantime checking the thread.
> >> >> >> > >
> >> >> >> > > Perhaps start with the cover letter which includes an ASCII
> >> drawing?
> >> >> >> > >
> >> >> >> > > Using Mellanox nomenclature - PF port is a "representor"
> >> >> >> > > for the PF which may be on another Host (SmartNIC or
> multihost).
> >> >> >> > > It's pretty much the same thing as a VF port/"representor".
> >> >> >> > >
> >> >> >> > Yes. We are aligned here. :-) I see your point, where in
> >> >> >> > multi-host scenario, a physical port may be 1, but PF ports
> >> >> >> > are 4, because of 4 PFs for 4 hosts.
> >> >> >> > (just an example of 4 hosts with their own mac address
> >> >> >> > sharing 1 physical port).
> >> >> >> >
> >> >> >> > When there is no multihost and one to one mapping between a
> >> >> >> > PF and physical links, there is some overlap between PF port
> >> >> >> > and physical port attributes.
> >> >> >> > I believe, such overlap is fine as long as we have unique
> >> >> >> > indices for the
> >> >> ports.
> >> >> >> >
> >> >> >> > So I am ok to have flavours as
> >> physical/cpu/dsa/pf/vf/mdev/switchport.
> >> >> >> > (last 4 as new port flavours).
> >> >> >> >
> >> >> >> > > Physical port is the hole on the panel of the adapter where
> >> >> >> > > cable
> >> >> goes.
> >> >> >>
> >> >> >> So my take away from above discussion are:
> >> >> >> 1. Following new port flavours should be added
> >> >> pci_pf/pci_vf/mdev/switchport.
> >> >> >> a. Switchport indicates port on the eswitch. Normally this port
> >> >> >> has rep-
> >> >> netdev attached to it.
> >> >> >
> >> >> >I don't understand the "switchport". Surely physical ports are
> >> >> >also attached to the eswitch? And one of the main purpose of
> >> >> >adding the pci_pf/pci_vf flavours was to generate phys_port_name
> >> >> >for the port netdevs.
> >> >> >
> >> >> >Please don't use the term representor if possible. Representor
> >> >> >for most developers describes the way the netdev is implemented
> >> >> >in the driver, so for Mellanox and Netronome different ports will
> >> >> >be representors and non-representors. That's why I prefer port
> >> >> >netdev (attached to eswitch, has switch_id) and host netdev
> >> >> >(PF/VF netdev, vNIC, VSI, etc).
> >> >> >
> >> >> >> b. host side port flavours are pci_pf/pci_vf/mdev which may be
> >> >> >> connected to switchport
> >> >> >
> >> >> >See above, pci_pf/pci_vf are needed for phys_port_name generation.
> >> >>
> >> >> Yep, that makes sense.
> >> >>
> >> >>
> >> >> >
> >> >> >> 2. host side port flavours are not limited to Ethernet, as it
> >> >> >> is for devlink's
> >> >> port instance.
> >> >> >>
> >> >> >> 3. Each port is continue to be accessed using unique port index.
> >> >> >>
> >> >> >> 4. host side ports and switchport are control objects.
> >> >> >> a. switch side ports reside where current eswitch object of
> >> >> >> devlink instance reside b. for a given VF/PF/mdev such host
> >> >> >> side ports may be in hypervisor or VM or both depending on the
> >> >> >> privilege
> >> >> >>
> >> >> >> 5. eth.mac_address, rdma.port_guid can be programmed at host
> >> >> >> port flavours by extending as $ devlink port param set...
> >> >> >> (similar to devlink dev param set)
> >> >> >
> >> >> >You can keep restating that's your position, but I have *not*
> >> >> >conceded to that.
> >> >>
> >> >> I'm also not convinced that host dummy ports are good idea to hold
> >> these.
> >> >>
> >> >>
> >> >I didn't understand what do you mean my dummy port.
> >>
> >> It's a port for a VF host port which is not actually in the host but in the
> vm.
> >> Very confusing.
> >>
> >It is the vf_ctrl flavour. I don't see it any different than rep-netdev.
> >rep-netdev is not that confusing to us that represent eswitch vport.
> >Why vf_ctrl flavour port that represents otherside of the pipe as you have
> shown in example?
> >Why it that confusing?
>
> Because sometimes it is there only once (PF), sometimes twice (VF) - and one
> of these is kind-of zombie.
>
I gave the example in email that contains description yesterday.
You didn't respond to it.
So repeating here.
Can you please point what looks like zombie below?
$ devlink port show
pci/0000:05:00.0/0 eth netdev repndev_pf0_p0 flavour physical switch_id 00154d130d2f
pci/0000:05:00.0/1 eth netdev repndev_pf0_p1 flavour physical switch_id 00154d130d2f
pci/0000:05:00.0/10001 eth netdev repndev_pf0_vf_1 flavour switchport switch_id 00154d130d2f peer pci/0000:05:00.0/1
pci/0000:05:00.0/10002 eth netdev repndev_pf0_p0_mdev_8000 flavour switchport switch_id 00154d130d2f peer mdev/uuidX/0
pci/0000:05:00.0/1 eth netdev flavour vf_ctrl vf 1
mdev/uuidX/0 eth netdev flavour mdev_ctrl
> >
> >
> >> >Can you explain what is wrong in programming host port params using
> >> host_port object?
> >> >Few questions are unanswered in my past 2 or 3 emails.
> >> >Can you please go through them?
> >> >Can you point to some example switch API where you program host
> >> >params
> >> at switch?
> >> >
> >> >> >
> >> >> >> 6. more host port params can be added in future when user need
> >> >> >> arise
> >> >> >>
> >> >> >> 7. rep-netdev continue to be eswitch (switchport) representor
> >> >> >> at the
> >> >> switch side.
> >> >> >> a. Hence rep-netdev cannot be used for programming host port's
> >> >> parameters.
> >> >> >>
> >> >> >> 8. eswitch devlink instance knows when VF/PF/mdev's switchport
> >> >> >> are
> >> >> created/removed.
> >> >> >> Hence, those will be created/deleted by eswitch.
> >> >> >> Similarly for host port flavours too.
> >> >> >>
> >> >> >> Does it look fine? Did I miss something?
> >> >> >> We would like to progress on incremental patches for item-4 and
> >> >> >> any prep work needed to reach to item-4.
^ permalink raw reply [flat|nested] 100+ messages in thread
* Re: [PATCH net-next v2 4/7] devlink: allow subports on devlink PCI ports
2019-03-21 17:42 ` Parav Pandit
@ 2019-03-22 13:32 ` Jiri Pirko
2019-03-23 0:40 ` Parav Pandit
0 siblings, 1 reply; 100+ messages in thread
From: Jiri Pirko @ 2019-03-22 13:32 UTC (permalink / raw)
To: Parav Pandit
Cc: Jakub Kicinski, Samudrala, Sridhar, davem, netdev, oss-drivers
Thu, Mar 21, 2019 at 06:42:55PM CET, parav@mellanox.com wrote:
>
>
>> -----Original Message-----
>> From: Jiri Pirko <jiri@resnulli.us>
>> Sent: Thursday, March 21, 2019 12:24 PM
>> To: Parav Pandit <parav@mellanox.com>
>> Cc: Jakub Kicinski <jakub.kicinski@netronome.com>; Samudrala, Sridhar
>> <sridhar.samudrala@intel.com>; davem@davemloft.net;
>> netdev@vger.kernel.org; oss-drivers@netronome.com
>> Subject: Re: [PATCH net-next v2 4/7] devlink: allow subports on devlink PCI
>> ports
>>
>> Thu, Mar 21, 2019 at 05:50:37PM CET, parav@mellanox.com wrote:
>> >
>> >
>> >> -----Original Message-----
>> >> From: Jiri Pirko <jiri@resnulli.us>
>> >> Sent: Thursday, March 21, 2019 11:16 AM
>> >> To: Parav Pandit <parav@mellanox.com>
>> >> Cc: Jakub Kicinski <jakub.kicinski@netronome.com>; Samudrala, Sridhar
>> >> <sridhar.samudrala@intel.com>; davem@davemloft.net;
>> >> netdev@vger.kernel.org; oss-drivers@netronome.com
>> >> Subject: Re: [PATCH net-next v2 4/7] devlink: allow subports on
>> >> devlink PCI ports
>> >>
>> >> Thu, Mar 21, 2019 at 04:03:58PM CET, parav@mellanox.com wrote:
>> >> >Hi Jiri,
>> >> >
>> >> >> -----Original Message-----
>> >> >> From: Jiri Pirko <jiri@resnulli.us>
>> >> >> Sent: Thursday, March 21, 2019 4:08 AM
>> >> >> To: Jakub Kicinski <jakub.kicinski@netronome.com>
>> >> >> Cc: Parav Pandit <parav@mellanox.com>; Samudrala, Sridhar
>> >> >> <sridhar.samudrala@intel.com>; davem@davemloft.net;
>> >> >> netdev@vger.kernel.org; oss-drivers@netronome.com
>> >> >> Subject: Re: [PATCH net-next v2 4/7] devlink: allow subports on
>> >> >> devlink PCI ports
>> >> >>
>> >> >> Wed, Mar 20, 2019 at 09:22:57PM CET, jakub.kicinski@netronome.com
>> >> >> wrote:
>> >> >> >On Wed, 20 Mar 2019 18:24:15 +0000, Parav Pandit wrote:
>> >> >> >> Hi Jiri, Jakub, Samudrala Sridhar,
>> >> >> >> > > > > > And physical port in include/uapi/linux/devlink.h
>> >> >> >> > > > > > also describe that.
>> >> >> >> > > > >
>> >> >> >> > > > > By "that" you must mean that the physical is a user facing
>> port.
>> >> >> >> > > >
>> >> >> >> > > > Can you please describe the difference between 'PF port'
>> >> >> >> > > > and 'physical port of include/uapi/linux/devlink.h'? I
>> >> >> >> > > > must have missed this crisp definition in discussion
>> >> >> >> > > > between you and Jiri. I am in meantime checking the thread.
>> >> >> >> > >
>> >> >> >> > > Perhaps start with the cover letter which includes an ASCII
>> >> drawing?
>> >> >> >> > >
>> >> >> >> > > Using Mellanox nomenclature - PF port is a "representor"
>> >> >> >> > > for the PF which may be on another Host (SmartNIC or
>> multihost).
>> >> >> >> > > It's pretty much the same thing as a VF port/"representor".
>> >> >> >> > >
>> >> >> >> > Yes. We are aligned here. :-) I see your point, where in
>> >> >> >> > multi-host scenario, a physical port may be 1, but PF ports
>> >> >> >> > are 4, because of 4 PFs for 4 hosts.
>> >> >> >> > (just an example of 4 hosts with their own mac address
>> >> >> >> > sharing 1 physical port).
>> >> >> >> >
>> >> >> >> > When there is no multihost and one to one mapping between a
>> >> >> >> > PF and physical links, there is some overlap between PF port
>> >> >> >> > and physical port attributes.
>> >> >> >> > I believe, such overlap is fine as long as we have unique
>> >> >> >> > indices for the
>> >> >> ports.
>> >> >> >> >
>> >> >> >> > So I am ok to have flavours as
>> >> physical/cpu/dsa/pf/vf/mdev/switchport.
>> >> >> >> > (last 4 as new port flavours).
>> >> >> >> >
>> >> >> >> > > Physical port is the hole on the panel of the adapter where
>> >> >> >> > > cable
>> >> >> goes.
>> >> >> >>
>> >> >> >> So my take away from above discussion are:
>> >> >> >> 1. Following new port flavours should be added
>> >> >> pci_pf/pci_vf/mdev/switchport.
>> >> >> >> a. Switchport indicates port on the eswitch. Normally this port
>> >> >> >> has rep-
>> >> >> netdev attached to it.
>> >> >> >
>> >> >> >I don't understand the "switchport". Surely physical ports are
>> >> >> >also attached to the eswitch? And one of the main purpose of
>> >> >> >adding the pci_pf/pci_vf flavours was to generate phys_port_name
>> >> >> >for the port netdevs.
>> >> >> >
>> >> >> >Please don't use the term representor if possible. Representor
>> >> >> >for most developers describes the way the netdev is implemented
>> >> >> >in the driver, so for Mellanox and Netronome different ports will
>> >> >> >be representors and non-representors. That's why I prefer port
>> >> >> >netdev (attached to eswitch, has switch_id) and host netdev
>> >> >> >(PF/VF netdev, vNIC, VSI, etc).
>> >> >> >
>> >> >> >> b. host side port flavours are pci_pf/pci_vf/mdev which may be
>> >> >> >> connected to switchport
>> >> >> >
>> >> >> >See above, pci_pf/pci_vf are needed for phys_port_name generation.
>> >> >>
>> >> >> Yep, that makes sense.
>> >> >>
>> >> >>
>> >> >> >
>> >> >> >> 2. host side port flavours are not limited to Ethernet, as it
>> >> >> >> is for devlink's
>> >> >> port instance.
>> >> >> >>
>> >> >> >> 3. Each port is continue to be accessed using unique port index.
>> >> >> >>
>> >> >> >> 4. host side ports and switchport are control objects.
>> >> >> >> a. switch side ports reside where current eswitch object of
>> >> >> >> devlink instance reside b. for a given VF/PF/mdev such host
>> >> >> >> side ports may be in hypervisor or VM or both depending on the
>> >> >> >> privilege
>> >> >> >>
>> >> >> >> 5. eth.mac_address, rdma.port_guid can be programmed at host
>> >> >> >> port flavours by extending as $ devlink port param set...
>> >> >> >> (similar to devlink dev param set)
>> >> >> >
>> >> >> >You can keep restating that's your position, but I have *not*
>> >> >> >conceded to that.
>> >> >>
>> >> >> I'm also not convinced that host dummy ports are good idea to hold
>> >> these.
>> >> >>
>> >> >>
>> >> >I didn't understand what do you mean my dummy port.
>> >>
>> >> It's a port for a VF host port which is not actually in the host but in the
>> vm.
>> >> Very confusing.
>> >>
>> >It is the vf_ctrl flavour. I don't see it any different than rep-netdev.
>> >rep-netdev is not that confusing to us that represent eswitch vport.
>> >Why vf_ctrl flavour port that represents otherside of the pipe as you have
>> shown in example?
>> >Why it that confusing?
>>
>> Because sometimes it is there only once (PF), sometimes twice (VF) - and one
>> of these is kind-of zombie.
>>
>I gave the example in email that contains description yesterday.
>You didn't respond to it.
>So repeating here.
>Can you please point what looks like zombie below?
>
>$ devlink port show
>pci/0000:05:00.0/0 eth netdev repndev_pf0_p0 flavour physical switch_id 00154d130d2f
>pci/0000:05:00.0/1 eth netdev repndev_pf0_p1 flavour physical switch_id 00154d130d2f
>pci/0000:05:00.0/10001 eth netdev repndev_pf0_vf_1 flavour switchport switch_id 00154d130d2f peer pci/0000:05:00.0/1
>pci/0000:05:00.0/10002 eth netdev repndev_pf0_p0_mdev_8000 flavour switchport switch_id 00154d130d2f peer mdev/uuidX/0
>
>pci/0000:05:00.0/1 eth netdev flavour vf_ctrl vf 1
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ this one.
You are missing an actual VF instance.
>mdev/uuidX/0 eth netdev flavour mdev_ctrl
Why "ctrl"?
>
>> >
>> >
>> >> >Can you explain what is wrong in programming host port params using
>> >> host_port object?
>> >> >Few questions are unanswered in my past 2 or 3 emails.
>> >> >Can you please go through them?
>> >> >Can you point to some example switch API where you program host
>> >> >params
>> >> at switch?
>> >> >
>> >> >> >
>> >> >> >> 6. more host port params can be added in future when user need
>> >> >> >> arise
>> >> >> >>
>> >> >> >> 7. rep-netdev continue to be eswitch (switchport) representor
>> >> >> >> at the
>> >> >> switch side.
>> >> >> >> a. Hence rep-netdev cannot be used for programming host port's
>> >> >> parameters.
>> >> >> >>
>> >> >> >> 8. eswitch devlink instance knows when VF/PF/mdev's switchport
>> >> >> >> are
>> >> >> created/removed.
>> >> >> >> Hence, those will be created/deleted by eswitch.
>> >> >> >> Similarly for host port flavours too.
>> >> >> >>
>> >> >> >> Does it look fine? Did I miss something?
>> >> >> >> We would like to progress on incremental patches for item-4 and
>> >> >> >> any prep work needed to reach to item-4.
^ permalink raw reply [flat|nested] 100+ messages in thread
* Re: [PATCH net-next v2 4/7] devlink: allow subports on devlink PCI ports
2019-03-21 17:34 ` Parav Pandit
@ 2019-03-22 16:27 ` Jiri Pirko
2019-03-23 0:37 ` Parav Pandit
0 siblings, 1 reply; 100+ messages in thread
From: Jiri Pirko @ 2019-03-22 16:27 UTC (permalink / raw)
To: Parav Pandit
Cc: Jakub Kicinski, Samudrala, Sridhar, davem, netdev, oss-drivers
Thu, Mar 21, 2019 at 06:34:22PM CET, parav@mellanox.com wrote:
>
>
>> -----Original Message-----
>> From: Jiri Pirko <jiri@resnulli.us>
>> Sent: Thursday, March 21, 2019 12:21 PM
>> To: Parav Pandit <parav@mellanox.com>
>> Cc: Jakub Kicinski <jakub.kicinski@netronome.com>; Samudrala, Sridhar
>> <sridhar.samudrala@intel.com>; davem@davemloft.net;
>> netdev@vger.kernel.org; oss-drivers@netronome.com
>> Subject: Re: [PATCH net-next v2 4/7] devlink: allow subports on devlink PCI
>> ports
>>
>> Thu, Mar 21, 2019 at 05:52:09PM CET, parav@mellanox.com wrote:
>> >
>> >
>> >> -----Original Message-----
>> >> From: Jiri Pirko <jiri@resnulli.us>
>> >> Sent: Thursday, March 21, 2019 11:14 AM
>> >> To: Parav Pandit <parav@mellanox.com>
>> >> Cc: Jakub Kicinski <jakub.kicinski@netronome.com>; Samudrala, Sridhar
>> >> <sridhar.samudrala@intel.com>; davem@davemloft.net;
>> >> netdev@vger.kernel.org; oss-drivers@netronome.com
>> >> Subject: Re: [PATCH net-next v2 4/7] devlink: allow subports on
>> >> devlink PCI ports
>> >>
>> >> Thu, Mar 21, 2019 at 04:14:53PM CET, parav@mellanox.com wrote:
>> >> >
>> >> >
>> >> >> -----Original Message-----
>> >> >> From: Jiri Pirko <jiri@resnulli.us>
>> >> >> Sent: Thursday, March 21, 2019 3:45 AM
>> >> >> To: Jakub Kicinski <jakub.kicinski@netronome.com>
>> >> >> Cc: Parav Pandit <parav@mellanox.com>; Samudrala, Sridhar
>> >> >> <sridhar.samudrala@intel.com>; davem@davemloft.net;
>> >> >> netdev@vger.kernel.org; oss-drivers@netronome.com
>> >> >> Subject: Re: [PATCH net-next v2 4/7] devlink: allow subports on
>> >> >> devlink PCI ports
>> >> >>
>> >> >> Mon, Mar 18, 2019 at 08:16:42PM CET, jakub.kicinski@netronome.com
>> >> >> wrote:
>> >> >> >On Mon, 18 Mar 2019 13:11:54 +0100, Jiri Pirko wrote:
>> >> >> >> >> >2. flavour should not be vf/pf, flavour should be hostport,
>> >> switchport.
>> >> >> >> >> >Because switch is flat and agnostic of pf/vf/mdev.
>> >> >> >> >>
>> >> >> >> >> Not sure. It's good to have this kind of visibility.
>> >> >> >> >
>> >> >> >> >Yes, this subthread honestly makes me go from 60% sure to 95%
>> >> >> >> >sure we shouldn't do the dual object thing :( Seems like
>> >> >> >> >Parav is already confused by it and suggests host port can
>> >> >> >> >exist without switch port :(
>> >> >> >>
>> >> >> >> Although I understand your hesitation, the host ports are also
>> >> >> >> associated with the asic and should be under the devlink instance.
>> >> >> >> It is just a matter of proper documentation and clear code to
>> >> >> >> avoid confusions.
>> >> >> >
>> >> >> >They are certainly a part and belong to the ASIC, the question in
>> >> >> >my mind is more along the lines of do we want "one pipe/one port"
>> >> >> >or is it okay to have multiple software objects of the same kind
>> >> >> >for those objects.
>> >> >> >
>> >> >> >To put it differently - do want a port object for each port of
>> >> >> >the ASIC or do we want a port object for each netdev..
>> >> >>
>> >> >> Perhaps "port" name of the object is misleading. From the
>> >> >> beginning, I ment to have it for both switch ports and host ports.
>> >> >> I admit that "host port" is a bit misleading, as it is not really
>> >> >> a port of eswitch, but the counter part. But if we introduce
>> >> >> another object for that purpose in devlink (like "partititon"), it
>> >> >> would be a lot of duplication
>> >> I think.
>> >> >>
>> >> >> Question is, do we need the "host port"? Can't we just put a
>> >> >> relation to host netdev in the eswitch port.
>> >> >>
>> >> >Can you please explain how does it work for rdma for non sriov use
>> case?
>> >> >Do we have to create a fake eswitch object?
>> >>
>> >> Could you please provide details on "rdma for non sriov use case"?
>> >>
>> >There are multiple mdevs on PFs that happen to have link layer as IB and
>> those devlink instances have port that deserved to be configured same way
>> as that of Eth.
>>
>> Could you please describe it a bit more. There is still an eswitch through
>> which the traffic is going, isn't it?
>Yes, there is an eswitch but it doesn't have switch side of vports.
Why? They should have.
>It is equivalent to legacy mode.
>I hope you are not thinking to create fake eswitch vports. :-)
^ permalink raw reply [flat|nested] 100+ messages in thread
* RE: [PATCH net-next v2 4/7] devlink: allow subports on devlink PCI ports
2019-03-22 16:27 ` Jiri Pirko
@ 2019-03-23 0:37 ` Parav Pandit
0 siblings, 0 replies; 100+ messages in thread
From: Parav Pandit @ 2019-03-23 0:37 UTC (permalink / raw)
To: Jiri Pirko; +Cc: Jakub Kicinski, Samudrala, Sridhar, davem, netdev, oss-drivers
> -----Original Message-----
> From: Jiri Pirko <jiri@resnulli.us>
> Sent: Friday, March 22, 2019 11:28 AM
> To: Parav Pandit <parav@mellanox.com>
> Cc: Jakub Kicinski <jakub.kicinski@netronome.com>; Samudrala, Sridhar
> <sridhar.samudrala@intel.com>; davem@davemloft.net;
> netdev@vger.kernel.org; oss-drivers@netronome.com
> Subject: Re: [PATCH net-next v2 4/7] devlink: allow subports on devlink PCI
> ports
>
> Thu, Mar 21, 2019 at 06:34:22PM CET, parav@mellanox.com wrote:
> >
> >
> >> -----Original Message-----
> >> From: Jiri Pirko <jiri@resnulli.us>
> >> Sent: Thursday, March 21, 2019 12:21 PM
> >> To: Parav Pandit <parav@mellanox.com>
> >> Cc: Jakub Kicinski <jakub.kicinski@netronome.com>; Samudrala, Sridhar
> >> <sridhar.samudrala@intel.com>; davem@davemloft.net;
> >> netdev@vger.kernel.org; oss-drivers@netronome.com
> >> Subject: Re: [PATCH net-next v2 4/7] devlink: allow subports on
> >> devlink PCI ports
> >>
> >> Thu, Mar 21, 2019 at 05:52:09PM CET, parav@mellanox.com wrote:
> >> >
> >> >
> >> >> -----Original Message-----
> >> >> From: Jiri Pirko <jiri@resnulli.us>
> >> >> Sent: Thursday, March 21, 2019 11:14 AM
> >> >> To: Parav Pandit <parav@mellanox.com>
> >> >> Cc: Jakub Kicinski <jakub.kicinski@netronome.com>; Samudrala,
> >> >> Sridhar <sridhar.samudrala@intel.com>; davem@davemloft.net;
> >> >> netdev@vger.kernel.org; oss-drivers@netronome.com
> >> >> Subject: Re: [PATCH net-next v2 4/7] devlink: allow subports on
> >> >> devlink PCI ports
> >> >>
> >> >> Thu, Mar 21, 2019 at 04:14:53PM CET, parav@mellanox.com wrote:
> >> >> >
> >> >> >
> >> >> >> -----Original Message-----
> >> >> >> From: Jiri Pirko <jiri@resnulli.us>
> >> >> >> Sent: Thursday, March 21, 2019 3:45 AM
> >> >> >> To: Jakub Kicinski <jakub.kicinski@netronome.com>
> >> >> >> Cc: Parav Pandit <parav@mellanox.com>; Samudrala, Sridhar
> >> >> >> <sridhar.samudrala@intel.com>; davem@davemloft.net;
> >> >> >> netdev@vger.kernel.org; oss-drivers@netronome.com
> >> >> >> Subject: Re: [PATCH net-next v2 4/7] devlink: allow subports on
> >> >> >> devlink PCI ports
> >> >> >>
> >> >> >> Mon, Mar 18, 2019 at 08:16:42PM CET,
> >> >> >> jakub.kicinski@netronome.com
> >> >> >> wrote:
> >> >> >> >On Mon, 18 Mar 2019 13:11:54 +0100, Jiri Pirko wrote:
> >> >> >> >> >> >2. flavour should not be vf/pf, flavour should be
> >> >> >> >> >> >hostport,
> >> >> switchport.
> >> >> >> >> >> >Because switch is flat and agnostic of pf/vf/mdev.
> >> >> >> >> >>
> >> >> >> >> >> Not sure. It's good to have this kind of visibility.
> >> >> >> >> >
> >> >> >> >> >Yes, this subthread honestly makes me go from 60% sure to
> >> >> >> >> >95% sure we shouldn't do the dual object thing :( Seems
> >> >> >> >> >like Parav is already confused by it and suggests host port
> >> >> >> >> >can exist without switch port :(
> >> >> >> >>
> >> >> >> >> Although I understand your hesitation, the host ports are
> >> >> >> >> also associated with the asic and should be under the devlink
> instance.
> >> >> >> >> It is just a matter of proper documentation and clear code
> >> >> >> >> to avoid confusions.
> >> >> >> >
> >> >> >> >They are certainly a part and belong to the ASIC, the question
> >> >> >> >in my mind is more along the lines of do we want "one pipe/one
> port"
> >> >> >> >or is it okay to have multiple software objects of the same
> >> >> >> >kind for those objects.
> >> >> >> >
> >> >> >> >To put it differently - do want a port object for each port of
> >> >> >> >the ASIC or do we want a port object for each netdev..
> >> >> >>
> >> >> >> Perhaps "port" name of the object is misleading. From the
> >> >> >> beginning, I ment to have it for both switch ports and host ports.
> >> >> >> I admit that "host port" is a bit misleading, as it is not
> >> >> >> really a port of eswitch, but the counter part. But if we
> >> >> >> introduce another object for that purpose in devlink (like
> >> >> >> "partititon"), it would be a lot of duplication
> >> >> I think.
> >> >> >>
> >> >> >> Question is, do we need the "host port"? Can't we just put a
> >> >> >> relation to host netdev in the eswitch port.
> >> >> >>
> >> >> >Can you please explain how does it work for rdma for non sriov
> >> >> >use
> >> case?
> >> >> >Do we have to create a fake eswitch object?
> >> >>
> >> >> Could you please provide details on "rdma for non sriov use case"?
> >> >>
> >> >There are multiple mdevs on PFs that happen to have link layer as IB
> >> >and
> >> those devlink instances have port that deserved to be configured same
> >> way as that of Eth.
> >>
> >> Could you please describe it a bit more. There is still an eswitch
> >> through which the traffic is going, isn't it?
> >Yes, there is an eswitch but it doesn't have switch side of vports.
>
> Why? They should have.
>
It doesn't have.
>
> >It is equivalent to legacy mode.
> >I hope you are not thinking to create fake eswitch vports. :-)
^ permalink raw reply [flat|nested] 100+ messages in thread
* RE: [PATCH net-next v2 4/7] devlink: allow subports on devlink PCI ports
2019-03-22 13:32 ` Jiri Pirko
@ 2019-03-23 0:40 ` Parav Pandit
2019-03-25 20:34 ` Parav Pandit
0 siblings, 1 reply; 100+ messages in thread
From: Parav Pandit @ 2019-03-23 0:40 UTC (permalink / raw)
To: Jiri Pirko; +Cc: Jakub Kicinski, Samudrala, Sridhar, davem, netdev, oss-drivers
> -----Original Message-----
> From: Jiri Pirko <jiri@resnulli.us>
> Sent: Friday, March 22, 2019 8:33 AM
> To: Parav Pandit <parav@mellanox.com>
> Cc: Jakub Kicinski <jakub.kicinski@netronome.com>; Samudrala, Sridhar
> <sridhar.samudrala@intel.com>; davem@davemloft.net;
> netdev@vger.kernel.org; oss-drivers@netronome.com
> Subject: Re: [PATCH net-next v2 4/7] devlink: allow subports on devlink PCI
> ports
>
> Thu, Mar 21, 2019 at 06:42:55PM CET, parav@mellanox.com wrote:
> >
> >
> >> -----Original Message-----
> >> From: Jiri Pirko <jiri@resnulli.us>
> >> Sent: Thursday, March 21, 2019 12:24 PM
> >> To: Parav Pandit <parav@mellanox.com>
> >> Cc: Jakub Kicinski <jakub.kicinski@netronome.com>; Samudrala, Sridhar
> >> <sridhar.samudrala@intel.com>; davem@davemloft.net;
> >> netdev@vger.kernel.org; oss-drivers@netronome.com
> >> Subject: Re: [PATCH net-next v2 4/7] devlink: allow subports on
> >> devlink PCI ports
> >>
> >> Thu, Mar 21, 2019 at 05:50:37PM CET, parav@mellanox.com wrote:
> >> >
> >> >
> >> >> -----Original Message-----
> >> >> From: Jiri Pirko <jiri@resnulli.us>
> >> >> Sent: Thursday, March 21, 2019 11:16 AM
> >> >> To: Parav Pandit <parav@mellanox.com>
> >> >> Cc: Jakub Kicinski <jakub.kicinski@netronome.com>; Samudrala,
> >> >> Sridhar <sridhar.samudrala@intel.com>; davem@davemloft.net;
> >> >> netdev@vger.kernel.org; oss-drivers@netronome.com
> >> >> Subject: Re: [PATCH net-next v2 4/7] devlink: allow subports on
> >> >> devlink PCI ports
> >> >>
> >> >> Thu, Mar 21, 2019 at 04:03:58PM CET, parav@mellanox.com wrote:
> >> >> >Hi Jiri,
> >> >> >
> >> >> >> -----Original Message-----
> >> >> >> From: Jiri Pirko <jiri@resnulli.us>
> >> >> >> Sent: Thursday, March 21, 2019 4:08 AM
> >> >> >> To: Jakub Kicinski <jakub.kicinski@netronome.com>
> >> >> >> Cc: Parav Pandit <parav@mellanox.com>; Samudrala, Sridhar
> >> >> >> <sridhar.samudrala@intel.com>; davem@davemloft.net;
> >> >> >> netdev@vger.kernel.org; oss-drivers@netronome.com
> >> >> >> Subject: Re: [PATCH net-next v2 4/7] devlink: allow subports on
> >> >> >> devlink PCI ports
> >> >> >>
> >> >> >> Wed, Mar 20, 2019 at 09:22:57PM CET,
> >> >> >> jakub.kicinski@netronome.com
> >> >> >> wrote:
> >> >> >> >On Wed, 20 Mar 2019 18:24:15 +0000, Parav Pandit wrote:
> >> >> >> >> Hi Jiri, Jakub, Samudrala Sridhar,
> >> >> >> >> > > > > > And physical port in include/uapi/linux/devlink.h
> >> >> >> >> > > > > > also describe that.
> >> >> >> >> > > > >
> >> >> >> >> > > > > By "that" you must mean that the physical is a user
> >> >> >> >> > > > > facing
> >> port.
> >> >> >> >> > > >
> >> >> >> >> > > > Can you please describe the difference between 'PF port'
> >> >> >> >> > > > and 'physical port of include/uapi/linux/devlink.h'? I
> >> >> >> >> > > > must have missed this crisp definition in discussion
> >> >> >> >> > > > between you and Jiri. I am in meantime checking the
> thread.
> >> >> >> >> > >
> >> >> >> >> > > Perhaps start with the cover letter which includes an
> >> >> >> >> > > ASCII
> >> >> drawing?
> >> >> >> >> > >
> >> >> >> >> > > Using Mellanox nomenclature - PF port is a "representor"
> >> >> >> >> > > for the PF which may be on another Host (SmartNIC or
> >> multihost).
> >> >> >> >> > > It's pretty much the same thing as a VF port/"representor".
> >> >> >> >> > >
> >> >> >> >> > Yes. We are aligned here. :-) I see your point, where in
> >> >> >> >> > multi-host scenario, a physical port may be 1, but PF
> >> >> >> >> > ports are 4, because of 4 PFs for 4 hosts.
> >> >> >> >> > (just an example of 4 hosts with their own mac address
> >> >> >> >> > sharing 1 physical port).
> >> >> >> >> >
> >> >> >> >> > When there is no multihost and one to one mapping between
> >> >> >> >> > a PF and physical links, there is some overlap between PF
> >> >> >> >> > port and physical port attributes.
> >> >> >> >> > I believe, such overlap is fine as long as we have unique
> >> >> >> >> > indices for the
> >> >> >> ports.
> >> >> >> >> >
> >> >> >> >> > So I am ok to have flavours as
> >> >> physical/cpu/dsa/pf/vf/mdev/switchport.
> >> >> >> >> > (last 4 as new port flavours).
> >> >> >> >> >
> >> >> >> >> > > Physical port is the hole on the panel of the adapter
> >> >> >> >> > > where cable
> >> >> >> goes.
> >> >> >> >>
> >> >> >> >> So my take away from above discussion are:
> >> >> >> >> 1. Following new port flavours should be added
> >> >> >> pci_pf/pci_vf/mdev/switchport.
> >> >> >> >> a. Switchport indicates port on the eswitch. Normally this
> >> >> >> >> port has rep-
> >> >> >> netdev attached to it.
> >> >> >> >
> >> >> >> >I don't understand the "switchport". Surely physical ports
> >> >> >> >are also attached to the eswitch? And one of the main purpose
> >> >> >> >of adding the pci_pf/pci_vf flavours was to generate
> >> >> >> >phys_port_name for the port netdevs.
> >> >> >> >
> >> >> >> >Please don't use the term representor if possible.
> >> >> >> >Representor for most developers describes the way the netdev
> >> >> >> >is implemented in the driver, so for Mellanox and Netronome
> >> >> >> >different ports will be representors and non-representors.
> >> >> >> >That's why I prefer port netdev (attached to eswitch, has
> >> >> >> >switch_id) and host netdev (PF/VF netdev, vNIC, VSI, etc).
> >> >> >> >
> >> >> >> >> b. host side port flavours are pci_pf/pci_vf/mdev which may
> >> >> >> >> be connected to switchport
> >> >> >> >
> >> >> >> >See above, pci_pf/pci_vf are needed for phys_port_name
> generation.
> >> >> >>
> >> >> >> Yep, that makes sense.
> >> >> >>
> >> >> >>
> >> >> >> >
> >> >> >> >> 2. host side port flavours are not limited to Ethernet, as
> >> >> >> >> it is for devlink's
> >> >> >> port instance.
> >> >> >> >>
> >> >> >> >> 3. Each port is continue to be accessed using unique port index.
> >> >> >> >>
> >> >> >> >> 4. host side ports and switchport are control objects.
> >> >> >> >> a. switch side ports reside where current eswitch object of
> >> >> >> >> devlink instance reside b. for a given VF/PF/mdev such host
> >> >> >> >> side ports may be in hypervisor or VM or both depending on
> >> >> >> >> the privilege
> >> >> >> >>
> >> >> >> >> 5. eth.mac_address, rdma.port_guid can be programmed at host
> >> >> >> >> port flavours by extending as $ devlink port param set...
> >> >> >> >> (similar to devlink dev param set)
> >> >> >> >
> >> >> >> >You can keep restating that's your position, but I have *not*
> >> >> >> >conceded to that.
> >> >> >>
> >> >> >> I'm also not convinced that host dummy ports are good idea to
> >> >> >> hold
> >> >> these.
> >> >> >>
> >> >> >>
> >> >> >I didn't understand what do you mean my dummy port.
> >> >>
> >> >> It's a port for a VF host port which is not actually in the host
> >> >> but in the
> >> vm.
> >> >> Very confusing.
> >> >>
> >> >It is the vf_ctrl flavour. I don't see it any different than rep-netdev.
> >> >rep-netdev is not that confusing to us that represent eswitch vport.
> >> >Why vf_ctrl flavour port that represents otherside of the pipe as
> >> >you have
> >> shown in example?
> >> >Why it that confusing?
> >>
> >> Because sometimes it is there only once (PF), sometimes twice (VF) -
> >> and one of these is kind-of zombie.
> >>
> >I gave the example in email that contains description yesterday.
> >You didn't respond to it.
> >So repeating here.
> >Can you please point what looks like zombie below?
> >
> >$ devlink port show
> >pci/0000:05:00.0/0 eth netdev repndev_pf0_p0 flavour physical switch_id
> >00154d130d2f
> >pci/0000:05:00.0/1 eth netdev repndev_pf0_p1 flavour physical switch_id
> >00154d130d2f
> >pci/0000:05:00.0/10001 eth netdev repndev_pf0_vf_1 flavour switchport
> >switch_id 00154d130d2f peer pci/0000:05:00.0/1
> >pci/0000:05:00.0/10002 eth netdev repndev_pf0_p0_mdev_8000 flavour
> >switchport switch_id 00154d130d2f peer mdev/uuidX/0
> >
> >pci/0000:05:00.0/1 eth netdev flavour vf_ctrl vf 1
>
> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ this one.
> You are missing an actual VF instance.
>
VF instance is in VM. It is not visible here in Hypervisor. But if you prefer to see it,
It looks like below.
pci/0000:05:01.0/0 eth netdev eth0 flavour vf
> >mdev/uuidX/0 eth netdev flavour mdev_ctrl
> Why "ctrl"?
>
I suffixed it with ctrl to indicate you that it is used for control functionality.
Again, I described in previous email to Jakub' response in lot detail.
> >
> >> >
> >> >
> >> >> >Can you explain what is wrong in programming host port params
> >> >> >using
> >> >> host_port object?
> >> >> >Few questions are unanswered in my past 2 or 3 emails.
> >> >> >Can you please go through them?
> >> >> >Can you point to some example switch API where you program host
> >> >> >params
> >> >> at switch?
> >> >> >
> >> >> >> >
> >> >> >> >> 6. more host port params can be added in future when user
> >> >> >> >> need arise
> >> >> >> >>
> >> >> >> >> 7. rep-netdev continue to be eswitch (switchport)
> >> >> >> >> representor at the
> >> >> >> switch side.
> >> >> >> >> a. Hence rep-netdev cannot be used for programming host
> >> >> >> >> port's
> >> >> >> parameters.
> >> >> >> >>
> >> >> >> >> 8. eswitch devlink instance knows when VF/PF/mdev's
> >> >> >> >> switchport are
> >> >> >> created/removed.
> >> >> >> >> Hence, those will be created/deleted by eswitch.
> >> >> >> >> Similarly for host port flavours too.
> >> >> >> >>
> >> >> >> >> Does it look fine? Did I miss something?
> >> >> >> >> We would like to progress on incremental patches for item-4
> >> >> >> >> and any prep work needed to reach to item-4.
^ permalink raw reply [flat|nested] 100+ messages in thread
* RE: [PATCH net-next v2 4/7] devlink: allow subports on devlink PCI ports
2019-03-23 0:40 ` Parav Pandit
@ 2019-03-25 20:34 ` Parav Pandit
0 siblings, 0 replies; 100+ messages in thread
From: Parav Pandit @ 2019-03-25 20:34 UTC (permalink / raw)
To: Parav Pandit, Jiri Pirko
Cc: Jakub Kicinski, Samudrala, Sridhar, davem, netdev, oss-drivers
> -----Original Message-----
> From: netdev-owner@vger.kernel.org <netdev-owner@vger.kernel.org> On
> Behalf Of Parav Pandit
> Sent: Friday, March 22, 2019 7:40 PM
> To: Jiri Pirko <jiri@resnulli.us>
> Cc: Jakub Kicinski <jakub.kicinski@netronome.com>; Samudrala, Sridhar
> <sridhar.samudrala@intel.com>; davem@davemloft.net;
> netdev@vger.kernel.org; oss-drivers@netronome.com
> Subject: RE: [PATCH net-next v2 4/7] devlink: allow subports on devlink PCI
> ports
>
>
>
> > -----Original Message-----
> > From: Jiri Pirko <jiri@resnulli.us>
> > Sent: Friday, March 22, 2019 8:33 AM
> > To: Parav Pandit <parav@mellanox.com>
> > Cc: Jakub Kicinski <jakub.kicinski@netronome.com>; Samudrala, Sridhar
> > <sridhar.samudrala@intel.com>; davem@davemloft.net;
> > netdev@vger.kernel.org; oss-drivers@netronome.com
> > Subject: Re: [PATCH net-next v2 4/7] devlink: allow subports on
> > devlink PCI ports
> >
> > Thu, Mar 21, 2019 at 06:42:55PM CET, parav@mellanox.com wrote:
> > >
> > >
> > >> -----Original Message-----
> > >> From: Jiri Pirko <jiri@resnulli.us>
> > >> Sent: Thursday, March 21, 2019 12:24 PM
> > >> To: Parav Pandit <parav@mellanox.com>
> > >> Cc: Jakub Kicinski <jakub.kicinski@netronome.com>; Samudrala,
> > >> Sridhar <sridhar.samudrala@intel.com>; davem@davemloft.net;
> > >> netdev@vger.kernel.org; oss-drivers@netronome.com
> > >> Subject: Re: [PATCH net-next v2 4/7] devlink: allow subports on
> > >> devlink PCI ports
> > >>
> > >> Thu, Mar 21, 2019 at 05:50:37PM CET, parav@mellanox.com wrote:
> > >> >
> > >> >
> > >> >> -----Original Message-----
> > >> >> From: Jiri Pirko <jiri@resnulli.us>
> > >> >> Sent: Thursday, March 21, 2019 11:16 AM
> > >> >> To: Parav Pandit <parav@mellanox.com>
> > >> >> Cc: Jakub Kicinski <jakub.kicinski@netronome.com>; Samudrala,
> > >> >> Sridhar <sridhar.samudrala@intel.com>; davem@davemloft.net;
> > >> >> netdev@vger.kernel.org; oss-drivers@netronome.com
> > >> >> Subject: Re: [PATCH net-next v2 4/7] devlink: allow subports on
> > >> >> devlink PCI ports
> > >> >>
> > >> >> Thu, Mar 21, 2019 at 04:03:58PM CET, parav@mellanox.com wrote:
> > >> >> >Hi Jiri,
> > >> >> >
> > >> >> >> -----Original Message-----
> > >> >> >> From: Jiri Pirko <jiri@resnulli.us>
> > >> >> >> Sent: Thursday, March 21, 2019 4:08 AM
> > >> >> >> To: Jakub Kicinski <jakub.kicinski@netronome.com>
> > >> >> >> Cc: Parav Pandit <parav@mellanox.com>; Samudrala, Sridhar
> > >> >> >> <sridhar.samudrala@intel.com>; davem@davemloft.net;
> > >> >> >> netdev@vger.kernel.org; oss-drivers@netronome.com
> > >> >> >> Subject: Re: [PATCH net-next v2 4/7] devlink: allow subports
> > >> >> >> on devlink PCI ports
> > >> >> >>
> > >> >> >> Wed, Mar 20, 2019 at 09:22:57PM CET,
> > >> >> >> jakub.kicinski@netronome.com
> > >> >> >> wrote:
> > >> >> >> >On Wed, 20 Mar 2019 18:24:15 +0000, Parav Pandit wrote:
> > >> >> >> >> Hi Jiri, Jakub, Samudrala Sridhar,
> > >> >> >> >> > > > > > And physical port in
> > >> >> >> >> > > > > > include/uapi/linux/devlink.h also describe that.
> > >> >> >> >> > > > >
> > >> >> >> >> > > > > By "that" you must mean that the physical is a
> > >> >> >> >> > > > > user facing
> > >> port.
> > >> >> >> >> > > >
> > >> >> >> >> > > > Can you please describe the difference between 'PF port'
> > >> >> >> >> > > > and 'physical port of include/uapi/linux/devlink.h'?
> > >> >> >> >> > > > I must have missed this crisp definition in
> > >> >> >> >> > > > discussion between you and Jiri. I am in meantime
> > >> >> >> >> > > > checking the
> > thread.
> > >> >> >> >> > >
> > >> >> >> >> > > Perhaps start with the cover letter which includes an
> > >> >> >> >> > > ASCII
> > >> >> drawing?
> > >> >> >> >> > >
> > >> >> >> >> > > Using Mellanox nomenclature - PF port is a "representor"
> > >> >> >> >> > > for the PF which may be on another Host (SmartNIC or
> > >> multihost).
> > >> >> >> >> > > It's pretty much the same thing as a VF port/"representor".
> > >> >> >> >> > >
> > >> >> >> >> > Yes. We are aligned here. :-) I see your point, where in
> > >> >> >> >> > multi-host scenario, a physical port may be 1, but PF
> > >> >> >> >> > ports are 4, because of 4 PFs for 4 hosts.
> > >> >> >> >> > (just an example of 4 hosts with their own mac address
> > >> >> >> >> > sharing 1 physical port).
> > >> >> >> >> >
> > >> >> >> >> > When there is no multihost and one to one mapping
> > >> >> >> >> > between a PF and physical links, there is some overlap
> > >> >> >> >> > between PF port and physical port attributes.
> > >> >> >> >> > I believe, such overlap is fine as long as we have
> > >> >> >> >> > unique indices for the
> > >> >> >> ports.
> > >> >> >> >> >
> > >> >> >> >> > So I am ok to have flavours as
> > >> >> physical/cpu/dsa/pf/vf/mdev/switchport.
> > >> >> >> >> > (last 4 as new port flavours).
> > >> >> >> >> >
> > >> >> >> >> > > Physical port is the hole on the panel of the adapter
> > >> >> >> >> > > where cable
> > >> >> >> goes.
> > >> >> >> >>
> > >> >> >> >> So my take away from above discussion are:
> > >> >> >> >> 1. Following new port flavours should be added
> > >> >> >> pci_pf/pci_vf/mdev/switchport.
> > >> >> >> >> a. Switchport indicates port on the eswitch. Normally this
> > >> >> >> >> port has rep-
> > >> >> >> netdev attached to it.
> > >> >> >> >
> > >> >> >> >I don't understand the "switchport". Surely physical ports
> > >> >> >> >are also attached to the eswitch? And one of the main
> > >> >> >> >purpose of adding the pci_pf/pci_vf flavours was to generate
> > >> >> >> >phys_port_name for the port netdevs.
> > >> >> >> >
> > >> >> >> >Please don't use the term representor if possible.
> > >> >> >> >Representor for most developers describes the way the netdev
> > >> >> >> >is implemented in the driver, so for Mellanox and Netronome
> > >> >> >> >different ports will be representors and non-representors.
> > >> >> >> >That's why I prefer port netdev (attached to eswitch, has
> > >> >> >> >switch_id) and host netdev (PF/VF netdev, vNIC, VSI, etc).
> > >> >> >> >
> > >> >> >> >> b. host side port flavours are pci_pf/pci_vf/mdev which
> > >> >> >> >> may be connected to switchport
> > >> >> >> >
> > >> >> >> >See above, pci_pf/pci_vf are needed for phys_port_name
> > generation.
> > >> >> >>
> > >> >> >> Yep, that makes sense.
> > >> >> >>
> > >> >> >>
> > >> >> >> >
> > >> >> >> >> 2. host side port flavours are not limited to Ethernet, as
> > >> >> >> >> it is for devlink's
> > >> >> >> port instance.
> > >> >> >> >>
> > >> >> >> >> 3. Each port is continue to be accessed using unique port
> index.
> > >> >> >> >>
> > >> >> >> >> 4. host side ports and switchport are control objects.
> > >> >> >> >> a. switch side ports reside where current eswitch object
> > >> >> >> >> of devlink instance reside b. for a given VF/PF/mdev such
> > >> >> >> >> host side ports may be in hypervisor or VM or both
> > >> >> >> >> depending on the privilege
> > >> >> >> >>
> > >> >> >> >> 5. eth.mac_address, rdma.port_guid can be programmed at
> > >> >> >> >> host port flavours by extending as $ devlink port param set...
> > >> >> >> >> (similar to devlink dev param set)
> > >> >> >> >
> > >> >> >> >You can keep restating that's your position, but I have
> > >> >> >> >*not* conceded to that.
> > >> >> >>
> > >> >> >> I'm also not convinced that host dummy ports are good idea to
> > >> >> >> hold
> > >> >> these.
> > >> >> >>
> > >> >> >>
> > >> >> >I didn't understand what do you mean my dummy port.
> > >> >>
> > >> >> It's a port for a VF host port which is not actually in the host
> > >> >> but in the
> > >> vm.
> > >> >> Very confusing.
> > >> >>
> > >> >It is the vf_ctrl flavour. I don't see it any different than rep-netdev.
> > >> >rep-netdev is not that confusing to us that represent eswitch vport.
> > >> >Why vf_ctrl flavour port that represents otherside of the pipe as
> > >> >you have
> > >> shown in example?
> > >> >Why it that confusing?
> > >>
> > >> Because sometimes it is there only once (PF), sometimes twice (VF)
> > >> - and one of these is kind-of zombie.
> > >>
> > >I gave the example in email that contains description yesterday.
> > >You didn't respond to it.
> > >So repeating here.
> > >Can you please point what looks like zombie below?
> > >
> > >$ devlink port show
> > >pci/0000:05:00.0/0 eth netdev repndev_pf0_p0 flavour physical
> > >switch_id 00154d130d2f
> > >pci/0000:05:00.0/1 eth netdev repndev_pf0_p1 flavour physical
> > >switch_id 00154d130d2f
> > >pci/0000:05:00.0/10001 eth netdev repndev_pf0_vf_1 flavour switchport
> > >switch_id 00154d130d2f peer pci/0000:05:00.0/1
> > >pci/0000:05:00.0/10002 eth netdev repndev_pf0_p0_mdev_8000 flavour
> > >switchport switch_id 00154d130d2f peer mdev/uuidX/0
> > >
> > >pci/0000:05:00.0/1 eth netdev flavour vf_ctrl vf 1
> >
> > ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ this one.
> > You are missing an actual VF instance.
> >
> VF instance is in VM. It is not visible here in Hypervisor. But if you prefer to
> see it, It looks like below.
> pci/0000:05:01.0/0 eth netdev eth0 flavour vf
>
>
> > >mdev/uuidX/0 eth netdev flavour mdev_ctrl
> > Why "ctrl"?
> >
> I suffixed it with ctrl to indicate you that it is used for control functionality.
> Again, I described in previous email to Jakub' response in lot detail.
>
So we had offline discussion. Jiri and Jakub prefers to program hostport's parameters via 'peer' way.
This would require creating unmanaged switch port for rdma.
We concluded to expose host side property via below indirect way on eswitch side port.
pci/0000:05:00.0/1 type eth netdev repndev_pf0_p1 flavour physical switch_id 00154d130d2f
pci/0000:05:00.0/2 type eth netdev repndev_pf0_vf_1 flavour eswitch switch_id 00154d130d2f vf 1 pf 0
pci/0000:05:00.0/4 type eth netdev repndev_pf0_sp_3 flavour eswitch switch_id 00154d130d2f mdev/uuidA/0
+---+ +---+
vf| | | | mdev
+-+-+ +-+-+
physical link <---------+ | |
| | |
| | |
+-+-+ +-+-+ +-+-+
| 1 | | 2 | | 3 |
+--+---+-----+---+------+---+--+
| physical vf pfsub |
| port port port |
| |
| eswitch |
| |
+------------------------------+
Host port parameters such as ether.mac_addr, rdma.node_port_guid and more port internal parameters to be programmed via peer mode.
These ports are created by the driver code and not by a user.
This is very unusual way to program host params via switch.
No solid example provided to support devlink model...
Anyways, I dislike but I agree to Jiri and Jakob suggestion. :-)
Let's move forward this way. I will let future speak for the design choices made...
> > >
> > >> >
> > >> >
> > >> >> >Can you explain what is wrong in programming host port params
> > >> >> >using
> > >> >> host_port object?
> > >> >> >Few questions are unanswered in my past 2 or 3 emails.
> > >> >> >Can you please go through them?
> > >> >> >Can you point to some example switch API where you program host
> > >> >> >params
> > >> >> at switch?
> > >> >> >
> > >> >> >> >
> > >> >> >> >> 6. more host port params can be added in future when user
> > >> >> >> >> need arise
> > >> >> >> >>
> > >> >> >> >> 7. rep-netdev continue to be eswitch (switchport)
> > >> >> >> >> representor at the
> > >> >> >> switch side.
> > >> >> >> >> a. Hence rep-netdev cannot be used for programming host
> > >> >> >> >> port's
> > >> >> >> parameters.
> > >> >> >> >>
> > >> >> >> >> 8. eswitch devlink instance knows when VF/PF/mdev's
> > >> >> >> >> switchport are
> > >> >> >> created/removed.
> > >> >> >> >> Hence, those will be created/deleted by eswitch.
> > >> >> >> >> Similarly for host port flavours too.
> > >> >> >> >>
> > >> >> >> >> Does it look fine? Did I miss something?
> > >> >> >> >> We would like to progress on incremental patches for
> > >> >> >> >> item-4 and any prep work needed to reach to item-4.
^ permalink raw reply [flat|nested] 100+ messages in thread
end of thread, other threads:[~2019-03-25 20:34 UTC | newest]
Thread overview: 100+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2019-03-01 18:04 [PATCH net-next v2 0/7] devlink: expose PF and VF representors as ports Jakub Kicinski
2019-03-01 18:04 ` [PATCH net-next v2 1/7] nfp: split devlink port init from registration Jakub Kicinski
2019-03-01 18:04 ` [PATCH net-next v2 2/7] devlink: add PF and VF port flavours Jakub Kicinski
2019-03-01 18:04 ` [PATCH net-next v2 3/7] nfp: register devlink ports of all reprs Jakub Kicinski
2019-03-02 8:43 ` Jiri Pirko
2019-03-02 19:07 ` Jakub Kicinski
2019-03-04 7:36 ` Jiri Pirko
2019-03-04 23:32 ` Jakub Kicinski
2019-03-01 18:04 ` [PATCH net-next v2 4/7] devlink: allow subports on devlink PCI ports Jakub Kicinski
2019-03-02 9:41 ` Jiri Pirko
2019-03-02 19:48 ` Jakub Kicinski
2019-03-04 7:56 ` Jiri Pirko
2019-03-05 0:33 ` Jakub Kicinski
2019-03-05 11:06 ` Jiri Pirko
2019-03-05 17:15 ` Jakub Kicinski
2019-03-05 19:59 ` Parav Pandit
2019-03-06 12:20 ` Jiri Pirko
2019-03-06 17:56 ` Jakub Kicinski
2019-03-07 3:56 ` Parav Pandit
2019-03-07 9:48 ` Jiri Pirko
2019-03-08 2:52 ` Jakub Kicinski
2019-03-08 14:54 ` Jiri Pirko
2019-03-08 19:09 ` Jakub Kicinski
2019-03-11 8:52 ` Jiri Pirko
2019-03-12 2:10 ` Jakub Kicinski
2019-03-12 14:02 ` Jiri Pirko
2019-03-12 20:56 ` Jakub Kicinski
2019-03-13 6:07 ` Jiri Pirko
2019-03-13 16:17 ` Jakub Kicinski
2019-03-13 16:22 ` Jiri Pirko
2019-03-13 16:55 ` Jakub Kicinski
2019-03-14 7:38 ` Jiri Pirko
2019-03-14 22:09 ` Jakub Kicinski
2019-03-14 22:35 ` Parav Pandit
2019-03-14 23:39 ` Jakub Kicinski
2019-03-15 1:28 ` Parav Pandit
2019-03-15 1:31 ` Parav Pandit
2019-03-15 2:15 ` Samudrala, Sridhar
2019-03-15 2:40 ` Parav Pandit
[not found] ` <ae938b4f-5fa9-3c33-8ae6-eab2d3d9f1ec@intel.com>
2019-03-15 15:32 ` Parav Pandit
2019-03-15 20:08 ` Jiri Pirko
2019-03-15 20:44 ` Jakub Kicinski
2019-03-15 22:12 ` Parav Pandit
2019-03-16 1:16 ` Jakub Kicinski
2019-03-18 15:43 ` Parav Pandit
2019-03-18 19:29 ` Jakub Kicinski
2019-03-18 12:11 ` Jiri Pirko
2019-03-18 19:16 ` Jakub Kicinski
2019-03-21 8:45 ` Jiri Pirko
2019-03-21 15:14 ` Parav Pandit
2019-03-21 16:14 ` Jiri Pirko
2019-03-21 16:52 ` Parav Pandit
2019-03-21 17:20 ` Jiri Pirko
2019-03-21 17:34 ` Parav Pandit
2019-03-22 16:27 ` Jiri Pirko
2019-03-23 0:37 ` Parav Pandit
2019-03-15 21:59 ` Parav Pandit
2019-03-18 12:21 ` Jiri Pirko
2019-03-18 15:56 ` Parav Pandit
2019-03-18 16:22 ` Parav Pandit
2019-03-18 19:36 ` Jakub Kicinski
2019-03-18 19:44 ` Parav Pandit
2019-03-18 19:59 ` Jakub Kicinski
2019-03-18 20:35 ` Parav Pandit
2019-03-18 21:29 ` Jakub Kicinski
2019-03-18 22:11 ` Parav Pandit
2019-03-20 18:24 ` Parav Pandit
2019-03-20 20:22 ` Jakub Kicinski
2019-03-20 23:39 ` Parav Pandit
2019-03-21 9:08 ` Jiri Pirko
2019-03-21 15:03 ` Parav Pandit
2019-03-21 16:16 ` Jiri Pirko
2019-03-21 16:50 ` Parav Pandit
2019-03-21 17:23 ` Jiri Pirko
2019-03-21 17:42 ` Parav Pandit
2019-03-22 13:32 ` Jiri Pirko
2019-03-23 0:40 ` Parav Pandit
2019-03-25 20:34 ` Parav Pandit
2019-03-18 19:19 ` Jakub Kicinski
2019-03-18 19:38 ` Parav Pandit
2019-03-21 9:09 ` Jiri Pirko
2019-03-15 7:00 ` Jiri Pirko
[not found] ` <7227d58e-ac58-d549-b921-ca0a0dd3f4b0@intel.com>
2019-03-13 7:37 ` Jiri Pirko
2019-03-13 16:03 ` Samudrala, Sridhar
2019-03-13 16:24 ` Jiri Pirko
2019-03-04 11:19 ` Jiri Pirko
2019-03-05 0:40 ` Jakub Kicinski
2019-03-05 11:07 ` Jiri Pirko
2019-03-04 11:08 ` Jiri Pirko
2019-03-05 0:51 ` Jakub Kicinski
2019-03-05 11:09 ` Jiri Pirko
2019-03-01 18:04 ` [PATCH net-next v2 5/7] nfp: switch to devlink_port_get_phys_port_name() Jakub Kicinski
2019-03-01 18:04 ` [PATCH net-next v2 6/7] devlink: introduce port's peer netdevs Jakub Kicinski
2019-03-01 18:04 ` [PATCH net-next v2 7/7] nfp: expose PF " Jakub Kicinski
2019-03-02 10:13 ` [PATCH net-next v2 0/7] devlink: expose PF and VF representors as ports Jiri Pirko
2019-03-02 19:49 ` [oss-drivers] " Jakub Kicinski
2019-03-04 5:12 ` Parav Pandit
2019-03-04 18:22 ` David Miller
2019-03-20 20:25 ` Jakub Kicinski
2019-03-21 9:11 ` Jiri Pirko
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.