[PATCH net-next RFC v4 00/15] Add devlink reload action and

[PATCH net-next RFC v4 00/15] Add devlink reload action and

[Moshe Shemesh]

Introduce new options on devlink reload API to enable the user to select
the reload action required and contrains limits on these actions that he
may want to ensure. Complete support for reload actions in mlx5.
The following reload actions are supported:
  driver_reinit: driver entities re-initialization, applying devlink-param
                 and devlink-resource values.
  fw_activate: firmware activate.

The uAPI is backward compatible, if the reload action option is omitted
from the reload command, the driver reinit action will be used.
Note that when required to do firmware activation some drivers may need
to reload the driver. On the other hand some drivers may need to reset
the firmware to reinitialize the driver entities. Therefore, the devlink
reload command returns the actions which were actually performed.

By default reload actions are not limited and driver implementation may
include reset or downtime as needed to perform the actions.
However, if limit_level is selected, the driver should perform only if
it can do it while keeping the limit level constrains.
Reload action limit level added:
  no_reset: No reset allowed, no down time allowed, no link flap and no
            configuration is lost.

Each driver which supports devlink reload command should expose the
reload actions and limit levels supported.

Add reload action stats to hold the history per reload action per limit
level. For example, the number of times fw_activate has been done on
this device since the driver module was added or if the firmware
activation was done with or without reset.

Patch 1-2 add the new API reload action and reload action limit level
          option to devlink reload.
Patch 3 adds reload actions stats.
Patch 4 exposes the reload actions stats on devlink dev get.
Patches 5-10 add support on mlx5 for devlink reload action fw_activate
            and handle the firmware reset events.
Patches 11-12 add devlink enable remote dev reset parameter and use it
             in mlx5.
Patches 13-14 mlx5 add devlink reload action limit level no_reset
              support for fw_activate reload action.
Patch 14 adds documentation file devlink-reload.rst 

Moshe Shemesh (15):
  devlink: Add reload action option to devlink reload command
  devlink: Add reload action limit level
  devlink: Add reload action stats
  devlink: Add reload actions stats to dev get
  net/mlx5: Add functions to set/query MFRL register
  net/mlx5: Set cap for pci sync for fw update event
  net/mlx5: Handle sync reset request event
  net/mlx5: Handle sync reset now event
  net/mlx5: Handle sync reset abort event
  net/mlx5: Add support for devlink reload action fw activate
  devlink: Add enable_remote_dev_reset generic parameter
  net/mlx5: Add devlink param enable_remote_dev_reset support
  net/mlx5: Add support for fw live patch event
  net/mlx5: Add support for devlink reload action limit level no reset
  devlink: Add Documentation/networking/devlink/devlink-reload.rst

 .../networking/devlink/devlink-params.rst     |   6 +
 .../networking/devlink/devlink-reload.rst     |  80 +++
 Documentation/networking/devlink/index.rst    |   1 +
 drivers/net/ethernet/mellanox/mlx4/main.c     |  17 +-
 .../net/ethernet/mellanox/mlx5/core/Makefile  |   2 +-
 .../net/ethernet/mellanox/mlx5/core/devlink.c | 124 ++++-
 .../mellanox/mlx5/core/diag/fw_tracer.c       |  31 ++
 .../mellanox/mlx5/core/diag/fw_tracer.h       |   1 +
 .../ethernet/mellanox/mlx5/core/fw_reset.c    | 454 ++++++++++++++++++
 .../ethernet/mellanox/mlx5/core/fw_reset.h    |  19 +
 .../net/ethernet/mellanox/mlx5/core/health.c  |  35 +-
 .../net/ethernet/mellanox/mlx5/core/main.c    |  13 +
 .../ethernet/mellanox/mlx5/core/mlx5_core.h   |   2 +
 drivers/net/ethernet/mellanox/mlxsw/core.c    |  28 +-
 drivers/net/netdevsim/dev.c                   |  18 +-
 include/linux/mlx5/device.h                   |   1 +
 include/linux/mlx5/driver.h                   |   4 +
 include/net/devlink.h                         |  20 +-
 include/uapi/linux/devlink.h                  |  39 ++
 net/core/devlink.c                            | 243 +++++++++-
 20 files changed, 1089 insertions(+), 49 deletions(-)
 create mode 100644 Documentation/networking/devlink/devlink-reload.rst
 create mode 100644 drivers/net/ethernet/mellanox/mlx5/core/fw_reset.c
 create mode 100644 drivers/net/ethernet/mellanox/mlx5/core/fw_reset.h

-- 
2.17.1

[PATCH net-next RFC v4 01/15] devlink: Add reload action option to devlink reload command

[Moshe Shemesh]

Add devlink reload action to allow the user to request a specific reload
action. The action parameter is optional, if not specified then devlink
driver re-init action is used (backward compatible).
Note that when required to do firmware activation some drivers may need
to reload the driver. On the other hand some drivers may need to reset
the firmware to reinitialize the driver entities. Therefore, the devlink
reload command returns the actions which were actually performed.
Reload actions supported are:
driver_reinit: driver entities re-initialization, applying devlink-param
               and devlink-resource values.
fw_activate: firmware activate.

command examples:
$devlink dev reload pci/0000:82:00.0 action driver_reinit
reload_actions_performed:
  driver_reinit

$devlink dev reload pci/0000:82:00.0 action fw_activate
reload_actions_performed:
  driver_reinit fw_activate

Signed-off-by: Moshe Shemesh <moshe@mellanox.com>
---
v3 -> v4:
- Removed fw_activate_no_reset as an action (next patch adds limit
  levels instead).
- Renamed actions_done to actions_performed
v2 -> v3:
- Replace fw_live_patch action by fw_activate_no_reset
- Devlink reload returns the actions done over netlink reply
v1 -> v2:
- Instead of reload levels driver,fw_reset,fw_live_patch have reload
  actions driver_reinit,fw_activate,fw_live_patch
- Remove driver default level, the action driver_reinit is the default
  action for all drivers
---
 drivers/net/ethernet/mellanox/mlx4/main.c     |  14 ++-
 .../net/ethernet/mellanox/mlx5/core/devlink.c |  15 ++-
 drivers/net/ethernet/mellanox/mlxsw/core.c    |  25 ++--
 drivers/net/netdevsim/dev.c                   |  16 ++-
 include/net/devlink.h                         |   7 +-
 include/uapi/linux/devlink.h                  |  19 +++
 net/core/devlink.c                            | 111 +++++++++++++++++-
 7 files changed, 180 insertions(+), 27 deletions(-)

diff --git a/drivers/net/ethernet/mellanox/mlx4/main.c b/drivers/net/ethernet/mellanox/mlx4/main.c
index 70cf24ba71e4..aadf1676a0ed 100644
--- a/drivers/net/ethernet/mellanox/mlx4/main.c
+++ b/drivers/net/ethernet/mellanox/mlx4/main.c
@@ -3946,6 +3946,7 @@ static int mlx4_restart_one_up(struct pci_dev *pdev, bool reload,
 			       struct devlink *devlink);
 
 static int mlx4_devlink_reload_down(struct devlink *devlink, bool netns_change,
+				    enum devlink_reload_action action,
 				    struct netlink_ext_ack *extack)
 {
 	struct mlx4_priv *priv = devlink_priv(devlink);
@@ -3962,8 +3963,8 @@ static int mlx4_devlink_reload_down(struct devlink *devlink, bool netns_change,
 	return 0;
 }
 
-static int mlx4_devlink_reload_up(struct devlink *devlink,
-				  struct netlink_ext_ack *extack)
+static int mlx4_devlink_reload_up(struct devlink *devlink, enum devlink_reload_action action,
+				  struct netlink_ext_ack *extack, unsigned long *actions_performed)
 {
 	struct mlx4_priv *priv = devlink_priv(devlink);
 	struct mlx4_dev *dev = &priv->dev;
@@ -3971,15 +3972,20 @@ static int mlx4_devlink_reload_up(struct devlink *devlink,
 	int err;
 
 	err = mlx4_restart_one_up(persist->pdev, true, devlink);
-	if (err)
+	if (err) {
 		mlx4_err(persist->dev, "mlx4_restart_one_up failed, ret=%d\n",
 			 err);
+		return err;
+	}
+	if (actions_performed)
+		*actions_performed = BIT(DEVLINK_RELOAD_ACTION_DRIVER_REINIT);
 
-	return err;
+	return 0;
 }
 
 static const struct devlink_ops mlx4_devlink_ops = {
 	.port_type_set	= mlx4_devlink_port_type_set,
+	.supported_reload_actions = BIT(DEVLINK_RELOAD_ACTION_DRIVER_REINIT),
 	.reload_down	= mlx4_devlink_reload_down,
 	.reload_up	= mlx4_devlink_reload_up,
 };
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/devlink.c b/drivers/net/ethernet/mellanox/mlx5/core/devlink.c
index c709e9a385f6..9cd6b6c884e3 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/devlink.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/devlink.c
@@ -89,6 +89,7 @@ mlx5_devlink_info_get(struct devlink *devlink, struct devlink_info_req *req,
 }
 
 static int mlx5_devlink_reload_down(struct devlink *devlink, bool netns_change,
+				    enum devlink_reload_action action,
 				    struct netlink_ext_ack *extack)
 {
 	struct mlx5_core_dev *dev = devlink_priv(devlink);
@@ -97,12 +98,19 @@ static int mlx5_devlink_reload_down(struct devlink *devlink, bool netns_change,
 	return 0;
 }
 
-static int mlx5_devlink_reload_up(struct devlink *devlink,
-				  struct netlink_ext_ack *extack)
+static int mlx5_devlink_reload_up(struct devlink *devlink, enum devlink_reload_action action,
+				  struct netlink_ext_ack *extack, unsigned long *actions_performed)
 {
 	struct mlx5_core_dev *dev = devlink_priv(devlink);
+	int err;
 
-	return mlx5_load_one(dev, false);
+	err = mlx5_load_one(dev, false);
+	if (err)
+		return err;
+	if (actions_performed)
+		*actions_performed = BIT(DEVLINK_RELOAD_ACTION_DRIVER_REINIT);
+
+	return 0;
 }
 
 static const struct devlink_ops mlx5_devlink_ops = {
@@ -118,6 +126,7 @@ static const struct devlink_ops mlx5_devlink_ops = {
 #endif
 	.flash_update = mlx5_devlink_flash_update,
 	.info_get = mlx5_devlink_info_get,
+	.supported_reload_actions = BIT(DEVLINK_RELOAD_ACTION_DRIVER_REINIT),
 	.reload_down = mlx5_devlink_reload_down,
 	.reload_up = mlx5_devlink_reload_up,
 };
diff --git a/drivers/net/ethernet/mellanox/mlxsw/core.c b/drivers/net/ethernet/mellanox/mlxsw/core.c
index ec45a03140d7..c0a32f685b85 100644
--- a/drivers/net/ethernet/mellanox/mlxsw/core.c
+++ b/drivers/net/ethernet/mellanox/mlxsw/core.c
@@ -1113,7 +1113,7 @@ mlxsw_devlink_info_get(struct devlink *devlink, struct devlink_info_req *req,
 
 static int
 mlxsw_devlink_core_bus_device_reload_down(struct devlink *devlink,
-					  bool netns_change,
+					  bool netns_change, enum devlink_reload_action action,
 					  struct netlink_ext_ack *extack)
 {
 	struct mlxsw_core *mlxsw_core = devlink_priv(devlink);
@@ -1126,15 +1126,24 @@ mlxsw_devlink_core_bus_device_reload_down(struct devlink *devlink,
 }
 
 static int
-mlxsw_devlink_core_bus_device_reload_up(struct devlink *devlink,
-					struct netlink_ext_ack *extack)
+mlxsw_devlink_core_bus_device_reload_up(struct devlink *devlink, enum devlink_reload_action action,
+					struct netlink_ext_ack *extack,
+					unsigned long *actions_performed)
 {
 	struct mlxsw_core *mlxsw_core = devlink_priv(devlink);
+	int err;
 
-	return mlxsw_core_bus_device_register(mlxsw_core->bus_info,
-					      mlxsw_core->bus,
-					      mlxsw_core->bus_priv, true,
-					      devlink, extack);
+	err = mlxsw_core_bus_device_register(mlxsw_core->bus_info,
+					     mlxsw_core->bus,
+					     mlxsw_core->bus_priv, true,
+					     devlink, extack);
+	if (err)
+		return err;
+	if (actions_performed)
+		*actions_performed = BIT(DEVLINK_RELOAD_ACTION_DRIVER_REINIT) |
+				     BIT(DEVLINK_RELOAD_ACTION_FW_ACTIVATE);
+
+	return 0;
 }
 
 static int mlxsw_devlink_flash_update(struct devlink *devlink,
@@ -1268,6 +1277,8 @@ mlxsw_devlink_trap_policer_counter_get(struct devlink *devlink,
 }
 
 static const struct devlink_ops mlxsw_devlink_ops = {
+	.supported_reload_actions	= BIT(DEVLINK_RELOAD_ACTION_DRIVER_REINIT) |
+					  BIT(DEVLINK_RELOAD_ACTION_FW_ACTIVATE),
 	.reload_down		= mlxsw_devlink_core_bus_device_reload_down,
 	.reload_up		= mlxsw_devlink_core_bus_device_reload_up,
 	.port_type_set			= mlxsw_devlink_port_type_set,
diff --git a/drivers/net/netdevsim/dev.c b/drivers/net/netdevsim/dev.c
index 32f339fedb21..f0919fa0cd8b 100644
--- a/drivers/net/netdevsim/dev.c
+++ b/drivers/net/netdevsim/dev.c
@@ -697,7 +697,7 @@ static int nsim_dev_reload_create(struct nsim_dev *nsim_dev,
 static void nsim_dev_reload_destroy(struct nsim_dev *nsim_dev);
 
 static int nsim_dev_reload_down(struct devlink *devlink, bool netns_change,
-				struct netlink_ext_ack *extack)
+				enum devlink_reload_action action, struct netlink_ext_ack *extack)
 {
 	struct nsim_dev *nsim_dev = devlink_priv(devlink);
 
@@ -713,10 +713,11 @@ static int nsim_dev_reload_down(struct devlink *devlink, bool netns_change,
 	return 0;
 }
 
-static int nsim_dev_reload_up(struct devlink *devlink,
-			      struct netlink_ext_ack *extack)
+static int nsim_dev_reload_up(struct devlink *devlink, enum devlink_reload_action action,
+			      struct netlink_ext_ack *extack, unsigned long *actions_performed)
 {
 	struct nsim_dev *nsim_dev = devlink_priv(devlink);
+	int err;
 
 	if (nsim_dev->fail_reload) {
 		/* For testing purposes, user set debugfs fail_reload
@@ -726,7 +727,13 @@ static int nsim_dev_reload_up(struct devlink *devlink,
 		return -EINVAL;
 	}
 
-	return nsim_dev_reload_create(nsim_dev, extack);
+	err = nsim_dev_reload_create(nsim_dev, extack);
+	if (err)
+		return err;
+	if (actions_performed)
+		*actions_performed = BIT(DEVLINK_RELOAD_ACTION_DRIVER_REINIT);
+
+	return 0;
 }
 
 static int nsim_dev_info_get(struct devlink *devlink,
@@ -875,6 +882,7 @@ nsim_dev_devlink_trap_policer_counter_get(struct devlink *devlink,
 }
 
 static const struct devlink_ops nsim_dev_devlink_ops = {
+	.supported_reload_actions = BIT(DEVLINK_RELOAD_ACTION_DRIVER_REINIT),
 	.reload_down = nsim_dev_reload_down,
 	.reload_up = nsim_dev_reload_up,
 	.info_get = nsim_dev_info_get,
diff --git a/include/net/devlink.h b/include/net/devlink.h
index eaec0a8cc5ef..b09db891db04 100644
--- a/include/net/devlink.h
+++ b/include/net/devlink.h
@@ -1011,10 +1011,11 @@ enum devlink_trap_group_generic_id {
 	}
 
 struct devlink_ops {
+	unsigned long supported_reload_actions;
 	int (*reload_down)(struct devlink *devlink, bool netns_change,
-			   struct netlink_ext_ack *extack);
-	int (*reload_up)(struct devlink *devlink,
-			 struct netlink_ext_ack *extack);
+			   enum devlink_reload_action action, struct netlink_ext_ack *extack);
+	int (*reload_up)(struct devlink *devlink, enum devlink_reload_action action,
+			 struct netlink_ext_ack *extack, unsigned long *actions_performed);
 	int (*port_type_set)(struct devlink_port *devlink_port,
 			     enum devlink_port_type port_type);
 	int (*port_split)(struct devlink *devlink, unsigned int port_index,
diff --git a/include/uapi/linux/devlink.h b/include/uapi/linux/devlink.h
index 40d35145c879..a6f64db0bdf3 100644
--- a/include/uapi/linux/devlink.h
+++ b/include/uapi/linux/devlink.h
@@ -272,6 +272,21 @@ enum {
 	DEVLINK_ATTR_TRAP_METADATA_TYPE_FA_COOKIE,
 };
 
+/**
+ * enum devlink_reload_action - Reload action.
+ * @DEVLINK_RELOAD_ACTION_DRIVER_REINIT: Driver entities re-instantiation.
+ * @DEVLINK_RELOAD_ACTION_FW_ACTIVATE: FW activate.
+ */
+enum devlink_reload_action {
+	DEVLINK_RELOAD_ACTION_UNSPEC,
+	DEVLINK_RELOAD_ACTION_DRIVER_REINIT,
+	DEVLINK_RELOAD_ACTION_FW_ACTIVATE,
+
+	/* Add new reload actions above */
+	__DEVLINK_RELOAD_ACTION_MAX,
+	DEVLINK_RELOAD_ACTION_MAX = __DEVLINK_RELOAD_ACTION_MAX - 1
+};
+
 enum devlink_attr {
 	/* don't change the order or add anything between, this is ABI! */
 	DEVLINK_ATTR_UNSPEC,
@@ -460,6 +475,10 @@ enum devlink_attr {
 
 	DEVLINK_ATTR_PORT_EXTERNAL,		/* u8 */
 	DEVLINK_ATTR_PORT_CONTROLLER_NUMBER,	/* u32 */
+
+	DEVLINK_ATTR_RELOAD_ACTION,		/* u8 */
+	DEVLINK_ATTR_RELOAD_ACTIONS_PERFORMED,	/* nested */
+
 	/* add new attributes above here, update the policy in devlink.c */
 
 	__DEVLINK_ATTR_MAX,
diff --git a/net/core/devlink.c b/net/core/devlink.c
index 19037f114307..f4be1e1bf864 100644
--- a/net/core/devlink.c
+++ b/net/core/devlink.c
@@ -462,6 +462,12 @@ static int devlink_nl_put_handle(struct sk_buff *msg, struct devlink *devlink)
 	return 0;
 }
 
+static bool
+devlink_reload_action_is_supported(struct devlink *devlink, enum devlink_reload_action action)
+{
+	return test_bit(action, &devlink->ops->supported_reload_actions);
+}
+
 static int devlink_nl_fill(struct sk_buff *msg, struct devlink *devlink,
 			   enum devlink_command cmd, u32 portid,
 			   u32 seq, int flags)
@@ -2969,29 +2975,72 @@ bool devlink_is_reload_failed(const struct devlink *devlink)
 EXPORT_SYMBOL_GPL(devlink_is_reload_failed);
 
 static int devlink_reload(struct devlink *devlink, struct net *dest_net,
-			  struct netlink_ext_ack *extack)
+			  enum devlink_reload_action action, struct netlink_ext_ack *extack,
+			  unsigned long *actions_performed)
 {
 	int err;
 
 	if (!devlink->reload_enabled)
 		return -EOPNOTSUPP;
 
-	err = devlink->ops->reload_down(devlink, !!dest_net, extack);
+	err = devlink->ops->reload_down(devlink, !!dest_net, action, extack);
 	if (err)
 		return err;
 
 	if (dest_net && !net_eq(dest_net, devlink_net(devlink)))
 		devlink_reload_netns_change(devlink, dest_net);
 
-	err = devlink->ops->reload_up(devlink, extack);
+	err = devlink->ops->reload_up(devlink, action, extack, actions_performed);
 	devlink_reload_failed_set(devlink, !!err);
 	return err;
 }
 
+static int
+devlink_nl_reload_actions_performed_fill(struct sk_buff *msg,
+					 struct devlink *devlink,
+					 unsigned long actions_performed,
+					 enum devlink_command cmd, u32 portid,
+					 u32 seq, int flags)
+{
+	struct nlattr *actions_performed_attr;
+	void *hdr;
+	int i;
+
+	hdr = genlmsg_put(msg, portid, seq, &devlink_nl_family, flags, cmd);
+	if (!hdr)
+		return -EMSGSIZE;
+
+	if (devlink_nl_put_handle(msg, devlink))
+		goto genlmsg_cancel;
+
+	actions_performed_attr = nla_nest_start(msg, DEVLINK_ATTR_RELOAD_ACTIONS_PERFORMED);
+	if (!actions_performed_attr)
+		goto genlmsg_cancel;
+
+	for (i = 0; i <= DEVLINK_RELOAD_ACTION_MAX; i++) {
+		if (!test_bit(i, &actions_performed))
+			continue;
+		if (nla_put_u8(msg, DEVLINK_ATTR_RELOAD_ACTION, i))
+			goto actions_performed_nest_cancel;
+	}
+	nla_nest_end(msg, actions_performed_attr);
+	genlmsg_end(msg, hdr);
+	return 0;
+
+actions_performed_nest_cancel:
+	nla_nest_cancel(msg, actions_performed_attr);
+genlmsg_cancel:
+	genlmsg_cancel(msg, hdr);
+	return -EMSGSIZE;
+}
+
 static int devlink_nl_cmd_reload(struct sk_buff *skb, struct genl_info *info)
 {
 	struct devlink *devlink = info->user_ptr[0];
+	enum devlink_reload_action action;
+	unsigned long actions_performed;
 	struct net *dest_net = NULL;
+	struct sk_buff *msg;
 	int err;
 
 	if (!devlink_reload_supported(devlink))
@@ -3011,12 +3060,41 @@ static int devlink_nl_cmd_reload(struct sk_buff *skb, struct genl_info *info)
 			return PTR_ERR(dest_net);
 	}
 
-	err = devlink_reload(devlink, dest_net, info->extack);
+	if (info->attrs[DEVLINK_ATTR_RELOAD_ACTION])
+		action = nla_get_u8(info->attrs[DEVLINK_ATTR_RELOAD_ACTION]);
+	else
+		action = DEVLINK_RELOAD_ACTION_DRIVER_REINIT;
+
+	if (action == DEVLINK_RELOAD_ACTION_UNSPEC || action > DEVLINK_RELOAD_ACTION_MAX) {
+		NL_SET_ERR_MSG_MOD(info->extack, "Invalid reload action");
+		return -EINVAL;
+	} else if (!devlink_reload_action_is_supported(devlink, action)) {
+		NL_SET_ERR_MSG_MOD(info->extack, "Requested reload action is not supported");
+		return -EOPNOTSUPP;
+	}
+
+	err = devlink_reload(devlink, dest_net, action, info->extack, &actions_performed);
 
 	if (dest_net)
 		put_net(dest_net);
 
-	return err;
+	if (err)
+		return err;
+
+	WARN_ON(!actions_performed);
+	msg = nlmsg_new(NLMSG_DEFAULT_SIZE, GFP_KERNEL);
+	if (!msg)
+		return -ENOMEM;
+
+	err = devlink_nl_reload_actions_performed_fill(msg, devlink, actions_performed,
+						       DEVLINK_CMD_RELOAD, info->snd_portid,
+						       info->snd_seq, 0);
+	if (err) {
+		nlmsg_free(msg);
+		return err;
+	}
+
+	return genlmsg_reply(msg, info);
 }
 
 static int devlink_nl_flash_update_fill(struct sk_buff *msg,
@@ -7047,6 +7125,7 @@ static const struct nla_policy devlink_nl_policy[DEVLINK_ATTR_MAX + 1] = {
 	[DEVLINK_ATTR_TRAP_POLICER_RATE] = { .type = NLA_U64 },
 	[DEVLINK_ATTR_TRAP_POLICER_BURST] = { .type = NLA_U64 },
 	[DEVLINK_ATTR_PORT_FUNCTION] = { .type = NLA_NESTED },
+	[DEVLINK_ATTR_RELOAD_ACTION] = { .type = NLA_U8 },
 };
 
 static const struct genl_ops devlink_nl_ops[] = {
@@ -7372,6 +7451,20 @@ static struct genl_family devlink_nl_family __ro_after_init = {
 	.n_mcgrps	= ARRAY_SIZE(devlink_nl_mcgrps),
 };
 
+static int devlink_reload_actions_verify(struct devlink *devlink)
+{
+	const struct devlink_ops *ops;
+
+	if (!devlink_reload_supported(devlink))
+		return 0;
+
+	ops = devlink->ops;
+	if (WARN_ON(ops->supported_reload_actions >= BIT(__DEVLINK_RELOAD_ACTION_MAX) ||
+		    ops->supported_reload_actions <= BIT(DEVLINK_RELOAD_ACTION_UNSPEC)))
+		return -EINVAL;
+	return 0;
+}
+
 /**
  *	devlink_alloc - Allocate new devlink instance resources
  *
@@ -7392,6 +7485,11 @@ struct devlink *devlink_alloc(const struct devlink_ops *ops, size_t priv_size)
 	if (!devlink)
 		return NULL;
 	devlink->ops = ops;
+	if (devlink_reload_actions_verify(devlink)) {
+		kfree(devlink);
+		return NULL;
+	}
+
 	xa_init_flags(&devlink->snapshot_ids, XA_FLAGS_ALLOC);
 	__devlink_net_set(devlink, &init_net);
 	INIT_LIST_HEAD(&devlink->port_list);
@@ -9657,7 +9755,8 @@ static void __net_exit devlink_pernet_pre_exit(struct net *net)
 		if (net_eq(devlink_net(devlink), net)) {
 			if (WARN_ON(!devlink_reload_supported(devlink)))
 				continue;
-			err = devlink_reload(devlink, &init_net, NULL);
+			err = devlink_reload(devlink, &init_net,
+					     DEVLINK_RELOAD_ACTION_DRIVER_REINIT, NULL, NULL);
 			if (err && err != -EOPNOTSUPP)
 				pr_warn("Failed to reload devlink instance into init_net\n");
 		}
-- 
2.17.1

[PATCH net-next RFC v4 02/15] devlink: Add reload action limit level

[Moshe Shemesh]

Add reload action limit level to demand restrictions on actions.
Reload action limit levels supported:
none (default): No constrains on actions. Driver implementation may
                include reset or downtime as needed to perform the
                actions.
no_reset: No reset allowed, no down time allowed, no link flap and no
          configuration is lost.

The no_reset limit level will have usecase in this patchset to
implement restricted fw_activate on mlx5.

Signed-off-by: Moshe Shemesh <moshe@mellanox.com>
---
v3 -> v4:
- New patch
---
 drivers/net/ethernet/mellanox/mlx4/main.c     |  3 ++
 .../net/ethernet/mellanox/mlx5/core/devlink.c |  3 ++
 drivers/net/ethernet/mellanox/mlxsw/core.c    |  3 ++
 drivers/net/netdevsim/dev.c                   |  6 ++-
 include/net/devlink.h                         |  6 ++-
 include/uapi/linux/devlink.h                  | 17 ++++++++
 net/core/devlink.c                            | 40 ++++++++++++++++---
 7 files changed, 70 insertions(+), 8 deletions(-)

diff --git a/drivers/net/ethernet/mellanox/mlx4/main.c b/drivers/net/ethernet/mellanox/mlx4/main.c
index aadf1676a0ed..681b4507b2ba 100644
--- a/drivers/net/ethernet/mellanox/mlx4/main.c
+++ b/drivers/net/ethernet/mellanox/mlx4/main.c
@@ -3947,6 +3947,7 @@ static int mlx4_restart_one_up(struct pci_dev *pdev, bool reload,
 
 static int mlx4_devlink_reload_down(struct devlink *devlink, bool netns_change,
 				    enum devlink_reload_action action,
+				    enum devlink_reload_action_limit_level limit_level,
 				    struct netlink_ext_ack *extack)
 {
 	struct mlx4_priv *priv = devlink_priv(devlink);
@@ -3964,6 +3965,7 @@ static int mlx4_devlink_reload_down(struct devlink *devlink, bool netns_change,
 }
 
 static int mlx4_devlink_reload_up(struct devlink *devlink, enum devlink_reload_action action,
+				  enum devlink_reload_action_limit_level limit_level,
 				  struct netlink_ext_ack *extack, unsigned long *actions_performed)
 {
 	struct mlx4_priv *priv = devlink_priv(devlink);
@@ -3986,6 +3988,7 @@ static int mlx4_devlink_reload_up(struct devlink *devlink, enum devlink_reload_a
 static const struct devlink_ops mlx4_devlink_ops = {
 	.port_type_set	= mlx4_devlink_port_type_set,
 	.supported_reload_actions = BIT(DEVLINK_RELOAD_ACTION_DRIVER_REINIT),
+	.supported_reload_action_limit_levels = BIT(DEVLINK_RELOAD_ACTION_LIMIT_LEVEL_NONE),
 	.reload_down	= mlx4_devlink_reload_down,
 	.reload_up	= mlx4_devlink_reload_up,
 };
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/devlink.c b/drivers/net/ethernet/mellanox/mlx5/core/devlink.c
index 9cd6b6c884e3..f6b29deaf02e 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/devlink.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/devlink.c
@@ -90,6 +90,7 @@ mlx5_devlink_info_get(struct devlink *devlink, struct devlink_info_req *req,
 
 static int mlx5_devlink_reload_down(struct devlink *devlink, bool netns_change,
 				    enum devlink_reload_action action,
+				    enum devlink_reload_action_limit_level limit_level,
 				    struct netlink_ext_ack *extack)
 {
 	struct mlx5_core_dev *dev = devlink_priv(devlink);
@@ -99,6 +100,7 @@ static int mlx5_devlink_reload_down(struct devlink *devlink, bool netns_change,
 }
 
 static int mlx5_devlink_reload_up(struct devlink *devlink, enum devlink_reload_action action,
+				  enum devlink_reload_action_limit_level limit_level,
 				  struct netlink_ext_ack *extack, unsigned long *actions_performed)
 {
 	struct mlx5_core_dev *dev = devlink_priv(devlink);
@@ -127,6 +129,7 @@ static const struct devlink_ops mlx5_devlink_ops = {
 	.flash_update = mlx5_devlink_flash_update,
 	.info_get = mlx5_devlink_info_get,
 	.supported_reload_actions = BIT(DEVLINK_RELOAD_ACTION_DRIVER_REINIT),
+	.supported_reload_action_limit_levels = BIT(DEVLINK_RELOAD_ACTION_LIMIT_LEVEL_NONE),
 	.reload_down = mlx5_devlink_reload_down,
 	.reload_up = mlx5_devlink_reload_up,
 };
diff --git a/drivers/net/ethernet/mellanox/mlxsw/core.c b/drivers/net/ethernet/mellanox/mlxsw/core.c
index c0a32f685b85..e2b21e5878f3 100644
--- a/drivers/net/ethernet/mellanox/mlxsw/core.c
+++ b/drivers/net/ethernet/mellanox/mlxsw/core.c
@@ -1114,6 +1114,7 @@ mlxsw_devlink_info_get(struct devlink *devlink, struct devlink_info_req *req,
 static int
 mlxsw_devlink_core_bus_device_reload_down(struct devlink *devlink,
 					  bool netns_change, enum devlink_reload_action action,
+					  enum devlink_reload_action_limit_level limit_level,
 					  struct netlink_ext_ack *extack)
 {
 	struct mlxsw_core *mlxsw_core = devlink_priv(devlink);
@@ -1127,6 +1128,7 @@ mlxsw_devlink_core_bus_device_reload_down(struct devlink *devlink,
 
 static int
 mlxsw_devlink_core_bus_device_reload_up(struct devlink *devlink, enum devlink_reload_action action,
+					enum devlink_reload_action_limit_level limit_level,
 					struct netlink_ext_ack *extack,
 					unsigned long *actions_performed)
 {
@@ -1279,6 +1281,7 @@ mlxsw_devlink_trap_policer_counter_get(struct devlink *devlink,
 static const struct devlink_ops mlxsw_devlink_ops = {
 	.supported_reload_actions	= BIT(DEVLINK_RELOAD_ACTION_DRIVER_REINIT) |
 					  BIT(DEVLINK_RELOAD_ACTION_FW_ACTIVATE),
+	.supported_reload_action_limit_levels = BIT(DEVLINK_RELOAD_ACTION_LIMIT_LEVEL_NONE),
 	.reload_down		= mlxsw_devlink_core_bus_device_reload_down,
 	.reload_up		= mlxsw_devlink_core_bus_device_reload_up,
 	.port_type_set			= mlxsw_devlink_port_type_set,
diff --git a/drivers/net/netdevsim/dev.c b/drivers/net/netdevsim/dev.c
index f0919fa0cd8b..12fbaf56f722 100644
--- a/drivers/net/netdevsim/dev.c
+++ b/drivers/net/netdevsim/dev.c
@@ -697,7 +697,9 @@ static int nsim_dev_reload_create(struct nsim_dev *nsim_dev,
 static void nsim_dev_reload_destroy(struct nsim_dev *nsim_dev);
 
 static int nsim_dev_reload_down(struct devlink *devlink, bool netns_change,
-				enum devlink_reload_action action, struct netlink_ext_ack *extack)
+				enum devlink_reload_action action,
+				enum devlink_reload_action_limit_level limit_level,
+				struct netlink_ext_ack *extack)
 {
 	struct nsim_dev *nsim_dev = devlink_priv(devlink);
 
@@ -714,6 +716,7 @@ static int nsim_dev_reload_down(struct devlink *devlink, bool netns_change,
 }
 
 static int nsim_dev_reload_up(struct devlink *devlink, enum devlink_reload_action action,
+			      enum devlink_reload_action_limit_level limit_level,
 			      struct netlink_ext_ack *extack, unsigned long *actions_performed)
 {
 	struct nsim_dev *nsim_dev = devlink_priv(devlink);
@@ -883,6 +886,7 @@ nsim_dev_devlink_trap_policer_counter_get(struct devlink *devlink,
 
 static const struct devlink_ops nsim_dev_devlink_ops = {
 	.supported_reload_actions = BIT(DEVLINK_RELOAD_ACTION_DRIVER_REINIT),
+	.supported_reload_action_limit_levels = BIT(DEVLINK_RELOAD_ACTION_LIMIT_LEVEL_NONE),
 	.reload_down = nsim_dev_reload_down,
 	.reload_up = nsim_dev_reload_up,
 	.info_get = nsim_dev_info_get,
diff --git a/include/net/devlink.h b/include/net/devlink.h
index b09db891db04..dddd9ee5b8a9 100644
--- a/include/net/devlink.h
+++ b/include/net/devlink.h
@@ -1012,9 +1012,13 @@ enum devlink_trap_group_generic_id {
 
 struct devlink_ops {
 	unsigned long supported_reload_actions;
+	unsigned long supported_reload_action_limit_levels;
 	int (*reload_down)(struct devlink *devlink, bool netns_change,
-			   enum devlink_reload_action action, struct netlink_ext_ack *extack);
+			   enum devlink_reload_action action,
+			   enum devlink_reload_action_limit_level limit_level,
+			   struct netlink_ext_ack *extack);
 	int (*reload_up)(struct devlink *devlink, enum devlink_reload_action action,
+			 enum devlink_reload_action_limit_level limit_level,
 			 struct netlink_ext_ack *extack, unsigned long *actions_performed);
 	int (*port_type_set)(struct devlink_port *devlink_port,
 			     enum devlink_port_type port_type);
diff --git a/include/uapi/linux/devlink.h b/include/uapi/linux/devlink.h
index a6f64db0bdf3..b19686fd80ff 100644
--- a/include/uapi/linux/devlink.h
+++ b/include/uapi/linux/devlink.h
@@ -287,6 +287,22 @@ enum devlink_reload_action {
 	DEVLINK_RELOAD_ACTION_MAX = __DEVLINK_RELOAD_ACTION_MAX - 1
 };
 
+/**
+ * enum devlink_reload_action_limit_level - Reload action limit level.
+ * @DEVLINK_RELOAD_ACTION_LIMIT_LEVEL_NONE: No constrains on action. Action may include
+ *                                          reset or downtime as needed.
+ * @DEVLINK_RELOAD_ACTION_LIMIT_LEVEL_NO_RESET: No reset allowed, no down time allowed,
+ *                                              no link flap and no configuration is lost.
+ */
+enum devlink_reload_action_limit_level {
+	DEVLINK_RELOAD_ACTION_LIMIT_LEVEL_NONE,
+	DEVLINK_RELOAD_ACTION_LIMIT_LEVEL_NO_RESET,
+
+	/* Add new reload actions limit level above */
+	__DEVLINK_RELOAD_ACTION_LIMIT_LEVEL_MAX,
+	DEVLINK_RELOAD_ACTION_LIMIT_LEVEL_MAX = __DEVLINK_RELOAD_ACTION_LIMIT_LEVEL_MAX - 1
+};
+
 enum devlink_attr {
 	/* don't change the order or add anything between, this is ABI! */
 	DEVLINK_ATTR_UNSPEC,
@@ -478,6 +494,7 @@ enum devlink_attr {
 
 	DEVLINK_ATTR_RELOAD_ACTION,		/* u8 */
 	DEVLINK_ATTR_RELOAD_ACTIONS_PERFORMED,	/* nested */
+	DEVLINK_ATTR_RELOAD_ACTION_LIMIT_LEVEL,	/* u8 */
 
 	/* add new attributes above here, update the policy in devlink.c */
 
diff --git a/net/core/devlink.c b/net/core/devlink.c
index f4be1e1bf864..60aa0c4a3726 100644
--- a/net/core/devlink.c
+++ b/net/core/devlink.c
@@ -468,6 +468,13 @@ devlink_reload_action_is_supported(struct devlink *devlink, enum devlink_reload_
 	return test_bit(action, &devlink->ops->supported_reload_actions);
 }
 
+static bool
+devlink_reload_action_limit_level_is_supported(struct devlink *devlink,
+					       enum devlink_reload_action_limit_level limit_level)
+{
+	return test_bit(limit_level, &devlink->ops->supported_reload_action_limit_levels);
+}
+
 static int devlink_nl_fill(struct sk_buff *msg, struct devlink *devlink,
 			   enum devlink_command cmd, u32 portid,
 			   u32 seq, int flags)
@@ -2975,22 +2982,23 @@ bool devlink_is_reload_failed(const struct devlink *devlink)
 EXPORT_SYMBOL_GPL(devlink_is_reload_failed);
 
 static int devlink_reload(struct devlink *devlink, struct net *dest_net,
-			  enum devlink_reload_action action, struct netlink_ext_ack *extack,
-			  unsigned long *actions_performed)
+			  enum devlink_reload_action action,
+			  enum devlink_reload_action_limit_level limit_level,
+			  struct netlink_ext_ack *extack, unsigned long *actions_performed)
 {
 	int err;
 
 	if (!devlink->reload_enabled)
 		return -EOPNOTSUPP;
 
-	err = devlink->ops->reload_down(devlink, !!dest_net, action, extack);
+	err = devlink->ops->reload_down(devlink, !!dest_net, action, limit_level, extack);
 	if (err)
 		return err;
 
 	if (dest_net && !net_eq(dest_net, devlink_net(devlink)))
 		devlink_reload_netns_change(devlink, dest_net);
 
-	err = devlink->ops->reload_up(devlink, action, extack, actions_performed);
+	err = devlink->ops->reload_up(devlink, action, limit_level, extack, actions_performed);
 	devlink_reload_failed_set(devlink, !!err);
 	return err;
 }
@@ -3036,6 +3044,7 @@ devlink_nl_reload_actions_performed_fill(struct sk_buff *msg,
 
 static int devlink_nl_cmd_reload(struct sk_buff *skb, struct genl_info *info)
 {
+	enum devlink_reload_action_limit_level limit_level;
 	struct devlink *devlink = info->user_ptr[0];
 	enum devlink_reload_action action;
 	unsigned long actions_performed;
@@ -3073,7 +3082,20 @@ static int devlink_nl_cmd_reload(struct sk_buff *skb, struct genl_info *info)
 		return -EOPNOTSUPP;
 	}
 
-	err = devlink_reload(devlink, dest_net, action, info->extack, &actions_performed);
+	if (info->attrs[DEVLINK_ATTR_RELOAD_ACTION_LIMIT_LEVEL])
+		limit_level = nla_get_u8(info->attrs[DEVLINK_ATTR_RELOAD_ACTION_LIMIT_LEVEL]);
+	else
+		limit_level = DEVLINK_RELOAD_ACTION_LIMIT_LEVEL_NONE;
+
+	if (limit_level > DEVLINK_RELOAD_ACTION_LIMIT_LEVEL_MAX) {
+		NL_SET_ERR_MSG_MOD(info->extack, "Invalid limit level");
+		return -EINVAL;
+	} else if (!devlink_reload_action_limit_level_is_supported(devlink, limit_level)) {
+		NL_SET_ERR_MSG_MOD(info->extack, "Requested limit level is not supported");
+		return -EOPNOTSUPP;
+	}
+	err = devlink_reload(devlink, dest_net, action, limit_level, info->extack,
+			     &actions_performed);
 
 	if (dest_net)
 		put_net(dest_net);
@@ -7126,6 +7148,7 @@ static const struct nla_policy devlink_nl_policy[DEVLINK_ATTR_MAX + 1] = {
 	[DEVLINK_ATTR_TRAP_POLICER_BURST] = { .type = NLA_U64 },
 	[DEVLINK_ATTR_PORT_FUNCTION] = { .type = NLA_NESTED },
 	[DEVLINK_ATTR_RELOAD_ACTION] = { .type = NLA_U8 },
+	[DEVLINK_ATTR_RELOAD_ACTION_LIMIT_LEVEL] = { .type = NLA_U8 },
 };
 
 static const struct genl_ops devlink_nl_ops[] = {
@@ -7462,6 +7485,10 @@ static int devlink_reload_actions_verify(struct devlink *devlink)
 	if (WARN_ON(ops->supported_reload_actions >= BIT(__DEVLINK_RELOAD_ACTION_MAX) ||
 		    ops->supported_reload_actions <= BIT(DEVLINK_RELOAD_ACTION_UNSPEC)))
 		return -EINVAL;
+	if (WARN_ON(!ops->supported_reload_action_limit_levels ||
+		    ops->supported_reload_action_limit_levels >=
+		    BIT(__DEVLINK_RELOAD_ACTION_LIMIT_LEVEL_MAX)))
+		return -EINVAL;
 	return 0;
 }
 
@@ -9756,7 +9783,8 @@ static void __net_exit devlink_pernet_pre_exit(struct net *net)
 			if (WARN_ON(!devlink_reload_supported(devlink)))
 				continue;
 			err = devlink_reload(devlink, &init_net,
-					     DEVLINK_RELOAD_ACTION_DRIVER_REINIT, NULL, NULL);
+					     DEVLINK_RELOAD_ACTION_DRIVER_REINIT,
+					     DEVLINK_RELOAD_ACTION_LIMIT_LEVEL_NONE, NULL, NULL);
 			if (err && err != -EOPNOTSUPP)
 				pr_warn("Failed to reload devlink instance into init_net\n");
 		}
-- 
2.17.1

[PATCH net-next RFC v4 03/15] devlink: Add reload action stats

[Moshe Shemesh]

Add reload action stats to hold the history per reload action type and
limit level.
For example, the number of times fw_activate has been performed on this
device since the driver module was added or if the firmware activation
was performed with or without reset.
Add devlink notification on stats update.

The function devlink_reload_actions_implicit_actions_performed() is
exported to enable also drivers update on reload actions performed,
for example in case firmware activation with reset finished
successfully but was initiated by remote host.

Signed-off-by: Moshe Shemesh <moshe@mellanox.com>
---
v3 -> v4:
- Renamed reload_actions_cnts to reload_action_stats
- Add devlink notifications on stats update
- Renamed devlink_reload_actions_implicit_actions_performed() and add
  function comment in code
v2 -> v3:
- New patch
---
 include/net/devlink.h |  7 ++++++
 net/core/devlink.c    | 58 ++++++++++++++++++++++++++++++++++++++++---
 2 files changed, 62 insertions(+), 3 deletions(-)

diff --git a/include/net/devlink.h b/include/net/devlink.h
index dddd9ee5b8a9..b4feb92e0269 100644
--- a/include/net/devlink.h
+++ b/include/net/devlink.h
@@ -20,6 +20,9 @@
 #include <uapi/linux/devlink.h>
 #include <linux/xarray.h>
 
+#define DEVLINK_RELOAD_ACTION_STATS_ARRAY_SIZE \
+	(__DEVLINK_RELOAD_ACTION_LIMIT_LEVEL_MAX * __DEVLINK_RELOAD_ACTION_MAX)
+
 struct devlink_ops;
 
 struct devlink {
@@ -38,6 +41,7 @@ struct devlink {
 	struct list_head trap_policer_list;
 	const struct devlink_ops *ops;
 	struct xarray snapshot_ids;
+	u32 reload_action_stats[DEVLINK_RELOAD_ACTION_STATS_ARRAY_SIZE];
 	struct device *dev;
 	possible_net_t _net;
 	struct mutex lock; /* Serializes access to devlink instance specific objects such as
@@ -1397,6 +1401,9 @@ void
 devlink_health_reporter_recovery_done(struct devlink_health_reporter *reporter);
 
 bool devlink_is_reload_failed(const struct devlink *devlink);
+void devlink_reload_implicit_actions_performed(struct devlink *devlink,
+					       enum devlink_reload_action_limit_level limit_level,
+					       unsigned long actions_performed);
 
 void devlink_flash_update_begin_notify(struct devlink *devlink);
 void devlink_flash_update_end_notify(struct devlink *devlink);
diff --git a/net/core/devlink.c b/net/core/devlink.c
index 60aa0c4a3726..cbf746966913 100644
--- a/net/core/devlink.c
+++ b/net/core/devlink.c
@@ -2981,11 +2981,58 @@ bool devlink_is_reload_failed(const struct devlink *devlink)
 }
 EXPORT_SYMBOL_GPL(devlink_is_reload_failed);
 
+static void
+devlink_reload_action_stats_update(struct devlink *devlink,
+				   enum devlink_reload_action_limit_level limit_level,
+				   unsigned long actions_performed)
+{
+	int stat_idx;
+	int action;
+
+	if (!actions_performed)
+		return;
+
+	if (WARN_ON(limit_level > DEVLINK_RELOAD_ACTION_LIMIT_LEVEL_MAX))
+		return;
+	for (action = 0; action <= DEVLINK_RELOAD_ACTION_MAX; action++) {
+		if (!test_bit(action, &actions_performed))
+			continue;
+		stat_idx = limit_level * __DEVLINK_RELOAD_ACTION_MAX + action;
+		devlink->reload_action_stats[stat_idx]++;
+	}
+	devlink_notify(devlink, DEVLINK_CMD_NEW);
+}
+
+/**
+ *	devlink_reload_implicit_actions_performed - Update devlink on reload actions
+ *	  performed which are not a direct result of devlink reload call.
+ *
+ *	This should be called by a driver after performing reload actions in case it was not
+ *	a result of devlink reload call. For example fw_activate was performed as a result
+ *	of devlink reload triggered fw_activate on another host.
+ *	The motivation for this function is to keep data on reload actions performed on this
+ *	function whether it was done due to direct devlink reload call or not.
+ *
+ *	@devlink: devlink
+ *	@limit_level: reload action limit level
+ *	@actions_performed: bitmask of actions performed
+ */
+void devlink_reload_implicit_actions_performed(struct devlink *devlink,
+					       enum devlink_reload_action_limit_level limit_level,
+					       unsigned long actions_performed)
+{
+	if (!devlink_reload_supported(devlink))
+		return;
+	devlink_reload_action_stats_update(devlink, limit_level, actions_performed);
+}
+EXPORT_SYMBOL_GPL(devlink_reload_implicit_actions_performed);
+
 static int devlink_reload(struct devlink *devlink, struct net *dest_net,
 			  enum devlink_reload_action action,
 			  enum devlink_reload_action_limit_level limit_level,
-			  struct netlink_ext_ack *extack, unsigned long *actions_performed)
+			  struct netlink_ext_ack *extack, unsigned long *actions_performed_out)
 {
+	unsigned long actions_performed;
 	int err;
 
 	if (!devlink->reload_enabled)
@@ -2998,9 +3045,14 @@ static int devlink_reload(struct devlink *devlink, struct net *dest_net,
 	if (dest_net && !net_eq(dest_net, devlink_net(devlink)))
 		devlink_reload_netns_change(devlink, dest_net);
 
-	err = devlink->ops->reload_up(devlink, action, limit_level, extack, actions_performed);
+	err = devlink->ops->reload_up(devlink, action, limit_level, extack, &actions_performed);
 	devlink_reload_failed_set(devlink, !!err);
-	return err;
+	if (err)
+		return err;
+	devlink_reload_action_stats_update(devlink, limit_level, actions_performed);
+	if (actions_performed_out)
+		*actions_performed_out = actions_performed;
+	return 0;
 }
 
 static int
-- 
2.17.1

[PATCH net-next RFC v4 04/15] devlink: Add reload actions stats to dev get

[Moshe Shemesh]

Expose devlink reload actions stats to the user through devlink dev
get command.

Examples:
$ devlink dev show
pci/0000:82:00.0:
  reload_action_stats:
    driver_reinit 2
    fw_activate 1
    driver_reinit_no_reset 0
    fw_activate_no_reset 0
pci/0000:82:00.1:
  reload_action_stats:
    driver_reinit 1
    fw_activate 1
    driver_reinit_no_reset 0
    fw_activate_no_reset 0

$ devlink dev show -jp
{
    "dev": {
        "pci/0000:82:00.0": {
            "reload_action_stats": [ {
                    "driver_reinit": 2
                },{
                    "fw_activate": 1
                },{
                    "driver_reinit_no_reset": 0
                },{
                    "fw_activate_no_reset": 0
                } ]
        },
        "pci/0000:82:00.1": {
            "reload_action_stats": [ {
                    "driver_reinit": 1
                },{
                    "fw_activate": 1
                },{
                    "driver_reinit_no_reset": 0
                },{
                    "fw_activate_no_reset": 0
                } ]
        }
    }
}

Signed-off-by: Moshe Shemesh <moshe@mellanox.com>
---
v3 -> v4:
- Renamed DEVLINK_ATTR_RELOAD_ACTION_CNT to
  DEVLINK_ATTR_RELOAD_ACTION_STAT
- Add stats per action per limit level
v2 -> v3:
- Add reload actions counters instead of supported reload actions
  (reload actions counters are only for supported action so no need for
   both)
v1 -> v2:
- Removed DEVLINK_ATTR_RELOAD_DEFAULT_LEVEL
- Removed DEVLINK_ATTR_RELOAD_LEVELS_INFO
- Have actions instead of levels
---
 include/uapi/linux/devlink.h |  3 +++
 net/core/devlink.c           | 45 ++++++++++++++++++++++++++++++++----
 2 files changed, 43 insertions(+), 5 deletions(-)

diff --git a/include/uapi/linux/devlink.h b/include/uapi/linux/devlink.h
index b19686fd80ff..ac9be467d243 100644
--- a/include/uapi/linux/devlink.h
+++ b/include/uapi/linux/devlink.h
@@ -495,6 +495,9 @@ enum devlink_attr {
 	DEVLINK_ATTR_RELOAD_ACTION,		/* u8 */
 	DEVLINK_ATTR_RELOAD_ACTIONS_PERFORMED,	/* nested */
 	DEVLINK_ATTR_RELOAD_ACTION_LIMIT_LEVEL,	/* u8 */
+	DEVLINK_ATTR_RELOAD_ACTION_STATS,	/* nested */
+	DEVLINK_ATTR_RELOAD_ACTION_STAT,	/* nested */
+	DEVLINK_ATTR_RELOAD_ACTION_STAT_VALUE,	/* u32 */
 
 	/* add new attributes above here, update the policy in devlink.c */
 
diff --git a/net/core/devlink.c b/net/core/devlink.c
index cbf746966913..1063b7a4123a 100644
--- a/net/core/devlink.c
+++ b/net/core/devlink.c
@@ -462,6 +462,11 @@ static int devlink_nl_put_handle(struct sk_buff *msg, struct devlink *devlink)
 	return 0;
 }
 
+static bool devlink_reload_supported(struct devlink *devlink)
+{
+	return devlink->ops->reload_down && devlink->ops->reload_up;
+}
+
 static bool
 devlink_reload_action_is_supported(struct devlink *devlink, enum devlink_reload_action action)
 {
@@ -479,7 +484,9 @@ static int devlink_nl_fill(struct sk_buff *msg, struct devlink *devlink,
 			   enum devlink_command cmd, u32 portid,
 			   u32 seq, int flags)
 {
+	struct nlattr *reload_action_stats, *reload_action_stat;
+	int i, j, stat_idx;
 	void *hdr;
 
 	hdr = genlmsg_put(msg, portid, seq, &devlink_nl_family, flags, cmd);
 	if (!hdr)
@@ -490,9 +497,42 @@ static int devlink_nl_fill(struct sk_buff *msg, struct devlink *devlink,
 	if (nla_put_u8(msg, DEVLINK_ATTR_RELOAD_FAILED, devlink->reload_failed))
 		goto nla_put_failure;
 
+	if (!devlink_reload_supported(devlink))
+		goto out;
+
+	reload_action_stats = nla_nest_start(msg, DEVLINK_ATTR_RELOAD_ACTION_STATS);
+	if (!reload_action_stats)
+		goto nla_put_failure;
+
+	for (j = 0; j <= DEVLINK_RELOAD_ACTION_LIMIT_LEVEL_MAX; j++) {
+		if (!devlink_reload_action_limit_level_is_supported(devlink, j))
+			continue;
+		for (i = 0; i <= DEVLINK_RELOAD_ACTION_MAX; i++) {
+			if (!devlink_reload_action_is_supported(devlink, i))
+				continue;
+			reload_action_stat = nla_nest_start(msg, DEVLINK_ATTR_RELOAD_ACTION_STAT);
+			if (!reload_action_stat)
+				goto reload_action_stats_nest_cancel;
+			if (nla_put_u8(msg, DEVLINK_ATTR_RELOAD_ACTION, i))
+				goto reload_action_stat_nest_cancel;
+			if (nla_put_u8(msg, DEVLINK_ATTR_RELOAD_ACTION_LIMIT_LEVEL, j))
+				goto reload_action_stat_nest_cancel;
+			stat_idx = j * __DEVLINK_RELOAD_ACTION_MAX + i;
+			if (nla_put_u32(msg, DEVLINK_ATTR_RELOAD_ACTION_STAT_VALUE,
+					devlink->reload_action_stats[stat_idx]))
+				goto reload_action_stat_nest_cancel;
+			nla_nest_end(msg, reload_action_stat);
+		}
+		nla_nest_end(msg, reload_action_stats);
+	}
+out:
 	genlmsg_end(msg, hdr);
 	return 0;
 
+reload_action_stat_nest_cancel:
+	nla_nest_cancel(msg, reload_action_stat);
+reload_action_stats_nest_cancel:
+	nla_nest_cancel(msg, reload_action_stats);
 nla_put_failure:
 	genlmsg_cancel(msg, hdr);
 	return -EMSGSIZE;
@@ -2961,11 +3001,6 @@ static void devlink_reload_netns_change(struct devlink *devlink,
 				     DEVLINK_CMD_PARAM_NEW);
 }
 
-static bool devlink_reload_supported(const struct devlink *devlink)
-{
-	return devlink->ops->reload_down && devlink->ops->reload_up;
-}
-
 static void devlink_reload_failed_set(struct devlink *devlink,
 				      bool reload_failed)
 {
-- 
2.17.1

[PATCH net-next RFC v4 05/15] net/mlx5: Add functions to set/query MFRL register

[Moshe Shemesh]

Add functions to query and set the MFRL reset options supported by
firmware.

Signed-off-by: Moshe Shemesh <moshe@mellanox.com>
---
 .../net/ethernet/mellanox/mlx5/core/Makefile  |  2 +-
 .../ethernet/mellanox/mlx5/core/fw_reset.c    | 46 +++++++++++++++++++
 .../ethernet/mellanox/mlx5/core/fw_reset.h    | 13 ++++++
 3 files changed, 60 insertions(+), 1 deletion(-)
 create mode 100644 drivers/net/ethernet/mellanox/mlx5/core/fw_reset.c
 create mode 100644 drivers/net/ethernet/mellanox/mlx5/core/fw_reset.h

diff --git a/drivers/net/ethernet/mellanox/mlx5/core/Makefile b/drivers/net/ethernet/mellanox/mlx5/core/Makefile
index 9826a041e407..380a51747599 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/Makefile
+++ b/drivers/net/ethernet/mellanox/mlx5/core/Makefile
@@ -16,7 +16,7 @@ mlx5_core-y :=	main.o cmd.o debugfs.o fw.o eq.o uar.o pagealloc.o \
 		transobj.o vport.o sriov.o fs_cmd.o fs_core.o pci_irq.o \
 		fs_counters.o rl.o lag.o dev.o events.o wq.o lib/gid.o \
 		lib/devcom.o lib/pci_vsc.o lib/dm.o diag/fs_tracepoint.o \
-		diag/fw_tracer.o diag/crdump.o devlink.o diag/rsc_dump.o
+		diag/fw_tracer.o diag/crdump.o devlink.o diag/rsc_dump.o fw_reset.o
 
 #
 # Netdev basic
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/fw_reset.c b/drivers/net/ethernet/mellanox/mlx5/core/fw_reset.c
new file mode 100644
index 000000000000..76d2cece29ac
--- /dev/null
+++ b/drivers/net/ethernet/mellanox/mlx5/core/fw_reset.c
@@ -0,0 +1,46 @@
+// SPDX-License-Identifier: GPL-2.0 OR Linux-OpenIB
+/* Copyright (c) 2020, Mellanox Technologies inc.  All rights reserved. */
+
+#include "fw_reset.h"
+
+static int mlx5_reg_mfrl_set(struct mlx5_core_dev *dev, u8 reset_level,
+			     u8 reset_type_sel, u8 sync_resp, bool sync_start)
+{
+	u32 out[MLX5_ST_SZ_DW(mfrl_reg)] = {};
+	u32 in[MLX5_ST_SZ_DW(mfrl_reg)] = {};
+
+	MLX5_SET(mfrl_reg, in, reset_level, reset_level);
+	MLX5_SET(mfrl_reg, in, rst_type_sel, reset_type_sel);
+	MLX5_SET(mfrl_reg, in, pci_sync_for_fw_update_resp, sync_resp);
+	MLX5_SET(mfrl_reg, in, pci_sync_for_fw_update_start, sync_start);
+
+	return mlx5_core_access_reg(dev, in, sizeof(in), out, sizeof(out), MLX5_REG_MFRL, 0, 1);
+}
+
+int mlx5_reg_mfrl_query(struct mlx5_core_dev *dev, u8 *reset_level, u8 *reset_type)
+{
+	u32 out[MLX5_ST_SZ_DW(mfrl_reg)] = {};
+	u32 in[MLX5_ST_SZ_DW(mfrl_reg)] = {};
+	int err;
+
+	err = mlx5_core_access_reg(dev, in, sizeof(in), out, sizeof(out), MLX5_REG_MFRL, 0, 0);
+	if (err)
+		return err;
+
+	if (reset_level)
+		*reset_level = MLX5_GET(mfrl_reg, out, reset_level);
+	if (reset_type)
+		*reset_type = MLX5_GET(mfrl_reg, out, reset_type);
+
+	return 0;
+}
+
+int mlx5_fw_set_reset_sync(struct mlx5_core_dev *dev, u8 reset_type_sel)
+{
+	return mlx5_reg_mfrl_set(dev, MLX5_MFRL_REG_RESET_LEVEL3, reset_type_sel, 0, true);
+}
+
+int mlx5_fw_set_live_patch(struct mlx5_core_dev *dev)
+{
+	return mlx5_reg_mfrl_set(dev, MLX5_MFRL_REG_RESET_LEVEL0, 0, 0, false);
+}
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/fw_reset.h b/drivers/net/ethernet/mellanox/mlx5/core/fw_reset.h
new file mode 100644
index 000000000000..1bbd95182ca6
--- /dev/null
+++ b/drivers/net/ethernet/mellanox/mlx5/core/fw_reset.h
@@ -0,0 +1,13 @@
+/* SPDX-License-Identifier: GPL-2.0 OR Linux-OpenIB */
+/* Copyright (c) 2020, Mellanox Technologies inc.  All rights reserved. */
+
+#ifndef __MLX5_FW_RESET_H
+#define __MLX5_FW_RESET_H
+
+#include "mlx5_core.h"
+
+int mlx5_reg_mfrl_query(struct mlx5_core_dev *dev, u8 *reset_level, u8 *reset_type);
+int mlx5_fw_set_reset_sync(struct mlx5_core_dev *dev, u8 reset_type_sel);
+int mlx5_fw_set_live_patch(struct mlx5_core_dev *dev);
+
+#endif
-- 
2.17.1

[PATCH net-next RFC v4 06/15] net/mlx5: Set cap for pci sync for fw update event

[Moshe Shemesh]

Set capability to notify the firmware that this host driver is capable
of handling pci sync for firmware update events.

Signed-off-by: Moshe Shemesh <moshe@mellanox.com>
---
 drivers/net/ethernet/mellanox/mlx5/core/main.c | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/drivers/net/ethernet/mellanox/mlx5/core/main.c b/drivers/net/ethernet/mellanox/mlx5/core/main.c
index ce43e3feccd9..871d28b09f8a 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/main.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/main.c
@@ -548,6 +548,9 @@ static int handle_hca_cap(struct mlx5_core_dev *dev, void *set_ctx)
 	if (MLX5_CAP_GEN_MAX(dev, dct))
 		MLX5_SET(cmd_hca_cap, set_hca_cap, dct, 1);
 
+	if (MLX5_CAP_GEN_MAX(dev, pci_sync_for_fw_update_event))
+		MLX5_SET(cmd_hca_cap, set_hca_cap, pci_sync_for_fw_update_event, 1);
+
 	if (MLX5_CAP_GEN_MAX(dev, num_vhca_ports))
 		MLX5_SET(cmd_hca_cap,
 			 set_hca_cap,
-- 
2.17.1

[PATCH net-next RFC v4 07/15] net/mlx5: Handle sync reset request event

[Moshe Shemesh]

Once the driver gets sync_reset_request from firmware it prepares for the
coming reset and sends acknowledge.
After getting this event the driver expects device reset, either it will
trigger PCI reset on sync_reset_now event or such PCI reset will be
triggered by another PF of the same device. So it moves to reset
requested mode and if it gets PCI reset triggered by the other PF it
detect the reset and reloads.

Signed-off-by: Moshe Shemesh <moshe@mellanox.com>
---
v1 -> v2:
- Moved handling of sync reset recovery from health to fw_reset
---
 .../ethernet/mellanox/mlx5/core/fw_reset.c    | 167 ++++++++++++++++++
 .../ethernet/mellanox/mlx5/core/fw_reset.h    |   3 +
 .../net/ethernet/mellanox/mlx5/core/health.c  |  35 ++--
 .../net/ethernet/mellanox/mlx5/core/main.c    |  10 ++
 .../ethernet/mellanox/mlx5/core/mlx5_core.h   |   2 +
 include/linux/mlx5/driver.h                   |   4 +
 6 files changed, 206 insertions(+), 15 deletions(-)

diff --git a/drivers/net/ethernet/mellanox/mlx5/core/fw_reset.c b/drivers/net/ethernet/mellanox/mlx5/core/fw_reset.c
index 76d2cece29ac..0f224454b4a2 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/fw_reset.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/fw_reset.c
@@ -3,6 +3,20 @@
 
 #include "fw_reset.h"
 
+enum {
+	MLX5_FW_RESET_FLAGS_RESET_REQUESTED,
+};
+
+struct mlx5_fw_reset {
+	struct mlx5_core_dev *dev;
+	struct mlx5_nb nb;
+	struct workqueue_struct *wq;
+	struct work_struct reset_request_work;
+	struct work_struct reset_reload_work;
+	unsigned long reset_flags;
+	struct timer_list timer;
+};
+
 static int mlx5_reg_mfrl_set(struct mlx5_core_dev *dev, u8 reset_level,
 			     u8 reset_type_sel, u8 sync_resp, bool sync_start)
 {
@@ -44,3 +58,156 @@ int mlx5_fw_set_live_patch(struct mlx5_core_dev *dev)
 {
 	return mlx5_reg_mfrl_set(dev, MLX5_MFRL_REG_RESET_LEVEL0, 0, 0, false);
 }
+
+static int mlx5_fw_set_reset_sync_ack(struct mlx5_core_dev *dev)
+{
+	return mlx5_reg_mfrl_set(dev, MLX5_MFRL_REG_RESET_LEVEL3, 0, 1, false);
+}
+
+static void mlx5_sync_reset_reload_work(struct work_struct *work)
+{
+	struct mlx5_fw_reset *fw_reset = container_of(work, struct mlx5_fw_reset,
+						      reset_reload_work);
+	struct mlx5_core_dev *dev = fw_reset->dev;
+
+	mlx5_enter_error_state(dev, true);
+	mlx5_unload_one(dev, false);
+	if (mlx5_health_wait_pci_up(dev)) {
+		mlx5_core_err(dev, "reset reload flow aborted, PCI reads still not working\n");
+		return;
+	}
+	mlx5_load_one(dev, false);
+}
+
+static void mlx5_stop_sync_reset_poll(struct mlx5_core_dev *dev)
+{
+	struct mlx5_fw_reset *fw_reset = dev->priv.fw_reset;
+
+	del_timer(&fw_reset->timer);
+}
+
+static void mlx5_sync_reset_clear_reset_requested(struct mlx5_core_dev *dev, bool poll_health)
+{
+	struct mlx5_fw_reset *fw_reset = dev->priv.fw_reset;
+
+	mlx5_stop_sync_reset_poll(dev);
+	clear_bit(MLX5_FW_RESET_FLAGS_RESET_REQUESTED, &fw_reset->reset_flags);
+	if (poll_health)
+		mlx5_start_health_poll(dev);
+}
+
+#define MLX5_RESET_POLL_INTERVAL	(HZ / 10)
+static void poll_sync_reset(struct timer_list *t)
+{
+	struct mlx5_fw_reset *fw_reset = from_timer(fw_reset, t, timer);
+	struct mlx5_core_dev *dev = fw_reset->dev;
+	u32 fatal_error;
+
+	if (!test_bit(MLX5_FW_RESET_FLAGS_RESET_REQUESTED, &fw_reset->reset_flags))
+		return;
+
+	fatal_error = mlx5_health_check_fatal_sensors(dev);
+
+	if (fatal_error) {
+		mlx5_core_warn(dev, "Got Device Reset\n");
+		mlx5_sync_reset_clear_reset_requested(dev, false);
+		queue_work(fw_reset->wq, &fw_reset->reset_reload_work);
+		return;
+	}
+
+	mod_timer(&fw_reset->timer, round_jiffies(jiffies + MLX5_RESET_POLL_INTERVAL));
+}
+
+static void mlx5_start_sync_reset_poll(struct mlx5_core_dev *dev)
+{
+	struct mlx5_fw_reset *fw_reset = dev->priv.fw_reset;
+
+	timer_setup(&fw_reset->timer, poll_sync_reset, 0);
+	fw_reset->timer.expires = round_jiffies(jiffies + MLX5_RESET_POLL_INTERVAL);
+	add_timer(&fw_reset->timer);
+}
+
+static void mlx5_sync_reset_set_reset_requested(struct mlx5_core_dev *dev)
+{
+	struct mlx5_fw_reset *fw_reset = dev->priv.fw_reset;
+
+	mlx5_stop_health_poll(dev, true);
+	set_bit(MLX5_FW_RESET_FLAGS_RESET_REQUESTED, &fw_reset->reset_flags);
+	mlx5_start_sync_reset_poll(dev);
+}
+
+static void mlx5_sync_reset_request_event(struct work_struct *work)
+{
+	struct mlx5_fw_reset *fw_reset = container_of(work, struct mlx5_fw_reset,
+						      reset_request_work);
+	struct mlx5_core_dev *dev = fw_reset->dev;
+
+	mlx5_sync_reset_set_reset_requested(dev);
+	if (mlx5_fw_set_reset_sync_ack(dev))
+		mlx5_core_warn(dev, "PCI Sync FW Update Reset Ack Failed.\n");
+	else
+		mlx5_core_warn(dev, "PCI Sync FW Update Reset Ack. Device reset is expected.\n");
+}
+
+static void mlx5_sync_reset_events_handle(struct mlx5_fw_reset *fw_reset, struct mlx5_eqe *eqe)
+{
+	struct mlx5_eqe_sync_fw_update *sync_fw_update_eqe;
+	u8 sync_event_rst_type;
+
+	sync_fw_update_eqe = &eqe->data.sync_fw_update;
+	sync_event_rst_type = sync_fw_update_eqe->sync_rst_state & SYNC_RST_STATE_MASK;
+	switch (sync_event_rst_type) {
+	case MLX5_SYNC_RST_STATE_RESET_REQUEST:
+		queue_work(fw_reset->wq, &fw_reset->reset_request_work);
+		break;
+	}
+}
+
+static int fw_reset_event_notifier(struct notifier_block *nb, unsigned long action, void *data)
+{
+	struct mlx5_fw_reset *fw_reset = mlx5_nb_cof(nb, struct mlx5_fw_reset, nb);
+	struct mlx5_eqe *eqe = data;
+
+	switch (eqe->sub_type) {
+	case MLX5_GENERAL_SUBTYPE_PCI_SYNC_FOR_FW_UPDATE_EVENT:
+		mlx5_sync_reset_events_handle(fw_reset, eqe);
+		break;
+	default:
+		return NOTIFY_DONE;
+	}
+
+	return NOTIFY_OK;
+}
+
+int mlx5_fw_reset_events_init(struct mlx5_core_dev *dev)
+{
+	struct mlx5_fw_reset *fw_reset = kzalloc(sizeof(*fw_reset), GFP_KERNEL);
+
+	if (!fw_reset)
+		return -ENOMEM;
+	fw_reset->wq = create_singlethread_workqueue("mlx5_fw_reset_events");
+	if (!fw_reset->wq) {
+		kfree(fw_reset);
+		return -ENOMEM;
+	}
+
+	fw_reset->dev = dev;
+	dev->priv.fw_reset = fw_reset;
+
+	INIT_WORK(&fw_reset->reset_request_work, mlx5_sync_reset_request_event);
+	INIT_WORK(&fw_reset->reset_reload_work, mlx5_sync_reset_reload_work);
+
+	MLX5_NB_INIT(&fw_reset->nb, fw_reset_event_notifier, GENERAL_EVENT);
+	mlx5_eq_notifier_register(dev, &fw_reset->nb);
+
+	return 0;
+}
+
+void mlx5_fw_reset_events_cleanup(struct mlx5_core_dev *dev)
+{
+	struct mlx5_fw_reset *fw_reset = dev->priv.fw_reset;
+
+	mlx5_eq_notifier_unregister(dev, &fw_reset->nb);
+	destroy_workqueue(fw_reset->wq);
+	kvfree(dev->priv.fw_reset);
+}
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/fw_reset.h b/drivers/net/ethernet/mellanox/mlx5/core/fw_reset.h
index 1bbd95182ca6..278f538ea92a 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/fw_reset.h
+++ b/drivers/net/ethernet/mellanox/mlx5/core/fw_reset.h
@@ -10,4 +10,7 @@ int mlx5_reg_mfrl_query(struct mlx5_core_dev *dev, u8 *reset_level, u8 *reset_ty
 int mlx5_fw_set_reset_sync(struct mlx5_core_dev *dev, u8 reset_type_sel);
 int mlx5_fw_set_live_patch(struct mlx5_core_dev *dev);
 
+int mlx5_fw_reset_events_init(struct mlx5_core_dev *dev);
+void mlx5_fw_reset_events_cleanup(struct mlx5_core_dev *dev);
+
 #endif
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/health.c b/drivers/net/ethernet/mellanox/mlx5/core/health.c
index b31f769d2df9..54523bed16cd 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/health.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/health.c
@@ -110,7 +110,7 @@ static bool sensor_fw_synd_rfr(struct mlx5_core_dev *dev)
 	return rfr && synd;
 }
 
-static u32 check_fatal_sensors(struct mlx5_core_dev *dev)
+u32 mlx5_health_check_fatal_sensors(struct mlx5_core_dev *dev)
 {
 	if (sensor_pci_not_working(dev))
 		return MLX5_SENSOR_PCI_COMM_ERR;
@@ -173,7 +173,7 @@ static bool reset_fw_if_needed(struct mlx5_core_dev *dev)
 	 * Check again to avoid a redundant 2nd reset. If the fatal erros was
 	 * PCI related a reset won't help.
 	 */
-	fatal_error = check_fatal_sensors(dev);
+	fatal_error = mlx5_health_check_fatal_sensors(dev);
 	if (fatal_error == MLX5_SENSOR_PCI_COMM_ERR ||
 	    fatal_error == MLX5_SENSOR_NIC_DISABLED ||
 	    fatal_error == MLX5_SENSOR_NIC_SW_RESET) {
@@ -195,7 +195,7 @@ void mlx5_enter_error_state(struct mlx5_core_dev *dev, bool force)
 	bool err_detected = false;
 
 	/* Mark the device as fatal in order to abort FW commands */
-	if ((check_fatal_sensors(dev) || force) &&
+	if ((mlx5_health_check_fatal_sensors(dev) || force) &&
 	    dev->state == MLX5_DEVICE_STATE_UP) {
 		dev->state = MLX5_DEVICE_STATE_INTERNAL_ERROR;
 		err_detected = true;
@@ -208,7 +208,7 @@ void mlx5_enter_error_state(struct mlx5_core_dev *dev, bool force)
 		goto unlock;
 	}
 
-	if (check_fatal_sensors(dev) || force) { /* protected state setting */
+	if (mlx5_health_check_fatal_sensors(dev) || force) { /* protected state setting */
 		dev->state = MLX5_DEVICE_STATE_INTERNAL_ERROR;
 		mlx5_cmd_flush(dev);
 	}
@@ -231,7 +231,7 @@ void mlx5_error_sw_reset(struct mlx5_core_dev *dev)
 
 	mlx5_core_err(dev, "start\n");
 
-	if (check_fatal_sensors(dev) == MLX5_SENSOR_FW_SYND_RFR) {
+	if (mlx5_health_check_fatal_sensors(dev) == MLX5_SENSOR_FW_SYND_RFR) {
 		/* Get cr-dump and reset FW semaphore */
 		lock = lock_sem_sw_reset(dev, true);
 
@@ -308,26 +308,31 @@ static void mlx5_handle_bad_state(struct mlx5_core_dev *dev)
 
 /* How much time to wait until health resetting the driver (in msecs) */
 #define MLX5_RECOVERY_WAIT_MSECS 60000
-static int mlx5_health_try_recover(struct mlx5_core_dev *dev)
+int mlx5_health_wait_pci_up(struct mlx5_core_dev *dev)
 {
 	unsigned long end;
 
-	mlx5_core_warn(dev, "handling bad device here\n");
-	mlx5_handle_bad_state(dev);
 	end = jiffies + msecs_to_jiffies(MLX5_RECOVERY_WAIT_MSECS);
 	while (sensor_pci_not_working(dev)) {
-		if (time_after(jiffies, end)) {
-			mlx5_core_err(dev,
-				      "health recovery flow aborted, PCI reads still not working\n");
-			return -EIO;
-		}
+		if (time_after(jiffies, end))
+			return -ETIMEDOUT;
 		msleep(100);
 	}
+	return 0;
+}
 
+static int mlx5_health_try_recover(struct mlx5_core_dev *dev)
+{
+	mlx5_core_warn(dev, "handling bad device here\n");
+	mlx5_handle_bad_state(dev);
+	if (mlx5_health_wait_pci_up(dev)) {
+		mlx5_core_err(dev, "health recovery flow aborted, PCI reads still not working\n");
+		return -EIO;
+	}
 	mlx5_core_err(dev, "starting health recovery flow\n");
 	mlx5_recover_device(dev);
 	if (!test_bit(MLX5_INTERFACE_STATE_UP, &dev->intf_state) ||
-	    check_fatal_sensors(dev)) {
+	    mlx5_health_check_fatal_sensors(dev)) {
 		mlx5_core_err(dev, "health recovery failed\n");
 		return -EIO;
 	}
@@ -696,7 +701,7 @@ static void poll_health(struct timer_list *t)
 	if (dev->state == MLX5_DEVICE_STATE_INTERNAL_ERROR)
 		goto out;
 
-	fatal_error = check_fatal_sensors(dev);
+	fatal_error = mlx5_health_check_fatal_sensors(dev);
 
 	if (fatal_error && !health->fatal_error) {
 		mlx5_core_err(dev, "Fatal error %u detected\n", fatal_error);
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/main.c b/drivers/net/ethernet/mellanox/mlx5/core/main.c
index 871d28b09f8a..e833db424f11 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/main.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/main.c
@@ -57,6 +57,7 @@
 #include "lib/mpfs.h"
 #include "eswitch.h"
 #include "devlink.h"
+#include "fw_reset.h"
 #include "lib/mlx5.h"
 #include "fpga/core.h"
 #include "fpga/ipsec.h"
@@ -835,6 +836,12 @@ static int mlx5_init_once(struct mlx5_core_dev *dev)
 		goto err_eq_cleanup;
 	}
 
+	err = mlx5_fw_reset_events_init(dev);
+	if (err) {
+		mlx5_core_err(dev, "failed to initialize fw reset events\n");
+		goto err_events_cleanup;
+	}
+
 	mlx5_cq_debugfs_init(dev);
 
 	mlx5_init_reserved_gids(dev);
@@ -896,6 +903,8 @@ static int mlx5_init_once(struct mlx5_core_dev *dev)
 	mlx5_geneve_destroy(dev->geneve);
 	mlx5_vxlan_destroy(dev->vxlan);
 	mlx5_cq_debugfs_cleanup(dev);
+	mlx5_fw_reset_events_cleanup(dev);
+err_events_cleanup:
 	mlx5_events_cleanup(dev);
 err_eq_cleanup:
 	mlx5_eq_table_cleanup(dev);
@@ -923,6 +932,7 @@ static void mlx5_cleanup_once(struct mlx5_core_dev *dev)
 	mlx5_cleanup_clock(dev);
 	mlx5_cleanup_reserved_gids(dev);
 	mlx5_cq_debugfs_cleanup(dev);
+	mlx5_fw_reset_events_cleanup(dev);
 	mlx5_events_cleanup(dev);
 	mlx5_eq_table_cleanup(dev);
 	mlx5_irq_table_cleanup(dev);
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/mlx5_core.h b/drivers/net/ethernet/mellanox/mlx5/core/mlx5_core.h
index fc1649dac11b..d07a32165792 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/mlx5_core.h
+++ b/drivers/net/ethernet/mellanox/mlx5/core/mlx5_core.h
@@ -123,6 +123,8 @@ int mlx5_cmd_force_teardown_hca(struct mlx5_core_dev *dev);
 int mlx5_cmd_fast_teardown_hca(struct mlx5_core_dev *dev);
 void mlx5_enter_error_state(struct mlx5_core_dev *dev, bool force);
 void mlx5_error_sw_reset(struct mlx5_core_dev *dev);
+u32 mlx5_health_check_fatal_sensors(struct mlx5_core_dev *dev);
+int mlx5_health_wait_pci_up(struct mlx5_core_dev *dev);
 void mlx5_disable_device(struct mlx5_core_dev *dev);
 void mlx5_recover_device(struct mlx5_core_dev *dev);
 int mlx5_sriov_init(struct mlx5_core_dev *dev);
diff --git a/include/linux/mlx5/driver.h b/include/linux/mlx5/driver.h
index 8dc3da6e6480..80e31a7684e0 100644
--- a/include/linux/mlx5/driver.h
+++ b/include/linux/mlx5/driver.h
@@ -501,6 +501,7 @@ struct mlx5_mpfs;
 struct mlx5_eswitch;
 struct mlx5_lag;
 struct mlx5_devcom;
+struct mlx5_fw_reset;
 struct mlx5_eq_table;
 struct mlx5_irq_table;
 
@@ -578,6 +579,7 @@ struct mlx5_priv {
 	struct mlx5_core_sriov	sriov;
 	struct mlx5_lag		*lag;
 	struct mlx5_devcom	*devcom;
+	struct mlx5_fw_reset	*fw_reset;
 	struct mlx5_core_roce	roce;
 	struct mlx5_fc_stats		fc_stats;
 	struct mlx5_rl_table            rl_table;
@@ -943,6 +945,8 @@ void mlx5_start_health_poll(struct mlx5_core_dev *dev);
 void mlx5_stop_health_poll(struct mlx5_core_dev *dev, bool disable_health);
 void mlx5_drain_health_wq(struct mlx5_core_dev *dev);
 void mlx5_trigger_health_work(struct mlx5_core_dev *dev);
+void mlx5_health_set_reset_requested_mode(struct mlx5_core_dev *dev);
+void mlx5_health_clear_reset_requested_mode(struct mlx5_core_dev *dev);
 int mlx5_buf_alloc(struct mlx5_core_dev *dev,
 		   int size, struct mlx5_frag_buf *buf);
 void mlx5_buf_free(struct mlx5_core_dev *dev, struct mlx5_frag_buf *buf);
-- 
2.17.1

[PATCH net-next RFC v4 08/15] net/mlx5: Handle sync reset now event

[Moshe Shemesh]

On sync_reset_now event the driver does reload and PCI link toggle to
activate firmware upgrade reset. When the firmware sends this event it
syncs the event on all PFs, so all PFs will do PCI link toggle at once.
To do PCI link toggle, the driver ensures that no other device ID under
the same bridge by checking that all the PF functions under the same PCI
bridge have same device ID. If no other device it uses PCI bridge link
control to turn link down and up.

Signed-off-by: Moshe Shemesh <moshe@mellanox.com>
---
 .../ethernet/mellanox/mlx5/core/fw_reset.c    | 121 ++++++++++++++++++
 1 file changed, 121 insertions(+)

diff --git a/drivers/net/ethernet/mellanox/mlx5/core/fw_reset.c b/drivers/net/ethernet/mellanox/mlx5/core/fw_reset.c
index 0f224454b4a2..f9e293de9de3 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/fw_reset.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/fw_reset.c
@@ -13,6 +13,7 @@ struct mlx5_fw_reset {
 	struct workqueue_struct *wq;
 	struct work_struct reset_request_work;
 	struct work_struct reset_reload_work;
+	struct work_struct reset_now_work;
 	unsigned long reset_flags;
 	struct timer_list timer;
 };
@@ -149,6 +150,122 @@ static void mlx5_sync_reset_request_event(struct work_struct *work)
 		mlx5_core_warn(dev, "PCI Sync FW Update Reset Ack. Device reset is expected.\n");
 }
 
+#define MLX5_PCI_LINK_UP_TIMEOUT 2000
+
+static int mlx5_pci_link_toggle(struct mlx5_core_dev *dev)
+{
+	struct pci_bus *bridge_bus = dev->pdev->bus;
+	struct pci_dev *bridge = bridge_bus->self;
+	u16 reg16, dev_id, sdev_id;
+	unsigned long timeout;
+	struct pci_dev *sdev;
+	int cap, err;
+	u32 reg32;
+
+	/* Check that all functions under the pci bridge are PFs of
+	 * this device otherwise fail this function.
+	 */
+	err = pci_read_config_word(dev->pdev, PCI_DEVICE_ID, &dev_id);
+	if (err)
+		return err;
+	list_for_each_entry(sdev, &bridge_bus->devices, bus_list) {
+		err = pci_read_config_word(sdev, PCI_DEVICE_ID, &sdev_id);
+		if (err)
+			return err;
+		if (sdev_id != dev_id)
+			return -EPERM;
+	}
+
+	cap = pci_find_capability(bridge, PCI_CAP_ID_EXP);
+	if (!cap)
+		return -EOPNOTSUPP;
+
+	list_for_each_entry(sdev, &bridge_bus->devices, bus_list) {
+		pci_save_state(sdev);
+		pci_cfg_access_lock(sdev);
+	}
+	/* PCI link toggle */
+	err = pci_read_config_word(bridge, cap + PCI_EXP_LNKCTL, &reg16);
+	if (err)
+		return err;
+	reg16 |= PCI_EXP_LNKCTL_LD;
+	err = pci_write_config_word(bridge, cap + PCI_EXP_LNKCTL, reg16);
+	if (err)
+		return err;
+	msleep(500);
+	reg16 &= ~PCI_EXP_LNKCTL_LD;
+	err = pci_write_config_word(bridge, cap + PCI_EXP_LNKCTL, reg16);
+	if (err)
+		return err;
+
+	/* Check link */
+	err = pci_read_config_dword(bridge, cap + PCI_EXP_LNKCAP, &reg32);
+	if (err)
+		return err;
+	if (!(reg32 & PCI_EXP_LNKCAP_DLLLARC)) {
+		mlx5_core_warn(dev, "No PCI link reporting capability (0x%08x)\n", reg32);
+		msleep(1000);
+		goto restore;
+	}
+
+	timeout = jiffies + msecs_to_jiffies(MLX5_PCI_LINK_UP_TIMEOUT);
+	do {
+		err = pci_read_config_word(bridge, cap + PCI_EXP_LNKSTA, &reg16);
+		if (err)
+			return err;
+		if (reg16 & PCI_EXP_LNKSTA_DLLLA)
+			break;
+		msleep(20);
+	} while (!time_after(jiffies, timeout));
+
+	if (reg16 & PCI_EXP_LNKSTA_DLLLA) {
+		mlx5_core_info(dev, "PCI Link up\n");
+	} else {
+		mlx5_core_err(dev, "PCI link not ready (0x%04x) after %d ms\n",
+			      reg16, MLX5_PCI_LINK_UP_TIMEOUT);
+		err = -ETIMEDOUT;
+	}
+
+restore:
+	list_for_each_entry(sdev, &bridge_bus->devices, bus_list) {
+		pci_cfg_access_unlock(sdev);
+		pci_restore_state(sdev);
+	}
+
+	return err;
+}
+
+static void mlx5_sync_reset_now_event(struct work_struct *work)
+{
+	struct mlx5_fw_reset *fw_reset = container_of(work, struct mlx5_fw_reset,
+						      reset_now_work);
+	struct mlx5_core_dev *dev = fw_reset->dev;
+	int err;
+
+	mlx5_sync_reset_clear_reset_requested(dev, false);
+
+	mlx5_core_warn(dev, "Sync Reset now. Device is going to reset.\n");
+
+	err = mlx5_cmd_fast_teardown_hca(dev);
+	if (err) {
+		mlx5_core_warn(dev, "Fast teardown failed, no reset done, err %d\n", err);
+		goto done;
+	}
+
+	err = mlx5_pci_link_toggle(dev);
+	if (err) {
+		mlx5_core_warn(dev, "mlx5_pci_link_toggle failed, no reset done, err %d\n", err);
+		goto done;
+	}
+
+	mlx5_enter_error_state(dev, true);
+	mlx5_unload_one(dev, false);
+done:
+	if (err)
+		mlx5_start_health_poll(dev);
+	mlx5_load_one(dev, false);
+}
+
 static void mlx5_sync_reset_events_handle(struct mlx5_fw_reset *fw_reset, struct mlx5_eqe *eqe)
 {
 	struct mlx5_eqe_sync_fw_update *sync_fw_update_eqe;
@@ -160,6 +277,9 @@ static void mlx5_sync_reset_events_handle(struct mlx5_fw_reset *fw_reset, struct
 	case MLX5_SYNC_RST_STATE_RESET_REQUEST:
 		queue_work(fw_reset->wq, &fw_reset->reset_request_work);
 		break;
+	case MLX5_SYNC_RST_STATE_RESET_NOW:
+		queue_work(fw_reset->wq, &fw_reset->reset_now_work);
+		break;
 	}
 }
 
@@ -196,6 +316,7 @@ int mlx5_fw_reset_events_init(struct mlx5_core_dev *dev)
 
 	INIT_WORK(&fw_reset->reset_request_work, mlx5_sync_reset_request_event);
 	INIT_WORK(&fw_reset->reset_reload_work, mlx5_sync_reset_reload_work);
+	INIT_WORK(&fw_reset->reset_now_work, mlx5_sync_reset_now_event);
 
 	MLX5_NB_INIT(&fw_reset->nb, fw_reset_event_notifier, GENERAL_EVENT);
 	mlx5_eq_notifier_register(dev, &fw_reset->nb);
-- 
2.17.1

[PATCH net-next RFC v4 09/15] net/mlx5: Handle sync reset abort event

[Moshe Shemesh]

If firmware sends sync_reset_abort to driver the driver should clear the
reset requested mode as reset is not expected any more.

Signed-off-by: Moshe Shemesh <moshe@mellanox.com>
---
 .../net/ethernet/mellanox/mlx5/core/fw_reset.c    | 15 +++++++++++++++
 1 file changed, 15 insertions(+)

diff --git a/drivers/net/ethernet/mellanox/mlx5/core/fw_reset.c b/drivers/net/ethernet/mellanox/mlx5/core/fw_reset.c
index f9e293de9de3..61237f4836cc 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/fw_reset.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/fw_reset.c
@@ -14,6 +14,7 @@ struct mlx5_fw_reset {
 	struct work_struct reset_request_work;
 	struct work_struct reset_reload_work;
 	struct work_struct reset_now_work;
+	struct work_struct reset_abort_work;
 	unsigned long reset_flags;
 	struct timer_list timer;
 };
@@ -266,6 +267,16 @@ static void mlx5_sync_reset_now_event(struct work_struct *work)
 	mlx5_load_one(dev, false);
 }
 
+static void mlx5_sync_reset_abort_event(struct work_struct *work)
+{
+	struct mlx5_fw_reset *fw_reset = container_of(work, struct mlx5_fw_reset,
+						      reset_abort_work);
+	struct mlx5_core_dev *dev = fw_reset->dev;
+
+	mlx5_sync_reset_clear_reset_requested(dev, true);
+	mlx5_core_warn(dev, "PCI Sync FW Update Reset Aborted.\n");
+}
+
 static void mlx5_sync_reset_events_handle(struct mlx5_fw_reset *fw_reset, struct mlx5_eqe *eqe)
 {
 	struct mlx5_eqe_sync_fw_update *sync_fw_update_eqe;
@@ -280,6 +291,9 @@ static void mlx5_sync_reset_events_handle(struct mlx5_fw_reset *fw_reset, struct
 	case MLX5_SYNC_RST_STATE_RESET_NOW:
 		queue_work(fw_reset->wq, &fw_reset->reset_now_work);
 		break;
+	case MLX5_SYNC_RST_STATE_RESET_ABORT:
+		queue_work(fw_reset->wq, &fw_reset->reset_abort_work);
+		break;
 	}
 }
 
@@ -317,6 +331,7 @@ int mlx5_fw_reset_events_init(struct mlx5_core_dev *dev)
 	INIT_WORK(&fw_reset->reset_request_work, mlx5_sync_reset_request_event);
 	INIT_WORK(&fw_reset->reset_reload_work, mlx5_sync_reset_reload_work);
 	INIT_WORK(&fw_reset->reset_now_work, mlx5_sync_reset_now_event);
+	INIT_WORK(&fw_reset->reset_abort_work, mlx5_sync_reset_abort_event);
 
 	MLX5_NB_INIT(&fw_reset->nb, fw_reset_event_notifier, GENERAL_EVENT);
 	mlx5_eq_notifier_register(dev, &fw_reset->nb);
-- 
2.17.1

[PATCH net-next RFC v4 10/15] net/mlx5: Add support for devlink reload action fw activate

[Moshe Shemesh]

Add support for devlink reload action fw_activate. To activate firmware
image the mlx5 driver resets the firmware and reloads it from flash. If
a new image was stored on flash it will be loaded. Once this reload
command is executed the driver initiates fw sync reset flow, where the
firmware synchronizes all PFs on coming reset and driver reload.

Signed-off-by: Moshe Shemesh <moshe@mellanox.com>
---
v3 -> v4:
- Renamed actions_done to actions_performed
v2 -> v3:
- Return the reload actions done
- Update reload action counters if reset initiated by remote host
v1 -> v2:
- Have fw_activate action instead of fw_reset level
---
 .../net/ethernet/mellanox/mlx5/core/devlink.c | 62 ++++++++++++++++---
 .../ethernet/mellanox/mlx5/core/fw_reset.c    | 60 ++++++++++++++++--
 .../ethernet/mellanox/mlx5/core/fw_reset.h    |  1 +
 3 files changed, 109 insertions(+), 14 deletions(-)

diff --git a/drivers/net/ethernet/mellanox/mlx5/core/devlink.c b/drivers/net/ethernet/mellanox/mlx5/core/devlink.c
index f6b29deaf02e..fa8f6abbea4e 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/devlink.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/devlink.c
@@ -4,6 +4,7 @@
 #include <devlink.h>
 
 #include "mlx5_core.h"
+#include "fw_reset.h"
 #include "fs_core.h"
 #include "eswitch.h"
 
@@ -88,6 +89,32 @@ mlx5_devlink_info_get(struct devlink *devlink, struct devlink_info_req *req,
 	return 0;
 }
 
+static int mlx5_devlink_reload_fw_activate(struct devlink *devlink, struct netlink_ext_ack *extack)
+{
+	struct mlx5_core_dev *dev = devlink_priv(devlink);
+	u8 reset_level, reset_type, net_port_alive;
+	int err;
+
+	err = mlx5_reg_mfrl_query(dev, &reset_level, &reset_type);
+	if (err)
+		return err;
+	if (!(reset_level & MLX5_MFRL_REG_RESET_LEVEL3)) {
+		NL_SET_ERR_MSG_MOD(extack, "FW activate requires reboot");
+		return -EINVAL;
+	}
+
+	net_port_alive = !!(reset_type & MLX5_MFRL_REG_RESET_TYPE_NET_PORT_ALIVE);
+	err = mlx5_fw_set_reset_sync(dev, net_port_alive);
+	if (err)
+		goto out;
+
+	err = mlx5_fw_wait_fw_reset_done(dev);
+out:
+	if (err)
+		NL_SET_ERR_MSG_MOD(extack, "FW activate command failed");
+	return err;
+}
+
 static int mlx5_devlink_reload_down(struct devlink *devlink, bool netns_change,
 				    enum devlink_reload_action action,
 				    enum devlink_reload_action_limit_level limit_level,
@@ -95,8 +122,17 @@ static int mlx5_devlink_reload_down(struct devlink *devlink, bool netns_change,
 {
 	struct mlx5_core_dev *dev = devlink_priv(devlink);
 
-	mlx5_unload_one(dev, false);
-	return 0;
+	switch (action) {
+	case DEVLINK_RELOAD_ACTION_DRIVER_REINIT:
+		mlx5_unload_one(dev, false);
+		return 0;
+	case DEVLINK_RELOAD_ACTION_FW_ACTIVATE:
+		return mlx5_devlink_reload_fw_activate(devlink, extack);
+	default:
+		/* Unsupported action should not get to this function */
+		WARN_ON(1);
+		return -EOPNOTSUPP;
+	}
 }
 
 static int mlx5_devlink_reload_up(struct devlink *devlink, enum devlink_reload_action action,
@@ -104,13 +140,22 @@ static int mlx5_devlink_reload_up(struct devlink *devlink, enum devlink_reload_a
 				  struct netlink_ext_ack *extack, unsigned long *actions_performed)
 {
 	struct mlx5_core_dev *dev = devlink_priv(devlink);
-	int err;
 
-	err = mlx5_load_one(dev, false);
-	if (err)
-		return err;
 	if (actions_performed)
-		*actions_performed = BIT(DEVLINK_RELOAD_ACTION_DRIVER_REINIT);
+		*actions_performed = BIT(action);
+
+	switch (action) {
+	case DEVLINK_RELOAD_ACTION_DRIVER_REINIT:
+	case DEVLINK_RELOAD_ACTION_FW_ACTIVATE:
+		/* On fw_activate action, also driver is reloaded and reinit performed */
+		if (actions_performed)
+			*actions_performed |= BIT(DEVLINK_RELOAD_ACTION_DRIVER_REINIT);
+		return mlx5_load_one(dev, false);
+	default:
+		/* Unsupported action should not get to this function */
+		WARN_ON(1);
+		return -EOPNOTSUPP;
+	}
 
 	return 0;
 }
@@ -128,7 +173,8 @@ static const struct devlink_ops mlx5_devlink_ops = {
 #endif
 	.flash_update = mlx5_devlink_flash_update,
 	.info_get = mlx5_devlink_info_get,
-	.supported_reload_actions = BIT(DEVLINK_RELOAD_ACTION_DRIVER_REINIT),
+	.supported_reload_actions = BIT(DEVLINK_RELOAD_ACTION_DRIVER_REINIT) |
+				    BIT(DEVLINK_RELOAD_ACTION_FW_ACTIVATE),
 	.supported_reload_action_limit_levels = BIT(DEVLINK_RELOAD_ACTION_LIMIT_LEVEL_NONE),
 	.reload_down = mlx5_devlink_reload_down,
 	.reload_up = mlx5_devlink_reload_up,
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/fw_reset.c b/drivers/net/ethernet/mellanox/mlx5/core/fw_reset.c
index 61237f4836cc..550f67b00473 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/fw_reset.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/fw_reset.c
@@ -5,6 +5,7 @@
 
 enum {
 	MLX5_FW_RESET_FLAGS_RESET_REQUESTED,
+	MLX5_FW_RESET_FLAGS_PENDING_COMP
 };
 
 struct mlx5_fw_reset {
@@ -17,6 +18,8 @@ struct mlx5_fw_reset {
 	struct work_struct reset_abort_work;
 	unsigned long reset_flags;
 	struct timer_list timer;
+	struct completion done;
+	int ret;
 };
 
 static int mlx5_reg_mfrl_set(struct mlx5_core_dev *dev, u8 reset_level,
@@ -53,7 +56,14 @@ int mlx5_reg_mfrl_query(struct mlx5_core_dev *dev, u8 *reset_level, u8 *reset_ty
 
 int mlx5_fw_set_reset_sync(struct mlx5_core_dev *dev, u8 reset_type_sel)
 {
-	return mlx5_reg_mfrl_set(dev, MLX5_MFRL_REG_RESET_LEVEL3, reset_type_sel, 0, true);
+	struct mlx5_fw_reset *fw_reset = dev->priv.fw_reset;
+	int err;
+
+	set_bit(MLX5_FW_RESET_FLAGS_PENDING_COMP, &fw_reset->reset_flags);
+	err =  mlx5_reg_mfrl_set(dev, MLX5_MFRL_REG_RESET_LEVEL3, reset_type_sel, 0, true);
+	if (err)
+		clear_bit(MLX5_FW_RESET_FLAGS_PENDING_COMP, &fw_reset->reset_flags);
+	return err;
 }
 
 int mlx5_fw_set_live_patch(struct mlx5_core_dev *dev)
@@ -66,19 +76,36 @@ static int mlx5_fw_set_reset_sync_ack(struct mlx5_core_dev *dev)
 	return mlx5_reg_mfrl_set(dev, MLX5_MFRL_REG_RESET_LEVEL3, 0, 1, false);
 }
 
+static void mlx5_fw_reset_complete_reload(struct mlx5_core_dev *dev)
+{
+	struct mlx5_fw_reset *fw_reset = dev->priv.fw_reset;
+
+	/* if this is the driver that initiated the fw reset, devlink completed the reload */
+	if (test_bit(MLX5_FW_RESET_FLAGS_PENDING_COMP, &fw_reset->reset_flags)) {
+		complete(&fw_reset->done);
+	} else {
+		mlx5_load_one(dev, false);
+		devlink_reload_implicit_actions_performed(priv_to_devlink(dev),
+							  DEVLINK_RELOAD_ACTION_LIMIT_LEVEL_NONE,
+							  BIT(DEVLINK_RELOAD_ACTION_DRIVER_REINIT) |
+							  BIT(DEVLINK_RELOAD_ACTION_FW_ACTIVATE));
+	}
+}
+
 static void mlx5_sync_reset_reload_work(struct work_struct *work)
 {
 	struct mlx5_fw_reset *fw_reset = container_of(work, struct mlx5_fw_reset,
 						      reset_reload_work);
 	struct mlx5_core_dev *dev = fw_reset->dev;
+	int err;
 
 	mlx5_enter_error_state(dev, true);
 	mlx5_unload_one(dev, false);
-	if (mlx5_health_wait_pci_up(dev)) {
+	err = mlx5_health_wait_pci_up(dev);
+	if (err)
 		mlx5_core_err(dev, "reset reload flow aborted, PCI reads still not working\n");
-		return;
-	}
-	mlx5_load_one(dev, false);
+	fw_reset->ret = err;
+	mlx5_fw_reset_complete_reload(dev);
 }
 
 static void mlx5_stop_sync_reset_poll(struct mlx5_core_dev *dev)
@@ -264,7 +291,8 @@ static void mlx5_sync_reset_now_event(struct work_struct *work)
 done:
 	if (err)
 		mlx5_start_health_poll(dev);
-	mlx5_load_one(dev, false);
+	fw_reset->ret = err;
+	mlx5_fw_reset_complete_reload(dev);
 }
 
 static void mlx5_sync_reset_abort_event(struct work_struct *work)
@@ -313,6 +341,25 @@ static int fw_reset_event_notifier(struct notifier_block *nb, unsigned long acti
 	return NOTIFY_OK;
 }
 
+#define MLX5_FW_RESET_TIMEOUT_MSEC 5000
+int mlx5_fw_wait_fw_reset_done(struct mlx5_core_dev *dev)
+{
+	unsigned long timeout = msecs_to_jiffies(MLX5_FW_RESET_TIMEOUT_MSEC);
+	struct mlx5_fw_reset *fw_reset = dev->priv.fw_reset;
+	int err;
+
+	if (!wait_for_completion_timeout(&fw_reset->done, timeout)) {
+		mlx5_core_warn(dev, "FW sync reset timeout after %d seconds\n",
+			       MLX5_FW_RESET_TIMEOUT_MSEC / 1000);
+		err = -ETIMEDOUT;
+		goto out;
+	}
+	err = fw_reset->ret;
+out:
+	clear_bit(MLX5_FW_RESET_FLAGS_PENDING_COMP, &fw_reset->reset_flags);
+	return err;
+}
+
 int mlx5_fw_reset_events_init(struct mlx5_core_dev *dev)
 {
 	struct mlx5_fw_reset *fw_reset = kzalloc(sizeof(*fw_reset), GFP_KERNEL);
@@ -336,6 +383,7 @@ int mlx5_fw_reset_events_init(struct mlx5_core_dev *dev)
 	MLX5_NB_INIT(&fw_reset->nb, fw_reset_event_notifier, GENERAL_EVENT);
 	mlx5_eq_notifier_register(dev, &fw_reset->nb);
 
+	init_completion(&fw_reset->done);
 	return 0;
 }
 
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/fw_reset.h b/drivers/net/ethernet/mellanox/mlx5/core/fw_reset.h
index 278f538ea92a..d7ee951a2258 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/fw_reset.h
+++ b/drivers/net/ethernet/mellanox/mlx5/core/fw_reset.h
@@ -10,6 +10,7 @@ int mlx5_reg_mfrl_query(struct mlx5_core_dev *dev, u8 *reset_level, u8 *reset_ty
 int mlx5_fw_set_reset_sync(struct mlx5_core_dev *dev, u8 reset_type_sel);
 int mlx5_fw_set_live_patch(struct mlx5_core_dev *dev);
 
+int mlx5_fw_wait_fw_reset_done(struct mlx5_core_dev *dev);
 int mlx5_fw_reset_events_init(struct mlx5_core_dev *dev);
 void mlx5_fw_reset_events_cleanup(struct mlx5_core_dev *dev);
 
-- 
2.17.1

[PATCH net-next RFC v4 11/15] devlink: Add enable_remote_dev_reset generic parameter

[Moshe Shemesh]

The enable_remote_dev_reset devlink param flags that the host admin
allows device resets that can be initiated by other hosts. This
parameter is useful for setups where a device is shared by different
hosts, such as multi-host setup. Once the user set this parameter to
false, the driver should NACK any attempt to reset the device while the
driver is loaded.

Signed-off-by: Moshe Shemesh <moshe@mellanox.com>
---
 Documentation/networking/devlink/devlink-params.rst | 6 ++++++
 include/net/devlink.h                               | 4 ++++
 net/core/devlink.c                                  | 5 +++++
 3 files changed, 15 insertions(+)

diff --git a/Documentation/networking/devlink/devlink-params.rst b/Documentation/networking/devlink/devlink-params.rst
index d075fd090b3d..54c9f107c4b0 100644
--- a/Documentation/networking/devlink/devlink-params.rst
+++ b/Documentation/networking/devlink/devlink-params.rst
@@ -108,3 +108,9 @@ own name.
    * - ``region_snapshot_enable``
      - Boolean
      - Enable capture of ``devlink-region`` snapshots.
+   * - ``enable_remote_dev_reset``
+     - Boolean
+     - Enable device reset by remote host. When cleared, the device driver
+       will NACK any attempt of other host to reset the device. This parameter
+       is useful for setups where a device is shared by different hosts, such
+       as multi-host setup.
diff --git a/include/net/devlink.h b/include/net/devlink.h
index b4feb92e0269..c5f26983a3a8 100644
--- a/include/net/devlink.h
+++ b/include/net/devlink.h
@@ -444,6 +444,7 @@ enum devlink_param_generic_id {
 	DEVLINK_PARAM_GENERIC_ID_FW_LOAD_POLICY,
 	DEVLINK_PARAM_GENERIC_ID_RESET_DEV_ON_DRV_PROBE,
 	DEVLINK_PARAM_GENERIC_ID_ENABLE_ROCE,
+	DEVLINK_PARAM_GENERIC_ID_ENABLE_REMOTE_DEV_RESET,
 
 	/* add new param generic ids above here*/
 	__DEVLINK_PARAM_GENERIC_ID_MAX,
@@ -481,6 +482,9 @@ enum devlink_param_generic_id {
 #define DEVLINK_PARAM_GENERIC_ENABLE_ROCE_NAME "enable_roce"
 #define DEVLINK_PARAM_GENERIC_ENABLE_ROCE_TYPE DEVLINK_PARAM_TYPE_BOOL
 
+#define DEVLINK_PARAM_GENERIC_ENABLE_REMOTE_DEV_RESET_NAME "enable_remote_dev_reset"
+#define DEVLINK_PARAM_GENERIC_ENABLE_REMOTE_DEV_RESET_TYPE DEVLINK_PARAM_TYPE_BOOL
+
 #define DEVLINK_PARAM_GENERIC(_id, _cmodes, _get, _set, _validate)	\
 {									\
 	.id = DEVLINK_PARAM_GENERIC_ID_##_id,				\
diff --git a/net/core/devlink.c b/net/core/devlink.c
index 1063b7a4123a..7afb4c8328b4 100644
--- a/net/core/devlink.c
+++ b/net/core/devlink.c
@@ -3380,6 +3380,11 @@ static const struct devlink_param devlink_param_generic[] = {
 		.name = DEVLINK_PARAM_GENERIC_ENABLE_ROCE_NAME,
 		.type = DEVLINK_PARAM_GENERIC_ENABLE_ROCE_TYPE,
 	},
+	{
+		.id = DEVLINK_PARAM_GENERIC_ID_ENABLE_REMOTE_DEV_RESET,
+		.name = DEVLINK_PARAM_GENERIC_ENABLE_REMOTE_DEV_RESET_NAME,
+		.type = DEVLINK_PARAM_GENERIC_ENABLE_REMOTE_DEV_RESET_TYPE,
+	},
 };
 
 static int devlink_param_generic_verify(const struct devlink_param *param)
-- 
2.17.1

[PATCH net-next RFC v4 12/15] net/mlx5: Add devlink param enable_remote_dev_reset support

[Moshe Shemesh]

The enable_remote_dev_reset devlink param flags that the host admin
allows resets by other hosts. In case it is cleared mlx5 host PF driver
will send NACK on pci sync for firmware update reset request and the
command will fail.
By default enable_remote_dev_reset parameter is true, so pci sync for
firmware update reset is enabled.

Signed-off-by: Moshe Shemesh <moshe@mellanox.com>
---
v1 -> v2:
- Have MLX5_FW_RESET_FLAGS_NACK_RESET_REQUEST instead of
  MLX5_HEALTH_RESET_FLAGS_NACK_RESET_REQUEST
---
 .../net/ethernet/mellanox/mlx5/core/devlink.c | 21 +++++++++++++
 .../ethernet/mellanox/mlx5/core/fw_reset.c    | 30 +++++++++++++++++++
 .../ethernet/mellanox/mlx5/core/fw_reset.h    |  2 ++
 3 files changed, 53 insertions(+)

diff --git a/drivers/net/ethernet/mellanox/mlx5/core/devlink.c b/drivers/net/ethernet/mellanox/mlx5/core/devlink.c
index fa8f6abbea4e..175e933cf19c 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/devlink.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/devlink.c
@@ -286,6 +286,24 @@ static int mlx5_devlink_large_group_num_validate(struct devlink *devlink, u32 id
 }
 #endif
 
+static int mlx5_devlink_enable_remote_dev_reset_set(struct devlink *devlink, u32 id,
+						    struct devlink_param_gset_ctx *ctx)
+{
+	struct mlx5_core_dev *dev = devlink_priv(devlink);
+
+	mlx5_fw_enable_remote_dev_reset_set(dev, ctx->val.vbool);
+	return 0;
+}
+
+static int mlx5_devlink_enable_remote_dev_reset_get(struct devlink *devlink, u32 id,
+						    struct devlink_param_gset_ctx *ctx)
+{
+	struct mlx5_core_dev *dev = devlink_priv(devlink);
+
+	ctx->val.vbool = mlx5_fw_enable_remote_dev_reset_get(dev);
+	return 0;
+}
+
 static const struct devlink_param mlx5_devlink_params[] = {
 	DEVLINK_PARAM_DRIVER(MLX5_DEVLINK_PARAM_ID_FLOW_STEERING_MODE,
 			     "flow_steering_mode", DEVLINK_PARAM_TYPE_STRING,
@@ -301,6 +319,9 @@ static const struct devlink_param mlx5_devlink_params[] = {
 			     NULL, NULL,
 			     mlx5_devlink_large_group_num_validate),
 #endif
+	DEVLINK_PARAM_GENERIC(ENABLE_REMOTE_DEV_RESET, BIT(DEVLINK_PARAM_CMODE_RUNTIME),
+			      mlx5_devlink_enable_remote_dev_reset_get,
+			      mlx5_devlink_enable_remote_dev_reset_set, NULL),
 };
 
 static void mlx5_devlink_set_params_init_values(struct devlink *devlink)
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/fw_reset.c b/drivers/net/ethernet/mellanox/mlx5/core/fw_reset.c
index 550f67b00473..88d59cc059bd 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/fw_reset.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/fw_reset.c
@@ -5,6 +5,7 @@
 
 enum {
 	MLX5_FW_RESET_FLAGS_RESET_REQUESTED,
+	MLX5_FW_RESET_FLAGS_NACK_RESET_REQUEST,
 	MLX5_FW_RESET_FLAGS_PENDING_COMP
 };
 
@@ -22,6 +23,23 @@ struct mlx5_fw_reset {
 	int ret;
 };
 
+void mlx5_fw_enable_remote_dev_reset_set(struct mlx5_core_dev *dev, bool enable)
+{
+	struct mlx5_fw_reset *fw_reset = dev->priv.fw_reset;
+
+	if (enable)
+		clear_bit(MLX5_FW_RESET_FLAGS_NACK_RESET_REQUEST, &fw_reset->reset_flags);
+	else
+		set_bit(MLX5_FW_RESET_FLAGS_NACK_RESET_REQUEST, &fw_reset->reset_flags);
+}
+
+bool mlx5_fw_enable_remote_dev_reset_get(struct mlx5_core_dev *dev)
+{
+	struct mlx5_fw_reset *fw_reset = dev->priv.fw_reset;
+
+	return !test_bit(MLX5_FW_RESET_FLAGS_NACK_RESET_REQUEST, &fw_reset->reset_flags);
+}
+
 static int mlx5_reg_mfrl_set(struct mlx5_core_dev *dev, u8 reset_level,
 			     u8 reset_type_sel, u8 sync_resp, bool sync_start)
 {
@@ -76,6 +94,11 @@ static int mlx5_fw_set_reset_sync_ack(struct mlx5_core_dev *dev)
 	return mlx5_reg_mfrl_set(dev, MLX5_MFRL_REG_RESET_LEVEL3, 0, 1, false);
 }
 
+static int mlx5_fw_set_reset_sync_nack(struct mlx5_core_dev *dev)
+{
+	return mlx5_reg_mfrl_set(dev, MLX5_MFRL_REG_RESET_LEVEL3, 0, 2, false);
+}
+
 static void mlx5_fw_reset_complete_reload(struct mlx5_core_dev *dev)
 {
 	struct mlx5_fw_reset *fw_reset = dev->priv.fw_reset;
@@ -170,7 +193,14 @@ static void mlx5_sync_reset_request_event(struct work_struct *work)
 	struct mlx5_fw_reset *fw_reset = container_of(work, struct mlx5_fw_reset,
 						      reset_request_work);
 	struct mlx5_core_dev *dev = fw_reset->dev;
+	int err;
 
+	if (test_bit(MLX5_FW_RESET_FLAGS_NACK_RESET_REQUEST, &fw_reset->reset_flags)) {
+		err = mlx5_fw_set_reset_sync_nack(dev);
+		mlx5_core_warn(dev, "PCI Sync FW Update Reset Nack %s",
+			       err ? "Failed" : "Sent");
+		return;
+	}
 	mlx5_sync_reset_set_reset_requested(dev);
 	if (mlx5_fw_set_reset_sync_ack(dev))
 		mlx5_core_warn(dev, "PCI Sync FW Update Reset Ack Failed.\n");
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/fw_reset.h b/drivers/net/ethernet/mellanox/mlx5/core/fw_reset.h
index d7ee951a2258..fd558dfe93fc 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/fw_reset.h
+++ b/drivers/net/ethernet/mellanox/mlx5/core/fw_reset.h
@@ -6,6 +6,8 @@
 
 #include "mlx5_core.h"
 
+void mlx5_fw_enable_remote_dev_reset_set(struct mlx5_core_dev *dev, bool enable);
+bool mlx5_fw_enable_remote_dev_reset_get(struct mlx5_core_dev *dev);
 int mlx5_reg_mfrl_query(struct mlx5_core_dev *dev, u8 *reset_level, u8 *reset_type);
 int mlx5_fw_set_reset_sync(struct mlx5_core_dev *dev, u8 reset_type_sel);
 int mlx5_fw_set_live_patch(struct mlx5_core_dev *dev);
-- 
2.17.1

[PATCH net-next RFC v4 13/15] net/mlx5: Add support for fw live patch event

[Moshe Shemesh]

Firmware live patch event notifies the driver that the firmware was just
updated using live patch. In such case the driver should not reload or
re-initiate entities, part to updating the firmware version and
re-initiate the firmware tracer which can be updated by live patch with
new strings database to help debugging an issue.

Signed-off-by: Moshe Shemesh <moshe@mellanox.com>
---
 .../mellanox/mlx5/core/diag/fw_tracer.c       | 31 +++++++++++++++++++
 .../mellanox/mlx5/core/diag/fw_tracer.h       |  1 +
 .../ethernet/mellanox/mlx5/core/fw_reset.c    | 27 ++++++++++++++++
 include/linux/mlx5/device.h                   |  1 +
 4 files changed, 60 insertions(+)

diff --git a/drivers/net/ethernet/mellanox/mlx5/core/diag/fw_tracer.c b/drivers/net/ethernet/mellanox/mlx5/core/diag/fw_tracer.c
index ad3594c4afcb..08dae045d185 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/diag/fw_tracer.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/diag/fw_tracer.c
@@ -1064,6 +1064,37 @@ void mlx5_fw_tracer_destroy(struct mlx5_fw_tracer *tracer)
 	kvfree(tracer);
 }
 
+int mlx5_fw_tracer_recreate_strings_db(struct mlx5_fw_tracer *tracer)
+{
+	struct mlx5_core_dev *dev;
+	int err;
+
+	if (IS_ERR_OR_NULL(tracer))
+		return -EINVAL;
+
+	cancel_work_sync(&tracer->read_fw_strings_work);
+	mlx5_fw_tracer_clean_ready_list(tracer);
+	mlx5_fw_tracer_clean_print_hash(tracer);
+	mlx5_fw_tracer_clean_saved_traces_array(tracer);
+	mlx5_fw_tracer_free_strings_db(tracer);
+
+	dev = tracer->dev;
+	err = mlx5_query_mtrc_caps(tracer);
+	if (err) {
+		mlx5_core_dbg(dev, "FWTracer: Failed to query capabilities %d\n", err);
+		return err;
+	}
+
+	err = mlx5_fw_tracer_allocate_strings_db(tracer);
+	if (err) {
+		mlx5_core_warn(dev, "FWTracer: Allocate strings DB failed %d\n", err);
+		return err;
+	}
+	mlx5_fw_tracer_init_saved_traces_array(tracer);
+
+	return 0;
+}
+
 static int fw_tracer_event(struct notifier_block *nb, unsigned long action, void *data)
 {
 	struct mlx5_fw_tracer *tracer = mlx5_nb_cof(nb, struct mlx5_fw_tracer, nb);
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/diag/fw_tracer.h b/drivers/net/ethernet/mellanox/mlx5/core/diag/fw_tracer.h
index 40601fba80ba..1a755098aeeb 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/diag/fw_tracer.h
+++ b/drivers/net/ethernet/mellanox/mlx5/core/diag/fw_tracer.h
@@ -191,5 +191,6 @@ void mlx5_fw_tracer_destroy(struct mlx5_fw_tracer *tracer);
 int mlx5_fw_tracer_trigger_core_dump_general(struct mlx5_core_dev *dev);
 int mlx5_fw_tracer_get_saved_traces_objects(struct mlx5_fw_tracer *tracer,
 					    struct devlink_fmsg *fmsg);
+int mlx5_fw_tracer_recreate_strings_db(struct mlx5_fw_tracer *tracer);
 
 #endif
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/fw_reset.c b/drivers/net/ethernet/mellanox/mlx5/core/fw_reset.c
index 88d59cc059bd..7a4563c43827 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/fw_reset.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/fw_reset.c
@@ -2,6 +2,7 @@
 /* Copyright (c) 2020, Mellanox Technologies inc.  All rights reserved. */
 
 #include "fw_reset.h"
+#include "diag/fw_tracer.h"
 
 enum {
 	MLX5_FW_RESET_FLAGS_RESET_REQUESTED,
@@ -13,6 +14,7 @@ struct mlx5_fw_reset {
 	struct mlx5_core_dev *dev;
 	struct mlx5_nb nb;
 	struct workqueue_struct *wq;
+	struct work_struct fw_live_patch_work;
 	struct work_struct reset_request_work;
 	struct work_struct reset_reload_work;
 	struct work_struct reset_now_work;
@@ -188,6 +190,27 @@ static void mlx5_sync_reset_set_reset_requested(struct mlx5_core_dev *dev)
 	mlx5_start_sync_reset_poll(dev);
 }
 
+static void mlx5_fw_live_patch_event(struct work_struct *work)
+{
+	struct mlx5_fw_reset *fw_reset = container_of(work, struct mlx5_fw_reset,
+						      fw_live_patch_work);
+	struct mlx5_core_dev *dev = fw_reset->dev;
+	struct mlx5_fw_tracer *tracer;
+
+	mlx5_core_info(dev, "Live patch updated firmware version: %d.%d.%d\n", fw_rev_maj(dev),
+		       fw_rev_min(dev), fw_rev_sub(dev));
+
+	tracer = dev->tracer;
+	if (IS_ERR_OR_NULL(tracer))
+		return;
+
+	mlx5_fw_tracer_cleanup(tracer);
+	if (mlx5_fw_tracer_recreate_strings_db(tracer))
+		mlx5_core_err(dev, "Failed to recreate FW tracer strings DB\n");
+	if (mlx5_fw_tracer_init(tracer))
+		mlx5_core_err(dev, "Failed to re-initialize FW tracer\n");
+}
+
 static void mlx5_sync_reset_request_event(struct work_struct *work)
 {
 	struct mlx5_fw_reset *fw_reset = container_of(work, struct mlx5_fw_reset,
@@ -361,6 +384,9 @@ static int fw_reset_event_notifier(struct notifier_block *nb, unsigned long acti
 	struct mlx5_eqe *eqe = data;
 
 	switch (eqe->sub_type) {
+	case MLX5_GENERAL_SUBTYPE_FW_LIVE_PATCH_EVENT:
+			queue_work(fw_reset->wq, &fw_reset->fw_live_patch_work);
+		break;
 	case MLX5_GENERAL_SUBTYPE_PCI_SYNC_FOR_FW_UPDATE_EVENT:
 		mlx5_sync_reset_events_handle(fw_reset, eqe);
 		break;
@@ -405,6 +431,7 @@ int mlx5_fw_reset_events_init(struct mlx5_core_dev *dev)
 	fw_reset->dev = dev;
 	dev->priv.fw_reset = fw_reset;
 
+	INIT_WORK(&fw_reset->fw_live_patch_work, mlx5_fw_live_patch_event);
 	INIT_WORK(&fw_reset->reset_request_work, mlx5_sync_reset_request_event);
 	INIT_WORK(&fw_reset->reset_reload_work, mlx5_sync_reset_reload_work);
 	INIT_WORK(&fw_reset->reset_now_work, mlx5_sync_reset_now_event);
diff --git a/include/linux/mlx5/device.h b/include/linux/mlx5/device.h
index 81ca5989009b..cf824366a7d1 100644
--- a/include/linux/mlx5/device.h
+++ b/include/linux/mlx5/device.h
@@ -366,6 +366,7 @@ enum {
 enum {
 	MLX5_GENERAL_SUBTYPE_DELAY_DROP_TIMEOUT = 0x1,
 	MLX5_GENERAL_SUBTYPE_PCI_POWER_CHANGE_EVENT = 0x5,
+	MLX5_GENERAL_SUBTYPE_FW_LIVE_PATCH_EVENT = 0x7,
 	MLX5_GENERAL_SUBTYPE_PCI_SYNC_FOR_FW_UPDATE_EVENT = 0x8,
 };
 
-- 
2.17.1

[PATCH net-next RFC v4 14/15] net/mlx5: Add support for devlink reload action limit level no reset

[Moshe Shemesh]

Add support for devlink reload action fw_activate with limit level
no_reset which does firmware live patching, updating the firmware image
without reset, no downtime and no configuration lose. The driver checks
if the firmware is capable of handling the pending firmware changes as a
live patch. If it is then it triggers firmware live patching flow.

Signed-off-by: Moshe Shemesh <moshe@mellanox.com>
---
v3 -> v4:
- Have action fw_activate with limit level no_reset instead of action
  fw_activate_no_reset
v2 -> v3:
- Replace fw_live_patch action by fw_activate_no_reset
v1 -> v2:
- Have fw_live_patch action instead of level
---
 .../net/ethernet/mellanox/mlx5/core/devlink.c | 37 ++++++++++++++++++-
 1 file changed, 36 insertions(+), 1 deletion(-)

diff --git a/drivers/net/ethernet/mellanox/mlx5/core/devlink.c b/drivers/net/ethernet/mellanox/mlx5/core/devlink.c
index 175e933cf19c..f2cc62b7232d 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/devlink.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/devlink.c
@@ -115,6 +115,29 @@ static int mlx5_devlink_reload_fw_activate(struct devlink *devlink, struct netli
 	return err;
 }
 
+static int mlx5_devlink_trigger_fw_live_patch(struct devlink *devlink,
+					      struct netlink_ext_ack *extack)
+{
+	struct mlx5_core_dev *dev = devlink_priv(devlink);
+	u8 reset_level;
+	int err;
+
+	err = mlx5_reg_mfrl_query(dev, &reset_level, NULL);
+	if (err)
+		return err;
+	if (!(reset_level & MLX5_MFRL_REG_RESET_LEVEL0)) {
+		NL_SET_ERR_MSG_MOD(extack,
+				   "FW upgrade to the stored FW can't be done by FW live patching");
+		return -EINVAL;
+	}
+
+	err = mlx5_fw_set_live_patch(dev);
+	if (err)
+		return err;
+
+	return 0;
+}
+
 static int mlx5_devlink_reload_down(struct devlink *devlink, bool netns_change,
 				    enum devlink_reload_action action,
 				    enum devlink_reload_action_limit_level limit_level,
@@ -122,11 +145,19 @@ static int mlx5_devlink_reload_down(struct devlink *devlink, bool netns_change,
 {
 	struct mlx5_core_dev *dev = devlink_priv(devlink);
 
+	if (limit_level == DEVLINK_RELOAD_ACTION_LIMIT_LEVEL_NO_RESET &&
+	    action != DEVLINK_RELOAD_ACTION_FW_ACTIVATE) {
+		NL_SET_ERR_MSG_MOD(extack, "Requested limit level is not supported");
+		return -EOPNOTSUPP;
+	}
+
 	switch (action) {
 	case DEVLINK_RELOAD_ACTION_DRIVER_REINIT:
 		mlx5_unload_one(dev, false);
 		return 0;
 	case DEVLINK_RELOAD_ACTION_FW_ACTIVATE:
+		if (limit_level == DEVLINK_RELOAD_ACTION_LIMIT_LEVEL_NO_RESET)
+			return mlx5_devlink_trigger_fw_live_patch(devlink, extack);
 		return mlx5_devlink_reload_fw_activate(devlink, extack);
 	default:
 		/* Unsupported action should not get to this function */
@@ -146,7 +177,10 @@ static int mlx5_devlink_reload_up(struct devlink *devlink, enum devlink_reload_a
 
 	switch (action) {
 	case DEVLINK_RELOAD_ACTION_DRIVER_REINIT:
+		return mlx5_load_one(dev, false);
 	case DEVLINK_RELOAD_ACTION_FW_ACTIVATE:
+		if (limit_level == DEVLINK_RELOAD_ACTION_LIMIT_LEVEL_NO_RESET)
+			break;
 		/* On fw_activate action, also driver is reloaded and reinit performed */
 		if (actions_performed)
 			*actions_performed |= BIT(DEVLINK_RELOAD_ACTION_DRIVER_REINIT);
@@ -175,7 +209,8 @@ static const struct devlink_ops mlx5_devlink_ops = {
 	.info_get = mlx5_devlink_info_get,
 	.supported_reload_actions = BIT(DEVLINK_RELOAD_ACTION_DRIVER_REINIT) |
 				    BIT(DEVLINK_RELOAD_ACTION_FW_ACTIVATE),
-	.supported_reload_action_limit_levels = BIT(DEVLINK_RELOAD_ACTION_LIMIT_LEVEL_NONE),
+	.supported_reload_action_limit_levels = BIT(DEVLINK_RELOAD_ACTION_LIMIT_LEVEL_NONE) |
+						BIT(DEVLINK_RELOAD_ACTION_LIMIT_LEVEL_NO_RESET),
 	.reload_down = mlx5_devlink_reload_down,
 	.reload_up = mlx5_devlink_reload_up,
 };
-- 
2.17.1

[PATCH net-next RFC v4 15/15] devlink: Add Documentation/networking/devlink/devlink-reload.rst

[Moshe Shemesh]

Add devlink reload rst documentation file.
Update index file to include it.

Signed-off-by: Moshe Shemesh <moshe@mellanox.com>
---
v3 -> v4:
- Remove reload action fw_activate_no_reset
- Add reload actions limit levels and document the no_reset limit level
  constrains
v2 -> v3:
- Devlink reload returns the actions done
- Replace fw_live_patch action by fw_activate_no_reset
- Explain fw_activate meaning
v1 -> v2:
- Instead of reload levels driver,fw_reset,fw_live_patch have reload
  actions driver_reinit,fw_activate,fw_live_patch
---
 .../networking/devlink/devlink-reload.rst     | 80 +++++++++++++++++++
 Documentation/networking/devlink/index.rst    |  1 +
 2 files changed, 81 insertions(+)
 create mode 100644 Documentation/networking/devlink/devlink-reload.rst

diff --git a/Documentation/networking/devlink/devlink-reload.rst b/Documentation/networking/devlink/devlink-reload.rst
new file mode 100644
index 000000000000..6ac9dddd2208
--- /dev/null
+++ b/Documentation/networking/devlink/devlink-reload.rst
@@ -0,0 +1,80 @@
+.. SPDX-License-Identifier: GPL-2.0
+
+==============
+Devlink Reload
+==============
+
+``devlink-reload`` provides mechanism to either reload driver entities,
+applying ``devlink-params`` and ``devlink-resources`` new values or firmware
+activation depends on reload action selected.
+
+Reload actions
+==============
+
+User may select a reload action.
+By default ``driver_reinit`` action is selected.
+
+.. list-table:: Possible reload actions
+   :widths: 5 90
+
+   * - Name
+     - Description
+   * - ``driver-reinit``
+     - Devlink driver entities re-initialization, including applying
+       new values to devlink entities which are used during driver
+       load such as ``devlink-params`` in configuration mode
+       ``driverinit`` or ``devlink-resources``
+   * - ``fw_activate``
+     - Firmware activate. Activates new firmware if such image is stored and
+       pending activation. This action involves firmware reset, if no new image
+       pending this action will reload current firmware image.
+
+Note that when required to do firmware activation some drivers may need
+to reload the driver. On the other hand some drivers may need to reset
+the firmware to reinitialize the driver entities. Therefore, the devlink
+reload command returns the actions which were actually performed.
+
+Reload action limit levels
+==========================
+
+By default reload actions are not limited and Driver implementation may
+include reset or downtime as needed to perform the actions.
+
+However, some drivers support action limit levels, which limits the action
+implementation to specific constrains.
+
+.. list-table:: Possible reload action limit levels
+   :widths: 5 90
+
+   * - Name
+     - Description
+   * - ``no_reset``
+     - No reset allowed, no down time allowed, no link flap and no
+       configuration is lost.
+
+Change namespace
+================
+
+All devlink instances are created in init_net and stay there for a
+lifetime. Allow user to be able to move devlink instances into
+namespaces during devlink reload operation. That ensures proper
+re-instantiation of driver objects, including netdevices.
+
+example usage
+-------------
+
+.. code:: shell
+
+    $ devlink dev reload help
+    $ devlink dev reload DEV [ netns { PID | NAME | ID } ] [ action { driver_reinit | fw_activate } ] [limit_level no_reset]
+
+    # Run reload command for devlink driver entities re-initialization:
+    $ devlink dev reload pci/0000:82:00.0 action driver_reinit
+    reload_actions_performed:
+      driver_reinit
+
+    # Run reload command to activate firmware:
+    # Note that mlx5 driver reloads the driver while activating firmware
+    $ devlink dev reload pci/0000:82:00.0 action fw_activate
+    reload_actions_performed:
+      driver_reinit fw_activate
diff --git a/Documentation/networking/devlink/index.rst b/Documentation/networking/devlink/index.rst
index 7684ae5c4a4a..d82874760ae2 100644
--- a/Documentation/networking/devlink/index.rst
+++ b/Documentation/networking/devlink/index.rst
@@ -20,6 +20,7 @@ general.
    devlink-params
    devlink-region
    devlink-resource
+   devlink-reload
    devlink-trap
 
 Driver-specific documentation
-- 
2.17.1

Re: [PATCH net-next RFC v4 01/15] devlink: Add reload action option to devlink reload command

[Vasundhara Volam]

On Mon, Sep 14, 2020 at 11:39 AM Moshe Shemesh <moshe@mellanox.com> wrote:
>
> Add devlink reload action to allow the user to request a specific reload
> action. The action parameter is optional, if not specified then devlink
> driver re-init action is used (backward compatible).
> Note that when required to do firmware activation some drivers may need
> to reload the driver. On the other hand some drivers may need to reset
> the firmware to reinitialize the driver entities. Therefore, the devlink
> reload command returns the actions which were actually performed.
> Reload actions supported are:
> driver_reinit: driver entities re-initialization, applying devlink-param
>                and devlink-resource values.
> fw_activate: firmware activate.
>
> command examples:
> $devlink dev reload pci/0000:82:00.0 action driver_reinit
> reload_actions_performed:
>   driver_reinit
>
> $devlink dev reload pci/0000:82:00.0 action fw_activate
> reload_actions_performed:
>   driver_reinit fw_activate
>
> Signed-off-by: Moshe Shemesh <moshe@mellanox.com>
> ---
> v3 -> v4:
> - Removed fw_activate_no_reset as an action (next patch adds limit
>   levels instead).
> - Renamed actions_done to actions_performed
> v2 -> v3:
> - Replace fw_live_patch action by fw_activate_no_reset
> - Devlink reload returns the actions done over netlink reply
> v1 -> v2:
> - Instead of reload levels driver,fw_reset,fw_live_patch have reload
>   actions driver_reinit,fw_activate,fw_live_patch
> - Remove driver default level, the action driver_reinit is the default
>   action for all drivers
> ---
>  drivers/net/ethernet/mellanox/mlx4/main.c     |  14 ++-
>  .../net/ethernet/mellanox/mlx5/core/devlink.c |  15 ++-
>  drivers/net/ethernet/mellanox/mlxsw/core.c    |  25 ++--
>  drivers/net/netdevsim/dev.c                   |  16 ++-
>  include/net/devlink.h                         |   7 +-
>  include/uapi/linux/devlink.h                  |  19 +++
>  net/core/devlink.c                            | 111 +++++++++++++++++-
>  7 files changed, 180 insertions(+), 27 deletions(-)
>
> diff --git a/drivers/net/ethernet/mellanox/mlx4/main.c b/drivers/net/ethernet/mellanox/mlx4/main.c
> index 70cf24ba71e4..aadf1676a0ed 100644
> --- a/drivers/net/ethernet/mellanox/mlx4/main.c
> +++ b/drivers/net/ethernet/mellanox/mlx4/main.c
> @@ -3946,6 +3946,7 @@ static int mlx4_restart_one_up(struct pci_dev *pdev, bool reload,
>                                struct devlink *devlink);
>
>  static int mlx4_devlink_reload_down(struct devlink *devlink, bool netns_change,
> +                                   enum devlink_reload_action action,
>                                     struct netlink_ext_ack *extack)
>  {
>         struct mlx4_priv *priv = devlink_priv(devlink);
> @@ -3962,8 +3963,8 @@ static int mlx4_devlink_reload_down(struct devlink *devlink, bool netns_change,
>         return 0;
>  }
>
> -static int mlx4_devlink_reload_up(struct devlink *devlink,
> -                                 struct netlink_ext_ack *extack)
> +static int mlx4_devlink_reload_up(struct devlink *devlink, enum devlink_reload_action action,
> +                                 struct netlink_ext_ack *extack, unsigned long *actions_performed)
>  {
>         struct mlx4_priv *priv = devlink_priv(devlink);
>         struct mlx4_dev *dev = &priv->dev;
> @@ -3971,15 +3972,20 @@ static int mlx4_devlink_reload_up(struct devlink *devlink,
>         int err;
>
>         err = mlx4_restart_one_up(persist->pdev, true, devlink);
> -       if (err)
> +       if (err) {
>                 mlx4_err(persist->dev, "mlx4_restart_one_up failed, ret=%d\n",
>                          err);
> +               return err;
> +       }
> +       if (actions_performed)
> +               *actions_performed = BIT(DEVLINK_RELOAD_ACTION_DRIVER_REINIT);
>
> -       return err;
> +       return 0;
>  }
>
>  static const struct devlink_ops mlx4_devlink_ops = {
>         .port_type_set  = mlx4_devlink_port_type_set,
> +       .supported_reload_actions = BIT(DEVLINK_RELOAD_ACTION_DRIVER_REINIT),
>         .reload_down    = mlx4_devlink_reload_down,
>         .reload_up      = mlx4_devlink_reload_up,
>  };
> diff --git a/drivers/net/ethernet/mellanox/mlx5/core/devlink.c b/drivers/net/ethernet/mellanox/mlx5/core/devlink.c
> index c709e9a385f6..9cd6b6c884e3 100644
> --- a/drivers/net/ethernet/mellanox/mlx5/core/devlink.c
> +++ b/drivers/net/ethernet/mellanox/mlx5/core/devlink.c
> @@ -89,6 +89,7 @@ mlx5_devlink_info_get(struct devlink *devlink, struct devlink_info_req *req,
>  }
>
>  static int mlx5_devlink_reload_down(struct devlink *devlink, bool netns_change,
> +                                   enum devlink_reload_action action,
>                                     struct netlink_ext_ack *extack)
>  {
>         struct mlx5_core_dev *dev = devlink_priv(devlink);
> @@ -97,12 +98,19 @@ static int mlx5_devlink_reload_down(struct devlink *devlink, bool netns_change,
>         return 0;
>  }
>
> -static int mlx5_devlink_reload_up(struct devlink *devlink,
> -                                 struct netlink_ext_ack *extack)
> +static int mlx5_devlink_reload_up(struct devlink *devlink, enum devlink_reload_action action,
> +                                 struct netlink_ext_ack *extack, unsigned long *actions_performed)
>  {
>         struct mlx5_core_dev *dev = devlink_priv(devlink);
> +       int err;
>
> -       return mlx5_load_one(dev, false);
> +       err = mlx5_load_one(dev, false);
> +       if (err)
> +               return err;
> +       if (actions_performed)
> +               *actions_performed = BIT(DEVLINK_RELOAD_ACTION_DRIVER_REINIT);
> +
> +       return 0;
>  }
>
>  static const struct devlink_ops mlx5_devlink_ops = {
> @@ -118,6 +126,7 @@ static const struct devlink_ops mlx5_devlink_ops = {
>  #endif
>         .flash_update = mlx5_devlink_flash_update,
>         .info_get = mlx5_devlink_info_get,
> +       .supported_reload_actions = BIT(DEVLINK_RELOAD_ACTION_DRIVER_REINIT),
>         .reload_down = mlx5_devlink_reload_down,
>         .reload_up = mlx5_devlink_reload_up,
>  };
> diff --git a/drivers/net/ethernet/mellanox/mlxsw/core.c b/drivers/net/ethernet/mellanox/mlxsw/core.c
> index ec45a03140d7..c0a32f685b85 100644
> --- a/drivers/net/ethernet/mellanox/mlxsw/core.c
> +++ b/drivers/net/ethernet/mellanox/mlxsw/core.c
> @@ -1113,7 +1113,7 @@ mlxsw_devlink_info_get(struct devlink *devlink, struct devlink_info_req *req,
>
>  static int
>  mlxsw_devlink_core_bus_device_reload_down(struct devlink *devlink,
> -                                         bool netns_change,
> +                                         bool netns_change, enum devlink_reload_action action,
>                                           struct netlink_ext_ack *extack)
>  {
>         struct mlxsw_core *mlxsw_core = devlink_priv(devlink);
> @@ -1126,15 +1126,24 @@ mlxsw_devlink_core_bus_device_reload_down(struct devlink *devlink,
>  }
>
>  static int
> -mlxsw_devlink_core_bus_device_reload_up(struct devlink *devlink,
> -                                       struct netlink_ext_ack *extack)
> +mlxsw_devlink_core_bus_device_reload_up(struct devlink *devlink, enum devlink_reload_action action,
> +                                       struct netlink_ext_ack *extack,
> +                                       unsigned long *actions_performed)
Sorry for repeating again, for fw_activate action on our device, all
the driver entities undergo reset asynchronously once user initiates
"devlink dev reload action fw_activate" and reload_up does not have
much to do except reporting actions that will be/being performed.

Once reset is complete, the health reporter will be notified using
devlink_health_reporter_recovery_done(). Returning from reload_up does
not guarantee successful activation of firmware. Status of reset will
be notified to the health reporter via
devlink_health_reporter_state_update().

I am just repeating this, so I want to know if I am on the same page.

Thanks.
>  {
>         struct mlxsw_core *mlxsw_core = devlink_priv(devlink);
> +       int err;
>
> -       return mlxsw_core_bus_device_register(mlxsw_core->bus_info,
> -                                             mlxsw_core->bus,
> -                                             mlxsw_core->bus_priv, true,
> -                                             devlink, extack);
> +       err = mlxsw_core_bus_device_register(mlxsw_core->bus_info,
> +                                            mlxsw_core->bus,
> +                                            mlxsw_core->bus_priv, true,
> +                                            devlink, extack);
> +       if (err)
> +               return err;
> +       if (actions_performed)
> +               *actions_performed = BIT(DEVLINK_RELOAD_ACTION_DRIVER_REINIT) |
> +                                    BIT(DEVLINK_RELOAD_ACTION_FW_ACTIVATE);
> +
> +       return 0;
>  }
>
>  static int mlxsw_devlink_flash_update(struct devlink *devlink,
> @@ -1268,6 +1277,8 @@ mlxsw_devlink_trap_policer_counter_get(struct devlink *devlink,
>  }
>
>  static const struct devlink_ops mlxsw_devlink_ops = {
> +       .supported_reload_actions       = BIT(DEVLINK_RELOAD_ACTION_DRIVER_REINIT) |
> +                                         BIT(DEVLINK_RELOAD_ACTION_FW_ACTIVATE),
What if drivers do not support DRIVER_REINIT operation directly but
when users issue FW_ACTIVATE, driver performs both FW_ACTIVATE and
DRIVER_REINIT? That particular driver has to advertise only
DRIVER_REINIT action, right?

>         .reload_down            = mlxsw_devlink_core_bus_device_reload_down,
>         .reload_up              = mlxsw_devlink_core_bus_device_reload_up,
>         .port_type_set                  = mlxsw_devlink_port_type_set,
> diff --git a/drivers/net/netdevsim/dev.c b/drivers/net/netdevsim/dev.c
> index 32f339fedb21..f0919fa0cd8b 100644
> --- a/drivers/net/netdevsim/dev.c
> +++ b/drivers/net/netdevsim/dev.c
> @@ -697,7 +697,7 @@ static int nsim_dev_reload_create(struct nsim_dev *nsim_dev,
>  static void nsim_dev_reload_destroy(struct nsim_dev *nsim_dev);
>
>  static int nsim_dev_reload_down(struct devlink *devlink, bool netns_change,
> -                               struct netlink_ext_ack *extack)
> +                               enum devlink_reload_action action, struct netlink_ext_ack *extack)
>  {
>         struct nsim_dev *nsim_dev = devlink_priv(devlink);
>
> @@ -713,10 +713,11 @@ static int nsim_dev_reload_down(struct devlink *devlink, bool netns_change,
>         return 0;
>  }
>
> -static int nsim_dev_reload_up(struct devlink *devlink,
> -                             struct netlink_ext_ack *extack)
> +static int nsim_dev_reload_up(struct devlink *devlink, enum devlink_reload_action action,
> +                             struct netlink_ext_ack *extack, unsigned long *actions_performed)
>  {
>         struct nsim_dev *nsim_dev = devlink_priv(devlink);
> +       int err;
>
>         if (nsim_dev->fail_reload) {
>                 /* For testing purposes, user set debugfs fail_reload
> @@ -726,7 +727,13 @@ static int nsim_dev_reload_up(struct devlink *devlink,
>                 return -EINVAL;
>         }
>
> -       return nsim_dev_reload_create(nsim_dev, extack);
> +       err = nsim_dev_reload_create(nsim_dev, extack);
> +       if (err)
> +               return err;
> +       if (actions_performed)
> +               *actions_performed = BIT(DEVLINK_RELOAD_ACTION_DRIVER_REINIT);
> +
> +       return 0;
>  }
>
>  static int nsim_dev_info_get(struct devlink *devlink,
> @@ -875,6 +882,7 @@ nsim_dev_devlink_trap_policer_counter_get(struct devlink *devlink,
>  }
>
>  static const struct devlink_ops nsim_dev_devlink_ops = {
> +       .supported_reload_actions = BIT(DEVLINK_RELOAD_ACTION_DRIVER_REINIT),
>         .reload_down = nsim_dev_reload_down,
>         .reload_up = nsim_dev_reload_up,
>         .info_get = nsim_dev_info_get,
> diff --git a/include/net/devlink.h b/include/net/devlink.h
> index eaec0a8cc5ef..b09db891db04 100644
> --- a/include/net/devlink.h
> +++ b/include/net/devlink.h
> @@ -1011,10 +1011,11 @@ enum devlink_trap_group_generic_id {
>         }
>
>  struct devlink_ops {
> +       unsigned long supported_reload_actions;
>         int (*reload_down)(struct devlink *devlink, bool netns_change,
> -                          struct netlink_ext_ack *extack);
> -       int (*reload_up)(struct devlink *devlink,
> -                        struct netlink_ext_ack *extack);
> +                          enum devlink_reload_action action, struct netlink_ext_ack *extack);
> +       int (*reload_up)(struct devlink *devlink, enum devlink_reload_action action,
> +                        struct netlink_ext_ack *extack, unsigned long *actions_performed);
>         int (*port_type_set)(struct devlink_port *devlink_port,
>                              enum devlink_port_type port_type);
>         int (*port_split)(struct devlink *devlink, unsigned int port_index,
> diff --git a/include/uapi/linux/devlink.h b/include/uapi/linux/devlink.h
> index 40d35145c879..a6f64db0bdf3 100644
> --- a/include/uapi/linux/devlink.h
> +++ b/include/uapi/linux/devlink.h
> @@ -272,6 +272,21 @@ enum {
>         DEVLINK_ATTR_TRAP_METADATA_TYPE_FA_COOKIE,
>  };
>
> +/**
> + * enum devlink_reload_action - Reload action.
> + * @DEVLINK_RELOAD_ACTION_DRIVER_REINIT: Driver entities re-instantiation.
> + * @DEVLINK_RELOAD_ACTION_FW_ACTIVATE: FW activate.
> + */
> +enum devlink_reload_action {
> +       DEVLINK_RELOAD_ACTION_UNSPEC,
> +       DEVLINK_RELOAD_ACTION_DRIVER_REINIT,
> +       DEVLINK_RELOAD_ACTION_FW_ACTIVATE,
> +
> +       /* Add new reload actions above */
> +       __DEVLINK_RELOAD_ACTION_MAX,
> +       DEVLINK_RELOAD_ACTION_MAX = __DEVLINK_RELOAD_ACTION_MAX - 1
> +};
> +
>  enum devlink_attr {
>         /* don't change the order or add anything between, this is ABI! */
>         DEVLINK_ATTR_UNSPEC,
> @@ -460,6 +475,10 @@ enum devlink_attr {
>
>         DEVLINK_ATTR_PORT_EXTERNAL,             /* u8 */
>         DEVLINK_ATTR_PORT_CONTROLLER_NUMBER,    /* u32 */
> +
> +       DEVLINK_ATTR_RELOAD_ACTION,             /* u8 */
> +       DEVLINK_ATTR_RELOAD_ACTIONS_PERFORMED,  /* nested */
> +
>         /* add new attributes above here, update the policy in devlink.c */
>
>         __DEVLINK_ATTR_MAX,
> diff --git a/net/core/devlink.c b/net/core/devlink.c
> index 19037f114307..f4be1e1bf864 100644
> --- a/net/core/devlink.c
> +++ b/net/core/devlink.c
> @@ -462,6 +462,12 @@ static int devlink_nl_put_handle(struct sk_buff *msg, struct devlink *devlink)
>         return 0;
>  }
>
> +static bool
> +devlink_reload_action_is_supported(struct devlink *devlink, enum devlink_reload_action action)
> +{
> +       return test_bit(action, &devlink->ops->supported_reload_actions);
> +}
> +
>  static int devlink_nl_fill(struct sk_buff *msg, struct devlink *devlink,
>                            enum devlink_command cmd, u32 portid,
>                            u32 seq, int flags)
> @@ -2969,29 +2975,72 @@ bool devlink_is_reload_failed(const struct devlink *devlink)
>  EXPORT_SYMBOL_GPL(devlink_is_reload_failed);
>
>  static int devlink_reload(struct devlink *devlink, struct net *dest_net,
> -                         struct netlink_ext_ack *extack)
> +                         enum devlink_reload_action action, struct netlink_ext_ack *extack,
> +                         unsigned long *actions_performed)
>  {
>         int err;
>
>         if (!devlink->reload_enabled)
>                 return -EOPNOTSUPP;
>
> -       err = devlink->ops->reload_down(devlink, !!dest_net, extack);
> +       err = devlink->ops->reload_down(devlink, !!dest_net, action, extack);
>         if (err)
>                 return err;
>
>         if (dest_net && !net_eq(dest_net, devlink_net(devlink)))
>                 devlink_reload_netns_change(devlink, dest_net);
>
> -       err = devlink->ops->reload_up(devlink, extack);
> +       err = devlink->ops->reload_up(devlink, action, extack, actions_performed);
>         devlink_reload_failed_set(devlink, !!err);
>         return err;
>  }
>
> +static int
> +devlink_nl_reload_actions_performed_fill(struct sk_buff *msg,
> +                                        struct devlink *devlink,
> +                                        unsigned long actions_performed,
> +                                        enum devlink_command cmd, u32 portid,
> +                                        u32 seq, int flags)
> +{
> +       struct nlattr *actions_performed_attr;
> +       void *hdr;
> +       int i;
> +
> +       hdr = genlmsg_put(msg, portid, seq, &devlink_nl_family, flags, cmd);
> +       if (!hdr)
> +               return -EMSGSIZE;
> +
> +       if (devlink_nl_put_handle(msg, devlink))
> +               goto genlmsg_cancel;
> +
> +       actions_performed_attr = nla_nest_start(msg, DEVLINK_ATTR_RELOAD_ACTIONS_PERFORMED);
> +       if (!actions_performed_attr)
> +               goto genlmsg_cancel;
> +
> +       for (i = 0; i <= DEVLINK_RELOAD_ACTION_MAX; i++) {
> +               if (!test_bit(i, &actions_performed))
> +                       continue;
> +               if (nla_put_u8(msg, DEVLINK_ATTR_RELOAD_ACTION, i))
> +                       goto actions_performed_nest_cancel;
> +       }
> +       nla_nest_end(msg, actions_performed_attr);
> +       genlmsg_end(msg, hdr);
> +       return 0;
> +
> +actions_performed_nest_cancel:
> +       nla_nest_cancel(msg, actions_performed_attr);
> +genlmsg_cancel:
> +       genlmsg_cancel(msg, hdr);
> +       return -EMSGSIZE;
> +}
> +
>  static int devlink_nl_cmd_reload(struct sk_buff *skb, struct genl_info *info)
>  {
>         struct devlink *devlink = info->user_ptr[0];
> +       enum devlink_reload_action action;
> +       unsigned long actions_performed;
>         struct net *dest_net = NULL;
> +       struct sk_buff *msg;
>         int err;
>
>         if (!devlink_reload_supported(devlink))
> @@ -3011,12 +3060,41 @@ static int devlink_nl_cmd_reload(struct sk_buff *skb, struct genl_info *info)
>                         return PTR_ERR(dest_net);
>         }
>
> -       err = devlink_reload(devlink, dest_net, info->extack);
> +       if (info->attrs[DEVLINK_ATTR_RELOAD_ACTION])
> +               action = nla_get_u8(info->attrs[DEVLINK_ATTR_RELOAD_ACTION]);
> +       else
> +               action = DEVLINK_RELOAD_ACTION_DRIVER_REINIT;
> +
> +       if (action == DEVLINK_RELOAD_ACTION_UNSPEC || action > DEVLINK_RELOAD_ACTION_MAX) {
> +               NL_SET_ERR_MSG_MOD(info->extack, "Invalid reload action");
> +               return -EINVAL;
> +       } else if (!devlink_reload_action_is_supported(devlink, action)) {
> +               NL_SET_ERR_MSG_MOD(info->extack, "Requested reload action is not supported");
> +               return -EOPNOTSUPP;
> +       }
> +
> +       err = devlink_reload(devlink, dest_net, action, info->extack, &actions_performed);
>
>         if (dest_net)
>                 put_net(dest_net);
>
> -       return err;
> +       if (err)
> +               return err;
> +
> +       WARN_ON(!actions_performed);
> +       msg = nlmsg_new(NLMSG_DEFAULT_SIZE, GFP_KERNEL);
> +       if (!msg)
> +               return -ENOMEM;
> +
> +       err = devlink_nl_reload_actions_performed_fill(msg, devlink, actions_performed,
> +                                                      DEVLINK_CMD_RELOAD, info->snd_portid,
> +                                                      info->snd_seq, 0);
> +       if (err) {
> +               nlmsg_free(msg);
> +               return err;
> +       }
> +
> +       return genlmsg_reply(msg, info);
>  }
>
>  static int devlink_nl_flash_update_fill(struct sk_buff *msg,
> @@ -7047,6 +7125,7 @@ static const struct nla_policy devlink_nl_policy[DEVLINK_ATTR_MAX + 1] = {
>         [DEVLINK_ATTR_TRAP_POLICER_RATE] = { .type = NLA_U64 },
>         [DEVLINK_ATTR_TRAP_POLICER_BURST] = { .type = NLA_U64 },
>         [DEVLINK_ATTR_PORT_FUNCTION] = { .type = NLA_NESTED },
> +       [DEVLINK_ATTR_RELOAD_ACTION] = { .type = NLA_U8 },
>  };
>
>  static const struct genl_ops devlink_nl_ops[] = {
> @@ -7372,6 +7451,20 @@ static struct genl_family devlink_nl_family __ro_after_init = {
>         .n_mcgrps       = ARRAY_SIZE(devlink_nl_mcgrps),
>  };
>
> +static int devlink_reload_actions_verify(struct devlink *devlink)
> +{
> +       const struct devlink_ops *ops;
> +
> +       if (!devlink_reload_supported(devlink))
> +               return 0;
> +
> +       ops = devlink->ops;
> +       if (WARN_ON(ops->supported_reload_actions >= BIT(__DEVLINK_RELOAD_ACTION_MAX) ||
> +                   ops->supported_reload_actions <= BIT(DEVLINK_RELOAD_ACTION_UNSPEC)))
> +               return -EINVAL;
> +       return 0;
> +}
> +
>  /**
>   *     devlink_alloc - Allocate new devlink instance resources
>   *
> @@ -7392,6 +7485,11 @@ struct devlink *devlink_alloc(const struct devlink_ops *ops, size_t priv_size)
>         if (!devlink)
>                 return NULL;
>         devlink->ops = ops;
> +       if (devlink_reload_actions_verify(devlink)) {
> +               kfree(devlink);
> +               return NULL;
> +       }
> +
>         xa_init_flags(&devlink->snapshot_ids, XA_FLAGS_ALLOC);
>         __devlink_net_set(devlink, &init_net);
>         INIT_LIST_HEAD(&devlink->port_list);
> @@ -9657,7 +9755,8 @@ static void __net_exit devlink_pernet_pre_exit(struct net *net)
>                 if (net_eq(devlink_net(devlink), net)) {
>                         if (WARN_ON(!devlink_reload_supported(devlink)))
>                                 continue;
> -                       err = devlink_reload(devlink, &init_net, NULL);
> +                       err = devlink_reload(devlink, &init_net,
> +                                            DEVLINK_RELOAD_ACTION_DRIVER_REINIT, NULL, NULL);
>                         if (err && err != -EOPNOTSUPP)
>                                 pr_warn("Failed to reload devlink instance into init_net\n");
>                 }
> --
> 2.17.1
>

Re: [PATCH net-next RFC v4 01/15] devlink: Add reload action option to devlink reload command

[Jiri Pirko]

Mon, Sep 14, 2020 at 09:08:58AM CEST, vasundhara-v.volam@broadcom.com wrote:
>On Mon, Sep 14, 2020 at 11:39 AM Moshe Shemesh <moshe@mellanox.com> wrote:

[...]


>> @@ -1126,15 +1126,24 @@ mlxsw_devlink_core_bus_device_reload_down(struct devlink *devlink,
>>  }
>>
>>  static int
>> -mlxsw_devlink_core_bus_device_reload_up(struct devlink *devlink,
>> -                                       struct netlink_ext_ack *extack)
>> +mlxsw_devlink_core_bus_device_reload_up(struct devlink *devlink, enum devlink_reload_action action,
>> +                                       struct netlink_ext_ack *extack,
>> +                                       unsigned long *actions_performed)
>Sorry for repeating again, for fw_activate action on our device, all
>the driver entities undergo reset asynchronously once user initiates
>"devlink dev reload action fw_activate" and reload_up does not have
>much to do except reporting actions that will be/being performed.
>
>Once reset is complete, the health reporter will be notified using

Hmm, how is the fw reset related to health reporter recovery? Recovery
happens after some error event. I don't believe it is wise to mix it.

Instead, why don't you block in reload_up() until the reset is complete?


>devlink_health_reporter_recovery_done(). Returning from reload_up does
>not guarantee successful activation of firmware. Status of reset will
>be notified to the health reporter via
>devlink_health_reporter_state_update().
>
>I am just repeating this, so I want to know if I am on the same page.
>
>Thanks.

[...]

Re: [PATCH net-next RFC v4 01/15] devlink: Add reload action option to devlink reload command

[Vasundhara Volam]

On Mon, Sep 14, 2020 at 3:02 PM Jiri Pirko <jiri@resnulli.us> wrote:
>
> Mon, Sep 14, 2020 at 09:08:58AM CEST, vasundhara-v.volam@broadcom.com wrote:
> >On Mon, Sep 14, 2020 at 11:39 AM Moshe Shemesh <moshe@mellanox.com> wrote:
>
> [...]
>
>
> >> @@ -1126,15 +1126,24 @@ mlxsw_devlink_core_bus_device_reload_down(struct devlink *devlink,
> >>  }
> >>
> >>  static int
> >> -mlxsw_devlink_core_bus_device_reload_up(struct devlink *devlink,
> >> -                                       struct netlink_ext_ack *extack)
> >> +mlxsw_devlink_core_bus_device_reload_up(struct devlink *devlink, enum devlink_reload_action action,
> >> +                                       struct netlink_ext_ack *extack,
> >> +                                       unsigned long *actions_performed)
> >Sorry for repeating again, for fw_activate action on our device, all
> >the driver entities undergo reset asynchronously once user initiates
> >"devlink dev reload action fw_activate" and reload_up does not have
> >much to do except reporting actions that will be/being performed.
> >
> >Once reset is complete, the health reporter will be notified using
>
> Hmm, how is the fw reset related to health reporter recovery? Recovery
> happens after some error event. I don't believe it is wise to mix it.
Our device has a fw_reset health reporter, which is updated on reset
events and firmware activation is one among them. All non-fatal
firmware reset events are reported on fw_reset health reporter.

>
> Instead, why don't you block in reload_up() until the reset is complete?

Though user initiate "devlink dev reload" event on a single interface,
all driver entities undergo reset and all entities recover
independently. I don't think we can block the reload_up() on the
interface(that user initiated the command), until whole reset is
complete.
>
>
> >devlink_health_reporter_recovery_done(). Returning from reload_up does
> >not guarantee successful activation of firmware. Status of reset will
> >be notified to the health reporter via
> >devlink_health_reporter_state_update().
> >
> >I am just repeating this, so I want to know if I am on the same page.
> >
> >Thanks.
>
> [...]

Re: [PATCH net-next RFC v4 01/15] devlink: Add reload action option to devlink reload command

[Jiri Pirko]

Mon, Sep 14, 2020 at 11:54:55AM CEST, vasundhara-v.volam@broadcom.com wrote:
>On Mon, Sep 14, 2020 at 3:02 PM Jiri Pirko <jiri@resnulli.us> wrote:
>>
>> Mon, Sep 14, 2020 at 09:08:58AM CEST, vasundhara-v.volam@broadcom.com wrote:
>> >On Mon, Sep 14, 2020 at 11:39 AM Moshe Shemesh <moshe@mellanox.com> wrote:
>>
>> [...]
>>
>>
>> >> @@ -1126,15 +1126,24 @@ mlxsw_devlink_core_bus_device_reload_down(struct devlink *devlink,
>> >>  }
>> >>
>> >>  static int
>> >> -mlxsw_devlink_core_bus_device_reload_up(struct devlink *devlink,
>> >> -                                       struct netlink_ext_ack *extack)
>> >> +mlxsw_devlink_core_bus_device_reload_up(struct devlink *devlink, enum devlink_reload_action action,
>> >> +                                       struct netlink_ext_ack *extack,
>> >> +                                       unsigned long *actions_performed)
>> >Sorry for repeating again, for fw_activate action on our device, all
>> >the driver entities undergo reset asynchronously once user initiates
>> >"devlink dev reload action fw_activate" and reload_up does not have
>> >much to do except reporting actions that will be/being performed.
>> >
>> >Once reset is complete, the health reporter will be notified using
>>
>> Hmm, how is the fw reset related to health reporter recovery? Recovery
>> happens after some error event. I don't believe it is wise to mix it.
>Our device has a fw_reset health reporter, which is updated on reset
>events and firmware activation is one among them. All non-fatal
>firmware reset events are reported on fw_reset health reporter.

Hmm, interesting. In that case, assuming this is fine, should we have
some standard in this. I mean, if the driver supports reset, should it
also define the "fw_reset" reporter to report such events?

Jakub, what is your take here?


>
>>
>> Instead, why don't you block in reload_up() until the reset is complete?
>
>Though user initiate "devlink dev reload" event on a single interface,
>all driver entities undergo reset and all entities recover
>independently. I don't think we can block the reload_up() on the
>interface(that user initiated the command), until whole reset is
>complete.

Why not? mlxsw reset takes up to like 10 seconds for example.


>>
>>
>> >devlink_health_reporter_recovery_done(). Returning from reload_up does
>> >not guarantee successful activation of firmware. Status of reset will
>> >be notified to the health reporter via
>> >devlink_health_reporter_state_update().
>> >
>> >I am just repeating this, so I want to know if I am on the same page.
>> >
>> >Thanks.
>>
>> [...]

Re: [PATCH net-next RFC v4 15/15] devlink: Add Documentation/networking/devlink/devlink-reload.rst

[Jiri Pirko]

Mon, Sep 14, 2020 at 08:08:02AM CEST, moshe@mellanox.com wrote:
>Add devlink reload rst documentation file.
>Update index file to include it.
>
>Signed-off-by: Moshe Shemesh <moshe@mellanox.com>
>---
>v3 -> v4:
>- Remove reload action fw_activate_no_reset
>- Add reload actions limit levels and document the no_reset limit level
>  constrains
>v2 -> v3:
>- Devlink reload returns the actions done
>- Replace fw_live_patch action by fw_activate_no_reset
>- Explain fw_activate meaning
>v1 -> v2:
>- Instead of reload levels driver,fw_reset,fw_live_patch have reload
>  actions driver_reinit,fw_activate,fw_live_patch
>---
> .../networking/devlink/devlink-reload.rst     | 80 +++++++++++++++++++
> Documentation/networking/devlink/index.rst    |  1 +
> 2 files changed, 81 insertions(+)
> create mode 100644 Documentation/networking/devlink/devlink-reload.rst
>
>diff --git a/Documentation/networking/devlink/devlink-reload.rst b/Documentation/networking/devlink/devlink-reload.rst
>new file mode 100644
>index 000000000000..6ac9dddd2208
>--- /dev/null
>+++ b/Documentation/networking/devlink/devlink-reload.rst
>@@ -0,0 +1,80 @@
>+.. SPDX-License-Identifier: GPL-2.0
>+
>+==============
>+Devlink Reload
>+==============
>+
>+``devlink-reload`` provides mechanism to either reload driver entities,
>+applying ``devlink-params`` and ``devlink-resources`` new values or firmware
>+activation depends on reload action selected.
>+
>+Reload actions
>+==============
>+
>+User may select a reload action.
>+By default ``driver_reinit`` action is selected.
>+
>+.. list-table:: Possible reload actions
>+   :widths: 5 90
>+
>+   * - Name
>+     - Description
>+   * - ``driver-reinit``
>+     - Devlink driver entities re-initialization, including applying
>+       new values to devlink entities which are used during driver
>+       load such as ``devlink-params`` in configuration mode
>+       ``driverinit`` or ``devlink-resources``
>+   * - ``fw_activate``
>+     - Firmware activate. Activates new firmware if such image is stored and
>+       pending activation. This action involves firmware reset, if no new image
>+       pending this action will reload current firmware image.
>+
>+Note that when required to do firmware activation some drivers may need
>+to reload the driver. On the other hand some drivers may need to reset

s/reload/reinit" ?

>+the firmware to reinitialize the driver entities. Therefore, the devlink
>+reload command returns the actions which were actually performed.

I would perhaps say something more generic like:
Note that even though user asks for a specific action, the driver
implementation might require to perform another action alongside with
it. For example, some driver do not support driver reinitialization
being performed without fw activation. Therefore, the devlink reload
command return the list of actions which were actrually performed.


>+
>+Reload action limit levels
>+==========================
>+
>+By default reload actions are not limited and Driver implementation may

Why capital "D"?


>+include reset or downtime as needed to perform the actions.
>+
>+However, some drivers support action limit levels, which limits the action
>+implementation to specific constrains.
>+
>+.. list-table:: Possible reload action limit levels
>+   :widths: 5 90
>+
>+   * - Name
>+     - Description
>+   * - ``no_reset``
>+     - No reset allowed, no down time allowed, no link flap and no
>+       configuration is lost.
>+
>+Change namespace
>+================
>+
>+All devlink instances are created in init_net and stay there for a
>+lifetime. Allow user to be able to move devlink instances into
>+namespaces during devlink reload operation. That ensures proper
>+re-instantiation of driver objects, including netdevices.

This sounds like a commit message :) Could you please re-phrase a bit?


>+
>+example usage
>+-------------
>+
>+.. code:: shell
>+
>+    $ devlink dev reload help
>+    $ devlink dev reload DEV [ netns { PID | NAME | ID } ] [ action { driver_reinit | fw_activate } ] [limit_level no_reset]
>+
>+    # Run reload command for devlink driver entities re-initialization:
>+    $ devlink dev reload pci/0000:82:00.0 action driver_reinit
>+    reload_actions_performed:
>+      driver_reinit
>+
>+    # Run reload command to activate firmware:
>+    # Note that mlx5 driver reloads the driver while activating firmware
>+    $ devlink dev reload pci/0000:82:00.0 action fw_activate
>+    reload_actions_performed:
>+      driver_reinit fw_activate

This looks fine to me.


>diff --git a/Documentation/networking/devlink/index.rst b/Documentation/networking/devlink/index.rst
>index 7684ae5c4a4a..d82874760ae2 100644
>--- a/Documentation/networking/devlink/index.rst
>+++ b/Documentation/networking/devlink/index.rst
>@@ -20,6 +20,7 @@ general.
>    devlink-params
>    devlink-region
>    devlink-resource
>+   devlink-reload
>    devlink-trap
> 
> Driver-specific documentation
>-- 
>2.17.1
>

Re: [PATCH net-next RFC v4 01/15] devlink: Add reload action option to devlink reload command

[Jiri Pirko]

Mon, Sep 14, 2020 at 08:07:48AM CEST, moshe@mellanox.com wrote:
>Add devlink reload action to allow the user to request a specific reload
>action. The action parameter is optional, if not specified then devlink
>driver re-init action is used (backward compatible).
>Note that when required to do firmware activation some drivers may need
>to reload the driver. On the other hand some drivers may need to reset
>the firmware to reinitialize the driver entities. Therefore, the devlink
>reload command returns the actions which were actually performed.
>Reload actions supported are:
>driver_reinit: driver entities re-initialization, applying devlink-param
>               and devlink-resource values.
>fw_activate: firmware activate.
>
>command examples:
>$devlink dev reload pci/0000:82:00.0 action driver_reinit
>reload_actions_performed:
>  driver_reinit
>
>$devlink dev reload pci/0000:82:00.0 action fw_activate
>reload_actions_performed:
>  driver_reinit fw_activate
>
>Signed-off-by: Moshe Shemesh <moshe@mellanox.com>
>---
>v3 -> v4:
>- Removed fw_activate_no_reset as an action (next patch adds limit
>  levels instead).
>- Renamed actions_done to actions_performed
>v2 -> v3:
>- Replace fw_live_patch action by fw_activate_no_reset
>- Devlink reload returns the actions done over netlink reply
>v1 -> v2:
>- Instead of reload levels driver,fw_reset,fw_live_patch have reload
>  actions driver_reinit,fw_activate,fw_live_patch
>- Remove driver default level, the action driver_reinit is the default
>  action for all drivers
>---
> drivers/net/ethernet/mellanox/mlx4/main.c     |  14 ++-
> .../net/ethernet/mellanox/mlx5/core/devlink.c |  15 ++-
> drivers/net/ethernet/mellanox/mlxsw/core.c    |  25 ++--
> drivers/net/netdevsim/dev.c                   |  16 ++-
> include/net/devlink.h                         |   7 +-
> include/uapi/linux/devlink.h                  |  19 +++
> net/core/devlink.c                            | 111 +++++++++++++++++-
> 7 files changed, 180 insertions(+), 27 deletions(-)
>
>diff --git a/drivers/net/ethernet/mellanox/mlx4/main.c b/drivers/net/ethernet/mellanox/mlx4/main.c
>index 70cf24ba71e4..aadf1676a0ed 100644
>--- a/drivers/net/ethernet/mellanox/mlx4/main.c
>+++ b/drivers/net/ethernet/mellanox/mlx4/main.c
>@@ -3946,6 +3946,7 @@ static int mlx4_restart_one_up(struct pci_dev *pdev, bool reload,
> 			       struct devlink *devlink);
> 
> static int mlx4_devlink_reload_down(struct devlink *devlink, bool netns_change,
>+				    enum devlink_reload_action action,
> 				    struct netlink_ext_ack *extack)
> {
> 	struct mlx4_priv *priv = devlink_priv(devlink);
>@@ -3962,8 +3963,8 @@ static int mlx4_devlink_reload_down(struct devlink *devlink, bool netns_change,
> 	return 0;
> }
> 
>-static int mlx4_devlink_reload_up(struct devlink *devlink,
>-				  struct netlink_ext_ack *extack)
>+static int mlx4_devlink_reload_up(struct devlink *devlink, enum devlink_reload_action action,
>+				  struct netlink_ext_ack *extack, unsigned long *actions_performed)
> {
> 	struct mlx4_priv *priv = devlink_priv(devlink);
> 	struct mlx4_dev *dev = &priv->dev;
>@@ -3971,15 +3972,20 @@ static int mlx4_devlink_reload_up(struct devlink *devlink,
> 	int err;
> 
> 	err = mlx4_restart_one_up(persist->pdev, true, devlink);
>-	if (err)
>+	if (err) {
> 		mlx4_err(persist->dev, "mlx4_restart_one_up failed, ret=%d\n",
> 			 err);
>+		return err;
>+	}
>+	if (actions_performed)

Nit, pass the unsigned long allways (even when it would be unused) and
avoid check in every driver.


>+		*actions_performed = BIT(DEVLINK_RELOAD_ACTION_DRIVER_REINIT);
> 
>-	return err;
>+	return 0;
> }
> 
> static const struct devlink_ops mlx4_devlink_ops = {
> 	.port_type_set	= mlx4_devlink_port_type_set,
>+	.supported_reload_actions = BIT(DEVLINK_RELOAD_ACTION_DRIVER_REINIT),
> 	.reload_down	= mlx4_devlink_reload_down,
> 	.reload_up	= mlx4_devlink_reload_up,
> };

[..]


>@@ -2969,29 +2975,72 @@ bool devlink_is_reload_failed(const struct devlink *devlink)
> EXPORT_SYMBOL_GPL(devlink_is_reload_failed);
> 
> static int devlink_reload(struct devlink *devlink, struct net *dest_net,
>-			  struct netlink_ext_ack *extack)
>+			  enum devlink_reload_action action, struct netlink_ext_ack *extack,
>+			  unsigned long *actions_performed)
> {
> 	int err;
> 
> 	if (!devlink->reload_enabled)
> 		return -EOPNOTSUPP;
> 
>-	err = devlink->ops->reload_down(devlink, !!dest_net, extack);
>+	err = devlink->ops->reload_down(devlink, !!dest_net, action, extack);
> 	if (err)
> 		return err;
> 
> 	if (dest_net && !net_eq(dest_net, devlink_net(devlink)))
> 		devlink_reload_netns_change(devlink, dest_net);
> 
>-	err = devlink->ops->reload_up(devlink, extack);
>+	err = devlink->ops->reload_up(devlink, action, extack, actions_performed);

Here, please add a WARN_ON() check:
	WARN_ON(!(*actions_performed & action));

The requested action should be always performed.


> 	devlink_reload_failed_set(devlink, !!err);
> 	return err;
> }
> 
>+static int
>+devlink_nl_reload_actions_performed_fill(struct sk_buff *msg,
>+					 struct devlink *devlink,
>+					 unsigned long actions_performed,
>+					 enum devlink_command cmd, u32 portid,
>+					 u32 seq, int flags)
>+{
>+	struct nlattr *actions_performed_attr;
>+	void *hdr;
>+	int i;
>+
>+	hdr = genlmsg_put(msg, portid, seq, &devlink_nl_family, flags, cmd);
>+	if (!hdr)
>+		return -EMSGSIZE;
>+
>+	if (devlink_nl_put_handle(msg, devlink))
>+		goto genlmsg_cancel;
>+
>+	actions_performed_attr = nla_nest_start(msg, DEVLINK_ATTR_RELOAD_ACTIONS_PERFORMED);
>+	if (!actions_performed_attr)
>+		goto genlmsg_cancel;
>+
>+	for (i = 0; i <= DEVLINK_RELOAD_ACTION_MAX; i++) {
>+		if (!test_bit(i, &actions_performed))
>+			continue;
>+		if (nla_put_u8(msg, DEVLINK_ATTR_RELOAD_ACTION, i))
>+			goto actions_performed_nest_cancel;
>+	}
>+	nla_nest_end(msg, actions_performed_attr);
>+	genlmsg_end(msg, hdr);
>+	return 0;
>+
>+actions_performed_nest_cancel:
>+	nla_nest_cancel(msg, actions_performed_attr);
>+genlmsg_cancel:
>+	genlmsg_cancel(msg, hdr);
>+	return -EMSGSIZE;
>+}
>+
> static int devlink_nl_cmd_reload(struct sk_buff *skb, struct genl_info *info)
> {
> 	struct devlink *devlink = info->user_ptr[0];
>+	enum devlink_reload_action action;
>+	unsigned long actions_performed;
> 	struct net *dest_net = NULL;
>+	struct sk_buff *msg;
> 	int err;
> 
> 	if (!devlink_reload_supported(devlink))
>@@ -3011,12 +3060,41 @@ static int devlink_nl_cmd_reload(struct sk_buff *skb, struct genl_info *info)
> 			return PTR_ERR(dest_net);
> 	}
> 
>-	err = devlink_reload(devlink, dest_net, info->extack);
>+	if (info->attrs[DEVLINK_ATTR_RELOAD_ACTION])
>+		action = nla_get_u8(info->attrs[DEVLINK_ATTR_RELOAD_ACTION]);
>+	else
>+		action = DEVLINK_RELOAD_ACTION_DRIVER_REINIT;
>+
>+	if (action == DEVLINK_RELOAD_ACTION_UNSPEC || action > DEVLINK_RELOAD_ACTION_MAX) {
>+		NL_SET_ERR_MSG_MOD(info->extack, "Invalid reload action");

Hmm, I understand the unspec check, but the max check is not needed. The
following check will take care of it.


>+		return -EINVAL;
>+	} else if (!devlink_reload_action_is_supported(devlink, action)) {
>+		NL_SET_ERR_MSG_MOD(info->extack, "Requested reload action is not supported");

".. by the driver" ?
	

>+		return -EOPNOTSUPP;
>+	}
>+
>+	err = devlink_reload(devlink, dest_net, action, info->extack, &actions_performed);
> 
> 	if (dest_net)
> 		put_net(dest_net);
> 
>-	return err;
>+	if (err)
>+		return err;
>+
>+	WARN_ON(!actions_performed);
>+	msg = nlmsg_new(NLMSG_DEFAULT_SIZE, GFP_KERNEL);
>+	if (!msg)
>+		return -ENOMEM;
>+
>+	err = devlink_nl_reload_actions_performed_fill(msg, devlink, actions_performed,
>+						       DEVLINK_CMD_RELOAD, info->snd_portid,
>+						       info->snd_seq, 0);
>+	if (err) {
>+		nlmsg_free(msg);
>+		return err;
>+	}
>+
>+	return genlmsg_reply(msg, info);
> }
> 
> static int devlink_nl_flash_update_fill(struct sk_buff *msg,
>@@ -7047,6 +7125,7 @@ static const struct nla_policy devlink_nl_policy[DEVLINK_ATTR_MAX + 1] = {
> 	[DEVLINK_ATTR_TRAP_POLICER_RATE] = { .type = NLA_U64 },
> 	[DEVLINK_ATTR_TRAP_POLICER_BURST] = { .type = NLA_U64 },
> 	[DEVLINK_ATTR_PORT_FUNCTION] = { .type = NLA_NESTED },
>+	[DEVLINK_ATTR_RELOAD_ACTION] = { .type = NLA_U8 },
> };
> 
> static const struct genl_ops devlink_nl_ops[] = {
>@@ -7372,6 +7451,20 @@ static struct genl_family devlink_nl_family __ro_after_init = {
> 	.n_mcgrps	= ARRAY_SIZE(devlink_nl_mcgrps),
> };
> 
>+static int devlink_reload_actions_verify(struct devlink *devlink)
>+{
>+	const struct devlink_ops *ops;
>+
>+	if (!devlink_reload_supported(devlink))

If reload is not supported, the supported_reload_actions should be 0.
Please check that with WARN_ON too.


>+		return 0;
>+
>+	ops = devlink->ops;
>+	if (WARN_ON(ops->supported_reload_actions >= BIT(__DEVLINK_RELOAD_ACTION_MAX) ||
>+		    ops->supported_reload_actions <= BIT(DEVLINK_RELOAD_ACTION_UNSPEC)))
>+		return -EINVAL;
>+	return 0;
>+}
>+
> /**
>  *	devlink_alloc - Allocate new devlink instance resources
>  *
>@@ -7392,6 +7485,11 @@ struct devlink *devlink_alloc(const struct devlink_ops *ops, size_t priv_size)
> 	if (!devlink)
> 		return NULL;
> 	devlink->ops = ops;
>+	if (devlink_reload_actions_verify(devlink)) {

Move this check to the beginning. You don't need devlink instance for
the check, just ops.

also, your devlink_reload_actions_verify() function returns
0/-ESOMETHING. Treat it accordingly here.


>+		kfree(devlink);
>+		return NULL;
>+	}
>+
> 	xa_init_flags(&devlink->snapshot_ids, XA_FLAGS_ALLOC);
> 	__devlink_net_set(devlink, &init_net);
> 	INIT_LIST_HEAD(&devlink->port_list);
>@@ -9657,7 +9755,8 @@ static void __net_exit devlink_pernet_pre_exit(struct net *net)
> 		if (net_eq(devlink_net(devlink), net)) {
> 			if (WARN_ON(!devlink_reload_supported(devlink)))
> 				continue;
>-			err = devlink_reload(devlink, &init_net, NULL);
>+			err = devlink_reload(devlink, &init_net,
>+					     DEVLINK_RELOAD_ACTION_DRIVER_REINIT, NULL, NULL);
> 			if (err && err != -EOPNOTSUPP)
> 				pr_warn("Failed to reload devlink instance into init_net\n");
> 		}
>-- 
>2.17.1
>

Re: [PATCH net-next RFC v4 02/15] devlink: Add reload action limit level

[Jiri Pirko]

Mon, Sep 14, 2020 at 08:07:49AM CEST, moshe@mellanox.com wrote:
		
[..]

	 
>diff --git a/include/net/devlink.h b/include/net/devlink.h
>index b09db891db04..dddd9ee5b8a9 100644
>--- a/include/net/devlink.h
>+++ b/include/net/devlink.h
>@@ -1012,9 +1012,13 @@ enum devlink_trap_group_generic_id {
> 
> struct devlink_ops {
> 	unsigned long supported_reload_actions;
>+	unsigned long supported_reload_action_limit_levels;
> 	int (*reload_down)(struct devlink *devlink, bool netns_change,
>-			   enum devlink_reload_action action, struct netlink_ext_ack *extack);
>+			   enum devlink_reload_action action,
>+			   enum devlink_reload_action_limit_level limit_level,
>+			   struct netlink_ext_ack *extack);
> 	int (*reload_up)(struct devlink *devlink, enum devlink_reload_action action,
>+			 enum devlink_reload_action_limit_level limit_level,
> 			 struct netlink_ext_ack *extack, unsigned long *actions_performed);
> 	int (*port_type_set)(struct devlink_port *devlink_port,
> 			     enum devlink_port_type port_type);
>diff --git a/include/uapi/linux/devlink.h b/include/uapi/linux/devlink.h
>index a6f64db0bdf3..b19686fd80ff 100644
>--- a/include/uapi/linux/devlink.h
>+++ b/include/uapi/linux/devlink.h
>@@ -287,6 +287,22 @@ enum devlink_reload_action {
> 	DEVLINK_RELOAD_ACTION_MAX = __DEVLINK_RELOAD_ACTION_MAX - 1
> };
> 
>+/**
>+ * enum devlink_reload_action_limit_level - Reload action limit level.
>+ * @DEVLINK_RELOAD_ACTION_LIMIT_LEVEL_NONE: No constrains on action. Action may include
>+ *                                          reset or downtime as needed.
>+ * @DEVLINK_RELOAD_ACTION_LIMIT_LEVEL_NO_RESET: No reset allowed, no down time allowed,
>+ *                                              no link flap and no configuration is lost.
>+ */
>+enum devlink_reload_action_limit_level {
>+	DEVLINK_RELOAD_ACTION_LIMIT_LEVEL_NONE,
>+	DEVLINK_RELOAD_ACTION_LIMIT_LEVEL_NO_RESET,
>+
>+	/* Add new reload actions limit level above */
>+	__DEVLINK_RELOAD_ACTION_LIMIT_LEVEL_MAX,
>+	DEVLINK_RELOAD_ACTION_LIMIT_LEVEL_MAX = __DEVLINK_RELOAD_ACTION_LIMIT_LEVEL_MAX - 1
>+};
>+
> enum devlink_attr {
> 	/* don't change the order or add anything between, this is ABI! */
> 	DEVLINK_ATTR_UNSPEC,
>@@ -478,6 +494,7 @@ enum devlink_attr {
> 
> 	DEVLINK_ATTR_RELOAD_ACTION,		/* u8 */
> 	DEVLINK_ATTR_RELOAD_ACTIONS_PERFORMED,	/* nested */
>+	DEVLINK_ATTR_RELOAD_ACTION_LIMIT_LEVEL,	/* u8 */
> 
> 	/* add new attributes above here, update the policy in devlink.c */
> 
>diff --git a/net/core/devlink.c b/net/core/devlink.c
>index f4be1e1bf864..60aa0c4a3726 100644
>--- a/net/core/devlink.c
>+++ b/net/core/devlink.c
>@@ -468,6 +468,13 @@ devlink_reload_action_is_supported(struct devlink *devlink, enum devlink_reload_
> 	return test_bit(action, &devlink->ops->supported_reload_actions);
> }
> 
>+static bool
>+devlink_reload_action_limit_level_is_supported(struct devlink *devlink,
>+					       enum devlink_reload_action_limit_level limit_level)
>+{
>+	return test_bit(limit_level, &devlink->ops->supported_reload_action_limit_levels);
>+}
>+
> static int devlink_nl_fill(struct sk_buff *msg, struct devlink *devlink,
> 			   enum devlink_command cmd, u32 portid,
> 			   u32 seq, int flags)
>@@ -2975,22 +2982,23 @@ bool devlink_is_reload_failed(const struct devlink *devlink)
> EXPORT_SYMBOL_GPL(devlink_is_reload_failed);
> 
> static int devlink_reload(struct devlink *devlink, struct net *dest_net,
>-			  enum devlink_reload_action action, struct netlink_ext_ack *extack,
>-			  unsigned long *actions_performed)
>+			  enum devlink_reload_action action,
>+			  enum devlink_reload_action_limit_level limit_level,
>+			  struct netlink_ext_ack *extack, unsigned long *actions_performed)
> {
> 	int err;
> 
> 	if (!devlink->reload_enabled)
> 		return -EOPNOTSUPP;
> 
>-	err = devlink->ops->reload_down(devlink, !!dest_net, action, extack);
>+	err = devlink->ops->reload_down(devlink, !!dest_net, action, limit_level, extack);
> 	if (err)
> 		return err;
> 
> 	if (dest_net && !net_eq(dest_net, devlink_net(devlink)))
> 		devlink_reload_netns_change(devlink, dest_net);
> 
>-	err = devlink->ops->reload_up(devlink, action, extack, actions_performed);
>+	err = devlink->ops->reload_up(devlink, action, limit_level, extack, actions_performed);
> 	devlink_reload_failed_set(devlink, !!err);
> 	return err;
> }
>@@ -3036,6 +3044,7 @@ devlink_nl_reload_actions_performed_fill(struct sk_buff *msg,
> 
> static int devlink_nl_cmd_reload(struct sk_buff *skb, struct genl_info *info)
> {
>+	enum devlink_reload_action_limit_level limit_level;
> 	struct devlink *devlink = info->user_ptr[0];
> 	enum devlink_reload_action action;
> 	unsigned long actions_performed;
>@@ -3073,7 +3082,20 @@ static int devlink_nl_cmd_reload(struct sk_buff *skb, struct genl_info *info)
> 		return -EOPNOTSUPP;
> 	}
> 
>-	err = devlink_reload(devlink, dest_net, action, info->extack, &actions_performed);
>+	if (info->attrs[DEVLINK_ATTR_RELOAD_ACTION_LIMIT_LEVEL])
>+		limit_level = nla_get_u8(info->attrs[DEVLINK_ATTR_RELOAD_ACTION_LIMIT_LEVEL]);
>+	else
>+		limit_level = DEVLINK_RELOAD_ACTION_LIMIT_LEVEL_NONE;
>+
>+	if (limit_level > DEVLINK_RELOAD_ACTION_LIMIT_LEVEL_MAX) {

Again, not needed, devlink_reload_action_limit_level_is_supported() will
take case of it.

>+		NL_SET_ERR_MSG_MOD(info->extack, "Invalid limit level");
>+		return -EINVAL;
>+	} else if (!devlink_reload_action_limit_level_is_supported(devlink, limit_level)) {
>+		NL_SET_ERR_MSG_MOD(info->extack, "Requested limit level is not supported");

"..by the driver"?


>+		return -EOPNOTSUPP;
>+	}
>+	err = devlink_reload(devlink, dest_net, action, limit_level, info->extack,
>+			     &actions_performed);
> 
> 	if (dest_net)
> 		put_net(dest_net);
>@@ -7126,6 +7148,7 @@ static const struct nla_policy devlink_nl_policy[DEVLINK_ATTR_MAX + 1] = {
> 	[DEVLINK_ATTR_TRAP_POLICER_BURST] = { .type = NLA_U64 },
> 	[DEVLINK_ATTR_PORT_FUNCTION] = { .type = NLA_NESTED },
> 	[DEVLINK_ATTR_RELOAD_ACTION] = { .type = NLA_U8 },
>+	[DEVLINK_ATTR_RELOAD_ACTION_LIMIT_LEVEL] = { .type = NLA_U8 },
> };
> 
> static const struct genl_ops devlink_nl_ops[] = {
>@@ -7462,6 +7485,10 @@ static int devlink_reload_actions_verify(struct devlink *devlink)
> 	if (WARN_ON(ops->supported_reload_actions >= BIT(__DEVLINK_RELOAD_ACTION_MAX) ||
> 		    ops->supported_reload_actions <= BIT(DEVLINK_RELOAD_ACTION_UNSPEC)))
> 		return -EINVAL;
>+	if (WARN_ON(!ops->supported_reload_action_limit_levels ||
>+		    ops->supported_reload_action_limit_levels >=
>+		    BIT(__DEVLINK_RELOAD_ACTION_LIMIT_LEVEL_MAX)))
>+		return -EINVAL;

I think that you can check some insane driver combinations like:
supports only driver-reinit, supports LEVEL_NO_RESET - that is
impossible and should be refused here.

Same goes to the actual user command call. If the user calls for
driver-reinit with LEVEL_NO_RESET, devlink should refuse with proper
extack


> 	return 0;
> }
> 
>@@ -9756,7 +9783,8 @@ static void __net_exit devlink_pernet_pre_exit(struct net *net)
> 			if (WARN_ON(!devlink_reload_supported(devlink)))
> 				continue;
> 			err = devlink_reload(devlink, &init_net,
>-					     DEVLINK_RELOAD_ACTION_DRIVER_REINIT, NULL, NULL);
>+					     DEVLINK_RELOAD_ACTION_DRIVER_REINIT,
>+					     DEVLINK_RELOAD_ACTION_LIMIT_LEVEL_NONE, NULL, NULL);
> 			if (err && err != -EOPNOTSUPP)
> 				pr_warn("Failed to reload devlink instance into init_net\n");
> 		}
>-- 
>2.17.1
>

Re: [PATCH net-next RFC v4 03/15] devlink: Add reload action stats

[Jiri Pirko]

Mon, Sep 14, 2020 at 08:07:50AM CEST, moshe@mellanox.com wrote:
>Add reload action stats to hold the history per reload action type and
>limit level.

Empty line missing.


>For example, the number of times fw_activate has been performed on this
>device since the driver module was added or if the firmware activation
>was performed with or without reset.
>Add devlink notification on stats update.
>
>The function devlink_reload_actions_implicit_actions_performed() is
>exported to enable also drivers update on reload actions performed,
>for example in case firmware activation with reset finished
>successfully but was initiated by remote host.
>
>Signed-off-by: Moshe Shemesh <moshe@mellanox.com>
>---
>v3 -> v4:
>- Renamed reload_actions_cnts to reload_action_stats
>- Add devlink notifications on stats update
>- Renamed devlink_reload_actions_implicit_actions_performed() and add
>  function comment in code
>v2 -> v3:
>- New patch
>---
> include/net/devlink.h |  7 ++++++
> net/core/devlink.c    | 58 ++++++++++++++++++++++++++++++++++++++++---
> 2 files changed, 62 insertions(+), 3 deletions(-)
>
>diff --git a/include/net/devlink.h b/include/net/devlink.h
>index dddd9ee5b8a9..b4feb92e0269 100644
>--- a/include/net/devlink.h
>+++ b/include/net/devlink.h
>@@ -20,6 +20,9 @@
> #include <uapi/linux/devlink.h>
> #include <linux/xarray.h>
> 
>+#define DEVLINK_RELOAD_ACTION_STATS_ARRAY_SIZE \
>+	(__DEVLINK_RELOAD_ACTION_LIMIT_LEVEL_MAX * __DEVLINK_RELOAD_ACTION_MAX)
>+
> struct devlink_ops;
> 
> struct devlink {
>@@ -38,6 +41,7 @@ struct devlink {
> 	struct list_head trap_policer_list;
> 	const struct devlink_ops *ops;
> 	struct xarray snapshot_ids;
>+	u32 reload_action_stats[DEVLINK_RELOAD_ACTION_STATS_ARRAY_SIZE];
> 	struct device *dev;
> 	possible_net_t _net;
> 	struct mutex lock; /* Serializes access to devlink instance specific objects such as
>@@ -1397,6 +1401,9 @@ void
> devlink_health_reporter_recovery_done(struct devlink_health_reporter *reporter);
> 
> bool devlink_is_reload_failed(const struct devlink *devlink);
>+void devlink_reload_implicit_actions_performed(struct devlink *devlink,
>+					       enum devlink_reload_action_limit_level limit_level,
>+					       unsigned long actions_performed);
> 
> void devlink_flash_update_begin_notify(struct devlink *devlink);
> void devlink_flash_update_end_notify(struct devlink *devlink);
>diff --git a/net/core/devlink.c b/net/core/devlink.c
>index 60aa0c4a3726..cbf746966913 100644
>--- a/net/core/devlink.c
>+++ b/net/core/devlink.c
>@@ -2981,11 +2981,58 @@ bool devlink_is_reload_failed(const struct devlink *devlink)
> }
> EXPORT_SYMBOL_GPL(devlink_is_reload_failed);
> 
>+static void
>+devlink_reload_action_stats_update(struct devlink *devlink,
>+				   enum devlink_reload_action_limit_level limit_level,
>+				   unsigned long actions_performed)
>+{
>+	int stat_idx;
>+	int action;
>+
>+	if (!actions_performed)
>+		return;
>+
>+	if (WARN_ON(limit_level > DEVLINK_RELOAD_ACTION_LIMIT_LEVEL_MAX))
>+		return;
>+	for (action = 0; action <= DEVLINK_RELOAD_ACTION_MAX; action++) {
>+		if (!test_bit(action, &actions_performed))
>+			continue;
>+		stat_idx = limit_level * __DEVLINK_RELOAD_ACTION_MAX + action;
>+		devlink->reload_action_stats[stat_idx]++;
>+	}
>+	devlink_notify(devlink, DEVLINK_CMD_NEW);
>+}
>+
>+/**
>+ *	devlink_reload_implicit_actions_performed - Update devlink on reload actions
>+ *	  performed which are not a direct result of devlink reload call.
>+ *
>+ *	This should be called by a driver after performing reload actions in case it was not
>+ *	a result of devlink reload call. For example fw_activate was performed as a result
>+ *	of devlink reload triggered fw_activate on another host.
>+ *	The motivation for this function is to keep data on reload actions performed on this
>+ *	function whether it was done due to direct devlink reload call or not.
>+ *
>+ *	@devlink: devlink
>+ *	@limit_level: reload action limit level
>+ *	@actions_performed: bitmask of actions performed
>+ */
>+void devlink_reload_implicit_actions_performed(struct devlink *devlink,
>+					       enum devlink_reload_action_limit_level limit_level,
>+					       unsigned long actions_performed)

What I'm a bit scarred of that the driver would call this from withing
reload_down()/up() ops. Perheps this could be WARN_ON'ed here (or in
devlink_reload())?


>+{
>+	if (!devlink_reload_supported(devlink))

Hmm. I think that the driver does not have to support the reload and can
still be reloaded by another instance and update the stats here. Why
not?


>+		return;
>+	devlink_reload_action_stats_update(devlink, limit_level, actions_performed);
>+}
>+EXPORT_SYMBOL_GPL(devlink_reload_implicit_actions_performed);
>+
> static int devlink_reload(struct devlink *devlink, struct net *dest_net,
> 			  enum devlink_reload_action action,
> 			  enum devlink_reload_action_limit_level limit_level,
>-			  struct netlink_ext_ack *extack, unsigned long *actions_performed)
>+			  struct netlink_ext_ack *extack, unsigned long *actions_performed_out)
> {
>+	unsigned long actions_performed;
> 	int err;
> 
> 	if (!devlink->reload_enabled)
>@@ -2998,9 +3045,14 @@ static int devlink_reload(struct devlink *devlink, struct net *dest_net,
> 	if (dest_net && !net_eq(dest_net, devlink_net(devlink)))
> 		devlink_reload_netns_change(devlink, dest_net);
> 
>-	err = devlink->ops->reload_up(devlink, action, limit_level, extack, actions_performed);
>+	err = devlink->ops->reload_up(devlink, action, limit_level, extack, &actions_performed);
> 	devlink_reload_failed_set(devlink, !!err);
>-	return err;
>+	if (err)
>+		return err;
>+	devlink_reload_action_stats_update(devlink, limit_level, actions_performed);
>+	if (actions_performed_out)

Just make the caller to provide valid pointer, as I suggested in the
other patch review.


>+		*actions_performed_out = actions_performed;
>+	return 0;
> }
> 
> static int
>-- 
>2.17.1
>

Re: [PATCH net-next RFC v4 04/15] devlink: Add reload actions stats to dev get

[Jiri Pirko]

Mon, Sep 14, 2020 at 08:07:51AM CEST, moshe@mellanox.com wrote:
>Expose devlink reload actions stats to the user through devlink dev
>get command.
>
>Examples:
>$ devlink dev show
>pci/0000:82:00.0:
>  reload_action_stats:
>    driver_reinit 2
>    fw_activate 1
>    driver_reinit_no_reset 0
>    fw_activate_no_reset 0
>pci/0000:82:00.1:
>  reload_action_stats:
>    driver_reinit 1
>    fw_activate 1
>    driver_reinit_no_reset 0
>    fw_activate_no_reset 0

I would rather have something like:
   stats:
     reload_action:
       driver_reinit 1
       fw_activate 1
       driver_reinit_no_reset 0
       fw_activate_no_reset 0

Then we can easily extend and add other stats in the tree.


Also, I wonder if these stats could be somehow merged with Ido's metrics
work:
https://github.com/idosch/linux/commits/submit/devlink_metric_rfc_v1

Ido, would it make sense?


>
>$ devlink dev show -jp
>{
>    "dev": {
>        "pci/0000:82:00.0": {
>            "reload_action_stats": [ {
>                    "driver_reinit": 2
>                },{
>                    "fw_activate": 1
>                },{
>                    "driver_reinit_no_reset": 0
>                },{
>                    "fw_activate_no_reset": 0
>                } ]
>        },
>        "pci/0000:82:00.1": {
>            "reload_action_stats": [ {
>                    "driver_reinit": 1
>                },{
>                    "fw_activate": 1
>                },{
>                    "driver_reinit_no_reset": 0
>                },{
>                    "fw_activate_no_reset": 0
>                } ]
>        }
>    }
>}
>

[..]

Re: [PATCH net-next RFC v4 10/15] net/mlx5: Add support for devlink reload action fw activate

[Jiri Pirko]

Mon, Sep 14, 2020 at 08:07:57AM CEST, moshe@mellanox.com wrote:
>Add support for devlink reload action fw_activate. To activate firmware
>image the mlx5 driver resets the firmware and reloads it from flash. If
>a new image was stored on flash it will be loaded. Once this reload
>command is executed the driver initiates fw sync reset flow, where the
>firmware synchronizes all PFs on coming reset and driver reload.
>
>Signed-off-by: Moshe Shemesh <moshe@mellanox.com>
>---
>v3 -> v4:
>- Renamed actions_done to actions_performed
>v2 -> v3:
>- Return the reload actions done
>- Update reload action counters if reset initiated by remote host
>v1 -> v2:
>- Have fw_activate action instead of fw_reset level
>---
> .../net/ethernet/mellanox/mlx5/core/devlink.c | 62 ++++++++++++++++---
> .../ethernet/mellanox/mlx5/core/fw_reset.c    | 60 ++++++++++++++++--
> .../ethernet/mellanox/mlx5/core/fw_reset.h    |  1 +
> 3 files changed, 109 insertions(+), 14 deletions(-)
>
>diff --git a/drivers/net/ethernet/mellanox/mlx5/core/devlink.c b/drivers/net/ethernet/mellanox/mlx5/core/devlink.c
>index f6b29deaf02e..fa8f6abbea4e 100644
>--- a/drivers/net/ethernet/mellanox/mlx5/core/devlink.c
>+++ b/drivers/net/ethernet/mellanox/mlx5/core/devlink.c
>@@ -4,6 +4,7 @@
> #include <devlink.h>
> 
> #include "mlx5_core.h"
>+#include "fw_reset.h"
> #include "fs_core.h"
> #include "eswitch.h"
> 
>@@ -88,6 +89,32 @@ mlx5_devlink_info_get(struct devlink *devlink, struct devlink_info_req *req,
> 	return 0;
> }
> 
>+static int mlx5_devlink_reload_fw_activate(struct devlink *devlink, struct netlink_ext_ack *extack)
>+{
>+	struct mlx5_core_dev *dev = devlink_priv(devlink);
>+	u8 reset_level, reset_type, net_port_alive;
>+	int err;
>+
>+	err = mlx5_reg_mfrl_query(dev, &reset_level, &reset_type);
>+	if (err)
>+		return err;
>+	if (!(reset_level & MLX5_MFRL_REG_RESET_LEVEL3)) {
>+		NL_SET_ERR_MSG_MOD(extack, "FW activate requires reboot");
>+		return -EINVAL;
>+	}
>+
>+	net_port_alive = !!(reset_type & MLX5_MFRL_REG_RESET_TYPE_NET_PORT_ALIVE);
>+	err = mlx5_fw_set_reset_sync(dev, net_port_alive);
>+	if (err)
>+		goto out;
>+
>+	err = mlx5_fw_wait_fw_reset_done(dev);
>+out:
>+	if (err)
>+		NL_SET_ERR_MSG_MOD(extack, "FW activate command failed");
>+	return err;
>+}
>+
> static int mlx5_devlink_reload_down(struct devlink *devlink, bool netns_change,
> 				    enum devlink_reload_action action,
> 				    enum devlink_reload_action_limit_level limit_level,
>@@ -95,8 +122,17 @@ static int mlx5_devlink_reload_down(struct devlink *devlink, bool netns_change,
> {
> 	struct mlx5_core_dev *dev = devlink_priv(devlink);
> 
>-	mlx5_unload_one(dev, false);
>-	return 0;
>+	switch (action) {
>+	case DEVLINK_RELOAD_ACTION_DRIVER_REINIT:
>+		mlx5_unload_one(dev, false);
>+		return 0;
>+	case DEVLINK_RELOAD_ACTION_FW_ACTIVATE:
>+		return mlx5_devlink_reload_fw_activate(devlink, extack);
>+	default:
>+		/* Unsupported action should not get to this function */
>+		WARN_ON(1);
>+		return -EOPNOTSUPP;
>+	}
> }
> 
> static int mlx5_devlink_reload_up(struct devlink *devlink, enum devlink_reload_action action,
>@@ -104,13 +140,22 @@ static int mlx5_devlink_reload_up(struct devlink *devlink, enum devlink_reload_a
> 				  struct netlink_ext_ack *extack, unsigned long *actions_performed)
> {
> 	struct mlx5_core_dev *dev = devlink_priv(devlink);
>-	int err;
> 
>-	err = mlx5_load_one(dev, false);
>-	if (err)
>-		return err;
> 	if (actions_performed)
>-		*actions_performed = BIT(DEVLINK_RELOAD_ACTION_DRIVER_REINIT);
>+		*actions_performed = BIT(action);
>+
>+	switch (action) {
>+	case DEVLINK_RELOAD_ACTION_DRIVER_REINIT:
>+	case DEVLINK_RELOAD_ACTION_FW_ACTIVATE:
>+		/* On fw_activate action, also driver is reloaded and reinit performed */
>+		if (actions_performed)
>+			*actions_performed |= BIT(DEVLINK_RELOAD_ACTION_DRIVER_REINIT);

You should set DEVLINK_RELOAD_ACTION_FW_ACTIVATE bit in actions_performed
upon activation.


>+		return mlx5_load_one(dev, false);
>+	default:
>+		/* Unsupported action should not get to this function */
>+		WARN_ON(1);
>+		return -EOPNOTSUPP;
>+	}
> 
> 	return 0;
> }
>@@ -128,7 +173,8 @@ static const struct devlink_ops mlx5_devlink_ops = {
> #endif
> 	.flash_update = mlx5_devlink_flash_update,
> 	.info_get = mlx5_devlink_info_get,
>-	.supported_reload_actions = BIT(DEVLINK_RELOAD_ACTION_DRIVER_REINIT),
>+	.supported_reload_actions = BIT(DEVLINK_RELOAD_ACTION_DRIVER_REINIT) |
>+				    BIT(DEVLINK_RELOAD_ACTION_FW_ACTIVATE),
> 	.supported_reload_action_limit_levels = BIT(DEVLINK_RELOAD_ACTION_LIMIT_LEVEL_NONE),
> 	.reload_down = mlx5_devlink_reload_down,
> 	.reload_up = mlx5_devlink_reload_up,
>diff --git a/drivers/net/ethernet/mellanox/mlx5/core/fw_reset.c b/drivers/net/ethernet/mellanox/mlx5/core/fw_reset.c
>index 61237f4836cc..550f67b00473 100644
>--- a/drivers/net/ethernet/mellanox/mlx5/core/fw_reset.c
>+++ b/drivers/net/ethernet/mellanox/mlx5/core/fw_reset.c
>@@ -5,6 +5,7 @@
> 
> enum {
> 	MLX5_FW_RESET_FLAGS_RESET_REQUESTED,
>+	MLX5_FW_RESET_FLAGS_PENDING_COMP
> };
> 
> struct mlx5_fw_reset {
>@@ -17,6 +18,8 @@ struct mlx5_fw_reset {
> 	struct work_struct reset_abort_work;
> 	unsigned long reset_flags;
> 	struct timer_list timer;
>+	struct completion done;
>+	int ret;
> };
> 
> static int mlx5_reg_mfrl_set(struct mlx5_core_dev *dev, u8 reset_level,
>@@ -53,7 +56,14 @@ int mlx5_reg_mfrl_query(struct mlx5_core_dev *dev, u8 *reset_level, u8 *reset_ty
> 
> int mlx5_fw_set_reset_sync(struct mlx5_core_dev *dev, u8 reset_type_sel)
> {
>-	return mlx5_reg_mfrl_set(dev, MLX5_MFRL_REG_RESET_LEVEL3, reset_type_sel, 0, true);
>+	struct mlx5_fw_reset *fw_reset = dev->priv.fw_reset;
>+	int err;
>+
>+	set_bit(MLX5_FW_RESET_FLAGS_PENDING_COMP, &fw_reset->reset_flags);
>+	err =  mlx5_reg_mfrl_set(dev, MLX5_MFRL_REG_RESET_LEVEL3, reset_type_sel, 0, true);
>+	if (err)
>+		clear_bit(MLX5_FW_RESET_FLAGS_PENDING_COMP, &fw_reset->reset_flags);
>+	return err;
> }
> 
> int mlx5_fw_set_live_patch(struct mlx5_core_dev *dev)
>@@ -66,19 +76,36 @@ static int mlx5_fw_set_reset_sync_ack(struct mlx5_core_dev *dev)
> 	return mlx5_reg_mfrl_set(dev, MLX5_MFRL_REG_RESET_LEVEL3, 0, 1, false);
> }
> 
>+static void mlx5_fw_reset_complete_reload(struct mlx5_core_dev *dev)
>+{
>+	struct mlx5_fw_reset *fw_reset = dev->priv.fw_reset;
>+
>+	/* if this is the driver that initiated the fw reset, devlink completed the reload */
>+	if (test_bit(MLX5_FW_RESET_FLAGS_PENDING_COMP, &fw_reset->reset_flags)) {
>+		complete(&fw_reset->done);
>+	} else {
>+		mlx5_load_one(dev, false);
>+		devlink_reload_implicit_actions_performed(priv_to_devlink(dev),
>+							  DEVLINK_RELOAD_ACTION_LIMIT_LEVEL_NONE,
>+							  BIT(DEVLINK_RELOAD_ACTION_DRIVER_REINIT) |
>+							  BIT(DEVLINK_RELOAD_ACTION_FW_ACTIVATE));
>+	}
>+}
>+
> static void mlx5_sync_reset_reload_work(struct work_struct *work)
> {
> 	struct mlx5_fw_reset *fw_reset = container_of(work, struct mlx5_fw_reset,
> 						      reset_reload_work);
> 	struct mlx5_core_dev *dev = fw_reset->dev;
>+	int err;
> 
> 	mlx5_enter_error_state(dev, true);
> 	mlx5_unload_one(dev, false);
>-	if (mlx5_health_wait_pci_up(dev)) {
>+	err = mlx5_health_wait_pci_up(dev);
>+	if (err)
> 		mlx5_core_err(dev, "reset reload flow aborted, PCI reads still not working\n");
>-		return;
>-	}
>-	mlx5_load_one(dev, false);
>+	fw_reset->ret = err;
>+	mlx5_fw_reset_complete_reload(dev);
> }
> 
> static void mlx5_stop_sync_reset_poll(struct mlx5_core_dev *dev)
>@@ -264,7 +291,8 @@ static void mlx5_sync_reset_now_event(struct work_struct *work)
> done:
> 	if (err)
> 		mlx5_start_health_poll(dev);
>-	mlx5_load_one(dev, false);
>+	fw_reset->ret = err;
>+	mlx5_fw_reset_complete_reload(dev);
> }
> 
> static void mlx5_sync_reset_abort_event(struct work_struct *work)
>@@ -313,6 +341,25 @@ static int fw_reset_event_notifier(struct notifier_block *nb, unsigned long acti
> 	return NOTIFY_OK;
> }
> 
>+#define MLX5_FW_RESET_TIMEOUT_MSEC 5000
>+int mlx5_fw_wait_fw_reset_done(struct mlx5_core_dev *dev)
>+{
>+	unsigned long timeout = msecs_to_jiffies(MLX5_FW_RESET_TIMEOUT_MSEC);
>+	struct mlx5_fw_reset *fw_reset = dev->priv.fw_reset;
>+	int err;
>+
>+	if (!wait_for_completion_timeout(&fw_reset->done, timeout)) {
>+		mlx5_core_warn(dev, "FW sync reset timeout after %d seconds\n",
>+			       MLX5_FW_RESET_TIMEOUT_MSEC / 1000);
>+		err = -ETIMEDOUT;
>+		goto out;
>+	}
>+	err = fw_reset->ret;
>+out:
>+	clear_bit(MLX5_FW_RESET_FLAGS_PENDING_COMP, &fw_reset->reset_flags);
>+	return err;
>+}
>+
> int mlx5_fw_reset_events_init(struct mlx5_core_dev *dev)
> {
> 	struct mlx5_fw_reset *fw_reset = kzalloc(sizeof(*fw_reset), GFP_KERNEL);
>@@ -336,6 +383,7 @@ int mlx5_fw_reset_events_init(struct mlx5_core_dev *dev)
> 	MLX5_NB_INIT(&fw_reset->nb, fw_reset_event_notifier, GENERAL_EVENT);
> 	mlx5_eq_notifier_register(dev, &fw_reset->nb);
> 
>+	init_completion(&fw_reset->done);
> 	return 0;
> }
> 
>diff --git a/drivers/net/ethernet/mellanox/mlx5/core/fw_reset.h b/drivers/net/ethernet/mellanox/mlx5/core/fw_reset.h
>index 278f538ea92a..d7ee951a2258 100644
>--- a/drivers/net/ethernet/mellanox/mlx5/core/fw_reset.h
>+++ b/drivers/net/ethernet/mellanox/mlx5/core/fw_reset.h
>@@ -10,6 +10,7 @@ int mlx5_reg_mfrl_query(struct mlx5_core_dev *dev, u8 *reset_level, u8 *reset_ty
> int mlx5_fw_set_reset_sync(struct mlx5_core_dev *dev, u8 reset_type_sel);
> int mlx5_fw_set_live_patch(struct mlx5_core_dev *dev);
> 
>+int mlx5_fw_wait_fw_reset_done(struct mlx5_core_dev *dev);
> int mlx5_fw_reset_events_init(struct mlx5_core_dev *dev);
> void mlx5_fw_reset_events_cleanup(struct mlx5_core_dev *dev);
> 
>-- 
>2.17.1
>

Re: [PATCH net-next RFC v4 10/15] net/mlx5: Add support for devlink reload action fw activate

[Jiri Pirko]

Mon, Sep 14, 2020 at 08:07:57AM CEST, moshe@mellanox.com wrote:

[..]

>+static void mlx5_fw_reset_complete_reload(struct mlx5_core_dev *dev)
>+{
>+	struct mlx5_fw_reset *fw_reset = dev->priv.fw_reset;
>+
>+	/* if this is the driver that initiated the fw reset, devlink completed the reload */
>+	if (test_bit(MLX5_FW_RESET_FLAGS_PENDING_COMP, &fw_reset->reset_flags)) {
>+		complete(&fw_reset->done);
>+	} else {
>+		mlx5_load_one(dev, false);
>+		devlink_reload_implicit_actions_performed(priv_to_devlink(dev),
>+							  DEVLINK_RELOAD_ACTION_LIMIT_LEVEL_NONE,
>+							  BIT(DEVLINK_RELOAD_ACTION_DRIVER_REINIT) |
>+							  BIT(DEVLINK_RELOAD_ACTION_FW_ACTIVATE));

Hmm, who originated the reset? Devlink_reload of the same devlink
instance?

[..]

Re: [PATCH net-next RFC v4 11/15] devlink: Add enable_remote_dev_reset generic parameter

[Jiri Pirko]

Mon, Sep 14, 2020 at 08:07:58AM CEST, moshe@mellanox.com wrote:
>The enable_remote_dev_reset devlink param flags that the host admin
>allows device resets that can be initiated by other hosts. This
>parameter is useful for setups where a device is shared by different
>hosts, such as multi-host setup. Once the user set this parameter to
>false, the driver should NACK any attempt to reset the device while the
>driver is loaded.
>
>Signed-off-by: Moshe Shemesh <moshe@mellanox.com>

Reviewed-by: Jiri Pirko <jiri@nvidia.com>

Re: [PATCH net-next RFC v4 01/15] devlink: Add reload action option to devlink reload command

[Jakub Kicinski]

On Mon, 14 Sep 2020 13:28:29 +0200 Jiri Pirko wrote:
> Mon, Sep 14, 2020 at 11:54:55AM CEST, vasundhara-v.volam@broadcom.com wrote:
> >On Mon, Sep 14, 2020 at 3:02 PM Jiri Pirko <jiri@resnulli.us> wrote:  
> >> >> +mlxsw_devlink_core_bus_device_reload_up(struct devlink *devlink, enum devlink_reload_action action,
> >> >> +                                       struct netlink_ext_ack *extack,
> >> >> +                                       unsigned long *actions_performed)  
> >> >Sorry for repeating again, for fw_activate action on our device, all
> >> >the driver entities undergo reset asynchronously once user initiates
> >> >"devlink dev reload action fw_activate" and reload_up does not have
> >> >much to do except reporting actions that will be/being performed.
> >> >
> >> >Once reset is complete, the health reporter will be notified using  
> >>
> >> Hmm, how is the fw reset related to health reporter recovery? Recovery
> >> happens after some error event. I don't believe it is wise to mix it.  
> >Our device has a fw_reset health reporter, which is updated on reset
> >events and firmware activation is one among them. All non-fatal
> >firmware reset events are reported on fw_reset health reporter.  
> 
> Hmm, interesting. In that case, assuming this is fine, should we have
> some standard in this. I mean, if the driver supports reset, should it
> also define the "fw_reset" reporter to report such events?
> 
> Jakub, what is your take here?

Sounds doubly wrong to me.

As you say health reporters should trigger on error events,
communicating completion of an action requested by the user
seems very wrong. IIUC operators should monitor and collect
health failures. In this case looks like all events from fw_reset 
would need to be discarded, since they are not meaningful
without the context of what triggered them.

And secondly, reporting the completion via some async mechanism
that user has to monitor is just plain lazy. That's pushing out
the work that has to be done out to user space. Wait for the 
completion in the driver.

> >> Instead, why don't you block in reload_up() until the reset is complete?  
> >
> >Though user initiate "devlink dev reload" event on a single interface,
> >all driver entities undergo reset and all entities recover
> >independently. I don't think we can block the reload_up() on the
> >interface(that user initiated the command), until whole reset is
> >complete.  
> 
> Why not? mlxsw reset takes up to like 10 seconds for example.

+1, why?

Re: [PATCH net-next RFC v4 01/15] devlink: Add reload action option to devlink reload command

[Jakub Kicinski]

On Mon, 14 Sep 2020 09:07:48 +0300 Moshe Shemesh wrote:
> @@ -3011,12 +3060,41 @@ static int devlink_nl_cmd_reload(struct sk_buff *skb, struct genl_info *info)
>  			return PTR_ERR(dest_net);
>  	}
>  
> -	err = devlink_reload(devlink, dest_net, info->extack);
> +	if (info->attrs[DEVLINK_ATTR_RELOAD_ACTION])
> +		action = nla_get_u8(info->attrs[DEVLINK_ATTR_RELOAD_ACTION]);
> +	else
> +		action = DEVLINK_RELOAD_ACTION_DRIVER_REINIT;
> +
> +	if (action == DEVLINK_RELOAD_ACTION_UNSPEC || action > DEVLINK_RELOAD_ACTION_MAX) {
> +		NL_SET_ERR_MSG_MOD(info->extack, "Invalid reload action");
> +		return -EINVAL;
> +	} else if (!devlink_reload_action_is_supported(devlink, action)) {
> +		NL_SET_ERR_MSG_MOD(info->extack, "Requested reload action is not supported");
> +		return -EOPNOTSUPP;
> +	}
> +
> +	err = devlink_reload(devlink, dest_net, action, info->extack, &actions_performed);
>  
>  	if (dest_net)
>  		put_net(dest_net);
>  
> -	return err;
> +	if (err)
> +		return err;
> +
> +	WARN_ON(!actions_performed);
> +	msg = nlmsg_new(NLMSG_DEFAULT_SIZE, GFP_KERNEL);
> +	if (!msg)
> +		return -ENOMEM;
> +
> +	err = devlink_nl_reload_actions_performed_fill(msg, devlink, actions_performed,
> +						       DEVLINK_CMD_RELOAD, info->snd_portid,
> +						       info->snd_seq, 0);
> +	if (err) {
> +		nlmsg_free(msg);
> +		return err;
> +	}
> +
> +	return genlmsg_reply(msg, info);

I think generating the reply may break existing users. Only generate
the reply if request contained DEVLINK_ATTR_RELOAD_ACTION (or any other
new attribute which existing users can't pass).

Re: [PATCH net-next RFC v4 01/15] devlink: Add reload action option to devlink reload command

[Michael Chan]

On Mon, Sep 14, 2020 at 2:31 PM Jakub Kicinski <kuba@kernel.org> wrote:
>
> On Mon, 14 Sep 2020 13:28:29 +0200 Jiri Pirko wrote:
> > >> Instead, why don't you block in reload_up() until the reset is complete?
> > >
> > >Though user initiate "devlink dev reload" event on a single interface,
> > >all driver entities undergo reset and all entities recover
> > >independently. I don't think we can block the reload_up() on the
> > >interface(that user initiated the command), until whole reset is
> > >complete.
> >
> > Why not? mlxsw reset takes up to like 10 seconds for example.
>
> +1, why?

Yes, we should be able to block until the reset sequence is complete.
I don't see any problem.  I will work with Vasundhara on this.

Re: [PATCH net-next RFC v4 01/15] devlink: Add reload action option to devlink reload command

[Jiri Pirko]

Tue, Sep 15, 2020 at 12:06:19AM CEST, michael.chan@broadcom.com wrote:
>On Mon, Sep 14, 2020 at 2:31 PM Jakub Kicinski <kuba@kernel.org> wrote:
>>
>> On Mon, 14 Sep 2020 13:28:29 +0200 Jiri Pirko wrote:
>> > >> Instead, why don't you block in reload_up() until the reset is complete?
>> > >
>> > >Though user initiate "devlink dev reload" event on a single interface,
>> > >all driver entities undergo reset and all entities recover
>> > >independently. I don't think we can block the reload_up() on the
>> > >interface(that user initiated the command), until whole reset is
>> > >complete.
>> >
>> > Why not? mlxsw reset takes up to like 10 seconds for example.
>>
>> +1, why?
>
>Yes, we should be able to block until the reset sequence is complete.
>I don't see any problem.  I will work with Vasundhara on this.

Could you please also remove fw_reset as it is apparently misuse of
devlink health mechanism?

Thanks!

Re: [PATCH net-next RFC v4 04/15] devlink: Add reload actions stats to dev get

[Ido Schimmel]

On Mon, Sep 14, 2020 at 03:45:00PM +0200, Jiri Pirko wrote:
> Mon, Sep 14, 2020 at 08:07:51AM CEST, moshe@mellanox.com wrote:
> >Expose devlink reload actions stats to the user through devlink dev
> >get command.
> >
> >Examples:
> >$ devlink dev show
> >pci/0000:82:00.0:
> >  reload_action_stats:
> >    driver_reinit 2
> >    fw_activate 1
> >    driver_reinit_no_reset 0
> >    fw_activate_no_reset 0
> >pci/0000:82:00.1:
> >  reload_action_stats:
> >    driver_reinit 1
> >    fw_activate 1
> >    driver_reinit_no_reset 0
> >    fw_activate_no_reset 0
> 
> I would rather have something like:
>    stats:
>      reload_action:
>        driver_reinit 1
>        fw_activate 1
>        driver_reinit_no_reset 0
>        fw_activate_no_reset 0
> 
> Then we can easily extend and add other stats in the tree.
> 
> 
> Also, I wonder if these stats could be somehow merged with Ido's metrics
> work:
> https://github.com/idosch/linux/commits/submit/devlink_metric_rfc_v1
> 
> Ido, would it make sense?

I guess. My original idea for devlink-metric was to expose
design-specific metrics to user space where the entity registering the
metrics is the device driver. In this case the entity would be devlink
itself and it would be auto-registered for each device.

> 
> 
> >
> >$ devlink dev show -jp
> >{
> >    "dev": {
> >        "pci/0000:82:00.0": {
> >            "reload_action_stats": [ {
> >                    "driver_reinit": 2
> >                },{
> >                    "fw_activate": 1
> >                },{
> >                    "driver_reinit_no_reset": 0
> >                },{
> >                    "fw_activate_no_reset": 0
> >                } ]
> >        },
> >        "pci/0000:82:00.1": {
> >            "reload_action_stats": [ {
> >                    "driver_reinit": 1
> >                },{
> >                    "fw_activate": 1
> >                },{
> >                    "driver_reinit_no_reset": 0
> >                },{
> >                    "fw_activate_no_reset": 0
> >                } ]
> >        }
> >    }
> >}
> >
> 
> [..]

Re: [PATCH net-next RFC v4 04/15] devlink: Add reload actions stats to dev get

[Jiri Pirko]

Tue, Sep 15, 2020 at 08:45:19AM CEST, idosch@idosch.org wrote:
>On Mon, Sep 14, 2020 at 03:45:00PM +0200, Jiri Pirko wrote:
>> Mon, Sep 14, 2020 at 08:07:51AM CEST, moshe@mellanox.com wrote:
>> >Expose devlink reload actions stats to the user through devlink dev
>> >get command.
>> >
>> >Examples:
>> >$ devlink dev show
>> >pci/0000:82:00.0:
>> >  reload_action_stats:
>> >    driver_reinit 2
>> >    fw_activate 1
>> >    driver_reinit_no_reset 0
>> >    fw_activate_no_reset 0
>> >pci/0000:82:00.1:
>> >  reload_action_stats:
>> >    driver_reinit 1
>> >    fw_activate 1
>> >    driver_reinit_no_reset 0
>> >    fw_activate_no_reset 0
>> 
>> I would rather have something like:
>>    stats:
>>      reload_action:
>>        driver_reinit 1
>>        fw_activate 1
>>        driver_reinit_no_reset 0
>>        fw_activate_no_reset 0
>> 
>> Then we can easily extend and add other stats in the tree.
>> 
>> 
>> Also, I wonder if these stats could be somehow merged with Ido's metrics
>> work:
>> https://github.com/idosch/linux/commits/submit/devlink_metric_rfc_v1
>> 
>> Ido, would it make sense?
>
>I guess. My original idea for devlink-metric was to expose
>design-specific metrics to user space where the entity registering the
>metrics is the device driver. In this case the entity would be devlink
>itself and it would be auto-registered for each device.

Yeah, the usecase is different, but it is still stats, right.


>
>> 
>> 
>> >
>> >$ devlink dev show -jp
>> >{
>> >    "dev": {
>> >        "pci/0000:82:00.0": {
>> >            "reload_action_stats": [ {
>> >                    "driver_reinit": 2
>> >                },{
>> >                    "fw_activate": 1
>> >                },{
>> >                    "driver_reinit_no_reset": 0
>> >                },{
>> >                    "fw_activate_no_reset": 0
>> >                } ]
>> >        },
>> >        "pci/0000:82:00.1": {
>> >            "reload_action_stats": [ {
>> >                    "driver_reinit": 1
>> >                },{
>> >                    "fw_activate": 1
>> >                },{
>> >                    "driver_reinit_no_reset": 0
>> >                },{
>> >                    "fw_activate_no_reset": 0
>> >                } ]
>> >        }
>> >    }
>> >}
>> >
>> 
>> [..]

Re: [PATCH net-next RFC v4 01/15] devlink: Add reload action option to devlink reload command

[Moshe Shemesh]

On 9/14/2020 3:27 PM, Jiri Pirko wrote:
> Mon, Sep 14, 2020 at 08:07:48AM CEST, moshe@mellanox.com wrote:
>> Add devlink reload action to allow the user to request a specific reload
>> action. The action parameter is optional, if not specified then devlink
>> driver re-init action is used (backward compatible).
>> Note that when required to do firmware activation some drivers may need
>> to reload the driver. On the other hand some drivers may need to reset
>> the firmware to reinitialize the driver entities. Therefore, the devlink
>> reload command returns the actions which were actually performed.
>> Reload actions supported are:
>> driver_reinit: driver entities re-initialization, applying devlink-param
>>                and devlink-resource values.
>> fw_activate: firmware activate.
>>
>> command examples:
>> $devlink dev reload pci/0000:82:00.0 action driver_reinit
>> reload_actions_performed:
>>   driver_reinit
>>
>> $devlink dev reload pci/0000:82:00.0 action fw_activate
>> reload_actions_performed:
>>   driver_reinit fw_activate
>>
>> Signed-off-by: Moshe Shemesh <moshe@mellanox.com>
>> ---
>> v3 -> v4:
>> - Removed fw_activate_no_reset as an action (next patch adds limit
>>   levels instead).
>> - Renamed actions_done to actions_performed
>> v2 -> v3:
>> - Replace fw_live_patch action by fw_activate_no_reset
>> - Devlink reload returns the actions done over netlink reply
>> v1 -> v2:
>> - Instead of reload levels driver,fw_reset,fw_live_patch have reload
>>   actions driver_reinit,fw_activate,fw_live_patch
>> - Remove driver default level, the action driver_reinit is the default
>>   action for all drivers
>> ---
>> drivers/net/ethernet/mellanox/mlx4/main.c     |  14 ++-
>> .../net/ethernet/mellanox/mlx5/core/devlink.c |  15 ++-
>> drivers/net/ethernet/mellanox/mlxsw/core.c    |  25 ++--
>> drivers/net/netdevsim/dev.c                   |  16 ++-
>> include/net/devlink.h                         |   7 +-
>> include/uapi/linux/devlink.h                  |  19 +++
>> net/core/devlink.c                            | 111 +++++++++++++++++-
>> 7 files changed, 180 insertions(+), 27 deletions(-)
>>
>> diff --git a/drivers/net/ethernet/mellanox/mlx4/main.c b/drivers/net/ethernet/mellanox/mlx4/main.c
>> index 70cf24ba71e4..aadf1676a0ed 100644
>> --- a/drivers/net/ethernet/mellanox/mlx4/main.c
>> +++ b/drivers/net/ethernet/mellanox/mlx4/main.c
>> @@ -3946,6 +3946,7 @@ static int mlx4_restart_one_up(struct pci_dev *pdev, bool reload,
>> 			       struct devlink *devlink);
>>
>> static int mlx4_devlink_reload_down(struct devlink *devlink, bool netns_change,
>> +				    enum devlink_reload_action action,
>> 				    struct netlink_ext_ack *extack)
>> {
>> 	struct mlx4_priv *priv = devlink_priv(devlink);
>> @@ -3962,8 +3963,8 @@ static int mlx4_devlink_reload_down(struct devlink *devlink, bool netns_change,
>> 	return 0;
>> }
>>
>> -static int mlx4_devlink_reload_up(struct devlink *devlink,
>> -				  struct netlink_ext_ack *extack)
>> +static int mlx4_devlink_reload_up(struct devlink *devlink, enum devlink_reload_action action,
>> +				  struct netlink_ext_ack *extack, unsigned long *actions_performed)
>> {
>> 	struct mlx4_priv *priv = devlink_priv(devlink);
>> 	struct mlx4_dev *dev = &priv->dev;
>> @@ -3971,15 +3972,20 @@ static int mlx4_devlink_reload_up(struct devlink *devlink,
>> 	int err;
>>
>> 	err = mlx4_restart_one_up(persist->pdev, true, devlink);
>> -	if (err)
>> +	if (err) {
>> 		mlx4_err(persist->dev, "mlx4_restart_one_up failed, ret=%d\n",
>> 			 err);
>> +		return err;
>> +	}
>> +	if (actions_performed)
> Nit, pass the unsigned long allways (even when it would be unused) and
> avoid check in every driver.
>

Ack.

>> +		*actions_performed = BIT(DEVLINK_RELOAD_ACTION_DRIVER_REINIT);
>>
>> -	return err;
>> +	return 0;
>> }
>>
>> static const struct devlink_ops mlx4_devlink_ops = {
>> 	.port_type_set	= mlx4_devlink_port_type_set,
>> +	.supported_reload_actions = BIT(DEVLINK_RELOAD_ACTION_DRIVER_REINIT),
>> 	.reload_down	= mlx4_devlink_reload_down,
>> 	.reload_up	= mlx4_devlink_reload_up,
>> };
> [..]
>
>
>> @@ -2969,29 +2975,72 @@ bool devlink_is_reload_failed(const struct devlink *devlink)
>> EXPORT_SYMBOL_GPL(devlink_is_reload_failed);
>>
>> static int devlink_reload(struct devlink *devlink, struct net *dest_net,
>> -			  struct netlink_ext_ack *extack)
>> +			  enum devlink_reload_action action, struct netlink_ext_ack *extack,
>> +			  unsigned long *actions_performed)
>> {
>> 	int err;
>>
>> 	if (!devlink->reload_enabled)
>> 		return -EOPNOTSUPP;
>>
>> -	err = devlink->ops->reload_down(devlink, !!dest_net, extack);
>> +	err = devlink->ops->reload_down(devlink, !!dest_net, action, extack);
>> 	if (err)
>> 		return err;
>>
>> 	if (dest_net && !net_eq(dest_net, devlink_net(devlink)))
>> 		devlink_reload_netns_change(devlink, dest_net);
>>
>> -	err = devlink->ops->reload_up(devlink, extack);
>> +	err = devlink->ops->reload_up(devlink, action, extack, actions_performed);
> Here, please add a WARN_ON() check:
> 	WARN_ON(!(*actions_performed & action));
>
> The requested action should be always performed.
>
Good point, I will add.
>> 	devlink_reload_failed_set(devlink, !!err);
>> 	return err;
>> }
>>
>> +static int
>> +devlink_nl_reload_actions_performed_fill(struct sk_buff *msg,
>> +					 struct devlink *devlink,
>> +					 unsigned long actions_performed,
>> +					 enum devlink_command cmd, u32 portid,
>> +					 u32 seq, int flags)
>> +{
>> +	struct nlattr *actions_performed_attr;
>> +	void *hdr;
>> +	int i;
>> +
>> +	hdr = genlmsg_put(msg, portid, seq, &devlink_nl_family, flags, cmd);
>> +	if (!hdr)
>> +		return -EMSGSIZE;
>> +
>> +	if (devlink_nl_put_handle(msg, devlink))
>> +		goto genlmsg_cancel;
>> +
>> +	actions_performed_attr = nla_nest_start(msg, DEVLINK_ATTR_RELOAD_ACTIONS_PERFORMED);
>> +	if (!actions_performed_attr)
>> +		goto genlmsg_cancel;
>> +
>> +	for (i = 0; i <= DEVLINK_RELOAD_ACTION_MAX; i++) {
>> +		if (!test_bit(i, &actions_performed))
>> +			continue;
>> +		if (nla_put_u8(msg, DEVLINK_ATTR_RELOAD_ACTION, i))
>> +			goto actions_performed_nest_cancel;
>> +	}
>> +	nla_nest_end(msg, actions_performed_attr);
>> +	genlmsg_end(msg, hdr);
>> +	return 0;
>> +
>> +actions_performed_nest_cancel:
>> +	nla_nest_cancel(msg, actions_performed_attr);
>> +genlmsg_cancel:
>> +	genlmsg_cancel(msg, hdr);
>> +	return -EMSGSIZE;
>> +}
>> +
>> static int devlink_nl_cmd_reload(struct sk_buff *skb, struct genl_info *info)
>> {
>> 	struct devlink *devlink = info->user_ptr[0];
>> +	enum devlink_reload_action action;
>> +	unsigned long actions_performed;
>> 	struct net *dest_net = NULL;
>> +	struct sk_buff *msg;
>> 	int err;
>>
>> 	if (!devlink_reload_supported(devlink))
>> @@ -3011,12 +3060,41 @@ static int devlink_nl_cmd_reload(struct sk_buff *skb, struct genl_info *info)
>> 			return PTR_ERR(dest_net);
>> 	}
>>
>> -	err = devlink_reload(devlink, dest_net, info->extack);
>> +	if (info->attrs[DEVLINK_ATTR_RELOAD_ACTION])
>> +		action = nla_get_u8(info->attrs[DEVLINK_ATTR_RELOAD_ACTION]);
>> +	else
>> +		action = DEVLINK_RELOAD_ACTION_DRIVER_REINIT;
>> +
>> +	if (action == DEVLINK_RELOAD_ACTION_UNSPEC || action > DEVLINK_RELOAD_ACTION_MAX) {
>> +		NL_SET_ERR_MSG_MOD(info->extack, "Invalid reload action");
> Hmm, I understand the unspec check, but the max check is not needed. The
> following check will take care of it.
>
Ack.
>> +		return -EINVAL;
>> +	} else if (!devlink_reload_action_is_supported(devlink, action)) {
>> +		NL_SET_ERR_MSG_MOD(info->extack, "Requested reload action is not supported");
> ".. by the driver" ?
> 	


Yes, will fix.

>> +		return -EOPNOTSUPP;
>> +	}
>> +
>> +	err = devlink_reload(devlink, dest_net, action, info->extack, &actions_performed);
>>
>> 	if (dest_net)
>> 		put_net(dest_net);
>>
>> -	return err;
>> +	if (err)
>> +		return err;
>> +
>> +	WARN_ON(!actions_performed);
>> +	msg = nlmsg_new(NLMSG_DEFAULT_SIZE, GFP_KERNEL);
>> +	if (!msg)
>> +		return -ENOMEM;
>> +
>> +	err = devlink_nl_reload_actions_performed_fill(msg, devlink, actions_performed,
>> +						       DEVLINK_CMD_RELOAD, info->snd_portid,
>> +						       info->snd_seq, 0);
>> +	if (err) {
>> +		nlmsg_free(msg);
>> +		return err;
>> +	}
>> +
>> +	return genlmsg_reply(msg, info);
>> }
>>
>> static int devlink_nl_flash_update_fill(struct sk_buff *msg,
>> @@ -7047,6 +7125,7 @@ static const struct nla_policy devlink_nl_policy[DEVLINK_ATTR_MAX + 1] = {
>> 	[DEVLINK_ATTR_TRAP_POLICER_RATE] = { .type = NLA_U64 },
>> 	[DEVLINK_ATTR_TRAP_POLICER_BURST] = { .type = NLA_U64 },
>> 	[DEVLINK_ATTR_PORT_FUNCTION] = { .type = NLA_NESTED },
>> +	[DEVLINK_ATTR_RELOAD_ACTION] = { .type = NLA_U8 },
>> };
>>
>> static const struct genl_ops devlink_nl_ops[] = {
>> @@ -7372,6 +7451,20 @@ static struct genl_family devlink_nl_family __ro_after_init = {
>> 	.n_mcgrps	= ARRAY_SIZE(devlink_nl_mcgrps),
>> };
>>
>> +static int devlink_reload_actions_verify(struct devlink *devlink)
>> +{
>> +	const struct devlink_ops *ops;
>> +
>> +	if (!devlink_reload_supported(devlink))
> If reload is not supported, the supported_reload_actions should be 0.
> Please check that with WARN_ON too.
>
OK, same goes to supported limit levels.
>> +		return 0;
>> +
>> +	ops = devlink->ops;
>> +	if (WARN_ON(ops->supported_reload_actions >= BIT(__DEVLINK_RELOAD_ACTION_MAX) ||
>> +		    ops->supported_reload_actions <= BIT(DEVLINK_RELOAD_ACTION_UNSPEC)))
>> +		return -EINVAL;
>> +	return 0;
>> +}
>> +
>> /**
>>   *	devlink_alloc - Allocate new devlink instance resources
>>   *
>> @@ -7392,6 +7485,11 @@ struct devlink *devlink_alloc(const struct devlink_ops *ops, size_t priv_size)
>> 	if (!devlink)
>> 		return NULL;
>> 	devlink->ops = ops;
>> +	if (devlink_reload_actions_verify(devlink)) {
> Move this check to the beginning. You don't need devlink instance for
> the check, just ops.


Right, will fix.

> also, your devlink_reload_actions_verify() function returns
> 0/-ESOMETHING. Treat it accordingly here.


Well, yes, but I rather return NULL here since devlink_alloc() failed. 
If devlink_reload_actions_verify() fails it has WARN_ON which will lead 
the driver developer to his bug.

>
>> +		kfree(devlink);
>> +		return NULL;
>> +	}
>> +
>> 	xa_init_flags(&devlink->snapshot_ids, XA_FLAGS_ALLOC);
>> 	__devlink_net_set(devlink, &init_net);
>> 	INIT_LIST_HEAD(&devlink->port_list);
>> @@ -9657,7 +9755,8 @@ static void __net_exit devlink_pernet_pre_exit(struct net *net)
>> 		if (net_eq(devlink_net(devlink), net)) {
>> 			if (WARN_ON(!devlink_reload_supported(devlink)))
>> 				continue;
>> -			err = devlink_reload(devlink, &init_net, NULL);
>> +			err = devlink_reload(devlink, &init_net,
>> +					     DEVLINK_RELOAD_ACTION_DRIVER_REINIT, NULL, NULL);
>> 			if (err && err != -EOPNOTSUPP)
>> 				pr_warn("Failed to reload devlink instance into init_net\n");
>> 		}
>> -- 
>> 2.17.1
>>

Re: [PATCH net-next RFC v4 02/15] devlink: Add reload action limit level

[Moshe Shemesh]

On 9/14/2020 4:10 PM, Jiri Pirko wrote:
> Mon, Sep 14, 2020 at 08:07:49AM CEST, moshe@mellanox.com wrote:
> 		
> [..]
>
> 	
>> diff --git a/include/net/devlink.h b/include/net/devlink.h
>> index b09db891db04..dddd9ee5b8a9 100644
>> --- a/include/net/devlink.h
>> +++ b/include/net/devlink.h
>> @@ -1012,9 +1012,13 @@ enum devlink_trap_group_generic_id {
>>
>> struct devlink_ops {
>> 	unsigned long supported_reload_actions;
>> +	unsigned long supported_reload_action_limit_levels;
>> 	int (*reload_down)(struct devlink *devlink, bool netns_change,
>> -			   enum devlink_reload_action action, struct netlink_ext_ack *extack);
>> +			   enum devlink_reload_action action,
>> +			   enum devlink_reload_action_limit_level limit_level,
>> +			   struct netlink_ext_ack *extack);
>> 	int (*reload_up)(struct devlink *devlink, enum devlink_reload_action action,
>> +			 enum devlink_reload_action_limit_level limit_level,
>> 			 struct netlink_ext_ack *extack, unsigned long *actions_performed);
>> 	int (*port_type_set)(struct devlink_port *devlink_port,
>> 			     enum devlink_port_type port_type);
>> diff --git a/include/uapi/linux/devlink.h b/include/uapi/linux/devlink.h
>> index a6f64db0bdf3..b19686fd80ff 100644
>> --- a/include/uapi/linux/devlink.h
>> +++ b/include/uapi/linux/devlink.h
>> @@ -287,6 +287,22 @@ enum devlink_reload_action {
>> 	DEVLINK_RELOAD_ACTION_MAX = __DEVLINK_RELOAD_ACTION_MAX - 1
>> };
>>
>> +/**
>> + * enum devlink_reload_action_limit_level - Reload action limit level.
>> + * @DEVLINK_RELOAD_ACTION_LIMIT_LEVEL_NONE: No constrains on action. Action may include
>> + *                                          reset or downtime as needed.
>> + * @DEVLINK_RELOAD_ACTION_LIMIT_LEVEL_NO_RESET: No reset allowed, no down time allowed,
>> + *                                              no link flap and no configuration is lost.
>> + */
>> +enum devlink_reload_action_limit_level {
>> +	DEVLINK_RELOAD_ACTION_LIMIT_LEVEL_NONE,
>> +	DEVLINK_RELOAD_ACTION_LIMIT_LEVEL_NO_RESET,
>> +
>> +	/* Add new reload actions limit level above */
>> +	__DEVLINK_RELOAD_ACTION_LIMIT_LEVEL_MAX,
>> +	DEVLINK_RELOAD_ACTION_LIMIT_LEVEL_MAX = __DEVLINK_RELOAD_ACTION_LIMIT_LEVEL_MAX - 1
>> +};
>> +
>> enum devlink_attr {
>> 	/* don't change the order or add anything between, this is ABI! */
>> 	DEVLINK_ATTR_UNSPEC,
>> @@ -478,6 +494,7 @@ enum devlink_attr {
>>
>> 	DEVLINK_ATTR_RELOAD_ACTION,		/* u8 */
>> 	DEVLINK_ATTR_RELOAD_ACTIONS_PERFORMED,	/* nested */
>> +	DEVLINK_ATTR_RELOAD_ACTION_LIMIT_LEVEL,	/* u8 */
>>
>> 	/* add new attributes above here, update the policy in devlink.c */
>>
>> diff --git a/net/core/devlink.c b/net/core/devlink.c
>> index f4be1e1bf864..60aa0c4a3726 100644
>> --- a/net/core/devlink.c
>> +++ b/net/core/devlink.c
>> @@ -468,6 +468,13 @@ devlink_reload_action_is_supported(struct devlink *devlink, enum devlink_reload_
>> 	return test_bit(action, &devlink->ops->supported_reload_actions);
>> }
>>
>> +static bool
>> +devlink_reload_action_limit_level_is_supported(struct devlink *devlink,
>> +					       enum devlink_reload_action_limit_level limit_level)
>> +{
>> +	return test_bit(limit_level, &devlink->ops->supported_reload_action_limit_levels);
>> +}
>> +
>> static int devlink_nl_fill(struct sk_buff *msg, struct devlink *devlink,
>> 			   enum devlink_command cmd, u32 portid,
>> 			   u32 seq, int flags)
>> @@ -2975,22 +2982,23 @@ bool devlink_is_reload_failed(const struct devlink *devlink)
>> EXPORT_SYMBOL_GPL(devlink_is_reload_failed);
>>
>> static int devlink_reload(struct devlink *devlink, struct net *dest_net,
>> -			  enum devlink_reload_action action, struct netlink_ext_ack *extack,
>> -			  unsigned long *actions_performed)
>> +			  enum devlink_reload_action action,
>> +			  enum devlink_reload_action_limit_level limit_level,
>> +			  struct netlink_ext_ack *extack, unsigned long *actions_performed)
>> {
>> 	int err;
>>
>> 	if (!devlink->reload_enabled)
>> 		return -EOPNOTSUPP;
>>
>> -	err = devlink->ops->reload_down(devlink, !!dest_net, action, extack);
>> +	err = devlink->ops->reload_down(devlink, !!dest_net, action, limit_level, extack);
>> 	if (err)
>> 		return err;
>>
>> 	if (dest_net && !net_eq(dest_net, devlink_net(devlink)))
>> 		devlink_reload_netns_change(devlink, dest_net);
>>
>> -	err = devlink->ops->reload_up(devlink, action, extack, actions_performed);
>> +	err = devlink->ops->reload_up(devlink, action, limit_level, extack, actions_performed);
>> 	devlink_reload_failed_set(devlink, !!err);
>> 	return err;
>> }
>> @@ -3036,6 +3044,7 @@ devlink_nl_reload_actions_performed_fill(struct sk_buff *msg,
>>
>> static int devlink_nl_cmd_reload(struct sk_buff *skb, struct genl_info *info)
>> {
>> +	enum devlink_reload_action_limit_level limit_level;
>> 	struct devlink *devlink = info->user_ptr[0];
>> 	enum devlink_reload_action action;
>> 	unsigned long actions_performed;
>> @@ -3073,7 +3082,20 @@ static int devlink_nl_cmd_reload(struct sk_buff *skb, struct genl_info *info)
>> 		return -EOPNOTSUPP;
>> 	}
>>
>> -	err = devlink_reload(devlink, dest_net, action, info->extack, &actions_performed);
>> +	if (info->attrs[DEVLINK_ATTR_RELOAD_ACTION_LIMIT_LEVEL])
>> +		limit_level = nla_get_u8(info->attrs[DEVLINK_ATTR_RELOAD_ACTION_LIMIT_LEVEL]);
>> +	else
>> +		limit_level = DEVLINK_RELOAD_ACTION_LIMIT_LEVEL_NONE;
>> +
>> +	if (limit_level > DEVLINK_RELOAD_ACTION_LIMIT_LEVEL_MAX) {
> Again, not needed, devlink_reload_action_limit_level_is_supported() will
> take case of it.
Ack.
>> +		NL_SET_ERR_MSG_MOD(info->extack, "Invalid limit level");
>> +		return -EINVAL;
>> +	} else if (!devlink_reload_action_limit_level_is_supported(devlink, limit_level)) {
>> +		NL_SET_ERR_MSG_MOD(info->extack, "Requested limit level is not supported");
> "..by the driver"?
Ack.
>
>> +		return -EOPNOTSUPP;
>> +	}
>> +	err = devlink_reload(devlink, dest_net, action, limit_level, info->extack,
>> +			     &actions_performed);
>>
>> 	if (dest_net)
>> 		put_net(dest_net);
>> @@ -7126,6 +7148,7 @@ static const struct nla_policy devlink_nl_policy[DEVLINK_ATTR_MAX + 1] = {
>> 	[DEVLINK_ATTR_TRAP_POLICER_BURST] = { .type = NLA_U64 },
>> 	[DEVLINK_ATTR_PORT_FUNCTION] = { .type = NLA_NESTED },
>> 	[DEVLINK_ATTR_RELOAD_ACTION] = { .type = NLA_U8 },
>> +	[DEVLINK_ATTR_RELOAD_ACTION_LIMIT_LEVEL] = { .type = NLA_U8 },
>> };
>>
>> static const struct genl_ops devlink_nl_ops[] = {
>> @@ -7462,6 +7485,10 @@ static int devlink_reload_actions_verify(struct devlink *devlink)
>> 	if (WARN_ON(ops->supported_reload_actions >= BIT(__DEVLINK_RELOAD_ACTION_MAX) ||
>> 		    ops->supported_reload_actions <= BIT(DEVLINK_RELOAD_ACTION_UNSPEC)))
>> 		return -EINVAL;
>> +	if (WARN_ON(!ops->supported_reload_action_limit_levels ||
>> +		    ops->supported_reload_action_limit_levels >=
>> +		    BIT(__DEVLINK_RELOAD_ACTION_LIMIT_LEVEL_MAX)))
>> +		return -EINVAL;
> I think that you can check some insane driver combinations like:
> supports only driver-reinit, supports LEVEL_NO_RESET - that is
> impossible and should be refused here.
>
> Same goes to the actual user command call. If the user calls for
> driver-reinit with LEVEL_NO_RESET, devlink should refuse with proper
> extack


I actually holds a counter for this combination too, we said no_reset 
can apply to any action, but not really.

>
>> 	return 0;
>> }
>>
>> @@ -9756,7 +9783,8 @@ static void __net_exit devlink_pernet_pre_exit(struct net *net)
>> 			if (WARN_ON(!devlink_reload_supported(devlink)))
>> 				continue;
>> 			err = devlink_reload(devlink, &init_net,
>> -					     DEVLINK_RELOAD_ACTION_DRIVER_REINIT, NULL, NULL);
>> +					     DEVLINK_RELOAD_ACTION_DRIVER_REINIT,
>> +					     DEVLINK_RELOAD_ACTION_LIMIT_LEVEL_NONE, NULL, NULL);
>> 			if (err && err != -EOPNOTSUPP)
>> 				pr_warn("Failed to reload devlink instance into init_net\n");
>> 		}
>> -- 
>> 2.17.1
>>

Re: [PATCH net-next RFC v4 03/15] devlink: Add reload action stats

[Moshe Shemesh]

On 9/14/2020 4:39 PM, Jiri Pirko wrote:
> Mon, Sep 14, 2020 at 08:07:50AM CEST, moshe@mellanox.com wrote:
>> Add reload action stats to hold the history per reload action type and
>> limit level.
> Empty line missing.
>

Ack.

>> For example, the number of times fw_activate has been performed on this
>> device since the driver module was added or if the firmware activation
>> was performed with or without reset.
>> Add devlink notification on stats update.
>>
>> The function devlink_reload_actions_implicit_actions_performed() is
>> exported to enable also drivers update on reload actions performed,
>> for example in case firmware activation with reset finished
>> successfully but was initiated by remote host.
>>
>> Signed-off-by: Moshe Shemesh <moshe@mellanox.com>
>> ---
>> v3 -> v4:
>> - Renamed reload_actions_cnts to reload_action_stats
>> - Add devlink notifications on stats update
>> - Renamed devlink_reload_actions_implicit_actions_performed() and add
>>   function comment in code
>> v2 -> v3:
>> - New patch
>> ---
>> include/net/devlink.h |  7 ++++++
>> net/core/devlink.c    | 58 ++++++++++++++++++++++++++++++++++++++++---
>> 2 files changed, 62 insertions(+), 3 deletions(-)
>>
>> diff --git a/include/net/devlink.h b/include/net/devlink.h
>> index dddd9ee5b8a9..b4feb92e0269 100644
>> --- a/include/net/devlink.h
>> +++ b/include/net/devlink.h
>> @@ -20,6 +20,9 @@
>> #include <uapi/linux/devlink.h>
>> #include <linux/xarray.h>
>>
>> +#define DEVLINK_RELOAD_ACTION_STATS_ARRAY_SIZE \
>> +	(__DEVLINK_RELOAD_ACTION_LIMIT_LEVEL_MAX * __DEVLINK_RELOAD_ACTION_MAX)
>> +
>> struct devlink_ops;
>>
>> struct devlink {
>> @@ -38,6 +41,7 @@ struct devlink {
>> 	struct list_head trap_policer_list;
>> 	const struct devlink_ops *ops;
>> 	struct xarray snapshot_ids;
>> +	u32 reload_action_stats[DEVLINK_RELOAD_ACTION_STATS_ARRAY_SIZE];
>> 	struct device *dev;
>> 	possible_net_t _net;
>> 	struct mutex lock; /* Serializes access to devlink instance specific objects such as
>> @@ -1397,6 +1401,9 @@ void
>> devlink_health_reporter_recovery_done(struct devlink_health_reporter *reporter);
>>
>> bool devlink_is_reload_failed(const struct devlink *devlink);
>> +void devlink_reload_implicit_actions_performed(struct devlink *devlink,
>> +					       enum devlink_reload_action_limit_level limit_level,
>> +					       unsigned long actions_performed);
>>
>> void devlink_flash_update_begin_notify(struct devlink *devlink);
>> void devlink_flash_update_end_notify(struct devlink *devlink);
>> diff --git a/net/core/devlink.c b/net/core/devlink.c
>> index 60aa0c4a3726..cbf746966913 100644
>> --- a/net/core/devlink.c
>> +++ b/net/core/devlink.c
>> @@ -2981,11 +2981,58 @@ bool devlink_is_reload_failed(const struct devlink *devlink)
>> }
>> EXPORT_SYMBOL_GPL(devlink_is_reload_failed);
>>
>> +static void
>> +devlink_reload_action_stats_update(struct devlink *devlink,
>> +				   enum devlink_reload_action_limit_level limit_level,
>> +				   unsigned long actions_performed)
>> +{
>> +	int stat_idx;
>> +	int action;
>> +
>> +	if (!actions_performed)
>> +		return;
>> +
>> +	if (WARN_ON(limit_level > DEVLINK_RELOAD_ACTION_LIMIT_LEVEL_MAX))
>> +		return;
>> +	for (action = 0; action <= DEVLINK_RELOAD_ACTION_MAX; action++) {
>> +		if (!test_bit(action, &actions_performed))
>> +			continue;
>> +		stat_idx = limit_level * __DEVLINK_RELOAD_ACTION_MAX + action;
>> +		devlink->reload_action_stats[stat_idx]++;
>> +	}
>> +	devlink_notify(devlink, DEVLINK_CMD_NEW);
>> +}
>> +
>> +/**
>> + *	devlink_reload_implicit_actions_performed - Update devlink on reload actions
>> + *	  performed which are not a direct result of devlink reload call.
>> + *
>> + *	This should be called by a driver after performing reload actions in case it was not
>> + *	a result of devlink reload call. For example fw_activate was performed as a result
>> + *	of devlink reload triggered fw_activate on another host.
>> + *	The motivation for this function is to keep data on reload actions performed on this
>> + *	function whether it was done due to direct devlink reload call or not.
>> + *
>> + *	@devlink: devlink
>> + *	@limit_level: reload action limit level
>> + *	@actions_performed: bitmask of actions performed
>> + */
>> +void devlink_reload_implicit_actions_performed(struct devlink *devlink,
>> +					       enum devlink_reload_action_limit_level limit_level,
>> +					       unsigned long actions_performed)
> What I'm a bit scarred of that the driver would call this from withing
> reload_down()/up() ops. Perheps this could be WARN_ON'ed here (or in
> devlink_reload())?
>

Not sure how I know if it was called from devlink_reload_down()/up() ? 
Maybe mutex ? So the warn will be actually mutex deadlock ?

>> +{
>> +	if (!devlink_reload_supported(devlink))
> Hmm. I think that the driver does not have to support the reload and can
> still be reloaded by another instance and update the stats here. Why
> not?
>

But I show counters only for supported reload actions and levels, 
otherwise we will have these counters on devlink dev show output for 
other drivers that don't have support for devlink reload and didn't 
implement any of these including this function and these drivers may do 
some actions like fw_activate in another way and don't update the stats 
and so that will make these stats misleading. They will show history 
"stats" but they don't update them as they didn't apply anything related 
to devlink reload.

>> +		return;
>> +	devlink_reload_action_stats_update(devlink, limit_level, actions_performed);
>> +}
>> +EXPORT_SYMBOL_GPL(devlink_reload_implicit_actions_performed);
>> +
>> static int devlink_reload(struct devlink *devlink, struct net *dest_net,
>> 			  enum devlink_reload_action action,
>> 			  enum devlink_reload_action_limit_level limit_level,
>> -			  struct netlink_ext_ack *extack, unsigned long *actions_performed)
>> +			  struct netlink_ext_ack *extack, unsigned long *actions_performed_out)
>> {
>> +	unsigned long actions_performed;
>> 	int err;
>>
>> 	if (!devlink->reload_enabled)
>> @@ -2998,9 +3045,14 @@ static int devlink_reload(struct devlink *devlink, struct net *dest_net,
>> 	if (dest_net && !net_eq(dest_net, devlink_net(devlink)))
>> 		devlink_reload_netns_change(devlink, dest_net);
>>
>> -	err = devlink->ops->reload_up(devlink, action, limit_level, extack, actions_performed);
>> +	err = devlink->ops->reload_up(devlink, action, limit_level, extack, &actions_performed);
>> 	devlink_reload_failed_set(devlink, !!err);
>> -	return err;
>> +	if (err)
>> +		return err;
>> +	devlink_reload_action_stats_update(devlink, limit_level, actions_performed);
>> +	if (actions_performed_out)
> Just make the caller to provide valid pointer, as I suggested in the
> other patch review.


Ack.

>
>> +		*actions_performed_out = actions_performed;
>> +	return 0;
>> }
>>
>> static int
>> -- 
>> 2.17.1
>>

Re: [PATCH net-next RFC v4 04/15] devlink: Add reload actions stats to dev get

[Moshe Shemesh]

On 9/15/2020 10:44 AM, Jiri Pirko wrote:
> Tue, Sep 15, 2020 at 08:45:19AM CEST, idosch@idosch.org wrote:
>> On Mon, Sep 14, 2020 at 03:45:00PM +0200, Jiri Pirko wrote:
>>> Mon, Sep 14, 2020 at 08:07:51AM CEST, moshe@mellanox.com wrote:
>>>> Expose devlink reload actions stats to the user through devlink dev
>>>> get command.
>>>>
>>>> Examples:
>>>> $ devlink dev show
>>>> pci/0000:82:00.0:
>>>>   reload_action_stats:
>>>>     driver_reinit 2
>>>>     fw_activate 1
>>>>     driver_reinit_no_reset 0
>>>>     fw_activate_no_reset 0
>>>> pci/0000:82:00.1:
>>>>   reload_action_stats:
>>>>     driver_reinit 1
>>>>     fw_activate 1
>>>>     driver_reinit_no_reset 0
>>>>     fw_activate_no_reset 0
>>> I would rather have something like:
>>>     stats:
>>>       reload_action:
>>>         driver_reinit 1
>>>         fw_activate 1
>>>         driver_reinit_no_reset 0
>>>         fw_activate_no_reset 0
>>>
>>> Then we can easily extend and add other stats in the tree.


Sure, I will add it.

>>>
>>> Also, I wonder if these stats could be somehow merged with Ido's metrics
>>> work:
>>> https://github.com/idosch/linux/commits/submit/devlink_metric_rfc_v1
>>>
>>> Ido, would it make sense?
>> I guess. My original idea for devlink-metric was to expose
>> design-specific metrics to user space where the entity registering the
>> metrics is the device driver. In this case the entity would be devlink
>> itself and it would be auto-registered for each device.
> Yeah, the usecase is different, but it is still stats, right.
>
>
>>>
>>>> $ devlink dev show -jp
>>>> {
>>>>     "dev": {
>>>>         "pci/0000:82:00.0": {
>>>>             "reload_action_stats": [ {
>>>>                     "driver_reinit": 2
>>>>                 },{
>>>>                     "fw_activate": 1
>>>>                 },{
>>>>                     "driver_reinit_no_reset": 0
>>>>                 },{
>>>>                     "fw_activate_no_reset": 0
>>>>                 } ]
>>>>         },
>>>>         "pci/0000:82:00.1": {
>>>>             "reload_action_stats": [ {
>>>>                     "driver_reinit": 1
>>>>                 },{
>>>>                     "fw_activate": 1
>>>>                 },{
>>>>                     "driver_reinit_no_reset": 0
>>>>                 },{
>>>>                     "fw_activate_no_reset": 0
>>>>                 } ]
>>>>         }
>>>>     }
>>>> }
>>>>
>>> [..]

Re: [PATCH net-next RFC v4 10/15] net/mlx5: Add support for devlink reload action fw activate

[Moshe Shemesh]

On 9/14/2020 4:52 PM, Jiri Pirko wrote:
> Mon, Sep 14, 2020 at 08:07:57AM CEST, moshe@mellanox.com wrote:
>> Add support for devlink reload action fw_activate. To activate firmware
>> image the mlx5 driver resets the firmware and reloads it from flash. If
>> a new image was stored on flash it will be loaded. Once this reload
>> command is executed the driver initiates fw sync reset flow, where the
>> firmware synchronizes all PFs on coming reset and driver reload.
>>
>> Signed-off-by: Moshe Shemesh <moshe@mellanox.com>
>> ---
>> v3 -> v4:
>> - Renamed actions_done to actions_performed
>> v2 -> v3:
>> - Return the reload actions done
>> - Update reload action counters if reset initiated by remote host
>> v1 -> v2:
>> - Have fw_activate action instead of fw_reset level
>> ---
>> .../net/ethernet/mellanox/mlx5/core/devlink.c | 62 ++++++++++++++++---
>> .../ethernet/mellanox/mlx5/core/fw_reset.c    | 60 ++++++++++++++++--
>> .../ethernet/mellanox/mlx5/core/fw_reset.h    |  1 +
>> 3 files changed, 109 insertions(+), 14 deletions(-)
>>
>> diff --git a/drivers/net/ethernet/mellanox/mlx5/core/devlink.c b/drivers/net/ethernet/mellanox/mlx5/core/devlink.c
>> index f6b29deaf02e..fa8f6abbea4e 100644
>> --- a/drivers/net/ethernet/mellanox/mlx5/core/devlink.c
>> +++ b/drivers/net/ethernet/mellanox/mlx5/core/devlink.c
>> @@ -4,6 +4,7 @@
>> #include <devlink.h>
>>
>> #include "mlx5_core.h"
>> +#include "fw_reset.h"
>> #include "fs_core.h"
>> #include "eswitch.h"
>>
>> @@ -88,6 +89,32 @@ mlx5_devlink_info_get(struct devlink *devlink, struct devlink_info_req *req,
>> 	return 0;
>> }
>>
>> +static int mlx5_devlink_reload_fw_activate(struct devlink *devlink, struct netlink_ext_ack *extack)
>> +{
>> +	struct mlx5_core_dev *dev = devlink_priv(devlink);
>> +	u8 reset_level, reset_type, net_port_alive;
>> +	int err;
>> +
>> +	err = mlx5_reg_mfrl_query(dev, &reset_level, &reset_type);
>> +	if (err)
>> +		return err;
>> +	if (!(reset_level & MLX5_MFRL_REG_RESET_LEVEL3)) {
>> +		NL_SET_ERR_MSG_MOD(extack, "FW activate requires reboot");
>> +		return -EINVAL;
>> +	}
>> +
>> +	net_port_alive = !!(reset_type & MLX5_MFRL_REG_RESET_TYPE_NET_PORT_ALIVE);
>> +	err = mlx5_fw_set_reset_sync(dev, net_port_alive);
>> +	if (err)
>> +		goto out;
>> +
>> +	err = mlx5_fw_wait_fw_reset_done(dev);
>> +out:
>> +	if (err)
>> +		NL_SET_ERR_MSG_MOD(extack, "FW activate command failed");
>> +	return err;
>> +}
>> +
>> static int mlx5_devlink_reload_down(struct devlink *devlink, bool netns_change,
>> 				    enum devlink_reload_action action,
>> 				    enum devlink_reload_action_limit_level limit_level,
>> @@ -95,8 +122,17 @@ static int mlx5_devlink_reload_down(struct devlink *devlink, bool netns_change,
>> {
>> 	struct mlx5_core_dev *dev = devlink_priv(devlink);
>>
>> -	mlx5_unload_one(dev, false);
>> -	return 0;
>> +	switch (action) {
>> +	case DEVLINK_RELOAD_ACTION_DRIVER_REINIT:
>> +		mlx5_unload_one(dev, false);
>> +		return 0;
>> +	case DEVLINK_RELOAD_ACTION_FW_ACTIVATE:
>> +		return mlx5_devlink_reload_fw_activate(devlink, extack);
>> +	default:
>> +		/* Unsupported action should not get to this function */
>> +		WARN_ON(1);
>> +		return -EOPNOTSUPP;
>> +	}
>> }
>>
>> static int mlx5_devlink_reload_up(struct devlink *devlink, enum devlink_reload_action action,
>> @@ -104,13 +140,22 @@ static int mlx5_devlink_reload_up(struct devlink *devlink, enum devlink_reload_a
>> 				  struct netlink_ext_ack *extack, unsigned long *actions_performed)
>> {
>> 	struct mlx5_core_dev *dev = devlink_priv(devlink);
>> -	int err;
>>
>> -	err = mlx5_load_one(dev, false);
>> -	if (err)
>> -		return err;
>> 	if (actions_performed)
>> -		*actions_performed = BIT(DEVLINK_RELOAD_ACTION_DRIVER_REINIT);
>> +		*actions_performed = BIT(action);
>> +
>> +	switch (action) {
>> +	case DEVLINK_RELOAD_ACTION_DRIVER_REINIT:
>> +	case DEVLINK_RELOAD_ACTION_FW_ACTIVATE:
>> +		/* On fw_activate action, also driver is reloaded and reinit performed */
>> +		if (actions_performed)
>> +			*actions_performed |= BIT(DEVLINK_RELOAD_ACTION_DRIVER_REINIT);
> You should set DEVLINK_RELOAD_ACTION_FW_ACTIVATE bit in actions_performed
> upon activation.
>

I have set it before the switch. As this is reload_up() in mlx5 only 
driver_up(), only driver up left to complete these actions. Also 
devlink_reload() will ignore the returned value of actions_performed if 
devlink_reload_up() failed.

>> +		return mlx5_load_one(dev, false);
>> +	default:
>> +		/* Unsupported action should not get to this function */
>> +		WARN_ON(1);
>> +		return -EOPNOTSUPP;
>> +	}
>>
>> 	return 0;
>> }
>> @@ -128,7 +173,8 @@ static const struct devlink_ops mlx5_devlink_ops = {
>> #endif
>> 	.flash_update = mlx5_devlink_flash_update,
>> 	.info_get = mlx5_devlink_info_get,
>> -	.supported_reload_actions = BIT(DEVLINK_RELOAD_ACTION_DRIVER_REINIT),
>> +	.supported_reload_actions = BIT(DEVLINK_RELOAD_ACTION_DRIVER_REINIT) |
>> +				    BIT(DEVLINK_RELOAD_ACTION_FW_ACTIVATE),
>> 	.supported_reload_action_limit_levels = BIT(DEVLINK_RELOAD_ACTION_LIMIT_LEVEL_NONE),
>> 	.reload_down = mlx5_devlink_reload_down,
>> 	.reload_up = mlx5_devlink_reload_up,
>> diff --git a/drivers/net/ethernet/mellanox/mlx5/core/fw_reset.c b/drivers/net/ethernet/mellanox/mlx5/core/fw_reset.c
>> index 61237f4836cc..550f67b00473 100644
>> --- a/drivers/net/ethernet/mellanox/mlx5/core/fw_reset.c
>> +++ b/drivers/net/ethernet/mellanox/mlx5/core/fw_reset.c
>> @@ -5,6 +5,7 @@
>>
>> enum {
>> 	MLX5_FW_RESET_FLAGS_RESET_REQUESTED,
>> +	MLX5_FW_RESET_FLAGS_PENDING_COMP
>> };
>>
>> struct mlx5_fw_reset {
>> @@ -17,6 +18,8 @@ struct mlx5_fw_reset {
>> 	struct work_struct reset_abort_work;
>> 	unsigned long reset_flags;
>> 	struct timer_list timer;
>> +	struct completion done;
>> +	int ret;
>> };
>>
>> static int mlx5_reg_mfrl_set(struct mlx5_core_dev *dev, u8 reset_level,
>> @@ -53,7 +56,14 @@ int mlx5_reg_mfrl_query(struct mlx5_core_dev *dev, u8 *reset_level, u8 *reset_ty
>>
>> int mlx5_fw_set_reset_sync(struct mlx5_core_dev *dev, u8 reset_type_sel)
>> {
>> -	return mlx5_reg_mfrl_set(dev, MLX5_MFRL_REG_RESET_LEVEL3, reset_type_sel, 0, true);
>> +	struct mlx5_fw_reset *fw_reset = dev->priv.fw_reset;
>> +	int err;
>> +
>> +	set_bit(MLX5_FW_RESET_FLAGS_PENDING_COMP, &fw_reset->reset_flags);
>> +	err =  mlx5_reg_mfrl_set(dev, MLX5_MFRL_REG_RESET_LEVEL3, reset_type_sel, 0, true);
>> +	if (err)
>> +		clear_bit(MLX5_FW_RESET_FLAGS_PENDING_COMP, &fw_reset->reset_flags);
>> +	return err;
>> }
>>
>> int mlx5_fw_set_live_patch(struct mlx5_core_dev *dev)
>> @@ -66,19 +76,36 @@ static int mlx5_fw_set_reset_sync_ack(struct mlx5_core_dev *dev)
>> 	return mlx5_reg_mfrl_set(dev, MLX5_MFRL_REG_RESET_LEVEL3, 0, 1, false);
>> }
>>
>> +static void mlx5_fw_reset_complete_reload(struct mlx5_core_dev *dev)
>> +{
>> +	struct mlx5_fw_reset *fw_reset = dev->priv.fw_reset;
>> +
>> +	/* if this is the driver that initiated the fw reset, devlink completed the reload */
>> +	if (test_bit(MLX5_FW_RESET_FLAGS_PENDING_COMP, &fw_reset->reset_flags)) {
>> +		complete(&fw_reset->done);
>> +	} else {
>> +		mlx5_load_one(dev, false);
>> +		devlink_reload_implicit_actions_performed(priv_to_devlink(dev),
>> +							  DEVLINK_RELOAD_ACTION_LIMIT_LEVEL_NONE,
>> +							  BIT(DEVLINK_RELOAD_ACTION_DRIVER_REINIT) |
>> +							  BIT(DEVLINK_RELOAD_ACTION_FW_ACTIVATE));
>> +	}
>> +}
>> +
>> static void mlx5_sync_reset_reload_work(struct work_struct *work)
>> {
>> 	struct mlx5_fw_reset *fw_reset = container_of(work, struct mlx5_fw_reset,
>> 						      reset_reload_work);
>> 	struct mlx5_core_dev *dev = fw_reset->dev;
>> +	int err;
>>
>> 	mlx5_enter_error_state(dev, true);
>> 	mlx5_unload_one(dev, false);
>> -	if (mlx5_health_wait_pci_up(dev)) {
>> +	err = mlx5_health_wait_pci_up(dev);
>> +	if (err)
>> 		mlx5_core_err(dev, "reset reload flow aborted, PCI reads still not working\n");
>> -		return;
>> -	}
>> -	mlx5_load_one(dev, false);
>> +	fw_reset->ret = err;
>> +	mlx5_fw_reset_complete_reload(dev);
>> }
>>
>> static void mlx5_stop_sync_reset_poll(struct mlx5_core_dev *dev)
>> @@ -264,7 +291,8 @@ static void mlx5_sync_reset_now_event(struct work_struct *work)
>> done:
>> 	if (err)
>> 		mlx5_start_health_poll(dev);
>> -	mlx5_load_one(dev, false);
>> +	fw_reset->ret = err;
>> +	mlx5_fw_reset_complete_reload(dev);
>> }
>>
>> static void mlx5_sync_reset_abort_event(struct work_struct *work)
>> @@ -313,6 +341,25 @@ static int fw_reset_event_notifier(struct notifier_block *nb, unsigned long acti
>> 	return NOTIFY_OK;
>> }
>>
>> +#define MLX5_FW_RESET_TIMEOUT_MSEC 5000
>> +int mlx5_fw_wait_fw_reset_done(struct mlx5_core_dev *dev)
>> +{
>> +	unsigned long timeout = msecs_to_jiffies(MLX5_FW_RESET_TIMEOUT_MSEC);
>> +	struct mlx5_fw_reset *fw_reset = dev->priv.fw_reset;
>> +	int err;
>> +
>> +	if (!wait_for_completion_timeout(&fw_reset->done, timeout)) {
>> +		mlx5_core_warn(dev, "FW sync reset timeout after %d seconds\n",
>> +			       MLX5_FW_RESET_TIMEOUT_MSEC / 1000);
>> +		err = -ETIMEDOUT;
>> +		goto out;
>> +	}
>> +	err = fw_reset->ret;
>> +out:
>> +	clear_bit(MLX5_FW_RESET_FLAGS_PENDING_COMP, &fw_reset->reset_flags);
>> +	return err;
>> +}
>> +
>> int mlx5_fw_reset_events_init(struct mlx5_core_dev *dev)
>> {
>> 	struct mlx5_fw_reset *fw_reset = kzalloc(sizeof(*fw_reset), GFP_KERNEL);
>> @@ -336,6 +383,7 @@ int mlx5_fw_reset_events_init(struct mlx5_core_dev *dev)
>> 	MLX5_NB_INIT(&fw_reset->nb, fw_reset_event_notifier, GENERAL_EVENT);
>> 	mlx5_eq_notifier_register(dev, &fw_reset->nb);
>>
>> +	init_completion(&fw_reset->done);
>> 	return 0;
>> }
>>
>> diff --git a/drivers/net/ethernet/mellanox/mlx5/core/fw_reset.h b/drivers/net/ethernet/mellanox/mlx5/core/fw_reset.h
>> index 278f538ea92a..d7ee951a2258 100644
>> --- a/drivers/net/ethernet/mellanox/mlx5/core/fw_reset.h
>> +++ b/drivers/net/ethernet/mellanox/mlx5/core/fw_reset.h
>> @@ -10,6 +10,7 @@ int mlx5_reg_mfrl_query(struct mlx5_core_dev *dev, u8 *reset_level, u8 *reset_ty
>> int mlx5_fw_set_reset_sync(struct mlx5_core_dev *dev, u8 reset_type_sel);
>> int mlx5_fw_set_live_patch(struct mlx5_core_dev *dev);
>>
>> +int mlx5_fw_wait_fw_reset_done(struct mlx5_core_dev *dev);
>> int mlx5_fw_reset_events_init(struct mlx5_core_dev *dev);
>> void mlx5_fw_reset_events_cleanup(struct mlx5_core_dev *dev);
>>
>> -- 
>> 2.17.1
>>

Re: [PATCH net-next RFC v4 10/15] net/mlx5: Add support for devlink reload action fw activate

[Moshe Shemesh]

On 9/14/2020 4:54 PM, Jiri Pirko wrote:
> Mon, Sep 14, 2020 at 08:07:57AM CEST, moshe@mellanox.com wrote:
>
> [..]
>
>> +static void mlx5_fw_reset_complete_reload(struct mlx5_core_dev *dev)
>> +{
>> +	struct mlx5_fw_reset *fw_reset = dev->priv.fw_reset;
>> +
>> +	/* if this is the driver that initiated the fw reset, devlink completed the reload */
>> +	if (test_bit(MLX5_FW_RESET_FLAGS_PENDING_COMP, &fw_reset->reset_flags)) {
>> +		complete(&fw_reset->done);
>> +	} else {
>> +		mlx5_load_one(dev, false);
>> +		devlink_reload_implicit_actions_performed(priv_to_devlink(dev),
>> +							  DEVLINK_RELOAD_ACTION_LIMIT_LEVEL_NONE,
>> +							  BIT(DEVLINK_RELOAD_ACTION_DRIVER_REINIT) |
>> +							  BIT(DEVLINK_RELOAD_ACTION_FW_ACTIVATE));
> Hmm, who originated the reset? Devlink_reload of the same devlink
> instance?


Not the same devlink instance for sure. I defer it by the flag above 
MLX5_FW_RESET_FLAG_PENDING_COMP. If the flag set, I set complete to the 
reload_down() waiting for it.


> [..]

Re: [PATCH net-next RFC v4 01/15] devlink: Add reload action option to devlink reload command

[Moshe Shemesh]

On 9/15/2020 12:33 AM, Jakub Kicinski wrote:
> External email: Use caution opening links or attachments
>
>
> On Mon, 14 Sep 2020 09:07:48 +0300 Moshe Shemesh wrote:
>> @@ -3011,12 +3060,41 @@ static int devlink_nl_cmd_reload(struct sk_buff *skb, struct genl_info *info)
>>                        return PTR_ERR(dest_net);
>>        }
>>
>> -     err = devlink_reload(devlink, dest_net, info->extack);
>> +     if (info->attrs[DEVLINK_ATTR_RELOAD_ACTION])
>> +             action = nla_get_u8(info->attrs[DEVLINK_ATTR_RELOAD_ACTION]);
>> +     else
>> +             action = DEVLINK_RELOAD_ACTION_DRIVER_REINIT;
>> +
>> +     if (action == DEVLINK_RELOAD_ACTION_UNSPEC || action > DEVLINK_RELOAD_ACTION_MAX) {
>> +             NL_SET_ERR_MSG_MOD(info->extack, "Invalid reload action");
>> +             return -EINVAL;
>> +     } else if (!devlink_reload_action_is_supported(devlink, action)) {
>> +             NL_SET_ERR_MSG_MOD(info->extack, "Requested reload action is not supported");
>> +             return -EOPNOTSUPP;
>> +     }
>> +
>> +     err = devlink_reload(devlink, dest_net, action, info->extack, &actions_performed);
>>
>>        if (dest_net)
>>                put_net(dest_net);
>>
>> -     return err;
>> +     if (err)
>> +             return err;
>> +
>> +     WARN_ON(!actions_performed);
>> +     msg = nlmsg_new(NLMSG_DEFAULT_SIZE, GFP_KERNEL);
>> +     if (!msg)
>> +             return -ENOMEM;
>> +
>> +     err = devlink_nl_reload_actions_performed_fill(msg, devlink, actions_performed,
>> +                                                    DEVLINK_CMD_RELOAD, info->snd_portid,
>> +                                                    info->snd_seq, 0);
>> +     if (err) {
>> +             nlmsg_free(msg);
>> +             return err;
>> +     }
>> +
>> +     return genlmsg_reply(msg, info);
> I think generating the reply may break existing users. Only generate
> the reply if request contained DEVLINK_ATTR_RELOAD_ACTION (or any other
> new attribute which existing users can't pass).


OK, I can do that. But I update stats and generate devlink notification 
anyway, that should fine, right ?

Re: [PATCH net-next RFC v4 01/15] devlink: Add reload action option to devlink reload command

[Jiri Pirko]

Tue, Sep 15, 2020 at 02:12:25PM CEST, moshe@nvidia.com wrote:
>
>On 9/14/2020 3:27 PM, Jiri Pirko wrote:
>> Mon, Sep 14, 2020 at 08:07:48AM CEST, moshe@mellanox.com wrote:

[..]	
	
>> > @@ -7392,6 +7485,11 @@ struct devlink *devlink_alloc(const struct devlink_ops *ops, size_t priv_size)
>> > 	if (!devlink)
>> > 		return NULL;
>> > 	devlink->ops = ops;
>> > +	if (devlink_reload_actions_verify(devlink)) {
>> Move this check to the beginning. You don't need devlink instance for
>> the check, just ops.
>
>
>Right, will fix.
>
>> also, your devlink_reload_actions_verify() function returns
>> 0/-ESOMETHING. Treat it accordingly here.
>
>
>Well, yes, but I rather return NULL here since devlink_alloc() failed. If
>devlink_reload_actions_verify() fails it has WARN_ON which will lead the
>driver developer to his bug.

So let the verify() return bool.
My point is, if a function return 0/-ESOMETHING, you should not check
the return value directly but you should use int err/ret.

>
>> 
>> > +		kfree(devlink);
>> > +		return NULL;
>> > +	}

[...]

Re: [PATCH net-next RFC v4 01/15] devlink: Add reload action option to devlink reload command

[Jiri Pirko]

Tue, Sep 15, 2020 at 02:56:48PM CEST, moshe@nvidia.com wrote:
>
>On 9/15/2020 12:33 AM, Jakub Kicinski wrote:
>> External email: Use caution opening links or attachments
>> 
>> 
>> On Mon, 14 Sep 2020 09:07:48 +0300 Moshe Shemesh wrote:
>> > @@ -3011,12 +3060,41 @@ static int devlink_nl_cmd_reload(struct sk_buff *skb, struct genl_info *info)
>> >                        return PTR_ERR(dest_net);
>> >        }
>> > 
>> > -     err = devlink_reload(devlink, dest_net, info->extack);
>> > +     if (info->attrs[DEVLINK_ATTR_RELOAD_ACTION])
>> > +             action = nla_get_u8(info->attrs[DEVLINK_ATTR_RELOAD_ACTION]);
>> > +     else
>> > +             action = DEVLINK_RELOAD_ACTION_DRIVER_REINIT;
>> > +
>> > +     if (action == DEVLINK_RELOAD_ACTION_UNSPEC || action > DEVLINK_RELOAD_ACTION_MAX) {
>> > +             NL_SET_ERR_MSG_MOD(info->extack, "Invalid reload action");
>> > +             return -EINVAL;
>> > +     } else if (!devlink_reload_action_is_supported(devlink, action)) {
>> > +             NL_SET_ERR_MSG_MOD(info->extack, "Requested reload action is not supported");
>> > +             return -EOPNOTSUPP;
>> > +     }
>> > +
>> > +     err = devlink_reload(devlink, dest_net, action, info->extack, &actions_performed);
>> > 
>> >        if (dest_net)
>> >                put_net(dest_net);
>> > 
>> > -     return err;
>> > +     if (err)
>> > +             return err;
>> > +
>> > +     WARN_ON(!actions_performed);
>> > +     msg = nlmsg_new(NLMSG_DEFAULT_SIZE, GFP_KERNEL);
>> > +     if (!msg)
>> > +             return -ENOMEM;
>> > +
>> > +     err = devlink_nl_reload_actions_performed_fill(msg, devlink, actions_performed,
>> > +                                                    DEVLINK_CMD_RELOAD, info->snd_portid,
>> > +                                                    info->snd_seq, 0);
>> > +     if (err) {
>> > +             nlmsg_free(msg);
>> > +             return err;
>> > +     }
>> > +
>> > +     return genlmsg_reply(msg, info);
>> I think generating the reply may break existing users. Only generate
>> the reply if request contained DEVLINK_ATTR_RELOAD_ACTION (or any other
>> new attribute which existing users can't pass).
>
>
>OK, I can do that. But I update stats and generate devlink notification
>anyway, that should fine, right ?

Yes.

>

Re: [PATCH net-next RFC v4 03/15] devlink: Add reload action stats

[Jiri Pirko]

Tue, Sep 15, 2020 at 02:30:19PM CEST, moshe@nvidia.com wrote:
>
>On 9/14/2020 4:39 PM, Jiri Pirko wrote:
>> Mon, Sep 14, 2020 at 08:07:50AM CEST, moshe@mellanox.com wrote:

[..]


>> > +/**
>> > + *	devlink_reload_implicit_actions_performed - Update devlink on reload actions
>> > + *	  performed which are not a direct result of devlink reload call.
>> > + *
>> > + *	This should be called by a driver after performing reload actions in case it was not
>> > + *	a result of devlink reload call. For example fw_activate was performed as a result
>> > + *	of devlink reload triggered fw_activate on another host.
>> > + *	The motivation for this function is to keep data on reload actions performed on this
>> > + *	function whether it was done due to direct devlink reload call or not.
>> > + *
>> > + *	@devlink: devlink
>> > + *	@limit_level: reload action limit level
>> > + *	@actions_performed: bitmask of actions performed
>> > + */
>> > +void devlink_reload_implicit_actions_performed(struct devlink *devlink,
>> > +					       enum devlink_reload_action_limit_level limit_level,
>> > +					       unsigned long actions_performed)
>> What I'm a bit scarred of that the driver would call this from withing
>> reload_down()/up() ops. Perheps this could be WARN_ON'ed here (or in
>> devlink_reload())?
>> 
>
>Not sure how I know if it was called from devlink_reload_down()/up() ? Maybe
>mutex ? So the warn will be actually mutex deadlock ?

No. Don't abuse mutex for this.
Just make sure that the counters do not move when you call
reload_down/up().


>
>> > +{
>> > +	if (!devlink_reload_supported(devlink))
>> Hmm. I think that the driver does not have to support the reload and can
>> still be reloaded by another instance and update the stats here. Why
>> not?
>> 
>
>But I show counters only for supported reload actions and levels, otherwise
>we will have these counters on devlink dev show output for other drivers that
>don't have support for devlink reload and didn't implement any of these
>including this function and these drivers may do some actions like
>fw_activate in another way and don't update the stats and so that will make
>these stats misleading. They will show history "stats" but they don't update
>them as they didn't apply anything related to devlink reload.

The case I tried to point at is the driver instance, that does not
implement reload ops itself, but still it can be reloaded by someone else -
the other driver instance outside.

The counters should work no matter if the driver implements reload ops
or not. Why wouldn't they? The user still likes to know that the devices
was reloaded.



>
>> > +		return;
>> > +	devlink_reload_action_stats_update(devlink, limit_level, actions_performed);
>> > +}
>> > +EXPORT_SYMBOL_GPL(devlink_reload_implicit_actions_performed);
>> > +
>> > static int devlink_reload(struct devlink *devlink, struct net *dest_net,
>> > 			  enum devlink_reload_action action,
>> > 			  enum devlink_reload_action_limit_level limit_level,
>> > -			  struct netlink_ext_ack *extack, unsigned long *actions_performed)
>> > +			  struct netlink_ext_ack *extack, unsigned long *actions_performed_out)
>> > {
>> > +	unsigned long actions_performed;
>> > 	int err;
>> > 
>> > 	if (!devlink->reload_enabled)
>> > @@ -2998,9 +3045,14 @@ static int devlink_reload(struct devlink *devlink, struct net *dest_net,
>> > 	if (dest_net && !net_eq(dest_net, devlink_net(devlink)))
>> > 		devlink_reload_netns_change(devlink, dest_net);
>> > 
>> > -	err = devlink->ops->reload_up(devlink, action, limit_level, extack, actions_performed);
>> > +	err = devlink->ops->reload_up(devlink, action, limit_level, extack, &actions_performed);
>> > 	devlink_reload_failed_set(devlink, !!err);
>> > -	return err;
>> > +	if (err)
>> > +		return err;
>> > +	devlink_reload_action_stats_update(devlink, limit_level, actions_performed);
>> > +	if (actions_performed_out)
>> Just make the caller to provide valid pointer, as I suggested in the
>> other patch review.
>
>
>Ack.
>
>> 
>> > +		*actions_performed_out = actions_performed;
>> > +	return 0;
>> > }
>> > 
>> > static int
>> > -- 
>> > 2.17.1
>> > 

Re: [PATCH net-next RFC v4 04/15] devlink: Add reload actions stats to dev get

[Jiri Pirko]

Tue, Sep 15, 2020 at 02:31:38PM CEST, moshe@nvidia.com wrote:
>
>On 9/15/2020 10:44 AM, Jiri Pirko wrote:
>> Tue, Sep 15, 2020 at 08:45:19AM CEST, idosch@idosch.org wrote:
>> > On Mon, Sep 14, 2020 at 03:45:00PM +0200, Jiri Pirko wrote:
>> > > Mon, Sep 14, 2020 at 08:07:51AM CEST, moshe@mellanox.com wrote:
>> > > > Expose devlink reload actions stats to the user through devlink dev
>> > > > get command.
>> > > > 
>> > > > Examples:
>> > > > $ devlink dev show
>> > > > pci/0000:82:00.0:
>> > > >   reload_action_stats:
>> > > >     driver_reinit 2
>> > > >     fw_activate 1
>> > > >     driver_reinit_no_reset 0
>> > > >     fw_activate_no_reset 0
>> > > > pci/0000:82:00.1:
>> > > >   reload_action_stats:
>> > > >     driver_reinit 1
>> > > >     fw_activate 1
>> > > >     driver_reinit_no_reset 0
>> > > >     fw_activate_no_reset 0
>> > > I would rather have something like:
>> > >     stats:
>> > >       reload_action:
>> > >         driver_reinit 1
>> > >         fw_activate 1
>> > >         driver_reinit_no_reset 0
>> > >         fw_activate_no_reset 0
>> > > 
>> > > Then we can easily extend and add other stats in the tree.
>
>
>Sure, I will add it.

Could you please checkout the metrics patchset and figure out how to
merge that with your usecase?


>
>> > > 
>> > > Also, I wonder if these stats could be somehow merged with Ido's metrics
>> > > work:
>> > > https://github.com/idosch/linux/commits/submit/devlink_metric_rfc_v1
>> > > 
>> > > Ido, would it make sense?
>> > I guess. My original idea for devlink-metric was to expose
>> > design-specific metrics to user space where the entity registering the
>> > metrics is the device driver. In this case the entity would be devlink
>> > itself and it would be auto-registered for each device.
>> Yeah, the usecase is different, but it is still stats, right.
>> 
>> 
>> > > 
>> > > > $ devlink dev show -jp
>> > > > {
>> > > >     "dev": {
>> > > >         "pci/0000:82:00.0": {
>> > > >             "reload_action_stats": [ {
>> > > >                     "driver_reinit": 2
>> > > >                 },{
>> > > >                     "fw_activate": 1
>> > > >                 },{
>> > > >                     "driver_reinit_no_reset": 0
>> > > >                 },{
>> > > >                     "fw_activate_no_reset": 0
>> > > >                 } ]
>> > > >         },
>> > > >         "pci/0000:82:00.1": {
>> > > >             "reload_action_stats": [ {
>> > > >                     "driver_reinit": 1
>> > > >                 },{
>> > > >                     "fw_activate": 1
>> > > >                 },{
>> > > >                     "driver_reinit_no_reset": 0
>> > > >                 },{
>> > > >                     "fw_activate_no_reset": 0
>> > > >                 } ]
>> > > >         }
>> > > >     }
>> > > > }
>> > > > 
>> > > [..]

Re: [PATCH net-next RFC v4 10/15] net/mlx5: Add support for devlink reload action fw activate

[Jiri Pirko]

Tue, Sep 15, 2020 at 02:44:02PM CEST, moshe@nvidia.com wrote:
>
>On 9/14/2020 4:54 PM, Jiri Pirko wrote:
>> Mon, Sep 14, 2020 at 08:07:57AM CEST, moshe@mellanox.com wrote:
>> 
>> [..]
>> 
>> > +static void mlx5_fw_reset_complete_reload(struct mlx5_core_dev *dev)
>> > +{
>> > +	struct mlx5_fw_reset *fw_reset = dev->priv.fw_reset;
>> > +
>> > +	/* if this is the driver that initiated the fw reset, devlink completed the reload */
>> > +	if (test_bit(MLX5_FW_RESET_FLAGS_PENDING_COMP, &fw_reset->reset_flags)) {
>> > +		complete(&fw_reset->done);
>> > +	} else {
>> > +		mlx5_load_one(dev, false);
>> > +		devlink_reload_implicit_actions_performed(priv_to_devlink(dev),
>> > +							  DEVLINK_RELOAD_ACTION_LIMIT_LEVEL_NONE,
>> > +							  BIT(DEVLINK_RELOAD_ACTION_DRIVER_REINIT) |
>> > +							  BIT(DEVLINK_RELOAD_ACTION_FW_ACTIVATE));
>> Hmm, who originated the reset? Devlink_reload of the same devlink
>> instance?
>
>
>Not the same devlink instance for sure. I defer it by the flag above
>MLX5_FW_RESET_FLAG_PENDING_COMP. If the flag set, I set complete to the
>reload_down() waiting for it.

Hmm, thinking about the stats, as
devlink_reload_implicit_actions_performed() is called only in case
another instance does the reload, shouldn't it be a separate set of
stats? I think that the user would like to distinguish local and remote
reload, don't you think?


>
>
>> [..]

Re: [PATCH net-next RFC v4 01/15] devlink: Add reload action option to devlink reload command

[Jakub Kicinski]

On Tue, 15 Sep 2020 15:56:48 +0300 Moshe Shemesh wrote:
> On 9/15/2020 12:33 AM, Jakub Kicinski wrote:
> >> +     if (err)
> >> +             return err;
> >> +
> >> +     WARN_ON(!actions_performed);
> >> +     msg = nlmsg_new(NLMSG_DEFAULT_SIZE, GFP_KERNEL);
> >> +     if (!msg)
> >> +             return -ENOMEM;
> >> +
> >> +     err = devlink_nl_reload_actions_performed_fill(msg, devlink, actions_performed,
> >> +                                                    DEVLINK_CMD_RELOAD, info->snd_portid,
> >> +                                                    info->snd_seq, 0);
> >> +     if (err) {
> >> +             nlmsg_free(msg);
> >> +             return err;
> >> +     }
> >> +
> >> +     return genlmsg_reply(msg, info);  
> > I think generating the reply may break existing users. Only generate
> > the reply if request contained DEVLINK_ATTR_RELOAD_ACTION (or any other
> > new attribute which existing users can't pass).  
> 
> OK, I can do that. But I update stats and generate devlink notification 
> anyway, that should fine, right ?

Yes, that should be fine.

Re: [PATCH net-next RFC v4 15/15] devlink: Add Documentation/networking/devlink/devlink-reload.rst

[Jakub Kicinski]

On Mon, 14 Sep 2020 09:08:02 +0300 Moshe Shemesh wrote:
> +   * - ``no_reset``
> +     - No reset allowed, no down time allowed, no link flap and no
> +       configuration is lost.

It still takes the PCI link down for up to 2sec. So there is down time,
right?

Re: [PATCH net-next RFC v4 15/15] devlink: Add Documentation/networking/devlink/devlink-reload.rst

[Moshe Shemesh]

On 9/15/2020 7:04 PM, Jakub Kicinski wrote:
> External email: Use caution opening links or attachments
>
>
> On Mon, 14 Sep 2020 09:08:02 +0300 Moshe Shemesh wrote:
>> +   * - ``no_reset``
>> +     - No reset allowed, no down time allowed, no link flap and no
>> +       configuration is lost.
> It still takes the PCI link down for up to 2sec. So there is down time,
> right?


No, the fw reset with PCI link down is categorized as fw_activate with 
limit level none.

fw_live_patch keeps the pci link up and it fits to "no_reset" limit level.

Re: [PATCH net-next RFC v4 01/15] devlink: Add reload action option to devlink reload command

[Moshe Shemesh]

On 9/15/2020 4:26 PM, Jiri Pirko wrote:
> Tue, Sep 15, 2020 at 02:12:25PM CEST, moshe@nvidia.com wrote:
>> On 9/14/2020 3:27 PM, Jiri Pirko wrote:
>>> Mon, Sep 14, 2020 at 08:07:48AM CEST, moshe@mellanox.com wrote:
> [..]	
> 	
>>>> @@ -7392,6 +7485,11 @@ struct devlink *devlink_alloc(const struct devlink_ops *ops, size_t priv_size)
>>>> 	if (!devlink)
>>>> 		return NULL;
>>>> 	devlink->ops = ops;
>>>> +	if (devlink_reload_actions_verify(devlink)) {
>>> Move this check to the beginning. You don't need devlink instance for
>>> the check, just ops.
>>
>> Right, will fix.
>>
>>> also, your devlink_reload_actions_verify() function returns
>>> 0/-ESOMETHING. Treat it accordingly here.
>>
>> Well, yes, but I rather return NULL here since devlink_alloc() failed. If
>> devlink_reload_actions_verify() fails it has WARN_ON which will lead the
>> driver developer to his bug.
> So let the verify() return bool.
> My point is, if a function return 0/-ESOMETHING, you should not check
> the return value directly but you should use int err/ret.


OK, will fix.

>>>> +		kfree(devlink);
>>>> +		return NULL;
>>>> +	}
> [...]

Re: [PATCH net-next RFC v4 03/15] devlink: Add reload action stats

[Moshe Shemesh]

On 9/15/2020 4:33 PM, Jiri Pirko wrote:
> Tue, Sep 15, 2020 at 02:30:19PM CEST, moshe@nvidia.com wrote:
>> On 9/14/2020 4:39 PM, Jiri Pirko wrote:
>>> Mon, Sep 14, 2020 at 08:07:50AM CEST, moshe@mellanox.com wrote:
> [..]
>
>
>>>> +/**
>>>> + *	devlink_reload_implicit_actions_performed - Update devlink on reload actions
>>>> + *	  performed which are not a direct result of devlink reload call.
>>>> + *
>>>> + *	This should be called by a driver after performing reload actions in case it was not
>>>> + *	a result of devlink reload call. For example fw_activate was performed as a result
>>>> + *	of devlink reload triggered fw_activate on another host.
>>>> + *	The motivation for this function is to keep data on reload actions performed on this
>>>> + *	function whether it was done due to direct devlink reload call or not.
>>>> + *
>>>> + *	@devlink: devlink
>>>> + *	@limit_level: reload action limit level
>>>> + *	@actions_performed: bitmask of actions performed
>>>> + */
>>>> +void devlink_reload_implicit_actions_performed(struct devlink *devlink,
>>>> +					       enum devlink_reload_action_limit_level limit_level,
>>>> +					       unsigned long actions_performed)
>>> What I'm a bit scarred of that the driver would call this from withing
>>> reload_down()/up() ops. Perheps this could be WARN_ON'ed here (or in
>>> devlink_reload())?
>>>
>> Not sure how I know if it was called from devlink_reload_down()/up() ? Maybe
>> mutex ? So the warn will be actually mutex deadlock ?
> No. Don't abuse mutex for this.
> Just make sure that the counters do not move when you call
> reload_down/up().
>

Can make that, but actually I better take devlink->lock anyway in this 
function to avoid races, WDYT ?

>>>> +{
>>>> +	if (!devlink_reload_supported(devlink))
>>> Hmm. I think that the driver does not have to support the reload and can
>>> still be reloaded by another instance and update the stats here. Why
>>> not?
>>>
>> But I show counters only for supported reload actions and levels, otherwise
>> we will have these counters on devlink dev show output for other drivers that
>> don't have support for devlink reload and didn't implement any of these
>> including this function and these drivers may do some actions like
>> fw_activate in another way and don't update the stats and so that will make
>> these stats misleading. They will show history "stats" but they don't update
>> them as they didn't apply anything related to devlink reload.
> The case I tried to point at is the driver instance, that does not
> implement reload ops itself, but still it can be reloaded by someone else -
> the other driver instance outside.
>
> The counters should work no matter if the driver implements reload ops
> or not. Why wouldn't they? The user still likes to know that the devices
> was reloaded.
>

OK, so you say that every driver should show all counters no matter what 
actions it supports and if it supports devlink reload at all, right ?

>
>>>> +		return;
>>>> +	devlink_reload_action_stats_update(devlink, limit_level, actions_performed);
>>>> +}
>>>> +EXPORT_SYMBOL_GPL(devlink_reload_implicit_actions_performed);
>>>> +
>>>> static int devlink_reload(struct devlink *devlink, struct net *dest_net,
>>>> 			  enum devlink_reload_action action,
>>>> 			  enum devlink_reload_action_limit_level limit_level,
>>>> -			  struct netlink_ext_ack *extack, unsigned long *actions_performed)
>>>> +			  struct netlink_ext_ack *extack, unsigned long *actions_performed_out)
>>>> {
>>>> +	unsigned long actions_performed;
>>>> 	int err;
>>>>
>>>> 	if (!devlink->reload_enabled)
>>>> @@ -2998,9 +3045,14 @@ static int devlink_reload(struct devlink *devlink, struct net *dest_net,
>>>> 	if (dest_net && !net_eq(dest_net, devlink_net(devlink)))
>>>> 		devlink_reload_netns_change(devlink, dest_net);
>>>>
>>>> -	err = devlink->ops->reload_up(devlink, action, limit_level, extack, actions_performed);
>>>> +	err = devlink->ops->reload_up(devlink, action, limit_level, extack, &actions_performed);
>>>> 	devlink_reload_failed_set(devlink, !!err);
>>>> -	return err;
>>>> +	if (err)
>>>> +		return err;
>>>> +	devlink_reload_action_stats_update(devlink, limit_level, actions_performed);
>>>> +	if (actions_performed_out)
>>> Just make the caller to provide valid pointer, as I suggested in the
>>> other patch review.
>>
>> Ack.
>>
>>>> +		*actions_performed_out = actions_performed;
>>>> +	return 0;
>>>> }
>>>>
>>>> static int
>>>> -- 
>>>> 2.17.1
>>>>

Re: [PATCH net-next RFC v4 10/15] net/mlx5: Add support for devlink reload action fw activate

[Moshe Shemesh]

On 9/15/2020 4:37 PM, Jiri Pirko wrote:
> Tue, Sep 15, 2020 at 02:44:02PM CEST, moshe@nvidia.com wrote:
>> On 9/14/2020 4:54 PM, Jiri Pirko wrote:
>>> Mon, Sep 14, 2020 at 08:07:57AM CEST, moshe@mellanox.com wrote:
>>>
>>> [..]
>>>
>>>> +static void mlx5_fw_reset_complete_reload(struct mlx5_core_dev *dev)
>>>> +{
>>>> +	struct mlx5_fw_reset *fw_reset = dev->priv.fw_reset;
>>>> +
>>>> +	/* if this is the driver that initiated the fw reset, devlink completed the reload */
>>>> +	if (test_bit(MLX5_FW_RESET_FLAGS_PENDING_COMP, &fw_reset->reset_flags)) {
>>>> +		complete(&fw_reset->done);
>>>> +	} else {
>>>> +		mlx5_load_one(dev, false);
>>>> +		devlink_reload_implicit_actions_performed(priv_to_devlink(dev),
>>>> +							  DEVLINK_RELOAD_ACTION_LIMIT_LEVEL_NONE,
>>>> +							  BIT(DEVLINK_RELOAD_ACTION_DRIVER_REINIT) |
>>>> +							  BIT(DEVLINK_RELOAD_ACTION_FW_ACTIVATE));
>>> Hmm, who originated the reset? Devlink_reload of the same devlink
>>> instance?
>>
>> Not the same devlink instance for sure. I defer it by the flag above
>> MLX5_FW_RESET_FLAG_PENDING_COMP. If the flag set, I set complete to the
>> reload_down() waiting for it.
> Hmm, thinking about the stats, as
> devlink_reload_implicit_actions_performed() is called only in case
> another instance does the reload, shouldn't it be a separate set of
> stats? I think that the user would like to distinguish local and remote
> reload, don't you think?
>

Possible, it will double the counters, but it will give more info.

So actually, if devlink_reload is not supported by driver, I should hold 
and show only the remote stats or all stats always ?

How such remote counter should look like ? something like 
remote_fw_activate  while the local is just fw_activate ?

>>
>>> [..]

Re: [PATCH net-next RFC v4 04/15] devlink: Add reload actions stats to dev get

[Moshe Shemesh]

On 9/15/2020 4:34 PM, Jiri Pirko wrote:
> Tue, Sep 15, 2020 at 02:31:38PM CEST, moshe@nvidia.com wrote:
>> On 9/15/2020 10:44 AM, Jiri Pirko wrote:
>>> Tue, Sep 15, 2020 at 08:45:19AM CEST, idosch@idosch.org wrote:
>>>> On Mon, Sep 14, 2020 at 03:45:00PM +0200, Jiri Pirko wrote:
>>>>> Mon, Sep 14, 2020 at 08:07:51AM CEST, moshe@mellanox.com wrote:
>>>>>> Expose devlink reload actions stats to the user through devlink dev
>>>>>> get command.
>>>>>>
>>>>>> Examples:
>>>>>> $ devlink dev show
>>>>>> pci/0000:82:00.0:
>>>>>>    reload_action_stats:
>>>>>>      driver_reinit 2
>>>>>>      fw_activate 1
>>>>>>      driver_reinit_no_reset 0
>>>>>>      fw_activate_no_reset 0
>>>>>> pci/0000:82:00.1:
>>>>>>    reload_action_stats:
>>>>>>      driver_reinit 1
>>>>>>      fw_activate 1
>>>>>>      driver_reinit_no_reset 0
>>>>>>      fw_activate_no_reset 0
>>>>> I would rather have something like:
>>>>>      stats:
>>>>>        reload_action:
>>>>>          driver_reinit 1
>>>>>          fw_activate 1
>>>>>          driver_reinit_no_reset 0
>>>>>          fw_activate_no_reset 0
>>>>>
>>>>> Then we can easily extend and add other stats in the tree.
>>
>> Sure, I will add it.
> Could you please checkout the metrics patchset and figure out how to
> merge that with your usecase?
>

I will check, I will discuss with Ido how it will fit.

>>>>> Also, I wonder if these stats could be somehow merged with Ido's metrics
>>>>> work:
>>>>> https://github.com/idosch/linux/commits/submit/devlink_metric_rfc_v1
>>>>>
>>>>> Ido, would it make sense?
>>>> I guess. My original idea for devlink-metric was to expose
>>>> design-specific metrics to user space where the entity registering the
>>>> metrics is the device driver. In this case the entity would be devlink
>>>> itself and it would be auto-registered for each device.
>>> Yeah, the usecase is different, but it is still stats, right.
>>>
>>>
>>>>>> $ devlink dev show -jp
>>>>>> {
>>>>>>      "dev": {
>>>>>>          "pci/0000:82:00.0": {
>>>>>>              "reload_action_stats": [ {
>>>>>>                      "driver_reinit": 2
>>>>>>                  },{
>>>>>>                      "fw_activate": 1
>>>>>>                  },{
>>>>>>                      "driver_reinit_no_reset": 0
>>>>>>                  },{
>>>>>>                      "fw_activate_no_reset": 0
>>>>>>                  } ]
>>>>>>          },
>>>>>>          "pci/0000:82:00.1": {
>>>>>>              "reload_action_stats": [ {
>>>>>>                      "driver_reinit": 1
>>>>>>                  },{
>>>>>>                      "fw_activate": 1
>>>>>>                  },{
>>>>>>                      "driver_reinit_no_reset": 0
>>>>>>                  },{
>>>>>>                      "fw_activate_no_reset": 0
>>>>>>                  } ]
>>>>>>          }
>>>>>>      }
>>>>>> }
>>>>>>
>>>>> [..]

Re: [PATCH net-next RFC v4 03/15] devlink: Add reload action stats

[Jiri Pirko]

Tue, Sep 15, 2020 at 10:20:39PM CEST, moshe@nvidia.com wrote:
>
>On 9/15/2020 4:33 PM, Jiri Pirko wrote:
>> Tue, Sep 15, 2020 at 02:30:19PM CEST, moshe@nvidia.com wrote:
>> > On 9/14/2020 4:39 PM, Jiri Pirko wrote:
>> > > Mon, Sep 14, 2020 at 08:07:50AM CEST, moshe@mellanox.com wrote:
>> [..]
>> 
>> 
>> > > > +/**
>> > > > + *	devlink_reload_implicit_actions_performed - Update devlink on reload actions
>> > > > + *	  performed which are not a direct result of devlink reload call.
>> > > > + *
>> > > > + *	This should be called by a driver after performing reload actions in case it was not
>> > > > + *	a result of devlink reload call. For example fw_activate was performed as a result
>> > > > + *	of devlink reload triggered fw_activate on another host.
>> > > > + *	The motivation for this function is to keep data on reload actions performed on this
>> > > > + *	function whether it was done due to direct devlink reload call or not.
>> > > > + *
>> > > > + *	@devlink: devlink
>> > > > + *	@limit_level: reload action limit level
>> > > > + *	@actions_performed: bitmask of actions performed
>> > > > + */
>> > > > +void devlink_reload_implicit_actions_performed(struct devlink *devlink,
>> > > > +					       enum devlink_reload_action_limit_level limit_level,
>> > > > +					       unsigned long actions_performed)
>> > > What I'm a bit scarred of that the driver would call this from withing
>> > > reload_down()/up() ops. Perheps this could be WARN_ON'ed here (or in
>> > > devlink_reload())?
>> > > 
>> > Not sure how I know if it was called from devlink_reload_down()/up() ? Maybe
>> > mutex ? So the warn will be actually mutex deadlock ?
>> No. Don't abuse mutex for this.
>> Just make sure that the counters do not move when you call
>> reload_down/up().
>> 
>
>Can make that, but actually I better take devlink->lock anyway in this
>function to avoid races, WDYT ?

Either you need to protect some data or not. So if you do, do it.


>
>> > > > +{
>> > > > +	if (!devlink_reload_supported(devlink))
>> > > Hmm. I think that the driver does not have to support the reload and can
>> > > still be reloaded by another instance and update the stats here. Why
>> > > not?
>> > > 
>> > But I show counters only for supported reload actions and levels, otherwise
>> > we will have these counters on devlink dev show output for other drivers that
>> > don't have support for devlink reload and didn't implement any of these
>> > including this function and these drivers may do some actions like
>> > fw_activate in another way and don't update the stats and so that will make
>> > these stats misleading. They will show history "stats" but they don't update
>> > them as they didn't apply anything related to devlink reload.
>> The case I tried to point at is the driver instance, that does not
>> implement reload ops itself, but still it can be reloaded by someone else -
>> the other driver instance outside.
>> 
>> The counters should work no matter if the driver implements reload ops
>> or not. Why wouldn't they? The user still likes to know that the devices
>> was reloaded.
>> 
>
>OK, so you say that every driver should show all counters no matter what
>actions it supports and if it supports devlink reload at all, right ?

Well, as I wrote in the other email, I think that there should be 2 sets
of stats for this.


>
>> 
>> > > > +		return;
>> > > > +	devlink_reload_action_stats_update(devlink, limit_level, actions_performed);
>> > > > +}
>> > > > +EXPORT_SYMBOL_GPL(devlink_reload_implicit_actions_performed);
>> > > > +
>> > > > static int devlink_reload(struct devlink *devlink, struct net *dest_net,
>> > > > 			  enum devlink_reload_action action,
>> > > > 			  enum devlink_reload_action_limit_level limit_level,
>> > > > -			  struct netlink_ext_ack *extack, unsigned long *actions_performed)
>> > > > +			  struct netlink_ext_ack *extack, unsigned long *actions_performed_out)
>> > > > {
>> > > > +	unsigned long actions_performed;
>> > > > 	int err;
>> > > > 
>> > > > 	if (!devlink->reload_enabled)
>> > > > @@ -2998,9 +3045,14 @@ static int devlink_reload(struct devlink *devlink, struct net *dest_net,
>> > > > 	if (dest_net && !net_eq(dest_net, devlink_net(devlink)))
>> > > > 		devlink_reload_netns_change(devlink, dest_net);
>> > > > 
>> > > > -	err = devlink->ops->reload_up(devlink, action, limit_level, extack, actions_performed);
>> > > > +	err = devlink->ops->reload_up(devlink, action, limit_level, extack, &actions_performed);
>> > > > 	devlink_reload_failed_set(devlink, !!err);
>> > > > -	return err;
>> > > > +	if (err)
>> > > > +		return err;
>> > > > +	devlink_reload_action_stats_update(devlink, limit_level, actions_performed);
>> > > > +	if (actions_performed_out)
>> > > Just make the caller to provide valid pointer, as I suggested in the
>> > > other patch review.
>> > 
>> > Ack.
>> > 
>> > > > +		*actions_performed_out = actions_performed;
>> > > > +	return 0;
>> > > > }
>> > > > 
>> > > > static int
>> > > > -- 
>> > > > 2.17.1
>> > > > 

Re: [PATCH net-next RFC v4 10/15] net/mlx5: Add support for devlink reload action fw activate

[Jiri Pirko]

Tue, Sep 15, 2020 at 10:28:44PM CEST, moshe@nvidia.com wrote:
>
>On 9/15/2020 4:37 PM, Jiri Pirko wrote:
>> Tue, Sep 15, 2020 at 02:44:02PM CEST, moshe@nvidia.com wrote:
>> > On 9/14/2020 4:54 PM, Jiri Pirko wrote:
>> > > Mon, Sep 14, 2020 at 08:07:57AM CEST, moshe@mellanox.com wrote:
>> > > 
>> > > [..]
>> > > 
>> > > > +static void mlx5_fw_reset_complete_reload(struct mlx5_core_dev *dev)
>> > > > +{
>> > > > +	struct mlx5_fw_reset *fw_reset = dev->priv.fw_reset;
>> > > > +
>> > > > +	/* if this is the driver that initiated the fw reset, devlink completed the reload */
>> > > > +	if (test_bit(MLX5_FW_RESET_FLAGS_PENDING_COMP, &fw_reset->reset_flags)) {
>> > > > +		complete(&fw_reset->done);
>> > > > +	} else {
>> > > > +		mlx5_load_one(dev, false);
>> > > > +		devlink_reload_implicit_actions_performed(priv_to_devlink(dev),
>> > > > +							  DEVLINK_RELOAD_ACTION_LIMIT_LEVEL_NONE,
>> > > > +							  BIT(DEVLINK_RELOAD_ACTION_DRIVER_REINIT) |
>> > > > +							  BIT(DEVLINK_RELOAD_ACTION_FW_ACTIVATE));
>> > > Hmm, who originated the reset? Devlink_reload of the same devlink
>> > > instance?
>> > 
>> > Not the same devlink instance for sure. I defer it by the flag above
>> > MLX5_FW_RESET_FLAG_PENDING_COMP. If the flag set, I set complete to the
>> > reload_down() waiting for it.
>> Hmm, thinking about the stats, as
>> devlink_reload_implicit_actions_performed() is called only in case
>> another instance does the reload, shouldn't it be a separate set of
>> stats? I think that the user would like to distinguish local and remote
>> reload, don't you think?
>> 
>
>Possible, it will double the counters, but it will give more info.
>
>So actually, if devlink_reload is not supported by driver, I should hold and
>show only the remote stats or all stats always ?

It would make sense to show just remote stats.

>
>How such remote counter should look like ? something like remote_fw_activate 
>while the local is just fw_activate ?

Sounds good.


>
>> > 
>> > > [..]

Re: [PATCH net-next RFC v4 04/15] devlink: Add reload actions stats to dev get

[Moshe Shemesh]

On 9/15/2020 11:33 PM, Moshe Shemesh wrote:
> External email: Use caution opening links or attachments
>
>
> On 9/15/2020 4:34 PM, Jiri Pirko wrote:
>> Tue, Sep 15, 2020 at 02:31:38PM CEST, moshe@nvidia.com wrote:
>>> On 9/15/2020 10:44 AM, Jiri Pirko wrote:
>>>> Tue, Sep 15, 2020 at 08:45:19AM CEST, idosch@idosch.org wrote:
>>>>> On Mon, Sep 14, 2020 at 03:45:00PM +0200, Jiri Pirko wrote:
>>>>>> Mon, Sep 14, 2020 at 08:07:51AM CEST, moshe@mellanox.com wrote:
>>>>>>> Expose devlink reload actions stats to the user through devlink dev
>>>>>>> get command.
>>>>>>>
>>>>>>> Examples:
>>>>>>> $ devlink dev show
>>>>>>> pci/0000:82:00.0:
>>>>>>>    reload_action_stats:
>>>>>>>      driver_reinit 2
>>>>>>>      fw_activate 1
>>>>>>>      driver_reinit_no_reset 0
>>>>>>>      fw_activate_no_reset 0
>>>>>>> pci/0000:82:00.1:
>>>>>>>    reload_action_stats:
>>>>>>>      driver_reinit 1
>>>>>>>      fw_activate 1
>>>>>>>      driver_reinit_no_reset 0
>>>>>>>      fw_activate_no_reset 0
>>>>>> I would rather have something like:
>>>>>>      stats:
>>>>>>        reload_action:
>>>>>>          driver_reinit 1
>>>>>>          fw_activate 1
>>>>>>          driver_reinit_no_reset 0
>>>>>>          fw_activate_no_reset 0
>>>>>>
>>>>>> Then we can easily extend and add other stats in the tree.
>>>
>>> Sure, I will add it.
>> Could you please checkout the metrics patchset and figure out how to
>> merge that with your usecase?
>>
>
> I will check, I will discuss with Ido how it will fit.
>

I have discussed it with Ido, it doesn't fit to merge with metrics:

1. These counters are maintained by devlink unlike metrics which are 
read by the driver from HW.

2. The metrics counters push string name, while here I use enum.

However, I did add another level as you suggested here for option to 
future stats that may fit.

>>>>>> Also, I wonder if these stats could be somehow merged with Ido's 
>>>>>> metrics
>>>>>> work:
>>>>>> https://github.com/idosch/linux/commits/submit/devlink_metric_rfc_v1
>>>>>>
>>>>>> Ido, would it make sense?
>>>>> I guess. My original idea for devlink-metric was to expose
>>>>> design-specific metrics to user space where the entity registering 
>>>>> the
>>>>> metrics is the device driver. In this case the entity would be 
>>>>> devlink
>>>>> itself and it would be auto-registered for each device.
>>>> Yeah, the usecase is different, but it is still stats, right.
>>>>
>>>>
>>>>>>> $ devlink dev show -jp
>>>>>>> {
>>>>>>>      "dev": {
>>>>>>>          "pci/0000:82:00.0": {
>>>>>>>              "reload_action_stats": [ {
>>>>>>>                      "driver_reinit": 2
>>>>>>>                  },{
>>>>>>>                      "fw_activate": 1
>>>>>>>                  },{
>>>>>>>                      "driver_reinit_no_reset": 0
>>>>>>>                  },{
>>>>>>>                      "fw_activate_no_reset": 0
>>>>>>>                  } ]
>>>>>>>          },
>>>>>>>          "pci/0000:82:00.1": {
>>>>>>>              "reload_action_stats": [ {
>>>>>>>                      "driver_reinit": 1
>>>>>>>                  },{
>>>>>>>                      "fw_activate": 1
>>>>>>>                  },{
>>>>>>>                      "driver_reinit_no_reset": 0
>>>>>>>                  },{
>>>>>>>                      "fw_activate_no_reset": 0
>>>>>>>                  } ]
>>>>>>>          }
>>>>>>>      }
>>>>>>> }
>>>>>>>
>>>>>> [..]

Re: [PATCH net-next RFC v4 04/15] devlink: Add reload actions stats to dev get

[Jiri Pirko]

Fri, Sep 18, 2020 at 06:13:59PM CEST, moshe@nvidia.com wrote:
>
>On 9/15/2020 11:33 PM, Moshe Shemesh wrote:
>> External email: Use caution opening links or attachments
>> 
>> 
>> On 9/15/2020 4:34 PM, Jiri Pirko wrote:
>> > Tue, Sep 15, 2020 at 02:31:38PM CEST, moshe@nvidia.com wrote:
>> > > On 9/15/2020 10:44 AM, Jiri Pirko wrote:
>> > > > Tue, Sep 15, 2020 at 08:45:19AM CEST, idosch@idosch.org wrote:
>> > > > > On Mon, Sep 14, 2020 at 03:45:00PM +0200, Jiri Pirko wrote:
>> > > > > > Mon, Sep 14, 2020 at 08:07:51AM CEST, moshe@mellanox.com wrote:
>> > > > > > > Expose devlink reload actions stats to the user through devlink dev
>> > > > > > > get command.
>> > > > > > > 
>> > > > > > > Examples:
>> > > > > > > $ devlink dev show
>> > > > > > > pci/0000:82:00.0:
>> > > > > > >    reload_action_stats:
>> > > > > > >      driver_reinit 2
>> > > > > > >      fw_activate 1
>> > > > > > >      driver_reinit_no_reset 0
>> > > > > > >      fw_activate_no_reset 0
>> > > > > > > pci/0000:82:00.1:
>> > > > > > >    reload_action_stats:
>> > > > > > >      driver_reinit 1
>> > > > > > >      fw_activate 1
>> > > > > > >      driver_reinit_no_reset 0
>> > > > > > >      fw_activate_no_reset 0
>> > > > > > I would rather have something like:
>> > > > > >      stats:
>> > > > > >        reload_action:
>> > > > > >          driver_reinit 1
>> > > > > >          fw_activate 1
>> > > > > >          driver_reinit_no_reset 0
>> > > > > >          fw_activate_no_reset 0
>> > > > > > 
>> > > > > > Then we can easily extend and add other stats in the tree.
>> > > 
>> > > Sure, I will add it.
>> > Could you please checkout the metrics patchset and figure out how to
>> > merge that with your usecase?
>> > 
>> 
>> I will check, I will discuss with Ido how it will fit.
>> 
>
>I have discussed it with Ido, it doesn't fit to merge with metrics:
>
>1. These counters are maintained by devlink unlike metrics which are read by
>the driver from HW.

Okay.

>
>2. The metrics counters push string name, while here I use enum.
>
>However, I did add another level as you suggested here for option to future
>stats that may fit.
>
>> > > > > > Also, I wonder if these stats could be somehow merged
>> > > > > > with Ido's metrics
>> > > > > > work:
>> > > > > > https://github.com/idosch/linux/commits/submit/devlink_metric_rfc_v1
>> > > > > > 
>> > > > > > Ido, would it make sense?
>> > > > > I guess. My original idea for devlink-metric was to expose
>> > > > > design-specific metrics to user space where the entity
>> > > > > registering the
>> > > > > metrics is the device driver. In this case the entity
>> > > > > would be devlink
>> > > > > itself and it would be auto-registered for each device.
>> > > > Yeah, the usecase is different, but it is still stats, right.
>> > > > 
>> > > > 
>> > > > > > > $ devlink dev show -jp
>> > > > > > > {
>> > > > > > >      "dev": {
>> > > > > > >          "pci/0000:82:00.0": {
>> > > > > > >              "reload_action_stats": [ {
>> > > > > > >                      "driver_reinit": 2
>> > > > > > >                  },{
>> > > > > > >                      "fw_activate": 1
>> > > > > > >                  },{
>> > > > > > >                      "driver_reinit_no_reset": 0
>> > > > > > >                  },{
>> > > > > > >                      "fw_activate_no_reset": 0
>> > > > > > >                  } ]
>> > > > > > >          },
>> > > > > > >          "pci/0000:82:00.1": {
>> > > > > > >              "reload_action_stats": [ {
>> > > > > > >                      "driver_reinit": 1
>> > > > > > >                  },{
>> > > > > > >                      "fw_activate": 1
>> > > > > > >                  },{
>> > > > > > >                      "driver_reinit_no_reset": 0
>> > > > > > >                  },{
>> > > > > > >                      "fw_activate_no_reset": 0
>> > > > > > >                  } ]
>> > > > > > >          }
>> > > > > > >      }
>> > > > > > > }
>> > > > > > > 
>> > > > > > [..]