[patch net-next 00/11] mlxsw: Implement dev info and dev flash for line cards

[patch net-next 00/11] mlxsw: Implement dev info and dev flash for line cards

[Jiri Pirko]

From: Jiri Pirko <jiri@nvidia.com>

This patchset implements two features:
1) "devlink dev info" is exposed for line card (patches 3-8)
2) "devlink dev flash" is implemented for line card gearbox
   flashing (patch 9)

For every line card, "a nested" auxiliary device is created which
allows to bind the features mentioned above (patch 2).

The relationship between line card and its auxiliary dev devlink
is carried over extra line card netlink attribute (patches 1 and 3).

Examples:

$ devlink lc show pci/0000:01:00.0 lc 1
pci/0000:01:00.0:
  lc 1 state active type 16x100G nested_devlink auxiliary/mlxsw_core.lc.0
    supported_types:
       16x100G

$ devlink dev show auxiliary/mlxsw_core.lc.0
auxiliary/mlxsw_core.lc.0

$ devlink dev info auxiliary/mlxsw_core.lc.0
auxiliary/mlxsw_core.lc.0:
  versions:
      fixed:
        hw.revision 0
        fw.psid MT_0000000749
      running:
        ini.version 4
        fw 19.2010.1312

$ devlink dev flash auxiliary/mlxsw_core.lc.0 file mellanox/fw-AGB-rel-19_2010_1312-022-EVB.mfa2

Jiri Pirko (11):
  devlink: introduce nested devlink entity for line card
  mlxsw: core_linecards: Introduce per line card auxiliary device
  mlxsw: core_linecard_dev: Set nested devlink relationship for a line
    card
  mlxsw: core_linecards: Expose HW revision and INI version
  mlxsw: reg: Extend MDDQ by device_info
  mlxsw: core_linecards: Probe provisioned line cards for devices and
    expose FW version
  mlxsw: reg: Add Management DownStream Device Tunneling Register
  mlxsw: core_linecards: Expose device PSID over device info
  mlxsw: core_linecards: Implement line card device flashing
  selftests: mlxsw: Check line card info on provisioned line card
  selftests: mlxsw: Check line card info on activated line card

 Documentation/networking/devlink/mlxsw.rst    |  24 ++
 drivers/net/ethernet/mellanox/mlxsw/Kconfig   |   1 +
 drivers/net/ethernet/mellanox/mlxsw/Makefile  |   2 +-
 drivers/net/ethernet/mellanox/mlxsw/core.c    |  44 +-
 drivers/net/ethernet/mellanox/mlxsw/core.h    |  35 ++
 .../mellanox/mlxsw/core_linecard_dev.c        | 180 ++++++++
 .../ethernet/mellanox/mlxsw/core_linecards.c  | 403 ++++++++++++++++++
 drivers/net/ethernet/mellanox/mlxsw/reg.h     | 173 +++++++-
 include/net/devlink.h                         |   2 +
 include/uapi/linux/devlink.h                  |   2 +
 net/core/devlink.c                            |  42 ++
 .../drivers/net/mlxsw/devlink_linecard.sh     |  54 +++
 12 files changed, 948 insertions(+), 14 deletions(-)
 create mode 100644 drivers/net/ethernet/mellanox/mlxsw/core_linecard_dev.c

-- 
2.35.3

[patch net-next 01/11] devlink: introduce nested devlink entity for line card

[Jiri Pirko]

From: Jiri Pirko <jiri@nvidia.com>

For the purpose of exposing device info and allow flash updated which is
going to be implemented in follow-up patches, introduce a possibility
for a line card to expose relation to nested devlink entity. The nested
devlink entity represents the line card.

Example:

$ devlink lc show pci/0000:01:00.0 lc 1
pci/0000:01:00.0:
  lc 1 state active type 16x100G nested_devlink auxiliary/mlxsw_core.lc.0
    supported_types:
       16x100G
$ devlink dev show auxiliary/mlxsw_core.lc.0
auxiliary/mlxsw_core.lc.0

Signed-off-by: Jiri Pirko <jiri@nvidia.com>
---
 include/net/devlink.h        |  2 ++
 include/uapi/linux/devlink.h |  2 ++
 net/core/devlink.c           | 42 ++++++++++++++++++++++++++++++++++++
 3 files changed, 46 insertions(+)

diff --git a/include/net/devlink.h b/include/net/devlink.h
index 2a2a2a0c93f7..83e62943e1d4 100644
--- a/include/net/devlink.h
+++ b/include/net/devlink.h
@@ -1584,6 +1584,8 @@ void devlink_linecard_provision_clear(struct devlink_linecard *linecard);
 void devlink_linecard_provision_fail(struct devlink_linecard *linecard);
 void devlink_linecard_activate(struct devlink_linecard *linecard);
 void devlink_linecard_deactivate(struct devlink_linecard *linecard);
+void devlink_linecard_nested_dl_set(struct devlink_linecard *linecard,
+				    struct devlink *nested_devlink);
 int devlink_sb_register(struct devlink *devlink, unsigned int sb_index,
 			u32 size, u16 ingress_pools_count,
 			u16 egress_pools_count, u16 ingress_tc_count,
diff --git a/include/uapi/linux/devlink.h b/include/uapi/linux/devlink.h
index b3d40a5d72ff..541321695f52 100644
--- a/include/uapi/linux/devlink.h
+++ b/include/uapi/linux/devlink.h
@@ -576,6 +576,8 @@ enum devlink_attr {
 	DEVLINK_ATTR_LINECARD_TYPE,		/* string */
 	DEVLINK_ATTR_LINECARD_SUPPORTED_TYPES,	/* nested */
 
+	DEVLINK_ATTR_NESTED_DEVLINK,		/* nested */
+
 	/* add new attributes above here, update the policy in devlink.c */
 
 	__DEVLINK_ATTR_MAX,
diff --git a/net/core/devlink.c b/net/core/devlink.c
index db61f3a341cb..a5953cfe1baa 100644
--- a/net/core/devlink.c
+++ b/net/core/devlink.c
@@ -87,6 +87,7 @@ struct devlink_linecard {
 	const char *type;
 	struct devlink_linecard_type *types;
 	unsigned int types_count;
+	struct devlink *nested_devlink;
 };
 
 /**
@@ -796,6 +797,24 @@ static int devlink_nl_put_handle(struct sk_buff *msg, struct devlink *devlink)
 	return 0;
 }
 
+static int devlink_nl_put_nested_handle(struct sk_buff *msg, struct devlink *devlink)
+{
+	struct nlattr *nested_attr;
+
+	nested_attr = nla_nest_start(msg, DEVLINK_ATTR_NESTED_DEVLINK);
+	if (!nested_attr)
+		return -EMSGSIZE;
+	if (devlink_nl_put_handle(msg, devlink))
+		goto nla_put_failure;
+
+	nla_nest_end(msg, nested_attr);
+	return 0;
+
+nla_put_failure:
+	nla_nest_cancel(msg, nested_attr);
+	return -EMSGSIZE;
+}
+
 struct devlink_reload_combination {
 	enum devlink_reload_action action;
 	enum devlink_reload_limit limit;
@@ -2100,6 +2119,10 @@ static int devlink_nl_linecard_fill(struct sk_buff *msg,
 		nla_nest_end(msg, attr);
 	}
 
+	if (linecard->nested_devlink &&
+	    devlink_nl_put_nested_handle(msg, linecard->nested_devlink))
+		goto nla_put_failure;
+
 	genlmsg_end(msg, hdr);
 	return 0;
 
@@ -10335,6 +10358,7 @@ EXPORT_SYMBOL_GPL(devlink_linecard_provision_set);
 void devlink_linecard_provision_clear(struct devlink_linecard *linecard)
 {
 	mutex_lock(&linecard->state_lock);
+	WARN_ON(linecard->nested_devlink);
 	linecard->state = DEVLINK_LINECARD_STATE_UNPROVISIONED;
 	linecard->type = NULL;
 	devlink_linecard_notify(linecard, DEVLINK_CMD_LINECARD_NEW);
@@ -10353,6 +10377,7 @@ EXPORT_SYMBOL_GPL(devlink_linecard_provision_clear);
 void devlink_linecard_provision_fail(struct devlink_linecard *linecard)
 {
 	mutex_lock(&linecard->state_lock);
+	WARN_ON(linecard->nested_devlink);
 	linecard->state = DEVLINK_LINECARD_STATE_PROVISIONING_FAILED;
 	devlink_linecard_notify(linecard, DEVLINK_CMD_LINECARD_NEW);
 	mutex_unlock(&linecard->state_lock);
@@ -10400,6 +10425,23 @@ void devlink_linecard_deactivate(struct devlink_linecard *linecard)
 }
 EXPORT_SYMBOL_GPL(devlink_linecard_deactivate);
 
+/**
+ *	devlink_linecard_nested_dl_set - Attach/detach nested delink
+ *					 instance to linecard.
+ *
+ *	@linecard: devlink linecard
+ *      @nested_devlink: devlink instance to attach or NULL to detach
+ */
+void devlink_linecard_nested_dl_set(struct devlink_linecard *linecard,
+				    struct devlink *nested_devlink)
+{
+	mutex_lock(&linecard->state_lock);
+	linecard->nested_devlink = nested_devlink;
+	devlink_linecard_notify(linecard, DEVLINK_CMD_LINECARD_NEW);
+	mutex_unlock(&linecard->state_lock);
+}
+EXPORT_SYMBOL_GPL(devlink_linecard_nested_dl_set);
+
 int devlink_sb_register(struct devlink *devlink, unsigned int sb_index,
 			u32 size, u16 ingress_pools_count,
 			u16 egress_pools_count, u16 ingress_tc_count,
-- 
2.35.3

Re: [patch net-next 01/11] devlink: introduce nested devlink entity for line card

[Ido Schimmel]

On Tue, Jun 14, 2022 at 02:33:16PM +0200, Jiri Pirko wrote:
> From: Jiri Pirko <jiri@nvidia.com>
> 
> For the purpose of exposing device info and allow flash updated which is

s/updated/update/

> going to be implemented in follow-up patches, introduce a possibility
> for a line card to expose relation to nested devlink entity. The nested
> devlink entity represents the line card.
> 
> Example:
> 
> $ devlink lc show pci/0000:01:00.0 lc 1
> pci/0000:01:00.0:
>   lc 1 state active type 16x100G nested_devlink auxiliary/mlxsw_core.lc.0
>     supported_types:
>        16x100G
> $ devlink dev show auxiliary/mlxsw_core.lc.0
> auxiliary/mlxsw_core.lc.0
> 
> Signed-off-by: Jiri Pirko <jiri@nvidia.com>
> ---
>  include/net/devlink.h        |  2 ++
>  include/uapi/linux/devlink.h |  2 ++
>  net/core/devlink.c           | 42 ++++++++++++++++++++++++++++++++++++
>  3 files changed, 46 insertions(+)
> 
> diff --git a/include/net/devlink.h b/include/net/devlink.h
> index 2a2a2a0c93f7..83e62943e1d4 100644
> --- a/include/net/devlink.h
> +++ b/include/net/devlink.h
> @@ -1584,6 +1584,8 @@ void devlink_linecard_provision_clear(struct devlink_linecard *linecard);
>  void devlink_linecard_provision_fail(struct devlink_linecard *linecard);
>  void devlink_linecard_activate(struct devlink_linecard *linecard);
>  void devlink_linecard_deactivate(struct devlink_linecard *linecard);
> +void devlink_linecard_nested_dl_set(struct devlink_linecard *linecard,
> +				    struct devlink *nested_devlink);
>  int devlink_sb_register(struct devlink *devlink, unsigned int sb_index,
>  			u32 size, u16 ingress_pools_count,
>  			u16 egress_pools_count, u16 ingress_tc_count,
> diff --git a/include/uapi/linux/devlink.h b/include/uapi/linux/devlink.h
> index b3d40a5d72ff..541321695f52 100644
> --- a/include/uapi/linux/devlink.h
> +++ b/include/uapi/linux/devlink.h
> @@ -576,6 +576,8 @@ enum devlink_attr {
>  	DEVLINK_ATTR_LINECARD_TYPE,		/* string */
>  	DEVLINK_ATTR_LINECARD_SUPPORTED_TYPES,	/* nested */
>  
> +	DEVLINK_ATTR_NESTED_DEVLINK,		/* nested */
> +
>  	/* add new attributes above here, update the policy in devlink.c */
>  
>  	__DEVLINK_ATTR_MAX,
> diff --git a/net/core/devlink.c b/net/core/devlink.c
> index db61f3a341cb..a5953cfe1baa 100644
> --- a/net/core/devlink.c
> +++ b/net/core/devlink.c
> @@ -87,6 +87,7 @@ struct devlink_linecard {
>  	const char *type;
>  	struct devlink_linecard_type *types;
>  	unsigned int types_count;
> +	struct devlink *nested_devlink;
>  };
>  
>  /**
> @@ -796,6 +797,24 @@ static int devlink_nl_put_handle(struct sk_buff *msg, struct devlink *devlink)
>  	return 0;
>  }
>  
> +static int devlink_nl_put_nested_handle(struct sk_buff *msg, struct devlink *devlink)
> +{
> +	struct nlattr *nested_attr;
> +
> +	nested_attr = nla_nest_start(msg, DEVLINK_ATTR_NESTED_DEVLINK);
> +	if (!nested_attr)
> +		return -EMSGSIZE;
> +	if (devlink_nl_put_handle(msg, devlink))
> +		goto nla_put_failure;
> +
> +	nla_nest_end(msg, nested_attr);
> +	return 0;
> +
> +nla_put_failure:
> +	nla_nest_cancel(msg, nested_attr);
> +	return -EMSGSIZE;
> +}
> +
>  struct devlink_reload_combination {
>  	enum devlink_reload_action action;
>  	enum devlink_reload_limit limit;
> @@ -2100,6 +2119,10 @@ static int devlink_nl_linecard_fill(struct sk_buff *msg,
>  		nla_nest_end(msg, attr);
>  	}
>  
> +	if (linecard->nested_devlink &&
> +	    devlink_nl_put_nested_handle(msg, linecard->nested_devlink))
> +		goto nla_put_failure;
> +
>  	genlmsg_end(msg, hdr);
>  	return 0;
>  
> @@ -10335,6 +10358,7 @@ EXPORT_SYMBOL_GPL(devlink_linecard_provision_set);
>  void devlink_linecard_provision_clear(struct devlink_linecard *linecard)
>  {
>  	mutex_lock(&linecard->state_lock);
> +	WARN_ON(linecard->nested_devlink);
>  	linecard->state = DEVLINK_LINECARD_STATE_UNPROVISIONED;
>  	linecard->type = NULL;
>  	devlink_linecard_notify(linecard, DEVLINK_CMD_LINECARD_NEW);
> @@ -10353,6 +10377,7 @@ EXPORT_SYMBOL_GPL(devlink_linecard_provision_clear);
>  void devlink_linecard_provision_fail(struct devlink_linecard *linecard)
>  {
>  	mutex_lock(&linecard->state_lock);
> +	WARN_ON(linecard->nested_devlink);
>  	linecard->state = DEVLINK_LINECARD_STATE_PROVISIONING_FAILED;
>  	devlink_linecard_notify(linecard, DEVLINK_CMD_LINECARD_NEW);
>  	mutex_unlock(&linecard->state_lock);
> @@ -10400,6 +10425,23 @@ void devlink_linecard_deactivate(struct devlink_linecard *linecard)
>  }
>  EXPORT_SYMBOL_GPL(devlink_linecard_deactivate);
>  
> +/**
> + *	devlink_linecard_nested_dl_set - Attach/detach nested delink

s/delink/devlink/

> + *					 instance to linecard.
> + *
> + *	@linecard: devlink linecard
> + *      @nested_devlink: devlink instance to attach or NULL to detach

The alignment looks off

> + */
> +void devlink_linecard_nested_dl_set(struct devlink_linecard *linecard,
> +				    struct devlink *nested_devlink)
> +{
> +	mutex_lock(&linecard->state_lock);
> +	linecard->nested_devlink = nested_devlink;
> +	devlink_linecard_notify(linecard, DEVLINK_CMD_LINECARD_NEW);
> +	mutex_unlock(&linecard->state_lock);
> +}
> +EXPORT_SYMBOL_GPL(devlink_linecard_nested_dl_set);
> +
>  int devlink_sb_register(struct devlink *devlink, unsigned int sb_index,
>  			u32 size, u16 ingress_pools_count,
>  			u16 egress_pools_count, u16 ingress_tc_count,
> -- 
> 2.35.3
> 

Re: [patch net-next 01/11] devlink: introduce nested devlink entity for line card

[Jiri Pirko]

Wed, Jun 15, 2022 at 04:05:13PM CEST, idosch@nvidia.com wrote:
>On Tue, Jun 14, 2022 at 02:33:16PM +0200, Jiri Pirko wrote:
>> From: Jiri Pirko <jiri@nvidia.com>
>> 
>> For the purpose of exposing device info and allow flash updated which is
>
>s/updated/update/
>
>> going to be implemented in follow-up patches, introduce a possibility
>> for a line card to expose relation to nested devlink entity. The nested
>> devlink entity represents the line card.
>> 
>> Example:
>> 
>> $ devlink lc show pci/0000:01:00.0 lc 1
>> pci/0000:01:00.0:
>>   lc 1 state active type 16x100G nested_devlink auxiliary/mlxsw_core.lc.0
>>     supported_types:
>>        16x100G
>> $ devlink dev show auxiliary/mlxsw_core.lc.0
>> auxiliary/mlxsw_core.lc.0
>> 
>> Signed-off-by: Jiri Pirko <jiri@nvidia.com>
>> ---
>>  include/net/devlink.h        |  2 ++
>>  include/uapi/linux/devlink.h |  2 ++
>>  net/core/devlink.c           | 42 ++++++++++++++++++++++++++++++++++++
>>  3 files changed, 46 insertions(+)
>> 
>> diff --git a/include/net/devlink.h b/include/net/devlink.h
>> index 2a2a2a0c93f7..83e62943e1d4 100644
>> --- a/include/net/devlink.h
>> +++ b/include/net/devlink.h
>> @@ -1584,6 +1584,8 @@ void devlink_linecard_provision_clear(struct devlink_linecard *linecard);
>>  void devlink_linecard_provision_fail(struct devlink_linecard *linecard);
>>  void devlink_linecard_activate(struct devlink_linecard *linecard);
>>  void devlink_linecard_deactivate(struct devlink_linecard *linecard);
>> +void devlink_linecard_nested_dl_set(struct devlink_linecard *linecard,
>> +				    struct devlink *nested_devlink);
>>  int devlink_sb_register(struct devlink *devlink, unsigned int sb_index,
>>  			u32 size, u16 ingress_pools_count,
>>  			u16 egress_pools_count, u16 ingress_tc_count,
>> diff --git a/include/uapi/linux/devlink.h b/include/uapi/linux/devlink.h
>> index b3d40a5d72ff..541321695f52 100644
>> --- a/include/uapi/linux/devlink.h
>> +++ b/include/uapi/linux/devlink.h
>> @@ -576,6 +576,8 @@ enum devlink_attr {
>>  	DEVLINK_ATTR_LINECARD_TYPE,		/* string */
>>  	DEVLINK_ATTR_LINECARD_SUPPORTED_TYPES,	/* nested */
>>  
>> +	DEVLINK_ATTR_NESTED_DEVLINK,		/* nested */
>> +
>>  	/* add new attributes above here, update the policy in devlink.c */
>>  
>>  	__DEVLINK_ATTR_MAX,
>> diff --git a/net/core/devlink.c b/net/core/devlink.c
>> index db61f3a341cb..a5953cfe1baa 100644
>> --- a/net/core/devlink.c
>> +++ b/net/core/devlink.c
>> @@ -87,6 +87,7 @@ struct devlink_linecard {
>>  	const char *type;
>>  	struct devlink_linecard_type *types;
>>  	unsigned int types_count;
>> +	struct devlink *nested_devlink;
>>  };
>>  
>>  /**
>> @@ -796,6 +797,24 @@ static int devlink_nl_put_handle(struct sk_buff *msg, struct devlink *devlink)
>>  	return 0;
>>  }
>>  
>> +static int devlink_nl_put_nested_handle(struct sk_buff *msg, struct devlink *devlink)
>> +{
>> +	struct nlattr *nested_attr;
>> +
>> +	nested_attr = nla_nest_start(msg, DEVLINK_ATTR_NESTED_DEVLINK);
>> +	if (!nested_attr)
>> +		return -EMSGSIZE;
>> +	if (devlink_nl_put_handle(msg, devlink))
>> +		goto nla_put_failure;
>> +
>> +	nla_nest_end(msg, nested_attr);
>> +	return 0;
>> +
>> +nla_put_failure:
>> +	nla_nest_cancel(msg, nested_attr);
>> +	return -EMSGSIZE;
>> +}
>> +
>>  struct devlink_reload_combination {
>>  	enum devlink_reload_action action;
>>  	enum devlink_reload_limit limit;
>> @@ -2100,6 +2119,10 @@ static int devlink_nl_linecard_fill(struct sk_buff *msg,
>>  		nla_nest_end(msg, attr);
>>  	}
>>  
>> +	if (linecard->nested_devlink &&
>> +	    devlink_nl_put_nested_handle(msg, linecard->nested_devlink))
>> +		goto nla_put_failure;
>> +
>>  	genlmsg_end(msg, hdr);
>>  	return 0;
>>  
>> @@ -10335,6 +10358,7 @@ EXPORT_SYMBOL_GPL(devlink_linecard_provision_set);
>>  void devlink_linecard_provision_clear(struct devlink_linecard *linecard)
>>  {
>>  	mutex_lock(&linecard->state_lock);
>> +	WARN_ON(linecard->nested_devlink);
>>  	linecard->state = DEVLINK_LINECARD_STATE_UNPROVISIONED;
>>  	linecard->type = NULL;
>>  	devlink_linecard_notify(linecard, DEVLINK_CMD_LINECARD_NEW);
>> @@ -10353,6 +10377,7 @@ EXPORT_SYMBOL_GPL(devlink_linecard_provision_clear);
>>  void devlink_linecard_provision_fail(struct devlink_linecard *linecard)
>>  {
>>  	mutex_lock(&linecard->state_lock);
>> +	WARN_ON(linecard->nested_devlink);
>>  	linecard->state = DEVLINK_LINECARD_STATE_PROVISIONING_FAILED;
>>  	devlink_linecard_notify(linecard, DEVLINK_CMD_LINECARD_NEW);
>>  	mutex_unlock(&linecard->state_lock);
>> @@ -10400,6 +10425,23 @@ void devlink_linecard_deactivate(struct devlink_linecard *linecard)
>>  }
>>  EXPORT_SYMBOL_GPL(devlink_linecard_deactivate);
>>  
>> +/**
>> + *	devlink_linecard_nested_dl_set - Attach/detach nested delink
>
>s/delink/devlink/
>
>> + *					 instance to linecard.
>> + *
>> + *	@linecard: devlink linecard
>> + *      @nested_devlink: devlink instance to attach or NULL to detach
>
>The alignment looks off

Will fix all in v2.

>
>> + */
>> +void devlink_linecard_nested_dl_set(struct devlink_linecard *linecard,
>> +				    struct devlink *nested_devlink)
>> +{
>> +	mutex_lock(&linecard->state_lock);
>> +	linecard->nested_devlink = nested_devlink;
>> +	devlink_linecard_notify(linecard, DEVLINK_CMD_LINECARD_NEW);
>> +	mutex_unlock(&linecard->state_lock);
>> +}
>> +EXPORT_SYMBOL_GPL(devlink_linecard_nested_dl_set);
>> +
>>  int devlink_sb_register(struct devlink *devlink, unsigned int sb_index,
>>  			u32 size, u16 ingress_pools_count,
>>  			u16 egress_pools_count, u16 ingress_tc_count,
>> -- 
>> 2.35.3
>> 

Re: [patch net-next 01/11] devlink: introduce nested devlink entity for line card

[Jakub Kicinski]

On Tue, 14 Jun 2022 14:33:16 +0200 Jiri Pirko wrote:
> From: Jiri Pirko <jiri@nvidia.com>
> 
> For the purpose of exposing device info and allow flash updated which is
> going to be implemented in follow-up patches, introduce a possibility
> for a line card to expose relation to nested devlink entity. The nested
> devlink entity represents the line card.
> 
> Example:
> 
> $ devlink lc show pci/0000:01:00.0 lc 1
> pci/0000:01:00.0:
>   lc 1 state active type 16x100G nested_devlink auxiliary/mlxsw_core.lc.0
>     supported_types:
>        16x100G
> $ devlink dev show auxiliary/mlxsw_core.lc.0
> auxiliary/mlxsw_core.lc.0
> 
> Signed-off-by: Jiri Pirko <jiri@nvidia.com>

Acked-by: Jakub Kicinski <kuba@kernel.org>

Thanks!

[patch net-next 02/11] mlxsw: core_linecards: Introduce per line card auxiliary device

[Jiri Pirko]

From: Jiri Pirko <jiri@nvidia.com>

In order to be eventually able to expose line card gearbox version and
possibility to flash FW, model the line card as a separate device on
auxiliary bus.

Signed-off-by: Jiri Pirko <jiri@nvidia.com>
---
 drivers/net/ethernet/mellanox/mlxsw/Kconfig   |   1 +
 drivers/net/ethernet/mellanox/mlxsw/Makefile  |   2 +-
 drivers/net/ethernet/mellanox/mlxsw/core.c    |  13 +-
 drivers/net/ethernet/mellanox/mlxsw/core.h    |  10 ++
 .../mellanox/mlxsw/core_linecard_dev.c        | 152 ++++++++++++++++++
 .../ethernet/mellanox/mlxsw/core_linecards.c  |  10 ++
 6 files changed, 185 insertions(+), 3 deletions(-)
 create mode 100644 drivers/net/ethernet/mellanox/mlxsw/core_linecard_dev.c

diff --git a/drivers/net/ethernet/mellanox/mlxsw/Kconfig b/drivers/net/ethernet/mellanox/mlxsw/Kconfig
index 4683312861ac..a510bf2cff2f 100644
--- a/drivers/net/ethernet/mellanox/mlxsw/Kconfig
+++ b/drivers/net/ethernet/mellanox/mlxsw/Kconfig
@@ -7,6 +7,7 @@ config MLXSW_CORE
 	tristate "Mellanox Technologies Switch ASICs support"
 	select NET_DEVLINK
 	select MLXFW
+	select AUXILIARY_BUS
 	help
 	  This driver supports Mellanox Technologies Switch ASICs family.
 
diff --git a/drivers/net/ethernet/mellanox/mlxsw/Makefile b/drivers/net/ethernet/mellanox/mlxsw/Makefile
index 1a465fd5d8b3..c9a773d3158b 100644
--- a/drivers/net/ethernet/mellanox/mlxsw/Makefile
+++ b/drivers/net/ethernet/mellanox/mlxsw/Makefile
@@ -2,7 +2,7 @@
 obj-$(CONFIG_MLXSW_CORE)	+= mlxsw_core.o
 mlxsw_core-objs			:= core.o core_acl_flex_keys.o \
 				   core_acl_flex_actions.o core_env.o \
-				   core_linecards.o
+				   core_linecards.o core_linecard_dev.o
 mlxsw_core-$(CONFIG_MLXSW_CORE_HWMON) += core_hwmon.o
 mlxsw_core-$(CONFIG_MLXSW_CORE_THERMAL) += core_thermal.o
 obj-$(CONFIG_MLXSW_PCI)		+= mlxsw_pci.o
diff --git a/drivers/net/ethernet/mellanox/mlxsw/core.c b/drivers/net/ethernet/mellanox/mlxsw/core.c
index fc52832241b3..8864533281bd 100644
--- a/drivers/net/ethernet/mellanox/mlxsw/core.c
+++ b/drivers/net/ethernet/mellanox/mlxsw/core.c
@@ -3331,9 +3331,15 @@ static int __init mlxsw_core_module_init(void)
 {
 	int err;
 
+	err = mlxsw_linecard_driver_register();
+	if (err)
+		return err;
+
 	mlxsw_wq = alloc_workqueue(mlxsw_core_driver_name, 0, 0);
-	if (!mlxsw_wq)
-		return -ENOMEM;
+	if (!mlxsw_wq) {
+		err = -ENOMEM;
+		goto err_alloc_workqueue;
+	}
 	mlxsw_owq = alloc_ordered_workqueue("%s_ordered", 0,
 					    mlxsw_core_driver_name);
 	if (!mlxsw_owq) {
@@ -3344,6 +3350,8 @@ static int __init mlxsw_core_module_init(void)
 
 err_alloc_ordered_workqueue:
 	destroy_workqueue(mlxsw_wq);
+err_alloc_workqueue:
+	mlxsw_linecard_driver_unregister();
 	return err;
 }
 
@@ -3351,6 +3359,7 @@ static void __exit mlxsw_core_module_exit(void)
 {
 	destroy_workqueue(mlxsw_owq);
 	destroy_workqueue(mlxsw_wq);
+	mlxsw_linecard_driver_unregister();
 }
 
 module_init(mlxsw_core_module_init);
diff --git a/drivers/net/ethernet/mellanox/mlxsw/core.h b/drivers/net/ethernet/mellanox/mlxsw/core.h
index c2a891287047..cda20a4fcbdb 100644
--- a/drivers/net/ethernet/mellanox/mlxsw/core.h
+++ b/drivers/net/ethernet/mellanox/mlxsw/core.h
@@ -12,6 +12,7 @@
 #include <linux/skbuff.h>
 #include <linux/workqueue.h>
 #include <linux/net_namespace.h>
+#include <linux/auxiliary_bus.h>
 #include <net/devlink.h>
 
 #include "trap.h"
@@ -567,6 +568,8 @@ enum mlxsw_linecard_status_event_type {
 	MLXSW_LINECARD_STATUS_EVENT_TYPE_UNPROVISION,
 };
 
+struct mlxsw_linecard_bdev;
+
 struct mlxsw_linecard {
 	u8 slot_index;
 	struct mlxsw_linecards *linecards;
@@ -581,6 +584,7 @@ struct mlxsw_linecard {
 	   active:1;
 	u16 hw_revision;
 	u16 ini_version;
+	struct mlxsw_linecard_bdev *bdev;
 };
 
 struct mlxsw_linecard_types_info;
@@ -620,4 +624,10 @@ void mlxsw_linecards_event_ops_unregister(struct mlxsw_core *mlxsw_core,
 					  struct mlxsw_linecards_event_ops *ops,
 					  void *priv);
 
+int mlxsw_linecard_bdev_add(struct mlxsw_linecard *linecard);
+void mlxsw_linecard_bdev_del(struct mlxsw_linecard *linecard);
+
+int mlxsw_linecard_driver_register(void);
+void mlxsw_linecard_driver_unregister(void);
+
 #endif
diff --git a/drivers/net/ethernet/mellanox/mlxsw/core_linecard_dev.c b/drivers/net/ethernet/mellanox/mlxsw/core_linecard_dev.c
new file mode 100644
index 000000000000..af70d3f7a177
--- /dev/null
+++ b/drivers/net/ethernet/mellanox/mlxsw/core_linecard_dev.c
@@ -0,0 +1,152 @@
+// SPDX-License-Identifier: BSD-3-Clause OR GPL-2.0
+/* Copyright (c) 2022 NVIDIA Corporation and Mellanox Technologies. All rights reserved */
+
+#include <linux/kernel.h>
+#include <linux/module.h>
+#include <linux/err.h>
+#include <linux/types.h>
+#include <linux/err.h>
+#include <linux/auxiliary_bus.h>
+#include <linux/idr.h>
+#include <linux/gfp.h>
+#include <linux/slab.h>
+#include <net/devlink.h>
+#include "core.h"
+
+#define MLXSW_LINECARD_DEV_ID_NAME "lc"
+
+struct mlxsw_linecard_dev {
+	struct mlxsw_linecard *linecard;
+};
+
+struct mlxsw_linecard_bdev {
+	struct auxiliary_device adev;
+	struct mlxsw_linecard *linecard;
+	struct mlxsw_linecard_dev *linecard_dev;
+};
+
+static DEFINE_IDA(mlxsw_linecard_bdev_ida);
+
+static int mlxsw_linecard_bdev_id_alloc(void)
+{
+	return ida_alloc(&mlxsw_linecard_bdev_ida, GFP_KERNEL);
+}
+
+static void mlxsw_linecard_bdev_id_free(int id)
+{
+	ida_free(&mlxsw_linecard_bdev_ida, id);
+}
+
+static void mlxsw_linecard_bdev_release(struct device *device)
+{
+	struct auxiliary_device *adev =
+			container_of(device, struct auxiliary_device, dev);
+	struct mlxsw_linecard_bdev *linecard_bdev =
+			container_of(adev, struct mlxsw_linecard_bdev, adev);
+
+	mlxsw_linecard_bdev_id_free(adev->id);
+	kfree(linecard_bdev);
+}
+
+int mlxsw_linecard_bdev_add(struct mlxsw_linecard *linecard)
+{
+	struct mlxsw_linecard_bdev *linecard_bdev;
+	int err;
+	int id;
+
+	id = mlxsw_linecard_bdev_id_alloc();
+	if (id < 0)
+		return id;
+
+	linecard_bdev = kzalloc(sizeof(*linecard_bdev), GFP_KERNEL);
+	if (!linecard_bdev) {
+		mlxsw_linecard_bdev_id_free(id);
+		return -ENOMEM;
+	}
+	linecard_bdev->adev.id = id;
+	linecard_bdev->adev.name = MLXSW_LINECARD_DEV_ID_NAME;
+	linecard_bdev->adev.dev.release = mlxsw_linecard_bdev_release;
+	linecard_bdev->adev.dev.parent = linecard->linecards->bus_info->dev;
+	linecard_bdev->linecard = linecard;
+
+	err = auxiliary_device_init(&linecard_bdev->adev);
+	if (err) {
+		mlxsw_linecard_bdev_id_free(id);
+		kfree(linecard_bdev);
+		return err;
+	}
+
+	err = auxiliary_device_add(&linecard_bdev->adev);
+	if (err) {
+		auxiliary_device_uninit(&linecard_bdev->adev);
+		return err;
+	}
+
+	linecard->bdev = linecard_bdev;
+	return 0;
+}
+
+void mlxsw_linecard_bdev_del(struct mlxsw_linecard *linecard)
+{
+	struct mlxsw_linecard_bdev *linecard_bdev = linecard->bdev;
+
+	auxiliary_device_delete(&linecard_bdev->adev);
+	auxiliary_device_uninit(&linecard_bdev->adev);
+}
+
+static const struct devlink_ops mlxsw_linecard_dev_devlink_ops = {
+};
+
+static int mlxsw_linecard_bdev_probe(struct auxiliary_device *adev,
+				     const struct auxiliary_device_id *id)
+{
+	struct mlxsw_linecard_bdev *linecard_bdev =
+			container_of(adev, struct mlxsw_linecard_bdev, adev);
+	struct mlxsw_linecard_dev *linecard_dev;
+	struct devlink *devlink;
+
+	devlink = devlink_alloc(&mlxsw_linecard_dev_devlink_ops,
+				sizeof(*linecard_dev), &adev->dev);
+	if (!devlink)
+		return -ENOMEM;
+	linecard_dev = devlink_priv(devlink);
+	linecard_dev->linecard = linecard_bdev->linecard;
+	linecard_bdev->linecard_dev = linecard_dev;
+
+	devlink_register(devlink);
+	return 0;
+}
+
+static void mlxsw_linecard_bdev_remove(struct auxiliary_device *adev)
+{
+	struct mlxsw_linecard_bdev *linecard_bdev =
+			container_of(adev, struct mlxsw_linecard_bdev, adev);
+	struct devlink *devlink = priv_to_devlink(linecard_bdev->linecard_dev);
+
+	devlink_unregister(devlink);
+	devlink_free(devlink);
+}
+
+static const struct auxiliary_device_id mlxsw_linecard_bdev_id_table[] = {
+	{ .name = KBUILD_MODNAME "." MLXSW_LINECARD_DEV_ID_NAME },
+	{},
+};
+
+MODULE_DEVICE_TABLE(auxiliary, mlxsw_linecard_bdev_id_table);
+
+static struct auxiliary_driver mlxsw_linecard_driver = {
+	.name = MLXSW_LINECARD_DEV_ID_NAME,
+	.probe = mlxsw_linecard_bdev_probe,
+	.remove = mlxsw_linecard_bdev_remove,
+	.id_table = mlxsw_linecard_bdev_id_table,
+};
+
+int mlxsw_linecard_driver_register(void)
+{
+	return auxiliary_driver_register(&mlxsw_linecard_driver);
+}
+
+void mlxsw_linecard_driver_unregister(void)
+{
+	auxiliary_driver_unregister(&mlxsw_linecard_driver);
+}
diff --git a/drivers/net/ethernet/mellanox/mlxsw/core_linecards.c b/drivers/net/ethernet/mellanox/mlxsw/core_linecards.c
index 5c9869dcf674..ae51944cde0c 100644
--- a/drivers/net/ethernet/mellanox/mlxsw/core_linecards.c
+++ b/drivers/net/ethernet/mellanox/mlxsw/core_linecards.c
@@ -232,6 +232,7 @@ mlxsw_linecard_provision_set(struct mlxsw_linecard *linecard, u8 card_type,
 {
 	struct mlxsw_linecards *linecards = linecard->linecards;
 	const char *type;
+	int err;
 
 	type = mlxsw_linecard_types_lookup(linecards, card_type);
 	mlxsw_linecard_status_event_done(linecard,
@@ -252,6 +253,14 @@ mlxsw_linecard_provision_set(struct mlxsw_linecard *linecard, u8 card_type,
 	linecard->provisioned = true;
 	linecard->hw_revision = hw_revision;
 	linecard->ini_version = ini_version;
+
+	err = mlxsw_linecard_bdev_add(linecard);
+	if (err) {
+		linecard->provisioned = false;
+		mlxsw_linecard_provision_fail(linecard);
+		return err;
+	}
+
 	devlink_linecard_provision_set(linecard->devlink_linecard, type);
 	return 0;
 }
@@ -260,6 +269,7 @@ static void mlxsw_linecard_provision_clear(struct mlxsw_linecard *linecard)
 {
 	mlxsw_linecard_status_event_done(linecard,
 					 MLXSW_LINECARD_STATUS_EVENT_TYPE_UNPROVISION);
+	mlxsw_linecard_bdev_del(linecard);
 	linecard->provisioned = false;
 	devlink_linecard_provision_clear(linecard->devlink_linecard);
 }
-- 
2.35.3

Re: [patch net-next 02/11] mlxsw: core_linecards: Introduce per line card auxiliary device

[Ido Schimmel]

> +int mlxsw_linecard_bdev_add(struct mlxsw_linecard *linecard)
> +{
> +	struct mlxsw_linecard_bdev *linecard_bdev;
> +	int err;
> +	int id;
> +
> +	id = mlxsw_linecard_bdev_id_alloc();
> +	if (id < 0)
> +		return id;
> +
> +	linecard_bdev = kzalloc(sizeof(*linecard_bdev), GFP_KERNEL);
> +	if (!linecard_bdev) {
> +		mlxsw_linecard_bdev_id_free(id);
> +		return -ENOMEM;
> +	}
> +	linecard_bdev->adev.id = id;
> +	linecard_bdev->adev.name = MLXSW_LINECARD_DEV_ID_NAME;
> +	linecard_bdev->adev.dev.release = mlxsw_linecard_bdev_release;
> +	linecard_bdev->adev.dev.parent = linecard->linecards->bus_info->dev;
> +	linecard_bdev->linecard = linecard;
> +
> +	err = auxiliary_device_init(&linecard_bdev->adev);
> +	if (err) {
> +		mlxsw_linecard_bdev_id_free(id);
> +		kfree(linecard_bdev);
> +		return err;
> +	}
> +
> +	err = auxiliary_device_add(&linecard_bdev->adev);
> +	if (err) {
> +		auxiliary_device_uninit(&linecard_bdev->adev);
> +		return err;
> +	}
> +
> +	linecard->bdev = linecard_bdev;
> +	return 0;
> +}

[...]

> +static int mlxsw_linecard_bdev_probe(struct auxiliary_device *adev,
> +				     const struct auxiliary_device_id *id)
> +{
> +	struct mlxsw_linecard_bdev *linecard_bdev =
> +			container_of(adev, struct mlxsw_linecard_bdev, adev);
> +	struct mlxsw_linecard_dev *linecard_dev;
> +	struct devlink *devlink;
> +
> +	devlink = devlink_alloc(&mlxsw_linecard_dev_devlink_ops,
> +				sizeof(*linecard_dev), &adev->dev);
> +	if (!devlink)
> +		return -ENOMEM;
> +	linecard_dev = devlink_priv(devlink);
> +	linecard_dev->linecard = linecard_bdev->linecard;
> +	linecard_bdev->linecard_dev = linecard_dev;
> +
> +	devlink_register(devlink);
> +	return 0;
> +}

[...]

> @@ -252,6 +253,14 @@ mlxsw_linecard_provision_set(struct mlxsw_linecard *linecard, u8 card_type,
>  	linecard->provisioned = true;
>  	linecard->hw_revision = hw_revision;
>  	linecard->ini_version = ini_version;
> +
> +	err = mlxsw_linecard_bdev_add(linecard);

If a line card is already provisioned and we are reloading the primary
devlink instance, isn't this going to deadlock on the global (not
per-instance) devlink mutex? It is held throughout the reload operation
and also taken in devlink_register()

My understanding of the auxiliary bus model is that after adding a
device to the bus via auxiliary_device_add(), the probe() function of
the auxiliary driver will be called. In our case, this function acquires
the global devlink mutex in devlink_register().

> +	if (err) {
> +		linecard->provisioned = false;
> +		mlxsw_linecard_provision_fail(linecard);
> +		return err;
> +	}
> +
>  	devlink_linecard_provision_set(linecard->devlink_linecard, type);
>  	return 0;
>  }

Re: [patch net-next 02/11] mlxsw: core_linecards: Introduce per line card auxiliary device

[Jiri Pirko]

Wed, Jun 15, 2022 at 04:52:13PM CEST, idosch@nvidia.com wrote:
>> +int mlxsw_linecard_bdev_add(struct mlxsw_linecard *linecard)
>> +{
>> +	struct mlxsw_linecard_bdev *linecard_bdev;
>> +	int err;
>> +	int id;
>> +
>> +	id = mlxsw_linecard_bdev_id_alloc();
>> +	if (id < 0)
>> +		return id;
>> +
>> +	linecard_bdev = kzalloc(sizeof(*linecard_bdev), GFP_KERNEL);
>> +	if (!linecard_bdev) {
>> +		mlxsw_linecard_bdev_id_free(id);
>> +		return -ENOMEM;
>> +	}
>> +	linecard_bdev->adev.id = id;
>> +	linecard_bdev->adev.name = MLXSW_LINECARD_DEV_ID_NAME;
>> +	linecard_bdev->adev.dev.release = mlxsw_linecard_bdev_release;
>> +	linecard_bdev->adev.dev.parent = linecard->linecards->bus_info->dev;
>> +	linecard_bdev->linecard = linecard;
>> +
>> +	err = auxiliary_device_init(&linecard_bdev->adev);
>> +	if (err) {
>> +		mlxsw_linecard_bdev_id_free(id);
>> +		kfree(linecard_bdev);
>> +		return err;
>> +	}
>> +
>> +	err = auxiliary_device_add(&linecard_bdev->adev);
>> +	if (err) {
>> +		auxiliary_device_uninit(&linecard_bdev->adev);
>> +		return err;
>> +	}
>> +
>> +	linecard->bdev = linecard_bdev;
>> +	return 0;
>> +}
>
>[...]
>
>> +static int mlxsw_linecard_bdev_probe(struct auxiliary_device *adev,
>> +				     const struct auxiliary_device_id *id)
>> +{
>> +	struct mlxsw_linecard_bdev *linecard_bdev =
>> +			container_of(adev, struct mlxsw_linecard_bdev, adev);
>> +	struct mlxsw_linecard_dev *linecard_dev;
>> +	struct devlink *devlink;
>> +
>> +	devlink = devlink_alloc(&mlxsw_linecard_dev_devlink_ops,
>> +				sizeof(*linecard_dev), &adev->dev);
>> +	if (!devlink)
>> +		return -ENOMEM;
>> +	linecard_dev = devlink_priv(devlink);
>> +	linecard_dev->linecard = linecard_bdev->linecard;
>> +	linecard_bdev->linecard_dev = linecard_dev;
>> +
>> +	devlink_register(devlink);
>> +	return 0;
>> +}
>
>[...]
>
>> @@ -252,6 +253,14 @@ mlxsw_linecard_provision_set(struct mlxsw_linecard *linecard, u8 card_type,
>>  	linecard->provisioned = true;
>>  	linecard->hw_revision = hw_revision;
>>  	linecard->ini_version = ini_version;
>> +
>> +	err = mlxsw_linecard_bdev_add(linecard);
>
>If a line card is already provisioned and we are reloading the primary
>devlink instance, isn't this going to deadlock on the global (not
>per-instance) devlink mutex? It is held throughout the reload operation
>and also taken in devlink_register()
>
>My understanding of the auxiliary bus model is that after adding a
>device to the bus via auxiliary_device_add(), the probe() function of
>the auxiliary driver will be called. In our case, this function acquires
>the global devlink mutex in devlink_register().

No, the line card auxdev is supposed to be removed during
linecard_fini(). This, I forgot to add, will do in v2.


>
>> +	if (err) {
>> +		linecard->provisioned = false;
>> +		mlxsw_linecard_provision_fail(linecard);
>> +		return err;
>> +	}
>> +
>>  	devlink_linecard_provision_set(linecard->devlink_linecard, type);
>>  	return 0;
>>  }

Re: [patch net-next 02/11] mlxsw: core_linecards: Introduce per line card auxiliary device

[Ido Schimmel]

On Wed, Jun 15, 2022 at 07:37:15PM +0200, Jiri Pirko wrote:
> Wed, Jun 15, 2022 at 04:52:13PM CEST, idosch@nvidia.com wrote:
> >> +int mlxsw_linecard_bdev_add(struct mlxsw_linecard *linecard)
> >> +{
> >> +	struct mlxsw_linecard_bdev *linecard_bdev;
> >> +	int err;
> >> +	int id;
> >> +
> >> +	id = mlxsw_linecard_bdev_id_alloc();
> >> +	if (id < 0)
> >> +		return id;
> >> +
> >> +	linecard_bdev = kzalloc(sizeof(*linecard_bdev), GFP_KERNEL);
> >> +	if (!linecard_bdev) {
> >> +		mlxsw_linecard_bdev_id_free(id);
> >> +		return -ENOMEM;
> >> +	}
> >> +	linecard_bdev->adev.id = id;
> >> +	linecard_bdev->adev.name = MLXSW_LINECARD_DEV_ID_NAME;
> >> +	linecard_bdev->adev.dev.release = mlxsw_linecard_bdev_release;
> >> +	linecard_bdev->adev.dev.parent = linecard->linecards->bus_info->dev;
> >> +	linecard_bdev->linecard = linecard;
> >> +
> >> +	err = auxiliary_device_init(&linecard_bdev->adev);
> >> +	if (err) {
> >> +		mlxsw_linecard_bdev_id_free(id);
> >> +		kfree(linecard_bdev);
> >> +		return err;
> >> +	}
> >> +
> >> +	err = auxiliary_device_add(&linecard_bdev->adev);
> >> +	if (err) {
> >> +		auxiliary_device_uninit(&linecard_bdev->adev);
> >> +		return err;
> >> +	}
> >> +
> >> +	linecard->bdev = linecard_bdev;
> >> +	return 0;
> >> +}
> >
> >[...]
> >
> >> +static int mlxsw_linecard_bdev_probe(struct auxiliary_device *adev,
> >> +				     const struct auxiliary_device_id *id)
> >> +{
> >> +	struct mlxsw_linecard_bdev *linecard_bdev =
> >> +			container_of(adev, struct mlxsw_linecard_bdev, adev);
> >> +	struct mlxsw_linecard_dev *linecard_dev;
> >> +	struct devlink *devlink;
> >> +
> >> +	devlink = devlink_alloc(&mlxsw_linecard_dev_devlink_ops,
> >> +				sizeof(*linecard_dev), &adev->dev);
> >> +	if (!devlink)
> >> +		return -ENOMEM;
> >> +	linecard_dev = devlink_priv(devlink);
> >> +	linecard_dev->linecard = linecard_bdev->linecard;
> >> +	linecard_bdev->linecard_dev = linecard_dev;
> >> +
> >> +	devlink_register(devlink);
> >> +	return 0;
> >> +}
> >
> >[...]
> >
> >> @@ -252,6 +253,14 @@ mlxsw_linecard_provision_set(struct mlxsw_linecard *linecard, u8 card_type,
> >>  	linecard->provisioned = true;
> >>  	linecard->hw_revision = hw_revision;
> >>  	linecard->ini_version = ini_version;
> >> +
> >> +	err = mlxsw_linecard_bdev_add(linecard);
> >
> >If a line card is already provisioned and we are reloading the primary
> >devlink instance, isn't this going to deadlock on the global (not
> >per-instance) devlink mutex? It is held throughout the reload operation
> >and also taken in devlink_register()
> >
> >My understanding of the auxiliary bus model is that after adding a
> >device to the bus via auxiliary_device_add(), the probe() function of
> >the auxiliary driver will be called. In our case, this function acquires
> >the global devlink mutex in devlink_register().
> 
> No, the line card auxdev is supposed to be removed during
> linecard_fini(). This, I forgot to add, will do in v2.

mlxsw_linecard_fini() is called as part of reload with the global
devlink mutex held. The removal of the auxdev should prompt the
unregistration of its devlink instance which also takes this mutex. If
this doesn't deadlock, then I'm probably missing something.

Can you test reload with lockdep when line cards are already
provisioned/active?

> 
> 
> >
> >> +	if (err) {
> >> +		linecard->provisioned = false;
> >> +		mlxsw_linecard_provision_fail(linecard);
> >> +		return err;
> >> +	}
> >> +
> >>  	devlink_linecard_provision_set(linecard->devlink_linecard, type);
> >>  	return 0;
> >>  }

Re: [patch net-next 02/11] mlxsw: core_linecards: Introduce per line card auxiliary device

[Jiri Pirko]

Thu, Jun 16, 2022 at 09:11:56AM CEST, idosch@nvidia.com wrote:
>On Wed, Jun 15, 2022 at 07:37:15PM +0200, Jiri Pirko wrote:
>> Wed, Jun 15, 2022 at 04:52:13PM CEST, idosch@nvidia.com wrote:
>> >> +int mlxsw_linecard_bdev_add(struct mlxsw_linecard *linecard)
>> >> +{
>> >> +	struct mlxsw_linecard_bdev *linecard_bdev;
>> >> +	int err;
>> >> +	int id;
>> >> +
>> >> +	id = mlxsw_linecard_bdev_id_alloc();
>> >> +	if (id < 0)
>> >> +		return id;
>> >> +
>> >> +	linecard_bdev = kzalloc(sizeof(*linecard_bdev), GFP_KERNEL);
>> >> +	if (!linecard_bdev) {
>> >> +		mlxsw_linecard_bdev_id_free(id);
>> >> +		return -ENOMEM;
>> >> +	}
>> >> +	linecard_bdev->adev.id = id;
>> >> +	linecard_bdev->adev.name = MLXSW_LINECARD_DEV_ID_NAME;
>> >> +	linecard_bdev->adev.dev.release = mlxsw_linecard_bdev_release;
>> >> +	linecard_bdev->adev.dev.parent = linecard->linecards->bus_info->dev;
>> >> +	linecard_bdev->linecard = linecard;
>> >> +
>> >> +	err = auxiliary_device_init(&linecard_bdev->adev);
>> >> +	if (err) {
>> >> +		mlxsw_linecard_bdev_id_free(id);
>> >> +		kfree(linecard_bdev);
>> >> +		return err;
>> >> +	}
>> >> +
>> >> +	err = auxiliary_device_add(&linecard_bdev->adev);
>> >> +	if (err) {
>> >> +		auxiliary_device_uninit(&linecard_bdev->adev);
>> >> +		return err;
>> >> +	}
>> >> +
>> >> +	linecard->bdev = linecard_bdev;
>> >> +	return 0;
>> >> +}
>> >
>> >[...]
>> >
>> >> +static int mlxsw_linecard_bdev_probe(struct auxiliary_device *adev,
>> >> +				     const struct auxiliary_device_id *id)
>> >> +{
>> >> +	struct mlxsw_linecard_bdev *linecard_bdev =
>> >> +			container_of(adev, struct mlxsw_linecard_bdev, adev);
>> >> +	struct mlxsw_linecard_dev *linecard_dev;
>> >> +	struct devlink *devlink;
>> >> +
>> >> +	devlink = devlink_alloc(&mlxsw_linecard_dev_devlink_ops,
>> >> +				sizeof(*linecard_dev), &adev->dev);
>> >> +	if (!devlink)
>> >> +		return -ENOMEM;
>> >> +	linecard_dev = devlink_priv(devlink);
>> >> +	linecard_dev->linecard = linecard_bdev->linecard;
>> >> +	linecard_bdev->linecard_dev = linecard_dev;
>> >> +
>> >> +	devlink_register(devlink);
>> >> +	return 0;
>> >> +}
>> >
>> >[...]
>> >
>> >> @@ -252,6 +253,14 @@ mlxsw_linecard_provision_set(struct mlxsw_linecard *linecard, u8 card_type,
>> >>  	linecard->provisioned = true;
>> >>  	linecard->hw_revision = hw_revision;
>> >>  	linecard->ini_version = ini_version;
>> >> +
>> >> +	err = mlxsw_linecard_bdev_add(linecard);
>> >
>> >If a line card is already provisioned and we are reloading the primary
>> >devlink instance, isn't this going to deadlock on the global (not
>> >per-instance) devlink mutex? It is held throughout the reload operation
>> >and also taken in devlink_register()
>> >
>> >My understanding of the auxiliary bus model is that after adding a
>> >device to the bus via auxiliary_device_add(), the probe() function of
>> >the auxiliary driver will be called. In our case, this function acquires
>> >the global devlink mutex in devlink_register().
>> 
>> No, the line card auxdev is supposed to be removed during
>> linecard_fini(). This, I forgot to add, will do in v2.
>
>mlxsw_linecard_fini() is called as part of reload with the global
>devlink mutex held. The removal of the auxdev should prompt the
>unregistration of its devlink instance which also takes this mutex. If
>this doesn't deadlock, then I'm probably missing something.

You don't miss anything, it really does. Need to remove devlink_mutex
first.


>
>Can you test reload with lockdep when line cards are already
>provisioned/active?
>
>> 
>> 
>> >
>> >> +	if (err) {
>> >> +		linecard->provisioned = false;
>> >> +		mlxsw_linecard_provision_fail(linecard);
>> >> +		return err;
>> >> +	}
>> >> +
>> >>  	devlink_linecard_provision_set(linecard->devlink_linecard, type);
>> >>  	return 0;
>> >>  }

Re: [patch net-next 02/11] mlxsw: core_linecards: Introduce per line card auxiliary device

[Ido Schimmel]

On Thu, Jun 16, 2022 at 06:39:59PM +0200, Jiri Pirko wrote:
> Thu, Jun 16, 2022 at 09:11:56AM CEST, idosch@nvidia.com wrote:
> >On Wed, Jun 15, 2022 at 07:37:15PM +0200, Jiri Pirko wrote:
> >> Wed, Jun 15, 2022 at 04:52:13PM CEST, idosch@nvidia.com wrote:
> >> >> +int mlxsw_linecard_bdev_add(struct mlxsw_linecard *linecard)
> >> >> +{
> >> >> +	struct mlxsw_linecard_bdev *linecard_bdev;
> >> >> +	int err;
> >> >> +	int id;
> >> >> +
> >> >> +	id = mlxsw_linecard_bdev_id_alloc();
> >> >> +	if (id < 0)
> >> >> +		return id;
> >> >> +
> >> >> +	linecard_bdev = kzalloc(sizeof(*linecard_bdev), GFP_KERNEL);
> >> >> +	if (!linecard_bdev) {
> >> >> +		mlxsw_linecard_bdev_id_free(id);
> >> >> +		return -ENOMEM;
> >> >> +	}
> >> >> +	linecard_bdev->adev.id = id;
> >> >> +	linecard_bdev->adev.name = MLXSW_LINECARD_DEV_ID_NAME;
> >> >> +	linecard_bdev->adev.dev.release = mlxsw_linecard_bdev_release;
> >> >> +	linecard_bdev->adev.dev.parent = linecard->linecards->bus_info->dev;
> >> >> +	linecard_bdev->linecard = linecard;
> >> >> +
> >> >> +	err = auxiliary_device_init(&linecard_bdev->adev);
> >> >> +	if (err) {
> >> >> +		mlxsw_linecard_bdev_id_free(id);
> >> >> +		kfree(linecard_bdev);
> >> >> +		return err;
> >> >> +	}
> >> >> +
> >> >> +	err = auxiliary_device_add(&linecard_bdev->adev);
> >> >> +	if (err) {
> >> >> +		auxiliary_device_uninit(&linecard_bdev->adev);
> >> >> +		return err;
> >> >> +	}
> >> >> +
> >> >> +	linecard->bdev = linecard_bdev;
> >> >> +	return 0;
> >> >> +}
> >> >
> >> >[...]
> >> >
> >> >> +static int mlxsw_linecard_bdev_probe(struct auxiliary_device *adev,
> >> >> +				     const struct auxiliary_device_id *id)
> >> >> +{
> >> >> +	struct mlxsw_linecard_bdev *linecard_bdev =
> >> >> +			container_of(adev, struct mlxsw_linecard_bdev, adev);
> >> >> +	struct mlxsw_linecard_dev *linecard_dev;
> >> >> +	struct devlink *devlink;
> >> >> +
> >> >> +	devlink = devlink_alloc(&mlxsw_linecard_dev_devlink_ops,
> >> >> +				sizeof(*linecard_dev), &adev->dev);
> >> >> +	if (!devlink)
> >> >> +		return -ENOMEM;
> >> >> +	linecard_dev = devlink_priv(devlink);
> >> >> +	linecard_dev->linecard = linecard_bdev->linecard;
> >> >> +	linecard_bdev->linecard_dev = linecard_dev;
> >> >> +
> >> >> +	devlink_register(devlink);
> >> >> +	return 0;
> >> >> +}
> >> >
> >> >[...]
> >> >
> >> >> @@ -252,6 +253,14 @@ mlxsw_linecard_provision_set(struct mlxsw_linecard *linecard, u8 card_type,
> >> >>  	linecard->provisioned = true;
> >> >>  	linecard->hw_revision = hw_revision;
> >> >>  	linecard->ini_version = ini_version;
> >> >> +
> >> >> +	err = mlxsw_linecard_bdev_add(linecard);
> >> >
> >> >If a line card is already provisioned and we are reloading the primary
> >> >devlink instance, isn't this going to deadlock on the global (not
> >> >per-instance) devlink mutex? It is held throughout the reload operation
> >> >and also taken in devlink_register()
> >> >
> >> >My understanding of the auxiliary bus model is that after adding a
> >> >device to the bus via auxiliary_device_add(), the probe() function of
> >> >the auxiliary driver will be called. In our case, this function acquires
> >> >the global devlink mutex in devlink_register().
> >> 
> >> No, the line card auxdev is supposed to be removed during
> >> linecard_fini(). This, I forgot to add, will do in v2.
> >
> >mlxsw_linecard_fini() is called as part of reload with the global
> >devlink mutex held. The removal of the auxdev should prompt the
> >unregistration of its devlink instance which also takes this mutex. If
> >this doesn't deadlock, then I'm probably missing something.
> 
> You don't miss anything, it really does. Need to remove devlink_mutex
> first.

Can you please send it separately? Will probably need thorough review
and testing...

The comment above devlink_mutex is: "An overall lock guarding every
operation coming from userspace. It also guards devlink devices list and
it is taken when driver registers/unregisters it.", but devlink does not
have "parallel_ops" enabled, so maybe it's enough to only use this lock
to protect the devlink devices list?

> 
> 
> >
> >Can you test reload with lockdep when line cards are already
> >provisioned/active?
> >
> >> 
> >> 
> >> >
> >> >> +	if (err) {
> >> >> +		linecard->provisioned = false;
> >> >> +		mlxsw_linecard_provision_fail(linecard);
> >> >> +		return err;
> >> >> +	}
> >> >> +
> >> >>  	devlink_linecard_provision_set(linecard->devlink_linecard, type);
> >> >>  	return 0;
> >> >>  }

Re: [patch net-next 02/11] mlxsw: core_linecards: Introduce per line card auxiliary device

[Jiri Pirko]

Thu, Jun 16, 2022 at 07:41:52PM CEST, idosch@nvidia.com wrote:
>On Thu, Jun 16, 2022 at 06:39:59PM +0200, Jiri Pirko wrote:
>> Thu, Jun 16, 2022 at 09:11:56AM CEST, idosch@nvidia.com wrote:
>> >On Wed, Jun 15, 2022 at 07:37:15PM +0200, Jiri Pirko wrote:
>> >> Wed, Jun 15, 2022 at 04:52:13PM CEST, idosch@nvidia.com wrote:
>> >> >> +int mlxsw_linecard_bdev_add(struct mlxsw_linecard *linecard)
>> >> >> +{
>> >> >> +	struct mlxsw_linecard_bdev *linecard_bdev;
>> >> >> +	int err;
>> >> >> +	int id;
>> >> >> +
>> >> >> +	id = mlxsw_linecard_bdev_id_alloc();
>> >> >> +	if (id < 0)
>> >> >> +		return id;
>> >> >> +
>> >> >> +	linecard_bdev = kzalloc(sizeof(*linecard_bdev), GFP_KERNEL);
>> >> >> +	if (!linecard_bdev) {
>> >> >> +		mlxsw_linecard_bdev_id_free(id);
>> >> >> +		return -ENOMEM;
>> >> >> +	}
>> >> >> +	linecard_bdev->adev.id = id;
>> >> >> +	linecard_bdev->adev.name = MLXSW_LINECARD_DEV_ID_NAME;
>> >> >> +	linecard_bdev->adev.dev.release = mlxsw_linecard_bdev_release;
>> >> >> +	linecard_bdev->adev.dev.parent = linecard->linecards->bus_info->dev;
>> >> >> +	linecard_bdev->linecard = linecard;
>> >> >> +
>> >> >> +	err = auxiliary_device_init(&linecard_bdev->adev);
>> >> >> +	if (err) {
>> >> >> +		mlxsw_linecard_bdev_id_free(id);
>> >> >> +		kfree(linecard_bdev);
>> >> >> +		return err;
>> >> >> +	}
>> >> >> +
>> >> >> +	err = auxiliary_device_add(&linecard_bdev->adev);
>> >> >> +	if (err) {
>> >> >> +		auxiliary_device_uninit(&linecard_bdev->adev);
>> >> >> +		return err;
>> >> >> +	}
>> >> >> +
>> >> >> +	linecard->bdev = linecard_bdev;
>> >> >> +	return 0;
>> >> >> +}
>> >> >
>> >> >[...]
>> >> >
>> >> >> +static int mlxsw_linecard_bdev_probe(struct auxiliary_device *adev,
>> >> >> +				     const struct auxiliary_device_id *id)
>> >> >> +{
>> >> >> +	struct mlxsw_linecard_bdev *linecard_bdev =
>> >> >> +			container_of(adev, struct mlxsw_linecard_bdev, adev);
>> >> >> +	struct mlxsw_linecard_dev *linecard_dev;
>> >> >> +	struct devlink *devlink;
>> >> >> +
>> >> >> +	devlink = devlink_alloc(&mlxsw_linecard_dev_devlink_ops,
>> >> >> +				sizeof(*linecard_dev), &adev->dev);
>> >> >> +	if (!devlink)
>> >> >> +		return -ENOMEM;
>> >> >> +	linecard_dev = devlink_priv(devlink);
>> >> >> +	linecard_dev->linecard = linecard_bdev->linecard;
>> >> >> +	linecard_bdev->linecard_dev = linecard_dev;
>> >> >> +
>> >> >> +	devlink_register(devlink);
>> >> >> +	return 0;
>> >> >> +}
>> >> >
>> >> >[...]
>> >> >
>> >> >> @@ -252,6 +253,14 @@ mlxsw_linecard_provision_set(struct mlxsw_linecard *linecard, u8 card_type,
>> >> >>  	linecard->provisioned = true;
>> >> >>  	linecard->hw_revision = hw_revision;
>> >> >>  	linecard->ini_version = ini_version;
>> >> >> +
>> >> >> +	err = mlxsw_linecard_bdev_add(linecard);
>> >> >
>> >> >If a line card is already provisioned and we are reloading the primary
>> >> >devlink instance, isn't this going to deadlock on the global (not
>> >> >per-instance) devlink mutex? It is held throughout the reload operation
>> >> >and also taken in devlink_register()
>> >> >
>> >> >My understanding of the auxiliary bus model is that after adding a
>> >> >device to the bus via auxiliary_device_add(), the probe() function of
>> >> >the auxiliary driver will be called. In our case, this function acquires
>> >> >the global devlink mutex in devlink_register().
>> >> 
>> >> No, the line card auxdev is supposed to be removed during
>> >> linecard_fini(). This, I forgot to add, will do in v2.
>> >
>> >mlxsw_linecard_fini() is called as part of reload with the global
>> >devlink mutex held. The removal of the auxdev should prompt the
>> >unregistration of its devlink instance which also takes this mutex. If
>> >this doesn't deadlock, then I'm probably missing something.
>> 
>> You don't miss anything, it really does. Need to remove devlink_mutex
>> first.
>
>Can you please send it separately? Will probably need thorough review
>and testing...

Sure.


>
>The comment above devlink_mutex is: "An overall lock guarding every
>operation coming from userspace. It also guards devlink devices list and
>it is taken when driver registers/unregisters it.", but devlink does not
>have "parallel_ops" enabled, so maybe it's enough to only use this lock
>to protect the devlink devices list?

I'm preparing a patchset that will answer all your questions, lets move
the discussion there.


>
>> 
>> 
>> >
>> >Can you test reload with lockdep when line cards are already
>> >provisioned/active?
>> >
>> >> 
>> >> 
>> >> >
>> >> >> +	if (err) {
>> >> >> +		linecard->provisioned = false;
>> >> >> +		mlxsw_linecard_provision_fail(linecard);
>> >> >> +		return err;
>> >> >> +	}
>> >> >> +
>> >> >>  	devlink_linecard_provision_set(linecard->devlink_linecard, type);
>> >> >>  	return 0;
>> >> >>  }

[patch net-next 03/11] mlxsw: core_linecard_dev: Set nested devlink relationship for a line card

[Jiri Pirko]

From: Jiri Pirko <jiri@nvidia.com>

For newly created line card device devlink instance, set the
relationship with the parent line card object.

Signed-off-by: Jiri Pirko <jiri@nvidia.com>
---
 drivers/net/ethernet/mellanox/mlxsw/core_linecard_dev.c | 4 ++++
 1 file changed, 4 insertions(+)

diff --git a/drivers/net/ethernet/mellanox/mlxsw/core_linecard_dev.c b/drivers/net/ethernet/mellanox/mlxsw/core_linecard_dev.c
index af70d3f7a177..d12abd935ded 100644
--- a/drivers/net/ethernet/mellanox/mlxsw/core_linecard_dev.c
+++ b/drivers/net/ethernet/mellanox/mlxsw/core_linecard_dev.c
@@ -102,6 +102,7 @@ static int mlxsw_linecard_bdev_probe(struct auxiliary_device *adev,
 {
 	struct mlxsw_linecard_bdev *linecard_bdev =
 			container_of(adev, struct mlxsw_linecard_bdev, adev);
+	struct mlxsw_linecard *linecard = linecard_bdev->linecard;
 	struct mlxsw_linecard_dev *linecard_dev;
 	struct devlink *devlink;
 
@@ -114,6 +115,7 @@ static int mlxsw_linecard_bdev_probe(struct auxiliary_device *adev,
 	linecard_bdev->linecard_dev = linecard_dev;
 
 	devlink_register(devlink);
+	devlink_linecard_nested_dl_set(linecard->devlink_linecard, devlink);
 	return 0;
 }
 
@@ -122,7 +124,9 @@ static void mlxsw_linecard_bdev_remove(struct auxiliary_device *adev)
 	struct mlxsw_linecard_bdev *linecard_bdev =
 			container_of(adev, struct mlxsw_linecard_bdev, adev);
 	struct devlink *devlink = priv_to_devlink(linecard_bdev->linecard_dev);
+	struct mlxsw_linecard *linecard = linecard_bdev->linecard;
 
+	devlink_linecard_nested_dl_set(linecard->devlink_linecard, NULL);
 	devlink_unregister(devlink);
 	devlink_free(devlink);
 }
-- 
2.35.3

[patch net-next 04/11] mlxsw: core_linecards: Expose HW revision and INI version

[Jiri Pirko]

From: Jiri Pirko <jiri@nvidia.com>

Implement info_get() to expose HW revision of a linecard and loaded INI
version.

Example:

$ devlink dev info auxiliary/mlxsw_core.lc.0
auxiliary/mlxsw_core.lc.0:
  versions:
      fixed:
        hw.revision 0
      running:
        ini.version 4

Signed-off-by: Jiri Pirko <jiri@nvidia.com>
---
 Documentation/networking/devlink/mlxsw.rst    | 18 ++++++++++++
 drivers/net/ethernet/mellanox/mlxsw/core.h    |  4 +++
 .../mellanox/mlxsw/core_linecard_dev.c        | 11 ++++++++
 .../ethernet/mellanox/mlxsw/core_linecards.c  | 28 +++++++++++++++++++
 4 files changed, 61 insertions(+)

diff --git a/Documentation/networking/devlink/mlxsw.rst b/Documentation/networking/devlink/mlxsw.rst
index cf857cb4ba8f..aededcf68df4 100644
--- a/Documentation/networking/devlink/mlxsw.rst
+++ b/Documentation/networking/devlink/mlxsw.rst
@@ -58,6 +58,24 @@ The ``mlxsw`` driver reports the following versions
      - running
      - Three digit firmware version
 
+Line card auxiliary device info versions
+========================================
+
+The ``mlxsw`` driver reports the following versions for line card auxiliary device
+
+.. list-table:: devlink info versions implemented
+   :widths: 5 5 90
+
+   * - Name
+     - Type
+     - Description
+   * - ``hw.revision``
+     - fixed
+     - The hardware revision for this line card
+   * - ``ini.version``
+     - running
+     - Version of line card INI loaded
+
 Driver-specific Traps
 =====================
 
diff --git a/drivers/net/ethernet/mellanox/mlxsw/core.h b/drivers/net/ethernet/mellanox/mlxsw/core.h
index cda20a4fcbdb..ea56bcc0a399 100644
--- a/drivers/net/ethernet/mellanox/mlxsw/core.h
+++ b/drivers/net/ethernet/mellanox/mlxsw/core.h
@@ -605,6 +605,10 @@ mlxsw_linecard_get(struct mlxsw_linecards *linecards, u8 slot_index)
 	return &linecards->linecards[slot_index - 1];
 }
 
+int mlxsw_linecard_devlink_info_get(struct mlxsw_linecard *linecard,
+				    struct devlink_info_req *req,
+				    struct netlink_ext_ack *extack);
+
 int mlxsw_linecards_init(struct mlxsw_core *mlxsw_core,
 			 const struct mlxsw_bus_info *bus_info);
 void mlxsw_linecards_fini(struct mlxsw_core *mlxsw_core);
diff --git a/drivers/net/ethernet/mellanox/mlxsw/core_linecard_dev.c b/drivers/net/ethernet/mellanox/mlxsw/core_linecard_dev.c
index d12abd935ded..30659f8be41c 100644
--- a/drivers/net/ethernet/mellanox/mlxsw/core_linecard_dev.c
+++ b/drivers/net/ethernet/mellanox/mlxsw/core_linecard_dev.c
@@ -94,7 +94,18 @@ void mlxsw_linecard_bdev_del(struct mlxsw_linecard *linecard)
 	auxiliary_device_uninit(&linecard_bdev->adev);
 }
 
+static int mlxsw_linecard_dev_devlink_info_get(struct devlink *devlink,
+					       struct devlink_info_req *req,
+					       struct netlink_ext_ack *extack)
+{
+	struct mlxsw_linecard_dev *linecard_dev = devlink_priv(devlink);
+	struct mlxsw_linecard *linecard = linecard_dev->linecard;
+
+	return mlxsw_linecard_devlink_info_get(linecard, req, extack);
+}
+
 static const struct devlink_ops mlxsw_linecard_dev_devlink_ops = {
+	.info_get			= mlxsw_linecard_dev_devlink_info_get,
 };
 
 static int mlxsw_linecard_bdev_probe(struct auxiliary_device *adev,
diff --git a/drivers/net/ethernet/mellanox/mlxsw/core_linecards.c b/drivers/net/ethernet/mellanox/mlxsw/core_linecards.c
index ae51944cde0c..c11d22708bba 100644
--- a/drivers/net/ethernet/mellanox/mlxsw/core_linecards.c
+++ b/drivers/net/ethernet/mellanox/mlxsw/core_linecards.c
@@ -226,6 +226,34 @@ void mlxsw_linecards_event_ops_unregister(struct mlxsw_core *mlxsw_core,
 }
 EXPORT_SYMBOL(mlxsw_linecards_event_ops_unregister);
 
+int mlxsw_linecard_devlink_info_get(struct mlxsw_linecard *linecard,
+				    struct devlink_info_req *req,
+				    struct netlink_ext_ack *extack)
+{
+	char buf[32];
+	int err;
+
+	mutex_lock(&linecard->lock);
+	if (WARN_ON(!linecard->provisioned)) {
+		err = 0;
+		goto unlock;
+	}
+
+	sprintf(buf, "%d", linecard->hw_revision);
+	err = devlink_info_version_fixed_put(req, "hw.revision", buf);
+	if (err)
+		goto unlock;
+
+	sprintf(buf, "%d", linecard->ini_version);
+	err = devlink_info_version_running_put(req, "ini.version", buf);
+	if (err)
+		goto unlock;
+
+unlock:
+	mutex_unlock(&linecard->lock);
+	return err;
+}
+
 static int
 mlxsw_linecard_provision_set(struct mlxsw_linecard *linecard, u8 card_type,
 			     u16 hw_revision, u16 ini_version)
-- 
2.35.3

[patch net-next 05/11] mlxsw: reg: Extend MDDQ by device_info

[Jiri Pirko]

From: Jiri Pirko <jiri@nvidia.com>

Extend existing MDDQ register by possibility to query information about
devices residing on a line card.

Signed-off-by: Jiri Pirko <jiri@nvidia.com>
---
 drivers/net/ethernet/mellanox/mlxsw/reg.h | 83 ++++++++++++++++++++++-
 1 file changed, 82 insertions(+), 1 deletion(-)

diff --git a/drivers/net/ethernet/mellanox/mlxsw/reg.h b/drivers/net/ethernet/mellanox/mlxsw/reg.h
index 93af6c974ece..c8d526669522 100644
--- a/drivers/net/ethernet/mellanox/mlxsw/reg.h
+++ b/drivers/net/ethernet/mellanox/mlxsw/reg.h
@@ -11643,7 +11643,11 @@ MLXSW_ITEM32(reg, mddq, sie, 0x00, 31, 1);
 
 enum mlxsw_reg_mddq_query_type {
 	MLXSW_REG_MDDQ_QUERY_TYPE_SLOT_INFO = 1,
-	MLXSW_REG_MDDQ_QUERY_TYPE_SLOT_NAME = 3,
+	MLXSW_REG_MDDQ_QUERY_TYPE_DEVICE_INFO, /* If there are no devices
+						* on the slot, data_valid
+						* will be '0'.
+						*/
+	MLXSW_REG_MDDQ_QUERY_TYPE_SLOT_NAME,
 };
 
 /* reg_mddq_query_type
@@ -11657,6 +11661,28 @@ MLXSW_ITEM32(reg, mddq, query_type, 0x00, 16, 8);
  */
 MLXSW_ITEM32(reg, mddq, slot_index, 0x00, 0, 4);
 
+/* reg_mddq_response_msg_seq
+ * Response message sequential number. For a specific request, the response
+ * message sequential number is the following one. In addition, the last
+ * message should be 0.
+ * Access: RO
+ */
+MLXSW_ITEM32(reg, mddq, response_msg_seq, 0x04, 16, 8);
+
+/* reg_mddq_request_msg_seq
+ * Request message sequential number.
+ * The first message number should be 0.
+ * Access: Index
+ */
+MLXSW_ITEM32(reg, mddq, request_msg_seq, 0x04, 0, 8);
+
+/* reg_mddq_data_valid
+ * If set, the data in the data field is valid and contain the information
+ * for the queried index.
+ * Access: RO
+ */
+MLXSW_ITEM32(reg, mddq, data_valid, 0x08, 31, 1);
+
 /* reg_mddq_slot_info_provisioned
  * If set, the INI file is applied and the card is provisioned.
  * Access: RO
@@ -11743,6 +11769,61 @@ mlxsw_reg_mddq_slot_info_unpack(const char *payload, u8 *p_slot_index,
 	*p_card_type = mlxsw_reg_mddq_slot_info_card_type_get(payload);
 }
 
+/* reg_mddq_device_info_flash_owner
+ * If set, the device is the flash owner. Otherwise, a shared flash
+ * is used by this device (another device is the flash owner).
+ * Access: RO
+ */
+MLXSW_ITEM32(reg, mddq, device_info_flash_owner, 0x10, 30, 1);
+
+/* reg_mddq_device_info_device_index
+ * Device index. The first device should number 0.
+ * Access: RO
+ */
+MLXSW_ITEM32(reg, mddq, device_info_device_index, 0x10, 0, 8);
+
+/* reg_mddq_device_info_fw_major
+ * Major FW version number.
+ * Access: RO
+ */
+MLXSW_ITEM32(reg, mddq, device_info_fw_major, 0x14, 16, 16);
+
+/* reg_mddq_device_info_fw_minor
+ * Minor FW version number.
+ * Access: RO
+ */
+MLXSW_ITEM32(reg, mddq, device_info_fw_minor, 0x18, 16, 16);
+
+/* reg_mddq_device_info_fw_sub_minor
+ * Sub-minor FW version number.
+ * Access: RO
+ */
+MLXSW_ITEM32(reg, mddq, device_info_fw_sub_minor, 0x18, 0, 16);
+
+static inline void
+mlxsw_reg_mddq_device_info_pack(char *payload, u8 slot_index,
+				u8 request_msg_seq)
+{
+	__mlxsw_reg_mddq_pack(payload, slot_index,
+			      MLXSW_REG_MDDQ_QUERY_TYPE_DEVICE_INFO);
+	mlxsw_reg_mddq_request_msg_seq_set(payload, request_msg_seq);
+}
+
+static inline void
+mlxsw_reg_mddq_device_info_unpack(const char *payload, u8 *p_response_msg_seq,
+				  bool *p_data_valid, bool *p_flash_owner,
+				  u8 *p_device_index, u16 *p_fw_major,
+				  u16 *p_fw_minor, u16 *p_fw_sub_minor)
+{
+	*p_response_msg_seq = mlxsw_reg_mddq_response_msg_seq_get(payload);
+	*p_data_valid = mlxsw_reg_mddq_data_valid_get(payload);
+	*p_flash_owner = mlxsw_reg_mddq_device_info_flash_owner_get(payload);
+	*p_device_index = mlxsw_reg_mddq_device_info_device_index_get(payload);
+	*p_fw_major = mlxsw_reg_mddq_device_info_fw_major_get(payload);
+	*p_fw_minor = mlxsw_reg_mddq_device_info_fw_minor_get(payload);
+	*p_fw_sub_minor = mlxsw_reg_mddq_device_info_fw_sub_minor_get(payload);
+}
+
 #define MLXSW_REG_MDDQ_SLOT_ASCII_NAME_LEN 20
 
 /* reg_mddq_slot_ascii_name
-- 
2.35.3

[patch net-next 06/11] mlxsw: core_linecards: Probe provisioned line cards for devices and expose FW version

[Jiri Pirko]

From: Jiri Pirko <jiri@nvidia.com>

In case the line card is provisioned, go over all possible existing
devices (gearboxes) on it and expose FW version of the flashable one.

Example:

$ devlink dev info auxiliary/mlxsw_core.lc.0
auxiliary/mlxsw_core.lc.0:
  versions:
      fixed:
        hw.revision 0
      running:
        ini.version 4
        fw 19.2010.1312

Signed-off-by: Jiri Pirko <jiri@nvidia.com>
---
 Documentation/networking/devlink/mlxsw.rst    |  3 +
 drivers/net/ethernet/mellanox/mlxsw/core.h    |  9 +++
 .../ethernet/mellanox/mlxsw/core_linecards.c  | 57 +++++++++++++++++++
 3 files changed, 69 insertions(+)

diff --git a/Documentation/networking/devlink/mlxsw.rst b/Documentation/networking/devlink/mlxsw.rst
index aededcf68df4..65ceed98f94d 100644
--- a/Documentation/networking/devlink/mlxsw.rst
+++ b/Documentation/networking/devlink/mlxsw.rst
@@ -75,6 +75,9 @@ The ``mlxsw`` driver reports the following versions for line card auxiliary devi
    * - ``ini.version``
      - running
      - Version of line card INI loaded
+   * - ``fw.version``
+     - running
+     - Three digit firmware version of line card device
 
 Driver-specific Traps
 =====================
diff --git a/drivers/net/ethernet/mellanox/mlxsw/core.h b/drivers/net/ethernet/mellanox/mlxsw/core.h
index ea56bcc0a399..edb0e1307d20 100644
--- a/drivers/net/ethernet/mellanox/mlxsw/core.h
+++ b/drivers/net/ethernet/mellanox/mlxsw/core.h
@@ -570,6 +570,12 @@ enum mlxsw_linecard_status_event_type {
 
 struct mlxsw_linecard_bdev;
 
+struct mlxsw_linecard_device_info {
+	u16 fw_major;
+	u16 fw_minor;
+	u16 fw_sub_minor;
+};
+
 struct mlxsw_linecard {
 	u8 slot_index;
 	struct mlxsw_linecards *linecards;
@@ -585,6 +591,9 @@ struct mlxsw_linecard {
 	u16 hw_revision;
 	u16 ini_version;
 	struct mlxsw_linecard_bdev *bdev;
+	struct {
+		struct mlxsw_linecard_device_info info;
+	} device;
 };
 
 struct mlxsw_linecard_types_info;
diff --git a/drivers/net/ethernet/mellanox/mlxsw/core_linecards.c b/drivers/net/ethernet/mellanox/mlxsw/core_linecards.c
index c11d22708bba..0f494ea755a9 100644
--- a/drivers/net/ethernet/mellanox/mlxsw/core_linecards.c
+++ b/drivers/net/ethernet/mellanox/mlxsw/core_linecards.c
@@ -87,6 +87,47 @@ static const char *mlxsw_linecard_type_name(struct mlxsw_linecard *linecard)
 	return linecard->name;
 }
 
+static int mlxsw_linecard_device_info_update(struct mlxsw_linecard *linecard)
+{
+	struct mlxsw_core *mlxsw_core = linecard->linecards->mlxsw_core;
+	bool flashable_found = false;
+	u8 msg_seq = 0;
+
+	do {
+		struct mlxsw_linecard_device_info info;
+		char mddq_pl[MLXSW_REG_MDDQ_LEN];
+		bool flash_owner;
+		bool data_valid;
+		u8 device_index;
+		int err;
+
+		mlxsw_reg_mddq_device_info_pack(mddq_pl, linecard->slot_index,
+						msg_seq);
+		err = mlxsw_reg_query(mlxsw_core, MLXSW_REG(mddq), mddq_pl);
+		if (err)
+			return err;
+		mlxsw_reg_mddq_device_info_unpack(mddq_pl, &msg_seq,
+						  &data_valid, &flash_owner,
+						  &device_index,
+						  &info.fw_major,
+						  &info.fw_minor,
+						  &info.fw_sub_minor);
+		if (!data_valid)
+			break;
+		if (!flash_owner) /* We care only about flashable ones. */
+			continue;
+		if (flashable_found) {
+			dev_warn_once(linecard->linecards->bus_info->dev, "linecard %u: More flashable devices present, exposing only the first one\n",
+				      linecard->slot_index);
+			return 0;
+		}
+		linecard->device.info = info;
+		flashable_found = true;
+	} while (msg_seq);
+
+	return 0;
+}
+
 static void mlxsw_linecard_provision_fail(struct mlxsw_linecard *linecard)
 {
 	linecard->provisioned = false;
@@ -249,6 +290,18 @@ int mlxsw_linecard_devlink_info_get(struct mlxsw_linecard *linecard,
 	if (err)
 		goto unlock;
 
+	if (linecard->ready) {
+		struct mlxsw_linecard_device_info *info = &linecard->device.info;
+
+		sprintf(buf, "%u.%u.%u", info->fw_major, info->fw_minor,
+			info->fw_sub_minor);
+		err = devlink_info_version_running_put(req,
+						       DEVLINK_INFO_VERSION_GENERIC_FW,
+						       buf);
+		if (err)
+			goto unlock;
+	}
+
 unlock:
 	mutex_unlock(&linecard->lock);
 	return err;
@@ -308,6 +361,10 @@ static int mlxsw_linecard_ready_set(struct mlxsw_linecard *linecard)
 	char mddc_pl[MLXSW_REG_MDDC_LEN];
 	int err;
 
+	err = mlxsw_linecard_device_info_update(linecard);
+	if (err)
+		return err;
+
 	mlxsw_reg_mddc_pack(mddc_pl, linecard->slot_index, false, true);
 	err = mlxsw_reg_write(mlxsw_core, MLXSW_REG(mddc), mddc_pl);
 	if (err)
-- 
2.35.3

[patch net-next 07/11] mlxsw: reg: Add Management DownStream Device Tunneling Register

[Jiri Pirko]

From: Jiri Pirko <jiri@nvidia.com>

The MDDT register allows to deliver query and request messages (PRM
registers, commands) to a DownStream device.

Signed-off-by: Jiri Pirko <jiri@nvidia.com>
Signed-off-by: Ido Schimmel <idosch@nvidia.com>
---
 drivers/net/ethernet/mellanox/mlxsw/reg.h | 90 +++++++++++++++++++++++
 1 file changed, 90 insertions(+)

diff --git a/drivers/net/ethernet/mellanox/mlxsw/reg.h b/drivers/net/ethernet/mellanox/mlxsw/reg.h
index c8d526669522..953790b9abd1 100644
--- a/drivers/net/ethernet/mellanox/mlxsw/reg.h
+++ b/drivers/net/ethernet/mellanox/mlxsw/reg.h
@@ -11622,6 +11622,95 @@ mlxsw_reg_mbct_unpack(const char *payload, u8 *p_slot_index,
 		*p_fsm_state = mlxsw_reg_mbct_fsm_state_get(payload);
 }
 
+/* MDDT - Management DownStream Device Tunneling Register
+ * ------------------------------------------------------
+ * This register allows to deliver query and request messages (PRM registers,
+ * commands) to a DownStream device.
+ */
+#define MLXSW_REG_MDDT_ID 0x9160
+#define MLXSW_REG_MDDT_LEN 0x110
+
+MLXSW_REG_DEFINE(mddt, MLXSW_REG_MDDT_ID, MLXSW_REG_MDDT_LEN);
+
+/* reg_mddt_slot_index
+ * Slot index.
+ * Access: Index
+ */
+MLXSW_ITEM32(reg, mddt, slot_index, 0x00, 8, 4);
+
+/* reg_mddt_device_index
+ * Device index.
+ * Access: Index
+ */
+MLXSW_ITEM32(reg, mddt, device_index, 0x00, 0, 8);
+
+/* reg_mddt_read_size
+ * Read size in D-Words.
+ * Access: OP
+ */
+MLXSW_ITEM32(reg, mddt, read_size, 0x04, 24, 8);
+
+/* reg_mddt_write_size
+ * Write size in D-Words.
+ * Access: OP
+ */
+MLXSW_ITEM32(reg, mddt, write_size, 0x04, 16, 8);
+
+enum mlxsw_reg_mddt_status {
+	MLXSW_REG_MDDT_STATUS_OK,
+};
+
+/* reg_mddt_status
+ * Return code of the Downstream Device to the register that was sent.
+ * Access: RO
+ */
+MLXSW_ITEM32(reg, mddt, status, 0x0C, 24, 8);
+
+enum mlxsw_reg_mddt_method {
+	MLXSW_REG_MDDT_METHOD_QUERY,
+	MLXSW_REG_MDDT_METHOD_WRITE,
+};
+
+/* reg_mddt_method
+ * Access: OP
+ */
+MLXSW_ITEM32(reg, mddt, method, 0x0C, 22, 2);
+
+/* reg_mddt_register_id
+ * Access: Index
+ */
+MLXSW_ITEM32(reg, mddt, register_id, 0x0C, 0, 16);
+
+#define MLXSW_REG_MDDT_PAYLOAD_OFFSET 0x0C
+#define MLXSW_REG_MDDT_PRM_REGISTER_HEADER_LEN 4
+
+static inline char *mlxsw_reg_mddt_inner_payload(char *payload)
+{
+	return payload + MLXSW_REG_MDDT_PAYLOAD_OFFSET +
+	       MLXSW_REG_MDDT_PRM_REGISTER_HEADER_LEN;
+}
+
+static inline void mlxsw_reg_mddt_pack(char *payload, u8 slot_index,
+				       u8 device_index,
+				       enum mlxsw_reg_mddt_method method,
+				       const struct mlxsw_reg_info *reg,
+				       char **inner_payload)
+{
+	int len = reg->len + MLXSW_REG_MDDT_PRM_REGISTER_HEADER_LEN;
+
+	if (WARN_ON(len + MLXSW_REG_MDDT_PAYLOAD_OFFSET > MLXSW_REG_MDDT_LEN))
+		len = MLXSW_REG_MDDT_LEN - MLXSW_REG_MDDT_PAYLOAD_OFFSET;
+
+	MLXSW_REG_ZERO(mddt, payload);
+	mlxsw_reg_mddt_slot_index_set(payload, slot_index);
+	mlxsw_reg_mddt_device_index_set(payload, device_index);
+	mlxsw_reg_mddt_method_set(payload, method);
+	mlxsw_reg_mddt_register_id_set(payload, reg->id);
+	mlxsw_reg_mddt_read_size_set(payload, len / 4);
+	mlxsw_reg_mddt_write_size_set(payload, len / 4);
+	*inner_payload = mlxsw_reg_mddt_inner_payload(payload);
+}
+
 /* MDDQ - Management DownStream Device Query Register
  * --------------------------------------------------
  * This register allows to query the DownStream device properties. The desired
@@ -13208,6 +13297,7 @@ static const struct mlxsw_reg_info *mlxsw_reg_infos[] = {
 	MLXSW_REG(mfgd),
 	MLXSW_REG(mgpir),
 	MLXSW_REG(mbct),
+	MLXSW_REG(mddt),
 	MLXSW_REG(mddq),
 	MLXSW_REG(mddc),
 	MLXSW_REG(mfde),
-- 
2.35.3

[patch net-next 08/11] mlxsw: core_linecards: Expose device PSID over device info

[Jiri Pirko]

From: Jiri Pirko <jiri@nvidia.com>

Used tunneled MGIR to obtain PSID of line card device and extend
device_info_get() op to fill up the info with that.

Example:

$ devlink dev info auxiliary/mlxsw_core.lc.0
auxiliary/mlxsw_core.lc.0:
  versions:
      fixed:
        hw.revision 0
        fw.psid MT_0000000749
      running:
        ini.version 4
        fw 19.2010.1312

Signed-off-by: Jiri Pirko <jiri@nvidia.com>
---
 Documentation/networking/devlink/mlxsw.rst    |  3 ++
 drivers/net/ethernet/mellanox/mlxsw/core.h    |  1 +
 .../ethernet/mellanox/mlxsw/core_linecards.c  | 31 +++++++++++++++++++
 3 files changed, 35 insertions(+)

diff --git a/Documentation/networking/devlink/mlxsw.rst b/Documentation/networking/devlink/mlxsw.rst
index 65ceed98f94d..433962225bd4 100644
--- a/Documentation/networking/devlink/mlxsw.rst
+++ b/Documentation/networking/devlink/mlxsw.rst
@@ -75,6 +75,9 @@ The ``mlxsw`` driver reports the following versions for line card auxiliary devi
    * - ``ini.version``
      - running
      - Version of line card INI loaded
+   * - ``fw.psid``
+     - fixed
+     - Line card device PSID
    * - ``fw.version``
      - running
      - Three digit firmware version of line card device
diff --git a/drivers/net/ethernet/mellanox/mlxsw/core.h b/drivers/net/ethernet/mellanox/mlxsw/core.h
index edb0e1307d20..6b8bafc66090 100644
--- a/drivers/net/ethernet/mellanox/mlxsw/core.h
+++ b/drivers/net/ethernet/mellanox/mlxsw/core.h
@@ -574,6 +574,7 @@ struct mlxsw_linecard_device_info {
 	u16 fw_major;
 	u16 fw_minor;
 	u16 fw_sub_minor;
+	char psid[MLXSW_REG_MGIR_FW_INFO_PSID_SIZE];
 };
 
 struct mlxsw_linecard {
diff --git a/drivers/net/ethernet/mellanox/mlxsw/core_linecards.c b/drivers/net/ethernet/mellanox/mlxsw/core_linecards.c
index 0f494ea755a9..fe3154c4f92f 100644
--- a/drivers/net/ethernet/mellanox/mlxsw/core_linecards.c
+++ b/drivers/net/ethernet/mellanox/mlxsw/core_linecards.c
@@ -87,6 +87,27 @@ static const char *mlxsw_linecard_type_name(struct mlxsw_linecard *linecard)
 	return linecard->name;
 }
 
+static int mlxsw_linecard_device_psid_get(struct mlxsw_linecard *linecard,
+					  u8 device_index, char *psid)
+{
+	struct mlxsw_core *mlxsw_core = linecard->linecards->mlxsw_core;
+	char mddt_pl[MLXSW_REG_MDDT_LEN];
+	char *mgir_pl;
+	int err;
+
+	mlxsw_reg_mddt_pack(mddt_pl, linecard->slot_index, device_index,
+			    MLXSW_REG_MDDT_METHOD_QUERY,
+			    MLXSW_REG(mgir), &mgir_pl);
+
+	mlxsw_reg_mgir_pack(mgir_pl);
+	err = mlxsw_reg_query(mlxsw_core, MLXSW_REG(mddt), mddt_pl);
+	if (err)
+		return err;
+
+	mlxsw_reg_mgir_fw_info_psid_memcpy_from(mgir_pl, psid);
+	return 0;
+}
+
 static int mlxsw_linecard_device_info_update(struct mlxsw_linecard *linecard)
 {
 	struct mlxsw_core *mlxsw_core = linecard->linecards->mlxsw_core;
@@ -121,6 +142,12 @@ static int mlxsw_linecard_device_info_update(struct mlxsw_linecard *linecard)
 				      linecard->slot_index);
 			return 0;
 		}
+
+		err = mlxsw_linecard_device_psid_get(linecard, device_index,
+						     info.psid);
+		if (err)
+			return err;
+
 		linecard->device.info = info;
 		flashable_found = true;
 	} while (msg_seq);
@@ -293,6 +320,10 @@ int mlxsw_linecard_devlink_info_get(struct mlxsw_linecard *linecard,
 	if (linecard->ready) {
 		struct mlxsw_linecard_device_info *info = &linecard->device.info;
 
+		err = devlink_info_version_fixed_put(req,
+						     DEVLINK_INFO_VERSION_GENERIC_FW_PSID,
+						     info->psid);
+
 		sprintf(buf, "%u.%u.%u", info->fw_major, info->fw_minor,
 			info->fw_sub_minor);
 		err = devlink_info_version_running_put(req,
-- 
2.35.3

[patch net-next 09/11] mlxsw: core_linecards: Implement line card device flashing

[Jiri Pirko]

From: Jiri Pirko <jiri@nvidia.com>

Implement flash_update() devlink op for the line card devlink instance
to allow user to update line card gearbox FW using MDDT register
and mlxfw.

Example:
$ devlink dev flash auxiliary/mlxsw_core.lc.0 file mellanox/fw-AGB-rel-19_2010_1312-022-EVB.mfa2

Signed-off-by: Jiri Pirko <jiri@nvidia.com>
---
 drivers/net/ethernet/mellanox/mlxsw/core.c    |  31 +-
 drivers/net/ethernet/mellanox/mlxsw/core.h    |  11 +
 .../mellanox/mlxsw/core_linecard_dev.c        |  13 +
 .../ethernet/mellanox/mlxsw/core_linecards.c  | 277 ++++++++++++++++++
 4 files changed, 322 insertions(+), 10 deletions(-)

diff --git a/drivers/net/ethernet/mellanox/mlxsw/core.c b/drivers/net/ethernet/mellanox/mlxsw/core.c
index 8864533281bd..cbde437ddec0 100644
--- a/drivers/net/ethernet/mellanox/mlxsw/core.c
+++ b/drivers/net/ethernet/mellanox/mlxsw/core.c
@@ -951,6 +951,20 @@ static struct mlxsw_driver *mlxsw_core_driver_get(const char *kind)
 	return mlxsw_driver;
 }
 
+int mlxsw_core_fw_flash(struct mlxsw_core *mlxsw_core,
+			struct mlxfw_dev *mlxfw_dev,
+			const struct firmware *firmware,
+			struct netlink_ext_ack *extack)
+{
+	int err;
+
+	mlxsw_core->fw_flash_in_progress = true;
+	err = mlxfw_firmware_flash(mlxfw_dev, firmware, extack);
+	mlxsw_core->fw_flash_in_progress = false;
+
+	return err;
+}
+
 struct mlxsw_core_fw_info {
 	struct mlxfw_dev mlxfw_dev;
 	struct mlxsw_core *mlxsw_core;
@@ -1105,8 +1119,9 @@ static const struct mlxfw_dev_ops mlxsw_core_fw_mlxsw_dev_ops = {
 	.fsm_release		= mlxsw_core_fw_fsm_release,
 };
 
-static int mlxsw_core_fw_flash(struct mlxsw_core *mlxsw_core, const struct firmware *firmware,
-			       struct netlink_ext_ack *extack)
+static int mlxsw_core_dev_fw_flash(struct mlxsw_core *mlxsw_core,
+				   const struct firmware *firmware,
+				   struct netlink_ext_ack *extack)
 {
 	struct mlxsw_core_fw_info mlxsw_core_fw_info = {
 		.mlxfw_dev = {
@@ -1117,13 +1132,9 @@ static int mlxsw_core_fw_flash(struct mlxsw_core *mlxsw_core, const struct firmw
 		},
 		.mlxsw_core = mlxsw_core
 	};
-	int err;
 
-	mlxsw_core->fw_flash_in_progress = true;
-	err = mlxfw_firmware_flash(&mlxsw_core_fw_info.mlxfw_dev, firmware, extack);
-	mlxsw_core->fw_flash_in_progress = false;
-
-	return err;
+	return mlxsw_core_fw_flash(mlxsw_core, &mlxsw_core_fw_info.mlxfw_dev,
+				   firmware, extack);
 }
 
 static int mlxsw_core_fw_rev_validate(struct mlxsw_core *mlxsw_core,
@@ -1169,7 +1180,7 @@ static int mlxsw_core_fw_rev_validate(struct mlxsw_core *mlxsw_core,
 		return err;
 	}
 
-	err = mlxsw_core_fw_flash(mlxsw_core, firmware, NULL);
+	err = mlxsw_core_dev_fw_flash(mlxsw_core, firmware, NULL);
 	release_firmware(firmware);
 	if (err)
 		dev_err(mlxsw_bus_info->dev, "Could not upgrade firmware\n");
@@ -1187,7 +1198,7 @@ static int mlxsw_core_fw_flash_update(struct mlxsw_core *mlxsw_core,
 				      struct devlink_flash_update_params *params,
 				      struct netlink_ext_ack *extack)
 {
-	return mlxsw_core_fw_flash(mlxsw_core, params->fw, extack);
+	return mlxsw_core_dev_fw_flash(mlxsw_core, params->fw, extack);
 }
 
 static int mlxsw_core_devlink_param_fw_load_policy_validate(struct devlink *devlink, u32 id,
diff --git a/drivers/net/ethernet/mellanox/mlxsw/core.h b/drivers/net/ethernet/mellanox/mlxsw/core.h
index 6b8bafc66090..e518956efbbc 100644
--- a/drivers/net/ethernet/mellanox/mlxsw/core.h
+++ b/drivers/net/ethernet/mellanox/mlxsw/core.h
@@ -19,6 +19,7 @@
 #include "reg.h"
 #include "cmd.h"
 #include "resources.h"
+#include "../mlxfw/mlxfw.h"
 
 enum mlxsw_core_resource_id {
 	MLXSW_CORE_RESOURCE_PORTS = 1,
@@ -48,6 +49,11 @@ mlxsw_core_fw_rev_minor_subminor_validate(const struct mlxsw_fw_rev *rev,
 int mlxsw_core_driver_register(struct mlxsw_driver *mlxsw_driver);
 void mlxsw_core_driver_unregister(struct mlxsw_driver *mlxsw_driver);
 
+int mlxsw_core_fw_flash(struct mlxsw_core *mlxsw_core,
+			struct mlxfw_dev *mlxfw_dev,
+			const struct firmware *firmware,
+			struct netlink_ext_ack *extack);
+
 int mlxsw_core_bus_device_register(const struct mlxsw_bus_info *mlxsw_bus_info,
 				   const struct mlxsw_bus *mlxsw_bus,
 				   void *bus_priv, bool reload,
@@ -594,6 +600,7 @@ struct mlxsw_linecard {
 	struct mlxsw_linecard_bdev *bdev;
 	struct {
 		struct mlxsw_linecard_device_info info;
+		u8 index;
 	} device;
 };
 
@@ -618,6 +625,10 @@ mlxsw_linecard_get(struct mlxsw_linecards *linecards, u8 slot_index)
 int mlxsw_linecard_devlink_info_get(struct mlxsw_linecard *linecard,
 				    struct devlink_info_req *req,
 				    struct netlink_ext_ack *extack);
+int mlxsw_linecard_flash_update(struct devlink *linecard_devlink,
+				struct mlxsw_linecard *linecard,
+				const struct firmware *firmware,
+				struct netlink_ext_ack *extack);
 
 int mlxsw_linecards_init(struct mlxsw_core *mlxsw_core,
 			 const struct mlxsw_bus_info *bus_info);
diff --git a/drivers/net/ethernet/mellanox/mlxsw/core_linecard_dev.c b/drivers/net/ethernet/mellanox/mlxsw/core_linecard_dev.c
index 30659f8be41c..17496ba51081 100644
--- a/drivers/net/ethernet/mellanox/mlxsw/core_linecard_dev.c
+++ b/drivers/net/ethernet/mellanox/mlxsw/core_linecard_dev.c
@@ -104,8 +104,21 @@ static int mlxsw_linecard_dev_devlink_info_get(struct devlink *devlink,
 	return mlxsw_linecard_devlink_info_get(linecard, req, extack);
 }
 
+static int
+mlxsw_linecard_dev_devlink_flash_update(struct devlink *devlink,
+					struct devlink_flash_update_params *params,
+					struct netlink_ext_ack *extack)
+{
+	struct mlxsw_linecard_dev *linecard_dev = devlink_priv(devlink);
+	struct mlxsw_linecard *linecard = linecard_dev->linecard;
+
+	return mlxsw_linecard_flash_update(devlink, linecard,
+					   params->fw, extack);
+}
+
 static const struct devlink_ops mlxsw_linecard_dev_devlink_ops = {
 	.info_get			= mlxsw_linecard_dev_devlink_info_get,
+	.flash_update			= mlxsw_linecard_dev_devlink_flash_update,
 };
 
 static int mlxsw_linecard_bdev_probe(struct auxiliary_device *adev,
diff --git a/drivers/net/ethernet/mellanox/mlxsw/core_linecards.c b/drivers/net/ethernet/mellanox/mlxsw/core_linecards.c
index fe3154c4f92f..7d90266094b2 100644
--- a/drivers/net/ethernet/mellanox/mlxsw/core_linecards.c
+++ b/drivers/net/ethernet/mellanox/mlxsw/core_linecards.c
@@ -13,6 +13,7 @@
 #include <linux/vmalloc.h>
 
 #include "core.h"
+#include "../mlxfw/mlxfw.h"
 
 struct mlxsw_linecard_ini_file {
 	__le16 size;
@@ -87,6 +88,281 @@ static const char *mlxsw_linecard_type_name(struct mlxsw_linecard *linecard)
 	return linecard->name;
 }
 
+struct mlxsw_linecard_device_fw_info {
+	struct mlxfw_dev mlxfw_dev;
+	struct mlxsw_core *mlxsw_core;
+	struct mlxsw_linecard *linecard;
+};
+
+static int mlxsw_linecard_device_fw_component_query(struct mlxfw_dev *mlxfw_dev,
+						    u16 component_index,
+						    u32 *p_max_size,
+						    u8 *p_align_bits,
+						    u16 *p_max_write_size)
+{
+	struct mlxsw_linecard_device_fw_info *info =
+		container_of(mlxfw_dev, struct mlxsw_linecard_device_fw_info,
+			     mlxfw_dev);
+	struct mlxsw_linecard *linecard = info->linecard;
+	struct mlxsw_core *mlxsw_core = info->mlxsw_core;
+	char mddt_pl[MLXSW_REG_MDDT_LEN];
+	char *mcqi_pl;
+	int err;
+
+	mlxsw_reg_mddt_pack(mddt_pl, linecard->slot_index,
+			    linecard->device.index,
+			    MLXSW_REG_MDDT_METHOD_QUERY,
+			    MLXSW_REG(mcqi), &mcqi_pl);
+
+	mlxsw_reg_mcqi_pack(mcqi_pl, component_index);
+	err = mlxsw_reg_query(mlxsw_core, MLXSW_REG(mddt), mddt_pl);
+	if (err)
+		return err;
+	mlxsw_reg_mcqi_unpack(mcqi_pl, p_max_size, p_align_bits,
+			      p_max_write_size);
+
+	*p_align_bits = max_t(u8, *p_align_bits, 2);
+	*p_max_write_size = min_t(u16, *p_max_write_size,
+				  MLXSW_REG_MCDA_MAX_DATA_LEN);
+	return 0;
+}
+
+static int mlxsw_linecard_device_fw_fsm_lock(struct mlxfw_dev *mlxfw_dev,
+					     u32 *fwhandle)
+{
+	struct mlxsw_linecard_device_fw_info *info =
+		container_of(mlxfw_dev, struct mlxsw_linecard_device_fw_info,
+			     mlxfw_dev);
+	struct mlxsw_linecard *linecard = info->linecard;
+	struct mlxsw_core *mlxsw_core = info->mlxsw_core;
+	char mddt_pl[MLXSW_REG_MDDT_LEN];
+	u8 control_state;
+	char *mcc_pl;
+	int err;
+
+	mlxsw_reg_mddt_pack(mddt_pl, linecard->slot_index,
+			    linecard->device.index,
+			    MLXSW_REG_MDDT_METHOD_QUERY,
+			    MLXSW_REG(mcc), &mcc_pl);
+	mlxsw_reg_mcc_pack(mcc_pl, 0, 0, 0, 0);
+	err = mlxsw_reg_query(mlxsw_core, MLXSW_REG(mddt), mddt_pl);
+	if (err)
+		return err;
+
+	mlxsw_reg_mcc_unpack(mcc_pl, fwhandle, NULL, &control_state);
+	if (control_state != MLXFW_FSM_STATE_IDLE)
+		return -EBUSY;
+
+	mlxsw_reg_mddt_pack(mddt_pl, linecard->slot_index,
+			    linecard->device.index,
+			    MLXSW_REG_MDDT_METHOD_WRITE,
+			    MLXSW_REG(mcc), &mcc_pl);
+	mlxsw_reg_mcc_pack(mcc_pl, MLXSW_REG_MCC_INSTRUCTION_LOCK_UPDATE_HANDLE,
+			   0, *fwhandle, 0);
+	return mlxsw_reg_write(mlxsw_core, MLXSW_REG(mddt), mddt_pl);
+}
+
+static int
+mlxsw_linecard_device_fw_fsm_component_update(struct mlxfw_dev *mlxfw_dev,
+					      u32 fwhandle,
+					      u16 component_index,
+					      u32 component_size)
+{
+	struct mlxsw_linecard_device_fw_info *info =
+		container_of(mlxfw_dev, struct mlxsw_linecard_device_fw_info,
+			     mlxfw_dev);
+	struct mlxsw_linecard *linecard = info->linecard;
+	struct mlxsw_core *mlxsw_core = info->mlxsw_core;
+	char mddt_pl[MLXSW_REG_MDDT_LEN];
+	char *mcc_pl;
+
+	mlxsw_reg_mddt_pack(mddt_pl, linecard->slot_index,
+			    linecard->device.index,
+			    MLXSW_REG_MDDT_METHOD_WRITE,
+			    MLXSW_REG(mcc), &mcc_pl);
+	mlxsw_reg_mcc_pack(mcc_pl, MLXSW_REG_MCC_INSTRUCTION_UPDATE_COMPONENT,
+			   component_index, fwhandle, component_size);
+	return mlxsw_reg_write(mlxsw_core, MLXSW_REG(mddt), mddt_pl);
+}
+
+static int
+mlxsw_linecard_device_fw_fsm_block_download(struct mlxfw_dev *mlxfw_dev,
+					    u32 fwhandle, u8 *data,
+					    u16 size, u32 offset)
+{
+	struct mlxsw_linecard_device_fw_info *info =
+		container_of(mlxfw_dev, struct mlxsw_linecard_device_fw_info,
+			     mlxfw_dev);
+	struct mlxsw_linecard *linecard = info->linecard;
+	struct mlxsw_core *mlxsw_core = info->mlxsw_core;
+	char mddt_pl[MLXSW_REG_MDDT_LEN];
+	char *mcda_pl;
+
+	mlxsw_reg_mddt_pack(mddt_pl, linecard->slot_index,
+			    linecard->device.index,
+			    MLXSW_REG_MDDT_METHOD_WRITE,
+			    MLXSW_REG(mcda), &mcda_pl);
+	mlxsw_reg_mcda_pack(mcda_pl, fwhandle, offset, size, data);
+	return mlxsw_reg_write(mlxsw_core, MLXSW_REG(mddt), mddt_pl);
+}
+
+static int
+mlxsw_linecard_device_fw_fsm_component_verify(struct mlxfw_dev *mlxfw_dev,
+					      u32 fwhandle, u16 component_index)
+{
+	struct mlxsw_linecard_device_fw_info *info =
+		container_of(mlxfw_dev, struct mlxsw_linecard_device_fw_info,
+			     mlxfw_dev);
+	struct mlxsw_linecard *linecard = info->linecard;
+	struct mlxsw_core *mlxsw_core = info->mlxsw_core;
+	char mddt_pl[MLXSW_REG_MDDT_LEN];
+	char *mcc_pl;
+
+	mlxsw_reg_mddt_pack(mddt_pl, linecard->slot_index,
+			    linecard->device.index,
+			    MLXSW_REG_MDDT_METHOD_WRITE,
+			    MLXSW_REG(mcc), &mcc_pl);
+	mlxsw_reg_mcc_pack(mcc_pl, MLXSW_REG_MCC_INSTRUCTION_VERIFY_COMPONENT,
+			   component_index, fwhandle, 0);
+	return mlxsw_reg_write(mlxsw_core, MLXSW_REG(mddt), mddt_pl);
+}
+
+static int mlxsw_linecard_device_fw_fsm_activate(struct mlxfw_dev *mlxfw_dev,
+						 u32 fwhandle)
+{
+	struct mlxsw_linecard_device_fw_info *info =
+		container_of(mlxfw_dev, struct mlxsw_linecard_device_fw_info,
+			     mlxfw_dev);
+	struct mlxsw_linecard *linecard = info->linecard;
+	struct mlxsw_core *mlxsw_core = info->mlxsw_core;
+	char mddt_pl[MLXSW_REG_MDDT_LEN];
+	char *mcc_pl;
+
+	mlxsw_reg_mddt_pack(mddt_pl, linecard->slot_index,
+			    linecard->device.index,
+			    MLXSW_REG_MDDT_METHOD_WRITE,
+			    MLXSW_REG(mcc), &mcc_pl);
+	mlxsw_reg_mcc_pack(mcc_pl, MLXSW_REG_MCC_INSTRUCTION_ACTIVATE,
+			   0, fwhandle, 0);
+	return mlxsw_reg_write(mlxsw_core, MLXSW_REG(mddt), mddt_pl);
+}
+
+static int
+mlxsw_linecard_device_fw_fsm_query_state(struct mlxfw_dev *mlxfw_dev,
+					 u32 fwhandle,
+					 enum mlxfw_fsm_state *fsm_state,
+					 enum mlxfw_fsm_state_err *fsm_state_err)
+{
+	struct mlxsw_linecard_device_fw_info *info =
+		container_of(mlxfw_dev, struct mlxsw_linecard_device_fw_info,
+			     mlxfw_dev);
+	struct mlxsw_linecard *linecard = info->linecard;
+	struct mlxsw_core *mlxsw_core = info->mlxsw_core;
+	char mddt_pl[MLXSW_REG_MDDT_LEN];
+	u8 control_state;
+	u8 error_code;
+	char *mcc_pl;
+	int err;
+
+	mlxsw_reg_mddt_pack(mddt_pl, linecard->slot_index,
+			    linecard->device.index,
+			    MLXSW_REG_MDDT_METHOD_QUERY,
+			    MLXSW_REG(mcc), &mcc_pl);
+	mlxsw_reg_mcc_pack(mcc_pl, 0, 0, fwhandle, 0);
+	err = mlxsw_reg_query(mlxsw_core, MLXSW_REG(mddt), mddt_pl);
+	if (err)
+		return err;
+
+	mlxsw_reg_mcc_unpack(mcc_pl, NULL, &error_code, &control_state);
+	*fsm_state = control_state;
+	*fsm_state_err = min_t(enum mlxfw_fsm_state_err, error_code,
+			       MLXFW_FSM_STATE_ERR_MAX);
+	return 0;
+}
+
+static void mlxsw_linecard_device_fw_fsm_cancel(struct mlxfw_dev *mlxfw_dev,
+						u32 fwhandle)
+{
+	struct mlxsw_linecard_device_fw_info *info =
+		container_of(mlxfw_dev, struct mlxsw_linecard_device_fw_info,
+			     mlxfw_dev);
+	struct mlxsw_linecard *linecard = info->linecard;
+	struct mlxsw_core *mlxsw_core = info->mlxsw_core;
+	char mddt_pl[MLXSW_REG_MDDT_LEN];
+	char *mcc_pl;
+
+	mlxsw_reg_mddt_pack(mddt_pl, linecard->slot_index,
+			    linecard->device.index,
+			    MLXSW_REG_MDDT_METHOD_WRITE,
+			    MLXSW_REG(mcc), &mcc_pl);
+	mlxsw_reg_mcc_pack(mcc_pl, MLXSW_REG_MCC_INSTRUCTION_CANCEL,
+			   0, fwhandle, 0);
+	mlxsw_reg_write(mlxsw_core, MLXSW_REG(mddt), mddt_pl);
+}
+
+static void mlxsw_linecard_device_fw_fsm_release(struct mlxfw_dev *mlxfw_dev,
+						 u32 fwhandle)
+{
+	struct mlxsw_linecard_device_fw_info *info =
+		container_of(mlxfw_dev, struct mlxsw_linecard_device_fw_info,
+			     mlxfw_dev);
+	struct mlxsw_linecard *linecard = info->linecard;
+	struct mlxsw_core *mlxsw_core = info->mlxsw_core;
+	char mddt_pl[MLXSW_REG_MDDT_LEN];
+	char *mcc_pl;
+
+	mlxsw_reg_mddt_pack(mddt_pl, linecard->slot_index,
+			    linecard->device.index,
+			    MLXSW_REG_MDDT_METHOD_WRITE,
+			    MLXSW_REG(mcc), &mcc_pl);
+	mlxsw_reg_mcc_pack(mcc_pl,
+			   MLXSW_REG_MCC_INSTRUCTION_RELEASE_UPDATE_HANDLE,
+			   0, fwhandle, 0);
+	mlxsw_reg_write(mlxsw_core, MLXSW_REG(mddt), mddt_pl);
+}
+
+static const struct mlxfw_dev_ops mlxsw_linecard_device_dev_ops = {
+	.component_query	= mlxsw_linecard_device_fw_component_query,
+	.fsm_lock		= mlxsw_linecard_device_fw_fsm_lock,
+	.fsm_component_update	= mlxsw_linecard_device_fw_fsm_component_update,
+	.fsm_block_download	= mlxsw_linecard_device_fw_fsm_block_download,
+	.fsm_component_verify	= mlxsw_linecard_device_fw_fsm_component_verify,
+	.fsm_activate		= mlxsw_linecard_device_fw_fsm_activate,
+	.fsm_query_state	= mlxsw_linecard_device_fw_fsm_query_state,
+	.fsm_cancel		= mlxsw_linecard_device_fw_fsm_cancel,
+	.fsm_release		= mlxsw_linecard_device_fw_fsm_release,
+};
+
+int mlxsw_linecard_flash_update(struct devlink *linecard_devlink,
+				struct mlxsw_linecard *linecard,
+				const struct firmware *firmware,
+				struct netlink_ext_ack *extack)
+{
+	struct mlxsw_core *mlxsw_core = linecard->linecards->mlxsw_core;
+	struct mlxsw_linecard_device_fw_info info = {
+		.mlxfw_dev = {
+			.ops = &mlxsw_linecard_device_dev_ops,
+			.psid = linecard->device.info.psid,
+			.psid_size = strlen(linecard->device.info.psid),
+			.devlink = linecard_devlink,
+		},
+		.mlxsw_core = mlxsw_core,
+		.linecard = linecard,
+	};
+	int err;
+
+	mutex_lock(&linecard->lock);
+	if (WARN_ON(!linecard->ready)) {
+		err = -EINVAL;
+		goto unlock;
+	}
+	err = mlxsw_core_fw_flash(mlxsw_core, &info.mlxfw_dev,
+				  firmware, extack);
+unlock:
+	mutex_unlock(&linecard->lock);
+	return err;
+}
+
 static int mlxsw_linecard_device_psid_get(struct mlxsw_linecard *linecard,
 					  u8 device_index, char *psid)
 {
@@ -149,6 +425,7 @@ static int mlxsw_linecard_device_info_update(struct mlxsw_linecard *linecard)
 			return err;
 
 		linecard->device.info = info;
+		linecard->device.index = device_index;
 		flashable_found = true;
 	} while (msg_seq);
 
-- 
2.35.3

[patch net-next 10/11] selftests: mlxsw: Check line card info on provisioned line card

[Jiri Pirko]

From: Jiri Pirko <jiri@nvidia.com>

Once line card is provisioned, check if HW revision and INI version
are exposed on associated nested auxiliary device.

Signed-off-by: Jiri Pirko <jiri@nvidia.com>
---
 .../drivers/net/mlxsw/devlink_linecard.sh     | 30 +++++++++++++++++++
 1 file changed, 30 insertions(+)

diff --git a/tools/testing/selftests/drivers/net/mlxsw/devlink_linecard.sh b/tools/testing/selftests/drivers/net/mlxsw/devlink_linecard.sh
index 08a922d8b86a..ca4e9b08a105 100755
--- a/tools/testing/selftests/drivers/net/mlxsw/devlink_linecard.sh
+++ b/tools/testing/selftests/drivers/net/mlxsw/devlink_linecard.sh
@@ -84,6 +84,13 @@ lc_wait_until_port_count_is()
 	busywait "$timeout" until_lc_port_count_is "$port_count" lc_port_count_get "$lc"
 }
 
+lc_nested_devlink_dev_get()
+{
+	local lc=$1
+
+	devlink lc show $DEVLINK_DEV lc $lc -j | jq -e -r ".[][][].nested_devlink"
+}
+
 PROV_UNPROV_TIMEOUT=8000 # ms
 POST_PROV_ACT_TIMEOUT=2000 # ms
 PROV_PORTS_INSTANTIATION_TIMEOUT=15000 # ms
@@ -191,12 +198,30 @@ ports_check()
 	check_err $? "Unexpected port count linecard $lc (got $port_count, expected $expected_port_count)"
 }
 
+lc_dev_info_provisioned_check()
+{
+	local lc=$1
+	local nested_devlink_dev=$2
+	local fixed_hw_revision
+	local running_ini_version
+
+	fixed_hw_revision=$(devlink dev info $nested_devlink_dev -j | \
+			    jq -e -r '.[][].versions.fixed."hw.revision"')
+	check_err $? "Failed to get linecard $lc fixed.hw.revision"
+	log_info "Linecard $lc fixed.hw.revision: \"$fixed_hw_revision\""
+	running_ini_version=$(devlink dev info $nested_devlink_dev -j | \
+			      jq -e -r '.[][].versions.running."ini.version"')
+	check_err $? "Failed to get linecard $lc running.ini.version"
+	log_info "Linecard $lc running.ini.version: \"$running_ini_version\""
+}
+
 provision_test()
 {
 	RET=0
 	local lc
 	local type
 	local state
+	local nested_devlink_dev
 
 	lc=$LC_SLOT
 	supported_types_check $lc
@@ -207,6 +232,11 @@ provision_test()
 	fi
 	provision_one $lc $LC_16X100G_TYPE
 	ports_check $lc $LC_16X100G_PORT_COUNT
+
+	nested_devlink_dev=$(lc_nested_devlink_dev_get $lc)
+	check_err $? "Failed to get nested devlink handle of linecard $lc"
+	lc_dev_info_provisioned_check $lc $nested_devlink_dev
+
 	log_test "Provision"
 }
 
-- 
2.35.3

[patch net-next 11/11] selftests: mlxsw: Check line card info on activated line card

[Jiri Pirko]

From: Jiri Pirko <jiri@nvidia.com>

Once line card is activated, check the FW version and PSID are exposed.

Signed-off-by: Jiri Pirko <jiri@nvidia.com>
---
 .../drivers/net/mlxsw/devlink_linecard.sh     | 24 +++++++++++++++++++
 1 file changed, 24 insertions(+)

diff --git a/tools/testing/selftests/drivers/net/mlxsw/devlink_linecard.sh b/tools/testing/selftests/drivers/net/mlxsw/devlink_linecard.sh
index ca4e9b08a105..224ca3695c89 100755
--- a/tools/testing/selftests/drivers/net/mlxsw/devlink_linecard.sh
+++ b/tools/testing/selftests/drivers/net/mlxsw/devlink_linecard.sh
@@ -250,12 +250,32 @@ interface_check()
 	setup_wait
 }
 
+lc_dev_info_active_check()
+{
+	local lc=$1
+	local nested_devlink_dev=$2
+	local fixed_device_fw_psid
+	local running_device_fw
+
+	fixed_device_fw_psid=$(devlink dev info $nested_devlink_dev -j | \
+			       jq -e -r ".[][].versions.fixed" | \
+			       jq -e -r '."fw.psid"')
+	check_err $? "Failed to get linecard $lc fixed fw PSID"
+	log_info "Linecard $lc fixed.fw.psid: \"$fixed_device_fw_psid\""
+
+	running_device_fw=$(devlink dev info $nested_devlink_dev -j | \
+			    jq -e -r ".[][].versions.running.fw")
+	check_err $? "Failed to get linecard $lc running.fw.version"
+	log_info "Linecard $lc running.fw: \"$running_device_fw\""
+}
+
 activation_16x100G_test()
 {
 	RET=0
 	local lc
 	local type
 	local state
+	local nested_devlink_dev
 
 	lc=$LC_SLOT
 	type=$LC_16X100G_TYPE
@@ -268,6 +288,10 @@ activation_16x100G_test()
 
 	interface_check
 
+	nested_devlink_dev=$(lc_nested_devlink_dev_get $lc)
+	check_err $? "Failed to get nested devlink handle of linecard $lc"
+	lc_dev_info_active_check $lc $nested_devlink_dev
+
 	log_test "Activation 16x100G"
 }
 
-- 
2.35.3

Re: [patch net-next 00/11] mlxsw: Implement dev info and dev flash for line cards

[Ido Schimmel]

On Tue, Jun 14, 2022 at 02:33:15PM +0200, Jiri Pirko wrote:
> From: Jiri Pirko <jiri@nvidia.com>
> 
> This patchset implements two features:
> 1) "devlink dev info" is exposed for line card (patches 3-8)
> 2) "devlink dev flash" is implemented for line card gearbox
>    flashing (patch 9)
> 
> For every line card, "a nested" auxiliary device is created which
> allows to bind the features mentioned above (patch 2).

The design choice to use an auxiliary device for each line card needs to
be explained in the cover letter. From what I can tell, the motivation
is to reuse the above devlink uAPI for line cards as opposed to using
the "component" attribute or adding new uAPI. This is achieved by
creating a devlink instance for each line card. The auxiliary device is
needed because each devlink instance is expected to be associated with a
device. Does this constitute proper use of the auxiliary bus?

> 
> The relationship between line card and its auxiliary dev devlink
> is carried over extra line card netlink attribute (patches 1 and 3).
> 
> Examples:
> 
> $ devlink lc show pci/0000:01:00.0 lc 1
> pci/0000:01:00.0:
>   lc 1 state active type 16x100G nested_devlink auxiliary/mlxsw_core.lc.0

Can we try to use the index of the line card as the identifier of the
auxiliary device?

>     supported_types:
>        16x100G
> 
> $ devlink dev show auxiliary/mlxsw_core.lc.0
> auxiliary/mlxsw_core.lc.0

I assume that the auxiliary device cannot outlive line card. Does that
mean that as part of reload of the primary devlink instance the nested
devlink instance is removed? If so, did you check the reload flow with
lockdep to ensure there aren't any problems?

> 
> $ devlink dev info auxiliary/mlxsw_core.lc.0
> auxiliary/mlxsw_core.lc.0:
>   versions:
>       fixed:
>         hw.revision 0
>         fw.psid MT_0000000749
>       running:
>         ini.version 4
>         fw 19.2010.1312
> 
> $ devlink dev flash auxiliary/mlxsw_core.lc.0 file mellanox/fw-AGB-rel-19_2010_1312-022-EVB.mfa2

How is this firmware activated? It is usually done after reload, but I
don't see reload implementation for the line card devlink instance.

> 
> Jiri Pirko (11):
>   devlink: introduce nested devlink entity for line card
>   mlxsw: core_linecards: Introduce per line card auxiliary device
>   mlxsw: core_linecard_dev: Set nested devlink relationship for a line
>     card
>   mlxsw: core_linecards: Expose HW revision and INI version
>   mlxsw: reg: Extend MDDQ by device_info
>   mlxsw: core_linecards: Probe provisioned line cards for devices and
>     expose FW version
>   mlxsw: reg: Add Management DownStream Device Tunneling Register
>   mlxsw: core_linecards: Expose device PSID over device info
>   mlxsw: core_linecards: Implement line card device flashing
>   selftests: mlxsw: Check line card info on provisioned line card
>   selftests: mlxsw: Check line card info on activated line card
> 
>  Documentation/networking/devlink/mlxsw.rst    |  24 ++
>  drivers/net/ethernet/mellanox/mlxsw/Kconfig   |   1 +
>  drivers/net/ethernet/mellanox/mlxsw/Makefile  |   2 +-
>  drivers/net/ethernet/mellanox/mlxsw/core.c    |  44 +-
>  drivers/net/ethernet/mellanox/mlxsw/core.h    |  35 ++
>  .../mellanox/mlxsw/core_linecard_dev.c        | 180 ++++++++
>  .../ethernet/mellanox/mlxsw/core_linecards.c  | 403 ++++++++++++++++++
>  drivers/net/ethernet/mellanox/mlxsw/reg.h     | 173 +++++++-
>  include/net/devlink.h                         |   2 +
>  include/uapi/linux/devlink.h                  |   2 +
>  net/core/devlink.c                            |  42 ++
>  .../drivers/net/mlxsw/devlink_linecard.sh     |  54 +++
>  12 files changed, 948 insertions(+), 14 deletions(-)
>  create mode 100644 drivers/net/ethernet/mellanox/mlxsw/core_linecard_dev.c
> 
> -- 
> 2.35.3
> 

Re: [patch net-next 00/11] mlxsw: Implement dev info and dev flash for line cards

[Jiri Pirko]

Wed, Jun 15, 2022 at 11:13:35AM CEST, idosch@nvidia.com wrote:
>On Tue, Jun 14, 2022 at 02:33:15PM +0200, Jiri Pirko wrote:
>> From: Jiri Pirko <jiri@nvidia.com>
>> 
>> This patchset implements two features:
>> 1) "devlink dev info" is exposed for line card (patches 3-8)
>> 2) "devlink dev flash" is implemented for line card gearbox
>>    flashing (patch 9)
>> 
>> For every line card, "a nested" auxiliary device is created which
>> allows to bind the features mentioned above (patch 2).
>
>The design choice to use an auxiliary device for each line card needs to
>be explained in the cover letter. From what I can tell, the motivation
>is to reuse the above devlink uAPI for line cards as opposed to using
>the "component" attribute or adding new uAPI. This is achieved by
>creating a devlink instance for each line card. The auxiliary device is
>needed because each devlink instance is expected to be associated with a
>device. Does this constitute proper use of the auxiliary bus?

Right, will describe this in cover letter.


>
>> 
>> The relationship between line card and its auxiliary dev devlink
>> is carried over extra line card netlink attribute (patches 1 and 3).
>> 
>> Examples:
>> 
>> $ devlink lc show pci/0000:01:00.0 lc 1
>> pci/0000:01:00.0:
>>   lc 1 state active type 16x100G nested_devlink auxiliary/mlxsw_core.lc.0
>
>Can we try to use the index of the line card as the identifier of the
>auxiliary device?

Not really. We would have a collision if there are 2 mlxsw instances.


>
>>     supported_types:
>>        16x100G
>> 
>> $ devlink dev show auxiliary/mlxsw_core.lc.0
>> auxiliary/mlxsw_core.lc.0
>
>I assume that the auxiliary device cannot outlive line card. Does that
>mean that as part of reload of the primary devlink instance the nested
>devlink instance is removed? If so, did you check the reload flow with
>lockdep to ensure there aren't any problems?

As I wrote in the other email, line card auxdev should be removed during
linecard_fini(), will add that to v2.


>
>> 
>> $ devlink dev info auxiliary/mlxsw_core.lc.0
>> auxiliary/mlxsw_core.lc.0:
>>   versions:
>>       fixed:
>>         hw.revision 0
>>         fw.psid MT_0000000749
>>       running:
>>         ini.version 4
>>         fw 19.2010.1312
>> 
>> $ devlink dev flash auxiliary/mlxsw_core.lc.0 file mellanox/fw-AGB-rel-19_2010_1312-022-EVB.mfa2
>
>How is this firmware activated? It is usually done after reload, but I
>don't see reload implementation for the line card devlink instance.

Currently, only devlink dev reload of the whole mlxsw instance or
unprovision/provision of a line card.

>
>> 
>> Jiri Pirko (11):
>>   devlink: introduce nested devlink entity for line card
>>   mlxsw: core_linecards: Introduce per line card auxiliary device
>>   mlxsw: core_linecard_dev: Set nested devlink relationship for a line
>>     card
>>   mlxsw: core_linecards: Expose HW revision and INI version
>>   mlxsw: reg: Extend MDDQ by device_info
>>   mlxsw: core_linecards: Probe provisioned line cards for devices and
>>     expose FW version
>>   mlxsw: reg: Add Management DownStream Device Tunneling Register
>>   mlxsw: core_linecards: Expose device PSID over device info
>>   mlxsw: core_linecards: Implement line card device flashing
>>   selftests: mlxsw: Check line card info on provisioned line card
>>   selftests: mlxsw: Check line card info on activated line card
>> 
>>  Documentation/networking/devlink/mlxsw.rst    |  24 ++
>>  drivers/net/ethernet/mellanox/mlxsw/Kconfig   |   1 +
>>  drivers/net/ethernet/mellanox/mlxsw/Makefile  |   2 +-
>>  drivers/net/ethernet/mellanox/mlxsw/core.c    |  44 +-
>>  drivers/net/ethernet/mellanox/mlxsw/core.h    |  35 ++
>>  .../mellanox/mlxsw/core_linecard_dev.c        | 180 ++++++++
>>  .../ethernet/mellanox/mlxsw/core_linecards.c  | 403 ++++++++++++++++++
>>  drivers/net/ethernet/mellanox/mlxsw/reg.h     | 173 +++++++-
>>  include/net/devlink.h                         |   2 +
>>  include/uapi/linux/devlink.h                  |   2 +
>>  net/core/devlink.c                            |  42 ++
>>  .../drivers/net/mlxsw/devlink_linecard.sh     |  54 +++
>>  12 files changed, 948 insertions(+), 14 deletions(-)
>>  create mode 100644 drivers/net/ethernet/mellanox/mlxsw/core_linecard_dev.c
>> 
>> -- 
>> 2.35.3
>> 

Re: [patch net-next 00/11] mlxsw: Implement dev info and dev flash for line cards

[Ido Schimmel]

On Wed, Jun 15, 2022 at 07:40:34PM +0200, Jiri Pirko wrote:
> Wed, Jun 15, 2022 at 11:13:35AM CEST, idosch@nvidia.com wrote:
> >On Tue, Jun 14, 2022 at 02:33:15PM +0200, Jiri Pirko wrote:
> >> $ devlink dev flash auxiliary/mlxsw_core.lc.0 file mellanox/fw-AGB-rel-19_2010_1312-022-EVB.mfa2
> >
> >How is this firmware activated? It is usually done after reload, but I
> >don't see reload implementation for the line card devlink instance.
> 
> Currently, only devlink dev reload of the whole mlxsw instance or
> unprovision/provision of a line card.

OK, please at least mention it in the commit message that adds flashing
support.

What about implementing reload support as unprovision/provision?

Re: [patch net-next 00/11] mlxsw: Implement dev info and dev flash for line cards

[Jiri Pirko]

Thu, Jun 16, 2022 at 09:03:52AM CEST, idosch@nvidia.com wrote:
>On Wed, Jun 15, 2022 at 07:40:34PM +0200, Jiri Pirko wrote:
>> Wed, Jun 15, 2022 at 11:13:35AM CEST, idosch@nvidia.com wrote:
>> >On Tue, Jun 14, 2022 at 02:33:15PM +0200, Jiri Pirko wrote:
>> >> $ devlink dev flash auxiliary/mlxsw_core.lc.0 file mellanox/fw-AGB-rel-19_2010_1312-022-EVB.mfa2
>> >
>> >How is this firmware activated? It is usually done after reload, but I
>> >don't see reload implementation for the line card devlink instance.
>> 
>> Currently, only devlink dev reload of the whole mlxsw instance or
>> unprovision/provision of a line card.
>
>OK, please at least mention it in the commit message that adds flashing
>support.
>
>What about implementing reload support as unprovision/provision?

Yes, that can be done eventually. I was thinking about that as well.

Re: [patch net-next 00/11] mlxsw: Implement dev info and dev flash for line cards

[Ido Schimmel]

On Thu, Jun 16, 2022 at 03:11:57PM +0200, Jiri Pirko wrote:
> Thu, Jun 16, 2022 at 09:03:52AM CEST, idosch@nvidia.com wrote:
> >On Wed, Jun 15, 2022 at 07:40:34PM +0200, Jiri Pirko wrote:
> >> Wed, Jun 15, 2022 at 11:13:35AM CEST, idosch@nvidia.com wrote:
> >> >On Tue, Jun 14, 2022 at 02:33:15PM +0200, Jiri Pirko wrote:
> >> >> $ devlink dev flash auxiliary/mlxsw_core.lc.0 file mellanox/fw-AGB-rel-19_2010_1312-022-EVB.mfa2
> >> >
> >> >How is this firmware activated? It is usually done after reload, but I
> >> >don't see reload implementation for the line card devlink instance.
> >> 
> >> Currently, only devlink dev reload of the whole mlxsw instance or
> >> unprovision/provision of a line card.
> >
> >OK, please at least mention it in the commit message that adds flashing
> >support.
> >
> >What about implementing reload support as unprovision/provision?
> 
> Yes, that can be done eventually. I was thinking about that as well.

This patch should come before the one that adds flashing. Then both the
primary and nested devlink instances maintain the same semantics with
regards to firmware flashing / activation.

Re: [patch net-next 00/11] mlxsw: Implement dev info and dev flash for line cards

[Jiri Pirko]

Thu, Jun 16, 2022 at 04:47:50PM CEST, idosch@nvidia.com wrote:
>On Thu, Jun 16, 2022 at 03:11:57PM +0200, Jiri Pirko wrote:
>> Thu, Jun 16, 2022 at 09:03:52AM CEST, idosch@nvidia.com wrote:
>> >On Wed, Jun 15, 2022 at 07:40:34PM +0200, Jiri Pirko wrote:
>> >> Wed, Jun 15, 2022 at 11:13:35AM CEST, idosch@nvidia.com wrote:
>> >> >On Tue, Jun 14, 2022 at 02:33:15PM +0200, Jiri Pirko wrote:
>> >> >> $ devlink dev flash auxiliary/mlxsw_core.lc.0 file mellanox/fw-AGB-rel-19_2010_1312-022-EVB.mfa2
>> >> >
>> >> >How is this firmware activated? It is usually done after reload, but I
>> >> >don't see reload implementation for the line card devlink instance.
>> >> 
>> >> Currently, only devlink dev reload of the whole mlxsw instance or
>> >> unprovision/provision of a line card.
>> >
>> >OK, please at least mention it in the commit message that adds flashing
>> >support.
>> >
>> >What about implementing reload support as unprovision/provision?
>> 
>> Yes, that can be done eventually. I was thinking about that as well.
>
>This patch should come before the one that adds flashing. Then both the
>primary and nested devlink instances maintain the same semantics with
>regards to firmware flashing / activation.

Ok.

Re: [patch net-next 00/11] mlxsw: Implement dev info and dev flash for line cards

[Shannon Nelson]

On 6/15/22 10:40 AM, Jiri Pirko wrote:
> Wed, Jun 15, 2022 at 11:13:35AM CEST, idosch@nvidia.com wrote:
>> On Tue, Jun 14, 2022 at 02:33:15PM +0200, Jiri Pirko wrote:
>>> From: Jiri Pirko <jiri@nvidia.com>
>>>
>>> This patchset implements two features:
>>> 1) "devlink dev info" is exposed for line card (patches 3-8)
>>> 2) "devlink dev flash" is implemented for line card gearbox
>>>     flashing (patch 9)
>>>
>>> For every line card, "a nested" auxiliary device is created which
>>> allows to bind the features mentioned above (patch 2).
>>
[...]>>
>>>
>>> The relationship between line card and its auxiliary dev devlink
>>> is carried over extra line card netlink attribute (patches 1 and 3).
>>>
>>> Examples:
>>>
>>> $ devlink lc show pci/0000:01:00.0 lc 1
>>> pci/0000:01:00.0:
>>>    lc 1 state active type 16x100G nested_devlink auxiliary/mlxsw_core.lc.0
>>
>> Can we try to use the index of the line card as the identifier of the
>> auxiliary device?
> 
> Not really. We would have a collision if there are 2 mlxsw instances.
> 

Can you encode the base device's PCI info into the auxiliary device's id
to make it unique?  Or maybe have each mlxsw instance have a unique ida 
value to encode in the linecard auxiliary device id?

sln

Re: [patch net-next 00/11] mlxsw: Implement dev info and dev flash for line cards

[Jiri Pirko]

Sat, Jun 18, 2022 at 08:12:20AM CEST, snelson@pensando.io wrote:
>
>
>On 6/15/22 10:40 AM, Jiri Pirko wrote:
>> Wed, Jun 15, 2022 at 11:13:35AM CEST, idosch@nvidia.com wrote:
>> > On Tue, Jun 14, 2022 at 02:33:15PM +0200, Jiri Pirko wrote:
>> > > From: Jiri Pirko <jiri@nvidia.com>
>> > > 
>> > > This patchset implements two features:
>> > > 1) "devlink dev info" is exposed for line card (patches 3-8)
>> > > 2) "devlink dev flash" is implemented for line card gearbox
>> > >     flashing (patch 9)
>> > > 
>> > > For every line card, "a nested" auxiliary device is created which
>> > > allows to bind the features mentioned above (patch 2).
>> > 
>[...]>>
>> > > 
>> > > The relationship between line card and its auxiliary dev devlink
>> > > is carried over extra line card netlink attribute (patches 1 and 3).
>> > > 
>> > > Examples:
>> > > 
>> > > $ devlink lc show pci/0000:01:00.0 lc 1
>> > > pci/0000:01:00.0:
>> > >    lc 1 state active type 16x100G nested_devlink auxiliary/mlxsw_core.lc.0
>> > 
>> > Can we try to use the index of the line card as the identifier of the
>> > auxiliary device?
>> 
>> Not really. We would have a collision if there are 2 mlxsw instances.
>> 
>
>Can you encode the base device's PCI info into the auxiliary device's id

Would look odd to he PCI BDF in auxdev addsess, wouldn't it?


>to make it unique?  Or maybe have each mlxsw instance have a unique ida value
>to encode in the linecard auxiliary device id?

Well, which value would that bring? It would be dynamic random number.
How the use would use that tho figure out the relation to the mlxsw
instance?

>
>sln

Re: [patch net-next 00/11] mlxsw: Implement dev info and dev flash for line cards

[Shannon Nelson]

On 6/27/22 1:43 AM, Jiri Pirko wrote:
> Sat, Jun 18, 2022 at 08:12:20AM CEST, snelson@pensando.io wrote:
>>
>>
>> On 6/15/22 10:40 AM, Jiri Pirko wrote:
>>> Wed, Jun 15, 2022 at 11:13:35AM CEST, idosch@nvidia.com wrote:
>>>> On Tue, Jun 14, 2022 at 02:33:15PM +0200, Jiri Pirko wrote:
>>>>> From: Jiri Pirko <jiri@nvidia.com>
>>>>>
>>>>> This patchset implements two features:
>>>>> 1) "devlink dev info" is exposed for line card (patches 3-8)
>>>>> 2) "devlink dev flash" is implemented for line card gearbox
>>>>>      flashing (patch 9)
>>>>>
>>>>> For every line card, "a nested" auxiliary device is created which
>>>>> allows to bind the features mentioned above (patch 2).
>>>>
>> [...]>>
>>>>>
>>>>> The relationship between line card and its auxiliary dev devlink
>>>>> is carried over extra line card netlink attribute (patches 1 and 3).
>>>>>
>>>>> Examples:
>>>>>
>>>>> $ devlink lc show pci/0000:01:00.0 lc 1
>>>>> pci/0000:01:00.0:
>>>>>     lc 1 state active type 16x100G nested_devlink auxiliary/mlxsw_core.lc.0
>>>>
>>>> Can we try to use the index of the line card as the identifier of the
>>>> auxiliary device?
>>>
>>> Not really. We would have a collision if there are 2 mlxsw instances.
>>>
>>
>> Can you encode the base device's PCI info into the auxiliary device's id
> 
> Would look odd to he PCI BDF in auxdev addsess, wouldn't it?

Sure, it looks a little odd to see something like mycore.app.1281, but 
it does afford the auxiliary driver, and any other observer, a way to 
figure out which device it is representing.  This also works nicely when 
trying to associate an auxiliary driver instance for a VF with the 
matching VF PCI driver instance.

sln

Re: [patch net-next 00/11] mlxsw: Implement dev info and dev flash for line cards

[Jakub Kicinski]

On Mon, 27 Jun 2022 11:38:50 -0700 Shannon Nelson wrote:
> >> Can you encode the base device's PCI info into the auxiliary device's id  
> > 
> > Would look odd to he PCI BDF in auxdev addsess, wouldn't it?  
> 
> Sure, it looks a little odd to see something like mycore.app.1281, but 
> it does afford the auxiliary driver, and any other observer, a way to 
> figure out which device it is representing.  This also works nicely when 
> trying to associate an auxiliary driver instance for a VF with the 
> matching VF PCI driver instance.

I'd personally not mind divorcing devlink from bus devices a little
more. On one hand we have cases like this where there's naturally no
bus device, on the other we have multi-link PCI devices which want to
straddle NUMA nodes but otherwise are just a logical unit.

Re: [patch net-next 00/11] mlxsw: Implement dev info and dev flash for line cards

[Jiri Pirko]

Mon, Jun 27, 2022 at 08:52:09PM CEST, kuba@kernel.org wrote:
>On Mon, 27 Jun 2022 11:38:50 -0700 Shannon Nelson wrote:
>> >> Can you encode the base device's PCI info into the auxiliary device's id  
>> > 
>> > Would look odd to he PCI BDF in auxdev addsess, wouldn't it?  
>> 
>> Sure, it looks a little odd to see something like mycore.app.1281, but 
>> it does afford the auxiliary driver, and any other observer, a way to 
>> figure out which device it is representing.  This also works nicely when 
>> trying to associate an auxiliary driver instance for a VF with the 
>> matching VF PCI driver instance.
>
>I'd personally not mind divorcing devlink from bus devices a little
>more. On one hand we have cases like this where there's naturally no

How exactly do you envision to do this? There is a good reason to have
the handle based on bus/name, as it is constant and predictable.


>bus device, on the other we have multi-link PCI devices which want to
>straddle NUMA nodes but otherwise are just a logical unit.