[PATCH net-next 0/9] mlxsw: spectrum_cnt: Expose counter resources

[PATCH net-next 0/9] mlxsw: spectrum_cnt: Expose counter resources

[Ido Schimmel]

From: Ido Schimmel <idosch@mellanox.com>

Jiri says:

Capacity and utilization of existing flow and RIF counters are currently
unavailable to be seen by the user. Use the existing devlink resources
API to expose the information:

$ sudo devlink resource show pci/0000:00:10.0 -v
pci/0000:00:10.0:
  name kvd resource_path /kvd size 524288 unit entry dpipe_tables none
  name span_agents resource_path /span_agents size 8 occ 0 unit entry dpipe_tables none
  name counters resource_path /counters size 79872 occ 44 unit entry dpipe_tables none
    resources:
      name flow resource_path /counters/flow size 61440 occ 4 unit entry dpipe_tables none
      name rif resource_path /counters/rif size 18432 occ 40 unit entry dpipe_tables none

Jiri Pirko (9):
  mlxsw: spectrum_cnt: Query bank size from FW resources
  selftests: spectrum-2: Adjust tc_flower_scale limit according to
    current counter count
  mlxsw: spectrum_cnt: Move sub_pools under per-instance pool struct
  mlxsw: spectrum_cnt: Add entry_size_res_id for each subpool and use it
    to query entry size
  mlxsw: spectrum_cnt: Expose subpool sizes over devlink resources
  mlxsw: spectrum_cnt: Move config validation along with resource
    register
  mlxsw: spectrum_cnt: Consolidate subpools initialization
  mlxsw: spectrum_cnt: Expose devlink resource occupancy for counters
  selftests: mlxsw: Add tc action hw_stats tests

 .../net/ethernet/mellanox/mlxsw/resources.h   |   2 +
 .../net/ethernet/mellanox/mlxsw/spectrum.c    |  10 +
 .../net/ethernet/mellanox/mlxsw/spectrum.h    |   7 +
 .../ethernet/mellanox/mlxsw/spectrum_cnt.c    | 243 +++++++++++++-----
 .../ethernet/mellanox/mlxsw/spectrum_cnt.h    |   2 +
 .../net/mlxsw/spectrum-2/tc_flower_scale.sh   |   4 +-
 .../drivers/net/mlxsw/tc_action_hw_stats.sh   | 130 ++++++++++
 tools/testing/selftests/net/forwarding/lib.sh |   9 +
 8 files changed, 343 insertions(+), 64 deletions(-)
 create mode 100755 tools/testing/selftests/drivers/net/mlxsw/tc_action_hw_stats.sh

-- 
2.24.1

[PATCH net-next 1/9] mlxsw: spectrum_cnt: Query bank size from FW resources

[Ido Schimmel]

From: Jiri Pirko <jiri@mellanox.com>

The bank size is different between Spectrum versions. Also it is
a resource that can be queried. So instead of hard coding the value in
code, query it from the firmware.

Signed-off-by: Jiri Pirko <jiri@mellanox.com>
Signed-off-by: Ido Schimmel <idosch@mellanox.com>
---
 drivers/net/ethernet/mellanox/mlxsw/resources.h   |  2 ++
 .../net/ethernet/mellanox/mlxsw/spectrum_cnt.c    | 15 +++++++++------
 2 files changed, 11 insertions(+), 6 deletions(-)

diff --git a/drivers/net/ethernet/mellanox/mlxsw/resources.h b/drivers/net/ethernet/mellanox/mlxsw/resources.h
index 6534184cb942..d62496ef299c 100644
--- a/drivers/net/ethernet/mellanox/mlxsw/resources.h
+++ b/drivers/net/ethernet/mellanox/mlxsw/resources.h
@@ -18,6 +18,7 @@ enum mlxsw_res_id {
 	MLXSW_RES_ID_CQE_V1,
 	MLXSW_RES_ID_CQE_V2,
 	MLXSW_RES_ID_COUNTER_POOL_SIZE,
+	MLXSW_RES_ID_COUNTER_BANK_SIZE,
 	MLXSW_RES_ID_MAX_SPAN,
 	MLXSW_RES_ID_COUNTER_SIZE_PACKETS_BYTES,
 	MLXSW_RES_ID_COUNTER_SIZE_ROUTER_BASIC,
@@ -75,6 +76,7 @@ static u16 mlxsw_res_ids[] = {
 	[MLXSW_RES_ID_CQE_V1] = 0x2211,
 	[MLXSW_RES_ID_CQE_V2] = 0x2212,
 	[MLXSW_RES_ID_COUNTER_POOL_SIZE] = 0x2410,
+	[MLXSW_RES_ID_COUNTER_BANK_SIZE] = 0x2411,
 	[MLXSW_RES_ID_MAX_SPAN] = 0x2420,
 	[MLXSW_RES_ID_COUNTER_SIZE_PACKETS_BYTES] = 0x2443,
 	[MLXSW_RES_ID_COUNTER_SIZE_ROUTER_BASIC] = 0x2449,
diff --git a/drivers/net/ethernet/mellanox/mlxsw/spectrum_cnt.c b/drivers/net/ethernet/mellanox/mlxsw/spectrum_cnt.c
index 6a02ef9ec00e..37811181586a 100644
--- a/drivers/net/ethernet/mellanox/mlxsw/spectrum_cnt.c
+++ b/drivers/net/ethernet/mellanox/mlxsw/spectrum_cnt.c
@@ -7,8 +7,6 @@
 
 #include "spectrum_cnt.h"
 
-#define MLXSW_SP_COUNTER_POOL_BANK_SIZE 4096
-
 struct mlxsw_sp_counter_sub_pool {
 	unsigned int base_index;
 	unsigned int size;
@@ -36,13 +34,15 @@ static int mlxsw_sp_counter_pool_validate(struct mlxsw_sp *mlxsw_sp)
 {
 	unsigned int total_bank_config = 0;
 	unsigned int pool_size;
+	unsigned int bank_size;
 	int i;
 
 	pool_size = MLXSW_CORE_RES_GET(mlxsw_sp->core, COUNTER_POOL_SIZE);
+	bank_size = MLXSW_CORE_RES_GET(mlxsw_sp->core, COUNTER_BANK_SIZE);
 	/* Check config is valid, no bank over subscription */
 	for (i = 0; i < ARRAY_SIZE(mlxsw_sp_counter_sub_pools); i++)
 		total_bank_config += mlxsw_sp_counter_sub_pools[i].bank_count;
-	if (total_bank_config > pool_size / MLXSW_SP_COUNTER_POOL_BANK_SIZE + 1)
+	if (total_bank_config > pool_size / bank_size + 1)
 		return -EINVAL;
 	return 0;
 }
@@ -71,11 +71,13 @@ int mlxsw_sp_counter_pool_init(struct mlxsw_sp *mlxsw_sp)
 	struct mlxsw_sp_counter_sub_pool *sub_pool;
 	struct mlxsw_sp_counter_pool *pool;
 	unsigned int base_index;
+	unsigned int bank_size;
 	unsigned int map_size;
 	int i;
 	int err;
 
-	if (!MLXSW_CORE_RES_VALID(mlxsw_sp->core, COUNTER_POOL_SIZE))
+	if (!MLXSW_CORE_RES_VALID(mlxsw_sp->core, COUNTER_POOL_SIZE) ||
+	    !MLXSW_CORE_RES_VALID(mlxsw_sp->core, COUNTER_BANK_SIZE))
 		return -EIO;
 
 	err = mlxsw_sp_counter_pool_validate(mlxsw_sp);
@@ -94,6 +96,8 @@ int mlxsw_sp_counter_pool_init(struct mlxsw_sp *mlxsw_sp)
 	pool->pool_size = MLXSW_CORE_RES_GET(mlxsw_sp->core, COUNTER_POOL_SIZE);
 	map_size = BITS_TO_LONGS(pool->pool_size) * sizeof(unsigned long);
 
+	bank_size = MLXSW_CORE_RES_GET(mlxsw_sp->core, COUNTER_BANK_SIZE);
+
 	pool->usage = kzalloc(map_size, GFP_KERNEL);
 	if (!pool->usage) {
 		err = -ENOMEM;
@@ -107,8 +111,7 @@ int mlxsw_sp_counter_pool_init(struct mlxsw_sp *mlxsw_sp)
 	base_index = 0;
 	for (i = 0; i < ARRAY_SIZE(mlxsw_sp_counter_sub_pools); i++) {
 		sub_pool = &pool->sub_pools[i];
-		sub_pool->size = sub_pool->bank_count *
-				 MLXSW_SP_COUNTER_POOL_BANK_SIZE;
+		sub_pool->size = sub_pool->bank_count * bank_size;
 		sub_pool->base_index = base_index;
 		base_index += sub_pool->size;
 		/* The last bank can't be fully used */
-- 
2.24.1

[PATCH net-next 2/9] selftests: spectrum-2: Adjust tc_flower_scale limit according to current counter count

[Ido Schimmel]

From: Jiri Pirko <jiri@mellanox.com>

With the change that made the code to query counter bank size from device
instead of using hard-coded value, the number of available counters
changed for Spectrum-2. Adjust the limit in the selftests.

Signed-off-by: Jiri Pirko <jiri@mellanox.com>
Signed-off-by: Ido Schimmel <idosch@mellanox.com>
---
 .../selftests/drivers/net/mlxsw/spectrum-2/tc_flower_scale.sh | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/tools/testing/selftests/drivers/net/mlxsw/spectrum-2/tc_flower_scale.sh b/tools/testing/selftests/drivers/net/mlxsw/spectrum-2/tc_flower_scale.sh
index a0795227216e..efd798a85931 100644
--- a/tools/testing/selftests/drivers/net/mlxsw/spectrum-2/tc_flower_scale.sh
+++ b/tools/testing/selftests/drivers/net/mlxsw/spectrum-2/tc_flower_scale.sh
@@ -8,9 +8,9 @@ tc_flower_get_target()
 	# The driver associates a counter with each tc filter, which means the
 	# number of supported filters is bounded by the number of available
 	# counters.
-	# Currently, the driver supports 12K (12,288) flow counters and six of
+	# Currently, the driver supports 30K (30,720) flow counters and six of
 	# these are used for multicast routing.
-	local target=12282
+	local target=30714
 
 	if ((! should_fail)); then
 		echo $target
-- 
2.24.1

[PATCH net-next 3/9] mlxsw: spectrum_cnt: Move sub_pools under per-instance pool struct

[Ido Schimmel]

From: Jiri Pirko <jiri@mellanox.com>

Currently, the global static array of subpools is used. Make it
per-instance as multiple instances of the mlxsw driver can have
different values.

Signed-off-by: Jiri Pirko <jiri@mellanox.com>
Signed-off-by: Ido Schimmel <idosch@mellanox.com>
---
 .../ethernet/mellanox/mlxsw/spectrum_cnt.c    | 45 +++++++++++--------
 1 file changed, 27 insertions(+), 18 deletions(-)

diff --git a/drivers/net/ethernet/mellanox/mlxsw/spectrum_cnt.c b/drivers/net/ethernet/mellanox/mlxsw/spectrum_cnt.c
index 37811181586a..4cdabde47dd0 100644
--- a/drivers/net/ethernet/mellanox/mlxsw/spectrum_cnt.c
+++ b/drivers/net/ethernet/mellanox/mlxsw/spectrum_cnt.c
@@ -18,10 +18,11 @@ struct mlxsw_sp_counter_pool {
 	unsigned int pool_size;
 	unsigned long *usage; /* Usage bitmap */
 	spinlock_t counter_pool_lock; /* Protects counter pool allocations */
-	struct mlxsw_sp_counter_sub_pool *sub_pools;
+	unsigned int sub_pools_count;
+	struct mlxsw_sp_counter_sub_pool sub_pools[];
 };
 
-static struct mlxsw_sp_counter_sub_pool mlxsw_sp_counter_sub_pools[] = {
+static const struct mlxsw_sp_counter_sub_pool mlxsw_sp_counter_sub_pools[] = {
 	[MLXSW_SP_COUNTER_SUB_POOL_FLOW] = {
 		.bank_count = 6,
 	},
@@ -32,6 +33,7 @@ static struct mlxsw_sp_counter_sub_pool mlxsw_sp_counter_sub_pools[] = {
 
 static int mlxsw_sp_counter_pool_validate(struct mlxsw_sp *mlxsw_sp)
 {
+	struct mlxsw_sp_counter_pool *pool = mlxsw_sp->counter_pool;
 	unsigned int total_bank_config = 0;
 	unsigned int pool_size;
 	unsigned int bank_size;
@@ -40,8 +42,8 @@ static int mlxsw_sp_counter_pool_validate(struct mlxsw_sp *mlxsw_sp)
 	pool_size = MLXSW_CORE_RES_GET(mlxsw_sp->core, COUNTER_POOL_SIZE);
 	bank_size = MLXSW_CORE_RES_GET(mlxsw_sp->core, COUNTER_BANK_SIZE);
 	/* Check config is valid, no bank over subscription */
-	for (i = 0; i < ARRAY_SIZE(mlxsw_sp_counter_sub_pools); i++)
-		total_bank_config += mlxsw_sp_counter_sub_pools[i].bank_count;
+	for (i = 0; i < pool->sub_pools_count; i++)
+		total_bank_config += pool->sub_pools[i].bank_count;
 	if (total_bank_config > pool_size / bank_size + 1)
 		return -EINVAL;
 	return 0;
@@ -49,16 +51,17 @@ static int mlxsw_sp_counter_pool_validate(struct mlxsw_sp *mlxsw_sp)
 
 static int mlxsw_sp_counter_sub_pools_prepare(struct mlxsw_sp *mlxsw_sp)
 {
+	struct mlxsw_sp_counter_pool *pool = mlxsw_sp->counter_pool;
 	struct mlxsw_sp_counter_sub_pool *sub_pool;
 
 	/* Prepare generic flow pool*/
-	sub_pool = &mlxsw_sp_counter_sub_pools[MLXSW_SP_COUNTER_SUB_POOL_FLOW];
+	sub_pool = &pool->sub_pools[MLXSW_SP_COUNTER_SUB_POOL_FLOW];
 	if (!MLXSW_CORE_RES_VALID(mlxsw_sp->core, COUNTER_SIZE_PACKETS_BYTES))
 		return -EIO;
 	sub_pool->entry_size = MLXSW_CORE_RES_GET(mlxsw_sp->core,
 						  COUNTER_SIZE_PACKETS_BYTES);
 	/* Prepare erif pool*/
-	sub_pool = &mlxsw_sp_counter_sub_pools[MLXSW_SP_COUNTER_SUB_POOL_RIF];
+	sub_pool = &pool->sub_pools[MLXSW_SP_COUNTER_SUB_POOL_RIF];
 	if (!MLXSW_CORE_RES_VALID(mlxsw_sp->core, COUNTER_SIZE_ROUTER_BASIC))
 		return -EIO;
 	sub_pool->entry_size = MLXSW_CORE_RES_GET(mlxsw_sp->core,
@@ -68,6 +71,7 @@ static int mlxsw_sp_counter_sub_pools_prepare(struct mlxsw_sp *mlxsw_sp)
 
 int mlxsw_sp_counter_pool_init(struct mlxsw_sp *mlxsw_sp)
 {
+	unsigned int sub_pools_count = ARRAY_SIZE(mlxsw_sp_counter_sub_pools);
 	struct mlxsw_sp_counter_sub_pool *sub_pool;
 	struct mlxsw_sp_counter_pool *pool;
 	unsigned int base_index;
@@ -80,18 +84,23 @@ int mlxsw_sp_counter_pool_init(struct mlxsw_sp *mlxsw_sp)
 	    !MLXSW_CORE_RES_VALID(mlxsw_sp->core, COUNTER_BANK_SIZE))
 		return -EIO;
 
+	pool = kzalloc(struct_size(pool, sub_pools, sub_pools_count),
+		       GFP_KERNEL);
+	if (!pool)
+		return -ENOMEM;
+	mlxsw_sp->counter_pool = pool;
+	memcpy(pool->sub_pools, mlxsw_sp_counter_sub_pools,
+	       sub_pools_count * sizeof(*sub_pool));
+	pool->sub_pools_count = sub_pools_count;
+	spin_lock_init(&pool->counter_pool_lock);
+
 	err = mlxsw_sp_counter_pool_validate(mlxsw_sp);
 	if (err)
-		return err;
+		goto err_pool_validate;
 
 	err = mlxsw_sp_counter_sub_pools_prepare(mlxsw_sp);
 	if (err)
-		return err;
-
-	pool = kzalloc(sizeof(*pool), GFP_KERNEL);
-	if (!pool)
-		return -ENOMEM;
-	spin_lock_init(&pool->counter_pool_lock);
+		goto err_sub_pools_prepare;
 
 	pool->pool_size = MLXSW_CORE_RES_GET(mlxsw_sp->core, COUNTER_POOL_SIZE);
 	map_size = BITS_TO_LONGS(pool->pool_size) * sizeof(unsigned long);
@@ -104,12 +113,11 @@ int mlxsw_sp_counter_pool_init(struct mlxsw_sp *mlxsw_sp)
 		goto err_usage_alloc;
 	}
 
-	pool->sub_pools = mlxsw_sp_counter_sub_pools;
 	/* Allocation is based on bank count which should be
 	 * specified for each sub pool statically.
 	 */
 	base_index = 0;
-	for (i = 0; i < ARRAY_SIZE(mlxsw_sp_counter_sub_pools); i++) {
+	for (i = 0; i < pool->sub_pools_count; i++) {
 		sub_pool = &pool->sub_pools[i];
 		sub_pool->size = sub_pool->bank_count * bank_size;
 		sub_pool->base_index = base_index;
@@ -119,10 +127,11 @@ int mlxsw_sp_counter_pool_init(struct mlxsw_sp *mlxsw_sp)
 			sub_pool->size = pool->pool_size - sub_pool->base_index;
 	}
 
-	mlxsw_sp->counter_pool = pool;
 	return 0;
 
 err_usage_alloc:
+err_sub_pools_prepare:
+err_pool_validate:
 	kfree(pool);
 	return err;
 }
@@ -147,7 +156,7 @@ int mlxsw_sp_counter_alloc(struct mlxsw_sp *mlxsw_sp,
 	unsigned int stop_index;
 	int i, err;
 
-	sub_pool = &mlxsw_sp_counter_sub_pools[sub_pool_id];
+	sub_pool = &pool->sub_pools[sub_pool_id];
 	stop_index = sub_pool->base_index + sub_pool->size;
 	entry_index = sub_pool->base_index;
 
@@ -186,7 +195,7 @@ void mlxsw_sp_counter_free(struct mlxsw_sp *mlxsw_sp,
 
 	if (WARN_ON(counter_index >= pool->pool_size))
 		return;
-	sub_pool = &mlxsw_sp_counter_sub_pools[sub_pool_id];
+	sub_pool = &pool->sub_pools[sub_pool_id];
 	spin_lock(&pool->counter_pool_lock);
 	for (i = 0; i < sub_pool->entry_size; i++)
 		__clear_bit(counter_index + i, pool->usage);
-- 
2.24.1

[PATCH net-next 4/9] mlxsw: spectrum_cnt: Add entry_size_res_id for each subpool and use it to query entry size

[Ido Schimmel]

From: Jiri Pirko <jiri@mellanox.com>

Add new field to subpool struct that would indicate which
resource id should be used to query the entry size for
the subpool from the device.

Signed-off-by: Jiri Pirko <jiri@mellanox.com>
Signed-off-by: Ido Schimmel <idosch@mellanox.com>
---
 .../ethernet/mellanox/mlxsw/spectrum_cnt.c    | 26 ++++++++++---------
 1 file changed, 14 insertions(+), 12 deletions(-)

diff --git a/drivers/net/ethernet/mellanox/mlxsw/spectrum_cnt.c b/drivers/net/ethernet/mellanox/mlxsw/spectrum_cnt.c
index 4cdabde47dd0..ef2c6c5c8b72 100644
--- a/drivers/net/ethernet/mellanox/mlxsw/spectrum_cnt.c
+++ b/drivers/net/ethernet/mellanox/mlxsw/spectrum_cnt.c
@@ -10,6 +10,7 @@
 struct mlxsw_sp_counter_sub_pool {
 	unsigned int base_index;
 	unsigned int size;
+	enum mlxsw_res_id entry_size_res_id;
 	unsigned int entry_size;
 	unsigned int bank_count;
 };
@@ -24,9 +25,11 @@ struct mlxsw_sp_counter_pool {
 
 static const struct mlxsw_sp_counter_sub_pool mlxsw_sp_counter_sub_pools[] = {
 	[MLXSW_SP_COUNTER_SUB_POOL_FLOW] = {
+		.entry_size_res_id = MLXSW_RES_ID_COUNTER_SIZE_PACKETS_BYTES,
 		.bank_count = 6,
 	},
 	[MLXSW_SP_COUNTER_SUB_POOL_RIF] = {
+		.entry_size_res_id = MLXSW_RES_ID_COUNTER_SIZE_ROUTER_BASIC,
 		.bank_count = 2,
 	}
 };
@@ -53,19 +56,18 @@ static int mlxsw_sp_counter_sub_pools_prepare(struct mlxsw_sp *mlxsw_sp)
 {
 	struct mlxsw_sp_counter_pool *pool = mlxsw_sp->counter_pool;
 	struct mlxsw_sp_counter_sub_pool *sub_pool;
+	enum mlxsw_res_id res_id;
+	int i;
 
-	/* Prepare generic flow pool*/
-	sub_pool = &pool->sub_pools[MLXSW_SP_COUNTER_SUB_POOL_FLOW];
-	if (!MLXSW_CORE_RES_VALID(mlxsw_sp->core, COUNTER_SIZE_PACKETS_BYTES))
-		return -EIO;
-	sub_pool->entry_size = MLXSW_CORE_RES_GET(mlxsw_sp->core,
-						  COUNTER_SIZE_PACKETS_BYTES);
-	/* Prepare erif pool*/
-	sub_pool = &pool->sub_pools[MLXSW_SP_COUNTER_SUB_POOL_RIF];
-	if (!MLXSW_CORE_RES_VALID(mlxsw_sp->core, COUNTER_SIZE_ROUTER_BASIC))
-		return -EIO;
-	sub_pool->entry_size = MLXSW_CORE_RES_GET(mlxsw_sp->core,
-						  COUNTER_SIZE_ROUTER_BASIC);
+	for (i = 0; i < pool->sub_pools_count; i++) {
+		sub_pool = &pool->sub_pools[i];
+		res_id = sub_pool->entry_size_res_id;
+
+		if (!mlxsw_core_res_valid(mlxsw_sp->core, res_id))
+			return -EIO;
+		sub_pool->entry_size = mlxsw_core_res_get(mlxsw_sp->core,
+							  res_id);
+	}
 	return 0;
 }
 
-- 
2.24.1

[PATCH net-next 5/9] mlxsw: spectrum_cnt: Expose subpool sizes over devlink resources

[Ido Schimmel]

From: Jiri Pirko <jiri@mellanox.com>

Implement devlink resources support for counter pools. Move the subpool
sizes calculations into the new resources register function.

Signed-off-by: Jiri Pirko <jiri@mellanox.com>
Signed-off-by: Ido Schimmel <idosch@mellanox.com>
---
 .../net/ethernet/mellanox/mlxsw/spectrum.c    | 10 ++
 .../net/ethernet/mellanox/mlxsw/spectrum.h    |  7 ++
 .../ethernet/mellanox/mlxsw/spectrum_cnt.c    | 99 +++++++++++++++----
 .../ethernet/mellanox/mlxsw/spectrum_cnt.h    |  2 +
 4 files changed, 101 insertions(+), 17 deletions(-)

diff --git a/drivers/net/ethernet/mellanox/mlxsw/spectrum.c b/drivers/net/ethernet/mellanox/mlxsw/spectrum.c
index 51709012593e..35d3a68ef4fd 100644
--- a/drivers/net/ethernet/mellanox/mlxsw/spectrum.c
+++ b/drivers/net/ethernet/mellanox/mlxsw/spectrum.c
@@ -5421,8 +5421,13 @@ static int mlxsw_sp1_resources_register(struct mlxsw_core *mlxsw_core)
 	if (err)
 		goto err_resources_span_register;
 
+	err = mlxsw_sp_counter_resources_register(mlxsw_core);
+	if (err)
+		goto err_resources_counter_register;
+
 	return 0;
 
+err_resources_counter_register:
 err_resources_span_register:
 	devlink_resources_unregister(priv_to_devlink(mlxsw_core), NULL);
 	return err;
@@ -5440,8 +5445,13 @@ static int mlxsw_sp2_resources_register(struct mlxsw_core *mlxsw_core)
 	if (err)
 		goto err_resources_span_register;
 
+	err = mlxsw_sp_counter_resources_register(mlxsw_core);
+	if (err)
+		goto err_resources_counter_register;
+
 	return 0;
 
+err_resources_counter_register:
 err_resources_span_register:
 	devlink_resources_unregister(priv_to_devlink(mlxsw_core), NULL);
 	return err;
diff --git a/drivers/net/ethernet/mellanox/mlxsw/spectrum.h b/drivers/net/ethernet/mellanox/mlxsw/spectrum.h
index 81801c6fb941..57d8c95e4f9f 100644
--- a/drivers/net/ethernet/mellanox/mlxsw/spectrum.h
+++ b/drivers/net/ethernet/mellanox/mlxsw/spectrum.h
@@ -46,6 +46,10 @@
 
 #define MLXSW_SP_RESOURCE_NAME_SPAN "span_agents"
 
+#define MLXSW_SP_RESOURCE_NAME_COUNTERS "counters"
+#define MLXSW_SP_RESOURCE_NAME_COUNTERS_FLOW "flow"
+#define MLXSW_SP_RESOURCE_NAME_COUNTERS_RIF "rif"
+
 enum mlxsw_sp_resource_id {
 	MLXSW_SP_RESOURCE_KVD = 1,
 	MLXSW_SP_RESOURCE_KVD_LINEAR,
@@ -55,6 +59,9 @@ enum mlxsw_sp_resource_id {
 	MLXSW_SP_RESOURCE_KVD_LINEAR_CHUNKS,
 	MLXSW_SP_RESOURCE_KVD_LINEAR_LARGE_CHUNKS,
 	MLXSW_SP_RESOURCE_SPAN,
+	MLXSW_SP_RESOURCE_COUNTERS,
+	MLXSW_SP_RESOURCE_COUNTERS_FLOW,
+	MLXSW_SP_RESOURCE_COUNTERS_RIF,
 };
 
 struct mlxsw_sp_port;
diff --git a/drivers/net/ethernet/mellanox/mlxsw/spectrum_cnt.c b/drivers/net/ethernet/mellanox/mlxsw/spectrum_cnt.c
index ef2c6c5c8b72..629355cf52a9 100644
--- a/drivers/net/ethernet/mellanox/mlxsw/spectrum_cnt.c
+++ b/drivers/net/ethernet/mellanox/mlxsw/spectrum_cnt.c
@@ -8,15 +8,17 @@
 #include "spectrum_cnt.h"
 
 struct mlxsw_sp_counter_sub_pool {
+	u64 size;
 	unsigned int base_index;
-	unsigned int size;
 	enum mlxsw_res_id entry_size_res_id;
+	const char *resource_name; /* devlink resource name */
+	u64 resource_id; /* devlink resource id */
 	unsigned int entry_size;
 	unsigned int bank_count;
 };
 
 struct mlxsw_sp_counter_pool {
-	unsigned int pool_size;
+	u64 pool_size;
 	unsigned long *usage; /* Usage bitmap */
 	spinlock_t counter_pool_lock; /* Protects counter pool allocations */
 	unsigned int sub_pools_count;
@@ -26,10 +28,14 @@ struct mlxsw_sp_counter_pool {
 static const struct mlxsw_sp_counter_sub_pool mlxsw_sp_counter_sub_pools[] = {
 	[MLXSW_SP_COUNTER_SUB_POOL_FLOW] = {
 		.entry_size_res_id = MLXSW_RES_ID_COUNTER_SIZE_PACKETS_BYTES,
+		.resource_name = MLXSW_SP_RESOURCE_NAME_COUNTERS_FLOW,
+		.resource_id = MLXSW_SP_RESOURCE_COUNTERS_FLOW,
 		.bank_count = 6,
 	},
 	[MLXSW_SP_COUNTER_SUB_POOL_RIF] = {
 		.entry_size_res_id = MLXSW_RES_ID_COUNTER_SIZE_ROUTER_BASIC,
+		.resource_name = MLXSW_SP_RESOURCE_NAME_COUNTERS_RIF,
+		.resource_id = MLXSW_SP_RESOURCE_COUNTERS_RIF,
 		.bank_count = 2,
 	}
 };
@@ -74,18 +80,14 @@ static int mlxsw_sp_counter_sub_pools_prepare(struct mlxsw_sp *mlxsw_sp)
 int mlxsw_sp_counter_pool_init(struct mlxsw_sp *mlxsw_sp)
 {
 	unsigned int sub_pools_count = ARRAY_SIZE(mlxsw_sp_counter_sub_pools);
+	struct devlink *devlink = priv_to_devlink(mlxsw_sp->core);
 	struct mlxsw_sp_counter_sub_pool *sub_pool;
 	struct mlxsw_sp_counter_pool *pool;
 	unsigned int base_index;
-	unsigned int bank_size;
 	unsigned int map_size;
 	int i;
 	int err;
 
-	if (!MLXSW_CORE_RES_VALID(mlxsw_sp->core, COUNTER_POOL_SIZE) ||
-	    !MLXSW_CORE_RES_VALID(mlxsw_sp->core, COUNTER_BANK_SIZE))
-		return -EIO;
-
 	pool = kzalloc(struct_size(pool, sub_pools, sub_pools_count),
 		       GFP_KERNEL);
 	if (!pool)
@@ -104,10 +106,12 @@ int mlxsw_sp_counter_pool_init(struct mlxsw_sp *mlxsw_sp)
 	if (err)
 		goto err_sub_pools_prepare;
 
-	pool->pool_size = MLXSW_CORE_RES_GET(mlxsw_sp->core, COUNTER_POOL_SIZE);
-	map_size = BITS_TO_LONGS(pool->pool_size) * sizeof(unsigned long);
+	err = devlink_resource_size_get(devlink, MLXSW_SP_RESOURCE_COUNTERS,
+					&pool->pool_size);
+	if (err)
+		goto err_pool_resource_size_get;
 
-	bank_size = MLXSW_CORE_RES_GET(mlxsw_sp->core, COUNTER_BANK_SIZE);
+	map_size = BITS_TO_LONGS(pool->pool_size) * sizeof(unsigned long);
 
 	pool->usage = kzalloc(map_size, GFP_KERNEL);
 	if (!pool->usage) {
@@ -115,23 +119,26 @@ int mlxsw_sp_counter_pool_init(struct mlxsw_sp *mlxsw_sp)
 		goto err_usage_alloc;
 	}
 
-	/* Allocation is based on bank count which should be
-	 * specified for each sub pool statically.
-	 */
 	base_index = 0;
 	for (i = 0; i < pool->sub_pools_count; i++) {
 		sub_pool = &pool->sub_pools[i];
-		sub_pool->size = sub_pool->bank_count * bank_size;
+
+		err = devlink_resource_size_get(devlink,
+						sub_pool->resource_id,
+						&sub_pool->size);
+		if (err)
+			goto err_sub_pool_resource_size_get;
+
 		sub_pool->base_index = base_index;
 		base_index += sub_pool->size;
-		/* The last bank can't be fully used */
-		if (sub_pool->base_index + sub_pool->size > pool->pool_size)
-			sub_pool->size = pool->pool_size - sub_pool->base_index;
 	}
 
 	return 0;
 
+err_sub_pool_resource_size_get:
+	kfree(pool->usage);
 err_usage_alloc:
+err_pool_resource_size_get:
 err_sub_pools_prepare:
 err_pool_validate:
 	kfree(pool);
@@ -203,3 +210,61 @@ void mlxsw_sp_counter_free(struct mlxsw_sp *mlxsw_sp,
 		__clear_bit(counter_index + i, pool->usage);
 	spin_unlock(&pool->counter_pool_lock);
 }
+
+int mlxsw_sp_counter_resources_register(struct mlxsw_core *mlxsw_core)
+{
+	static struct devlink_resource_size_params size_params;
+	struct devlink *devlink = priv_to_devlink(mlxsw_core);
+	const struct mlxsw_sp_counter_sub_pool *sub_pool;
+	u64 sub_pool_size;
+	u64 base_index;
+	u64 pool_size;
+	u64 bank_size;
+	int err;
+	int i;
+
+	if (!MLXSW_CORE_RES_VALID(mlxsw_core, COUNTER_POOL_SIZE) ||
+	    !MLXSW_CORE_RES_VALID(mlxsw_core, COUNTER_BANK_SIZE))
+		return -EIO;
+
+	pool_size = MLXSW_CORE_RES_GET(mlxsw_core, COUNTER_POOL_SIZE);
+	bank_size = MLXSW_CORE_RES_GET(mlxsw_core, COUNTER_BANK_SIZE);
+
+	devlink_resource_size_params_init(&size_params, pool_size,
+					  pool_size, bank_size,
+					  DEVLINK_RESOURCE_UNIT_ENTRY);
+	err = devlink_resource_register(devlink,
+					MLXSW_SP_RESOURCE_NAME_COUNTERS,
+					pool_size,
+					MLXSW_SP_RESOURCE_COUNTERS,
+					DEVLINK_RESOURCE_ID_PARENT_TOP,
+					&size_params);
+	if (err)
+		return err;
+
+	/* Allocation is based on bank count which should be
+	 * specified for each sub pool statically.
+	 */
+	base_index = 0;
+	for (i = 0; i < ARRAY_SIZE(mlxsw_sp_counter_sub_pools); i++) {
+		sub_pool = &mlxsw_sp_counter_sub_pools[i];
+		sub_pool_size = sub_pool->bank_count * bank_size;
+		/* The last bank can't be fully used */
+		if (base_index + sub_pool_size > pool_size)
+			sub_pool_size = pool_size - base_index;
+		base_index += sub_pool_size;
+
+		devlink_resource_size_params_init(&size_params, sub_pool_size,
+						  sub_pool_size, bank_size,
+						  DEVLINK_RESOURCE_UNIT_ENTRY);
+		err = devlink_resource_register(devlink,
+						sub_pool->resource_name,
+						sub_pool_size,
+						sub_pool->resource_id,
+						MLXSW_SP_RESOURCE_COUNTERS,
+						&size_params);
+		if (err)
+			return err;
+	}
+	return 0;
+}
diff --git a/drivers/net/ethernet/mellanox/mlxsw/spectrum_cnt.h b/drivers/net/ethernet/mellanox/mlxsw/spectrum_cnt.h
index 81465e267b10..a68d931090dd 100644
--- a/drivers/net/ethernet/mellanox/mlxsw/spectrum_cnt.h
+++ b/drivers/net/ethernet/mellanox/mlxsw/spectrum_cnt.h
@@ -4,6 +4,7 @@
 #ifndef _MLXSW_SPECTRUM_CNT_H
 #define _MLXSW_SPECTRUM_CNT_H
 
+#include "core.h"
 #include "spectrum.h"
 
 enum mlxsw_sp_counter_sub_pool_id {
@@ -19,5 +20,6 @@ void mlxsw_sp_counter_free(struct mlxsw_sp *mlxsw_sp,
 			   unsigned int counter_index);
 int mlxsw_sp_counter_pool_init(struct mlxsw_sp *mlxsw_sp);
 void mlxsw_sp_counter_pool_fini(struct mlxsw_sp *mlxsw_sp);
+int mlxsw_sp_counter_resources_register(struct mlxsw_core *mlxsw_core);
 
 #endif
-- 
2.24.1

[PATCH net-next 6/9] mlxsw: spectrum_cnt: Move config validation along with resource register

[Ido Schimmel]

From: Jiri Pirko <jiri@mellanox.com>

Move the validation of subpools configuration, to avoid possible over
commitment to resource registration. Add WARN_ON to indicate bug
in the code.

Signed-off-by: Jiri Pirko <jiri@mellanox.com>
Signed-off-by: Ido Schimmel <idosch@mellanox.com>
---
 .../ethernet/mellanox/mlxsw/spectrum_cnt.c    | 31 +++++--------------
 1 file changed, 8 insertions(+), 23 deletions(-)

diff --git a/drivers/net/ethernet/mellanox/mlxsw/spectrum_cnt.c b/drivers/net/ethernet/mellanox/mlxsw/spectrum_cnt.c
index 629355cf52a9..d36904143b10 100644
--- a/drivers/net/ethernet/mellanox/mlxsw/spectrum_cnt.c
+++ b/drivers/net/ethernet/mellanox/mlxsw/spectrum_cnt.c
@@ -40,24 +40,6 @@ static const struct mlxsw_sp_counter_sub_pool mlxsw_sp_counter_sub_pools[] = {
 	}
 };
 
-static int mlxsw_sp_counter_pool_validate(struct mlxsw_sp *mlxsw_sp)
-{
-	struct mlxsw_sp_counter_pool *pool = mlxsw_sp->counter_pool;
-	unsigned int total_bank_config = 0;
-	unsigned int pool_size;
-	unsigned int bank_size;
-	int i;
-
-	pool_size = MLXSW_CORE_RES_GET(mlxsw_sp->core, COUNTER_POOL_SIZE);
-	bank_size = MLXSW_CORE_RES_GET(mlxsw_sp->core, COUNTER_BANK_SIZE);
-	/* Check config is valid, no bank over subscription */
-	for (i = 0; i < pool->sub_pools_count; i++)
-		total_bank_config += pool->sub_pools[i].bank_count;
-	if (total_bank_config > pool_size / bank_size + 1)
-		return -EINVAL;
-	return 0;
-}
-
 static int mlxsw_sp_counter_sub_pools_prepare(struct mlxsw_sp *mlxsw_sp)
 {
 	struct mlxsw_sp_counter_pool *pool = mlxsw_sp->counter_pool;
@@ -98,10 +80,6 @@ int mlxsw_sp_counter_pool_init(struct mlxsw_sp *mlxsw_sp)
 	pool->sub_pools_count = sub_pools_count;
 	spin_lock_init(&pool->counter_pool_lock);
 
-	err = mlxsw_sp_counter_pool_validate(mlxsw_sp);
-	if (err)
-		goto err_pool_validate;
-
 	err = mlxsw_sp_counter_sub_pools_prepare(mlxsw_sp);
 	if (err)
 		goto err_sub_pools_prepare;
@@ -140,7 +118,6 @@ int mlxsw_sp_counter_pool_init(struct mlxsw_sp *mlxsw_sp)
 err_usage_alloc:
 err_pool_resource_size_get:
 err_sub_pools_prepare:
-err_pool_validate:
 	kfree(pool);
 	return err;
 }
@@ -216,6 +193,7 @@ int mlxsw_sp_counter_resources_register(struct mlxsw_core *mlxsw_core)
 	static struct devlink_resource_size_params size_params;
 	struct devlink *devlink = priv_to_devlink(mlxsw_core);
 	const struct mlxsw_sp_counter_sub_pool *sub_pool;
+	unsigned int total_bank_config;
 	u64 sub_pool_size;
 	u64 base_index;
 	u64 pool_size;
@@ -245,6 +223,7 @@ int mlxsw_sp_counter_resources_register(struct mlxsw_core *mlxsw_core)
 	/* Allocation is based on bank count which should be
 	 * specified for each sub pool statically.
 	 */
+	total_bank_config = 0;
 	base_index = 0;
 	for (i = 0; i < ARRAY_SIZE(mlxsw_sp_counter_sub_pools); i++) {
 		sub_pool = &mlxsw_sp_counter_sub_pools[i];
@@ -265,6 +244,12 @@ int mlxsw_sp_counter_resources_register(struct mlxsw_core *mlxsw_core)
 						&size_params);
 		if (err)
 			return err;
+		total_bank_config += sub_pool->bank_count;
 	}
+
+	/* Check config is valid, no bank over subscription */
+	if (WARN_ON(total_bank_config > pool_size / bank_size + 1))
+		return -EINVAL;
+
 	return 0;
 }
-- 
2.24.1

[PATCH net-next 7/9] mlxsw: spectrum_cnt: Consolidate subpools initialization

[Ido Schimmel]

From: Jiri Pirko <jiri@mellanox.com>

Put all init operations related to subpools into
mlxsw_sp_counter_sub_pools_init().

Signed-off-by: Jiri Pirko <jiri@mellanox.com>
Signed-off-by: Ido Schimmel <idosch@mellanox.com>
---
 .../ethernet/mellanox/mlxsw/spectrum_cnt.c    | 38 ++++++++-----------
 1 file changed, 16 insertions(+), 22 deletions(-)

diff --git a/drivers/net/ethernet/mellanox/mlxsw/spectrum_cnt.c b/drivers/net/ethernet/mellanox/mlxsw/spectrum_cnt.c
index d36904143b10..417c512bc7a2 100644
--- a/drivers/net/ethernet/mellanox/mlxsw/spectrum_cnt.c
+++ b/drivers/net/ethernet/mellanox/mlxsw/spectrum_cnt.c
@@ -40,11 +40,14 @@ static const struct mlxsw_sp_counter_sub_pool mlxsw_sp_counter_sub_pools[] = {
 	}
 };
 
-static int mlxsw_sp_counter_sub_pools_prepare(struct mlxsw_sp *mlxsw_sp)
+static int mlxsw_sp_counter_sub_pools_init(struct mlxsw_sp *mlxsw_sp)
 {
 	struct mlxsw_sp_counter_pool *pool = mlxsw_sp->counter_pool;
+	struct devlink *devlink = priv_to_devlink(mlxsw_sp->core);
 	struct mlxsw_sp_counter_sub_pool *sub_pool;
+	unsigned int base_index = 0;
 	enum mlxsw_res_id res_id;
+	int err;
 	int i;
 
 	for (i = 0; i < pool->sub_pools_count; i++) {
@@ -55,6 +58,14 @@ static int mlxsw_sp_counter_sub_pools_prepare(struct mlxsw_sp *mlxsw_sp)
 			return -EIO;
 		sub_pool->entry_size = mlxsw_core_res_get(mlxsw_sp->core,
 							  res_id);
+		err = devlink_resource_size_get(devlink,
+						sub_pool->resource_id,
+						&sub_pool->size);
+		if (err)
+			return err;
+
+		sub_pool->base_index = base_index;
+		base_index += sub_pool->size;
 	}
 	return 0;
 }
@@ -65,9 +76,7 @@ int mlxsw_sp_counter_pool_init(struct mlxsw_sp *mlxsw_sp)
 	struct devlink *devlink = priv_to_devlink(mlxsw_sp->core);
 	struct mlxsw_sp_counter_sub_pool *sub_pool;
 	struct mlxsw_sp_counter_pool *pool;
-	unsigned int base_index;
 	unsigned int map_size;
-	int i;
 	int err;
 
 	pool = kzalloc(struct_size(pool, sub_pools, sub_pools_count),
@@ -80,10 +89,6 @@ int mlxsw_sp_counter_pool_init(struct mlxsw_sp *mlxsw_sp)
 	pool->sub_pools_count = sub_pools_count;
 	spin_lock_init(&pool->counter_pool_lock);
 
-	err = mlxsw_sp_counter_sub_pools_prepare(mlxsw_sp);
-	if (err)
-		goto err_sub_pools_prepare;
-
 	err = devlink_resource_size_get(devlink, MLXSW_SP_RESOURCE_COUNTERS,
 					&pool->pool_size);
 	if (err)
@@ -97,27 +102,16 @@ int mlxsw_sp_counter_pool_init(struct mlxsw_sp *mlxsw_sp)
 		goto err_usage_alloc;
 	}
 
-	base_index = 0;
-	for (i = 0; i < pool->sub_pools_count; i++) {
-		sub_pool = &pool->sub_pools[i];
-
-		err = devlink_resource_size_get(devlink,
-						sub_pool->resource_id,
-						&sub_pool->size);
-		if (err)
-			goto err_sub_pool_resource_size_get;
-
-		sub_pool->base_index = base_index;
-		base_index += sub_pool->size;
-	}
+	err = mlxsw_sp_counter_sub_pools_init(mlxsw_sp);
+	if (err)
+		goto err_sub_pools_init;
 
 	return 0;
 
-err_sub_pool_resource_size_get:
+err_sub_pools_init:
 	kfree(pool->usage);
 err_usage_alloc:
 err_pool_resource_size_get:
-err_sub_pools_prepare:
 	kfree(pool);
 	return err;
 }
-- 
2.24.1

[PATCH net-next 8/9] mlxsw: spectrum_cnt: Expose devlink resource occupancy for counters

[Ido Schimmel]

From: Jiri Pirko <jiri@mellanox.com>

Implement occupancy counting for counters and expose over devlink
resource API.

Signed-off-by: Jiri Pirko <jiri@mellanox.com>
Signed-off-by: Ido Schimmel <idosch@mellanox.com>
---
 .../ethernet/mellanox/mlxsw/spectrum_cnt.c    | 63 ++++++++++++++++++-
 1 file changed, 62 insertions(+), 1 deletion(-)

diff --git a/drivers/net/ethernet/mellanox/mlxsw/spectrum_cnt.c b/drivers/net/ethernet/mellanox/mlxsw/spectrum_cnt.c
index 417c512bc7a2..0268f0a6662a 100644
--- a/drivers/net/ethernet/mellanox/mlxsw/spectrum_cnt.c
+++ b/drivers/net/ethernet/mellanox/mlxsw/spectrum_cnt.c
@@ -15,12 +15,14 @@ struct mlxsw_sp_counter_sub_pool {
 	u64 resource_id; /* devlink resource id */
 	unsigned int entry_size;
 	unsigned int bank_count;
+	atomic_t active_entries_count;
 };
 
 struct mlxsw_sp_counter_pool {
 	u64 pool_size;
 	unsigned long *usage; /* Usage bitmap */
 	spinlock_t counter_pool_lock; /* Protects counter pool allocations */
+	atomic_t active_entries_count;
 	unsigned int sub_pools_count;
 	struct mlxsw_sp_counter_sub_pool sub_pools[];
 };
@@ -40,6 +42,13 @@ static const struct mlxsw_sp_counter_sub_pool mlxsw_sp_counter_sub_pools[] = {
 	}
 };
 
+static u64 mlxsw_sp_counter_sub_pool_occ_get(void *priv)
+{
+	const struct mlxsw_sp_counter_sub_pool *sub_pool = priv;
+
+	return atomic_read(&sub_pool->active_entries_count);
+}
+
 static int mlxsw_sp_counter_sub_pools_init(struct mlxsw_sp *mlxsw_sp)
 {
 	struct mlxsw_sp_counter_pool *pool = mlxsw_sp->counter_pool;
@@ -62,12 +71,50 @@ static int mlxsw_sp_counter_sub_pools_init(struct mlxsw_sp *mlxsw_sp)
 						sub_pool->resource_id,
 						&sub_pool->size);
 		if (err)
-			return err;
+			goto err_resource_size_get;
+
+		devlink_resource_occ_get_register(devlink,
+						  sub_pool->resource_id,
+						  mlxsw_sp_counter_sub_pool_occ_get,
+						  sub_pool);
 
 		sub_pool->base_index = base_index;
 		base_index += sub_pool->size;
+		atomic_set(&sub_pool->active_entries_count, 0);
 	}
 	return 0;
+
+err_resource_size_get:
+	for (i--; i >= 0; i--) {
+		sub_pool = &pool->sub_pools[i];
+
+		devlink_resource_occ_get_unregister(devlink,
+						    sub_pool->resource_id);
+	}
+	return err;
+}
+
+static void mlxsw_sp_counter_sub_pools_fini(struct mlxsw_sp *mlxsw_sp)
+{
+	struct mlxsw_sp_counter_pool *pool = mlxsw_sp->counter_pool;
+	struct devlink *devlink = priv_to_devlink(mlxsw_sp->core);
+	struct mlxsw_sp_counter_sub_pool *sub_pool;
+	int i;
+
+	for (i = 0; i < pool->sub_pools_count; i++) {
+		sub_pool = &pool->sub_pools[i];
+
+		WARN_ON(atomic_read(&sub_pool->active_entries_count));
+		devlink_resource_occ_get_unregister(devlink,
+						    sub_pool->resource_id);
+	}
+}
+
+static u64 mlxsw_sp_counter_pool_occ_get(void *priv)
+{
+	const struct mlxsw_sp_counter_pool *pool = priv;
+
+	return atomic_read(&pool->active_entries_count);
 }
 
 int mlxsw_sp_counter_pool_init(struct mlxsw_sp *mlxsw_sp)
@@ -88,11 +135,14 @@ int mlxsw_sp_counter_pool_init(struct mlxsw_sp *mlxsw_sp)
 	       sub_pools_count * sizeof(*sub_pool));
 	pool->sub_pools_count = sub_pools_count;
 	spin_lock_init(&pool->counter_pool_lock);
+	atomic_set(&pool->active_entries_count, 0);
 
 	err = devlink_resource_size_get(devlink, MLXSW_SP_RESOURCE_COUNTERS,
 					&pool->pool_size);
 	if (err)
 		goto err_pool_resource_size_get;
+	devlink_resource_occ_get_register(devlink, MLXSW_SP_RESOURCE_COUNTERS,
+					  mlxsw_sp_counter_pool_occ_get, pool);
 
 	map_size = BITS_TO_LONGS(pool->pool_size) * sizeof(unsigned long);
 
@@ -111,6 +161,8 @@ int mlxsw_sp_counter_pool_init(struct mlxsw_sp *mlxsw_sp)
 err_sub_pools_init:
 	kfree(pool->usage);
 err_usage_alloc:
+	devlink_resource_occ_get_unregister(devlink,
+					    MLXSW_SP_RESOURCE_COUNTERS);
 err_pool_resource_size_get:
 	kfree(pool);
 	return err;
@@ -119,10 +171,15 @@ int mlxsw_sp_counter_pool_init(struct mlxsw_sp *mlxsw_sp)
 void mlxsw_sp_counter_pool_fini(struct mlxsw_sp *mlxsw_sp)
 {
 	struct mlxsw_sp_counter_pool *pool = mlxsw_sp->counter_pool;
+	struct devlink *devlink = priv_to_devlink(mlxsw_sp->core);
 
+	mlxsw_sp_counter_sub_pools_fini(mlxsw_sp);
 	WARN_ON(find_first_bit(pool->usage, pool->pool_size) !=
 			       pool->pool_size);
+	WARN_ON(atomic_read(&pool->active_entries_count));
 	kfree(pool->usage);
+	devlink_resource_occ_get_unregister(devlink,
+					    MLXSW_SP_RESOURCE_COUNTERS);
 	kfree(pool);
 }
 
@@ -158,6 +215,8 @@ int mlxsw_sp_counter_alloc(struct mlxsw_sp *mlxsw_sp,
 	spin_unlock(&pool->counter_pool_lock);
 
 	*p_counter_index = entry_index;
+	atomic_add(sub_pool->entry_size, &sub_pool->active_entries_count);
+	atomic_add(sub_pool->entry_size, &pool->active_entries_count);
 	return 0;
 
 err_alloc:
@@ -180,6 +239,8 @@ void mlxsw_sp_counter_free(struct mlxsw_sp *mlxsw_sp,
 	for (i = 0; i < sub_pool->entry_size; i++)
 		__clear_bit(counter_index + i, pool->usage);
 	spin_unlock(&pool->counter_pool_lock);
+	atomic_sub(sub_pool->entry_size, &sub_pool->active_entries_count);
+	atomic_sub(sub_pool->entry_size, &pool->active_entries_count);
 }
 
 int mlxsw_sp_counter_resources_register(struct mlxsw_core *mlxsw_core)
-- 
2.24.1

[PATCH net-next 9/9] selftests: mlxsw: Add tc action hw_stats tests

[Ido Schimmel]

From: Jiri Pirko <jiri@mellanox.com>

Add tests for mlxsw hw_stats types.

Signed-off-by: Jiri Pirko <jiri@mellanox.com>
Signed-off-by: Ido Schimmel <idosch@mellanox.com>
---
 .../drivers/net/mlxsw/tc_action_hw_stats.sh   | 130 ++++++++++++++++++
 tools/testing/selftests/net/forwarding/lib.sh |   9 ++
 2 files changed, 139 insertions(+)
 create mode 100755 tools/testing/selftests/drivers/net/mlxsw/tc_action_hw_stats.sh

diff --git a/tools/testing/selftests/drivers/net/mlxsw/tc_action_hw_stats.sh b/tools/testing/selftests/drivers/net/mlxsw/tc_action_hw_stats.sh
new file mode 100755
index 000000000000..20ed98fe5a60
--- /dev/null
+++ b/tools/testing/selftests/drivers/net/mlxsw/tc_action_hw_stats.sh
@@ -0,0 +1,130 @@
+#!/bin/bash
+# SPDX-License-Identifier: GPL-2.0
+
+lib_dir=$(dirname $0)/../../../net/forwarding
+
+ALL_TESTS="
+	default_hw_stats_test
+	immediate_hw_stats_test
+	delayed_hw_stats_test
+	disabled_hw_stats_test
+"
+NUM_NETIFS=2
+
+source $lib_dir/tc_common.sh
+source $lib_dir/lib.sh
+source $lib_dir/devlink_lib.sh
+
+h1_create()
+{
+	simple_if_init $h1 192.0.2.1/24
+}
+
+h1_destroy()
+{
+	simple_if_fini $h1 192.0.2.1/24
+}
+
+switch_create()
+{
+	simple_if_init $swp1 192.0.2.2/24
+	tc qdisc add dev $swp1 clsact
+}
+
+switch_destroy()
+{
+	tc qdisc del dev $swp1 clsact
+	simple_if_fini $swp1 192.0.2.2/24
+}
+
+hw_stats_test()
+{
+	RET=0
+
+	local name=$1
+	local action_hw_stats=$2
+	local occ_delta=$3
+	local expected_packet_count=$4
+
+	local orig_occ=$(devlink_resource_get "counters" "flow" | jq '.["occ"]')
+
+	tc filter add dev $swp1 ingress protocol ip pref 1 handle 101 flower \
+		skip_sw dst_ip 192.0.2.2 action drop $action_hw_stats
+	check_err $? "Failed to add rule with $name hw_stats"
+
+	local new_occ=$(devlink_resource_get "counters" "flow" | jq '.["occ"]')
+	local expected_occ=$((orig_occ + occ_delta))
+	[ "$new_occ" == "$expected_occ" ]
+	check_err $? "Expected occupancy of $expected_occ, got $new_occ"
+
+	$MZ $h1 -c 1 -p 64 -a $h1mac -b $swp1mac -A 192.0.2.1 -B 192.0.2.2 \
+		-t ip -q
+
+	tc_check_packets "dev $swp1 ingress" 101 $expected_packet_count
+	check_err $? "Did not match incoming packet"
+
+	tc filter del dev $swp1 ingress protocol ip pref 1 handle 101 flower
+
+	log_test "$name hw_stats"
+}
+
+default_hw_stats_test()
+{
+	hw_stats_test "default" "" 2 1
+}
+
+immediate_hw_stats_test()
+{
+	hw_stats_test "immediate" "hw_stats immediate" 2 1
+}
+
+delayed_hw_stats_test()
+{
+	RET=0
+
+	tc filter add dev $swp1 ingress protocol ip pref 1 handle 101 flower \
+		skip_sw dst_ip 192.0.2.2 action drop hw_stats delayed
+	check_fail $? "Unexpected success in adding rule with delayed hw_stats"
+
+	log_test "delayed hw_stats"
+}
+
+disabled_hw_stats_test()
+{
+	hw_stats_test "disabled" "hw_stats disabled" 0 0
+}
+
+setup_prepare()
+{
+	h1=${NETIFS[p1]}
+	swp1=${NETIFS[p2]}
+
+	h1mac=$(mac_get $h1)
+	swp1mac=$(mac_get $swp1)
+
+	vrf_prepare
+
+	h1_create
+	switch_create
+}
+
+cleanup()
+{
+	pre_cleanup
+
+	switch_destroy
+	h1_destroy
+
+	vrf_cleanup
+}
+
+check_tc_action_hw_stats_support
+
+trap cleanup EXIT
+
+setup_prepare
+setup_wait
+
+tests_run
+
+exit $EXIT_STATUS
diff --git a/tools/testing/selftests/net/forwarding/lib.sh b/tools/testing/selftests/net/forwarding/lib.sh
index a4a7879b3bb9..977fc2b326a2 100644
--- a/tools/testing/selftests/net/forwarding/lib.sh
+++ b/tools/testing/selftests/net/forwarding/lib.sh
@@ -60,6 +60,15 @@ check_tc_chain_support()
 	fi
 }
 
+check_tc_action_hw_stats_support()
+{
+	tc actions help 2>&1 | grep -q hw_stats
+	if [[ $? -ne 0 ]]; then
+		echo "SKIP: iproute2 too old; tc is missing action hw_stats support"
+		exit 1
+	fi
+}
+
 if [[ "$(id -u)" -ne 0 ]]; then
 	echo "SKIP: need root privileges"
 	exit 0
-- 
2.24.1

Re: [PATCH net-next 0/9] mlxsw: spectrum_cnt: Expose counter resources

[Jakub Kicinski]

On Wed, 18 Mar 2020 15:48:48 +0200 Ido Schimmel wrote:
> From: Ido Schimmel <idosch@mellanox.com>
> 
> Jiri says:
> 
> Capacity and utilization of existing flow and RIF counters are currently
> unavailable to be seen by the user. Use the existing devlink resources
> API to expose the information:
> 
> $ sudo devlink resource show pci/0000:00:10.0 -v
> pci/0000:00:10.0:
>   name kvd resource_path /kvd size 524288 unit entry dpipe_tables none
>   name span_agents resource_path /span_agents size 8 occ 0 unit entry dpipe_tables none
>   name counters resource_path /counters size 79872 occ 44 unit entry dpipe_tables none
>     resources:
>       name flow resource_path /counters/flow size 61440 occ 4 unit entry dpipe_tables none
>       name rif resource_path /counters/rif size 18432 occ 40 unit entry dpipe_tables none

Acked-by: Jakub Kicinski <kuba@kernel.org>

Nice!

Re: [PATCH net-next 0/9] mlxsw: spectrum_cnt: Expose counter resources

[David Miller]

From: Ido Schimmel <idosch@idosch.org>
Date: Wed, 18 Mar 2020 15:48:48 +0200

> From: Ido Schimmel <idosch@mellanox.com>
> 
> Jiri says:
> 
> Capacity and utilization of existing flow and RIF counters are currently
> unavailable to be seen by the user. Use the existing devlink resources
> API to expose the information:
> 
> $ sudo devlink resource show pci/0000:00:10.0 -v
> pci/0000:00:10.0:
>   name kvd resource_path /kvd size 524288 unit entry dpipe_tables none
>   name span_agents resource_path /span_agents size 8 occ 0 unit entry dpipe_tables none
>   name counters resource_path /counters size 79872 occ 44 unit entry dpipe_tables none
>     resources:
>       name flow resource_path /counters/flow size 61440 occ 4 unit entry dpipe_tables none
>       name rif resource_path /counters/rif size 18432 occ 40 unit entry dpipe_tables none

Series applied, thanks everyone.