* [PATCH 01/13] RDMA: Split the alloc_hw_stats() ops to port and device variants
2021-05-17 16:47 [PATCH 00/13] Reorganize sysfs file creation for struct ib_devices Jason Gunthorpe
@ 2021-05-17 16:47 ` Jason Gunthorpe
2021-05-17 23:06 ` Saleem, Shiraz
` (2 more replies)
2021-05-17 16:47 ` [PATCH 02/13] RDMA/core: Replace the ib_port_data hw_stats pointers with a ib_port pointer Jason Gunthorpe
` (12 subsequent siblings)
13 siblings, 3 replies; 35+ messages in thread
From: Jason Gunthorpe @ 2021-05-17 16:47 UTC (permalink / raw)
To: Potnuri Bharat Teja, Dennis Dalessandro, Devesh Sharma,
Doug Ledford, Faisal Latif, Gal Pressman, Leon Romanovsky,
linux-rdma, Mike Marciniszyn, Naresh Kumar PBS, Selvin Xavier,
Shiraz Saleem, Yossi Leybovich, Somnath Kotur,
Sriharsha Basavapatna, Yishai Hadas, Zhu Yanjun
Cc: Greg KH, Kees Cook, Nathan Chancellor
This is being used to implement both the port and device global stats,
which is causing some confusion in the drivers. For instance EFA and i40iw
both seem to be misusing the device stats.
Split it into two ops so drivers that don't support one or the other can
leave the op NULL'd, making the calling code a little simpler to
understand.
Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>
---
drivers/infiniband/core/counters.c | 4 +-
drivers/infiniband/core/device.c | 3 +-
drivers/infiniband/core/nldev.c | 2 +-
drivers/infiniband/core/sysfs.c | 15 +++-
drivers/infiniband/hw/bnxt_re/hw_counters.c | 7 +-
drivers/infiniband/hw/bnxt_re/hw_counters.h | 4 +-
drivers/infiniband/hw/bnxt_re/main.c | 2 +-
drivers/infiniband/hw/cxgb4/provider.c | 9 +--
drivers/infiniband/hw/efa/efa.h | 3 +-
drivers/infiniband/hw/efa/efa_main.c | 3 +-
drivers/infiniband/hw/efa/efa_verbs.c | 11 ++-
drivers/infiniband/hw/hfi1/verbs.c | 86 ++++++++++-----------
drivers/infiniband/hw/i40iw/i40iw_verbs.c | 19 ++++-
drivers/infiniband/hw/mlx4/main.c | 25 ++++--
drivers/infiniband/hw/mlx5/counters.c | 42 +++++++---
drivers/infiniband/sw/rxe/rxe_hw_counters.c | 7 +-
drivers/infiniband/sw/rxe/rxe_hw_counters.h | 4 +-
drivers/infiniband/sw/rxe/rxe_verbs.c | 2 +-
include/rdma/ib_verbs.h | 13 ++--
19 files changed, 158 insertions(+), 103 deletions(-)
diff --git a/drivers/infiniband/core/counters.c b/drivers/infiniband/core/counters.c
index 15493357cfef09..df9e6c5e4ddf7a 100644
--- a/drivers/infiniband/core/counters.c
+++ b/drivers/infiniband/core/counters.c
@@ -605,10 +605,10 @@ void rdma_counter_init(struct ib_device *dev)
port_counter->mode.mode = RDMA_COUNTER_MODE_NONE;
mutex_init(&port_counter->lock);
- if (!dev->ops.alloc_hw_stats)
+ if (!dev->ops.alloc_hw_port_stats)
continue;
- port_counter->hstats = dev->ops.alloc_hw_stats(dev, port);
+ port_counter->hstats = dev->ops.alloc_hw_port_stats(dev, port);
if (!port_counter->hstats)
goto fail;
}
diff --git a/drivers/infiniband/core/device.c b/drivers/infiniband/core/device.c
index c660cef66ac6ca..86a16cd7d7fdb2 100644
--- a/drivers/infiniband/core/device.c
+++ b/drivers/infiniband/core/device.c
@@ -2595,7 +2595,8 @@ void ib_set_device_ops(struct ib_device *dev, const struct ib_device_ops *ops)
SET_DEVICE_OP(dev_ops, add_gid);
SET_DEVICE_OP(dev_ops, advise_mr);
SET_DEVICE_OP(dev_ops, alloc_dm);
- SET_DEVICE_OP(dev_ops, alloc_hw_stats);
+ SET_DEVICE_OP(dev_ops, alloc_hw_device_stats);
+ SET_DEVICE_OP(dev_ops, alloc_hw_port_stats);
SET_DEVICE_OP(dev_ops, alloc_mr);
SET_DEVICE_OP(dev_ops, alloc_mr_integrity);
SET_DEVICE_OP(dev_ops, alloc_mw);
diff --git a/drivers/infiniband/core/nldev.c b/drivers/infiniband/core/nldev.c
index 34d0cc1a4147ff..01316926cef63d 100644
--- a/drivers/infiniband/core/nldev.c
+++ b/drivers/infiniband/core/nldev.c
@@ -2060,7 +2060,7 @@ static int stat_get_doit_default_counter(struct sk_buff *skb,
if (!device)
return -EINVAL;
- if (!device->ops.alloc_hw_stats || !device->ops.get_hw_stats) {
+ if (!device->ops.alloc_hw_port_stats || !device->ops.get_hw_stats) {
ret = -EINVAL;
goto err;
}
diff --git a/drivers/infiniband/core/sysfs.c b/drivers/infiniband/core/sysfs.c
index 05b702de00e89b..29082d8d45fc4f 100644
--- a/drivers/infiniband/core/sysfs.c
+++ b/drivers/infiniband/core/sysfs.c
@@ -981,8 +981,15 @@ static void setup_hw_stats(struct ib_device *device, struct ib_port *port,
struct rdma_hw_stats *stats;
int i, ret;
- stats = device->ops.alloc_hw_stats(device, port_num);
-
+ if (port_num) {
+ if (!device->ops.alloc_hw_port_stats)
+ return;
+ stats = device->ops.alloc_hw_port_stats(device, port_num);
+ } else {
+ if (!device->ops.alloc_hw_device_stats)
+ return;
+ stats = device->ops.alloc_hw_device_stats(device);
+ }
if (!stats)
return;
@@ -1165,7 +1172,7 @@ static int add_port(struct ib_core_device *coredev, int port_num)
* port, so holder should be device. Therefore skip per port conunter
* initialization.
*/
- if (device->ops.alloc_hw_stats && port_num && is_full_dev)
+ if (device->ops.alloc_hw_port_stats && port_num && is_full_dev)
setup_hw_stats(device, p, port_num);
list_add_tail(&p->kobj.entry, &coredev->port_list);
@@ -1409,7 +1416,7 @@ int ib_device_register_sysfs(struct ib_device *device)
if (ret)
return ret;
- if (device->ops.alloc_hw_stats)
+ if (device->ops.alloc_hw_device_stats)
setup_hw_stats(device, NULL, 0);
return 0;
diff --git a/drivers/infiniband/hw/bnxt_re/hw_counters.c b/drivers/infiniband/hw/bnxt_re/hw_counters.c
index 3e54e1ae75b499..7ba07797845c03 100644
--- a/drivers/infiniband/hw/bnxt_re/hw_counters.c
+++ b/drivers/infiniband/hw/bnxt_re/hw_counters.c
@@ -234,13 +234,10 @@ int bnxt_re_ib_get_hw_stats(struct ib_device *ibdev,
return ARRAY_SIZE(bnxt_re_stat_name);
}
-struct rdma_hw_stats *bnxt_re_ib_alloc_hw_stats(struct ib_device *ibdev,
- u32 port_num)
+struct rdma_hw_stats *bnxt_re_ib_alloc_hw_port_stats(struct ib_device *ibdev,
+ u32 port_num)
{
BUILD_BUG_ON(ARRAY_SIZE(bnxt_re_stat_name) != BNXT_RE_NUM_COUNTERS);
- /* We support only per port stats */
- if (!port_num)
- return NULL;
return rdma_alloc_hw_stats_struct(bnxt_re_stat_name,
ARRAY_SIZE(bnxt_re_stat_name),
diff --git a/drivers/infiniband/hw/bnxt_re/hw_counters.h b/drivers/infiniband/hw/bnxt_re/hw_counters.h
index ede048607d6c0d..6f2d2f91d9ff09 100644
--- a/drivers/infiniband/hw/bnxt_re/hw_counters.h
+++ b/drivers/infiniband/hw/bnxt_re/hw_counters.h
@@ -96,8 +96,8 @@ enum bnxt_re_hw_stats {
BNXT_RE_NUM_COUNTERS
};
-struct rdma_hw_stats *bnxt_re_ib_alloc_hw_stats(struct ib_device *ibdev,
- u32 port_num);
+struct rdma_hw_stats *bnxt_re_ib_alloc_hw_port_stats(struct ib_device *ibdev,
+ u32 port_num);
int bnxt_re_ib_get_hw_stats(struct ib_device *ibdev,
struct rdma_hw_stats *stats,
u32 port, int index);
diff --git a/drivers/infiniband/hw/bnxt_re/main.c b/drivers/infiniband/hw/bnxt_re/main.c
index 8bfbf0231a9ef6..c705f09e767036 100644
--- a/drivers/infiniband/hw/bnxt_re/main.c
+++ b/drivers/infiniband/hw/bnxt_re/main.c
@@ -662,7 +662,7 @@ static const struct ib_device_ops bnxt_re_dev_ops = {
.uverbs_abi_ver = BNXT_RE_ABI_VERSION,
.add_gid = bnxt_re_add_gid,
- .alloc_hw_stats = bnxt_re_ib_alloc_hw_stats,
+ .alloc_hw_port_stats = bnxt_re_ib_alloc_hw_port_stats,
.alloc_mr = bnxt_re_alloc_mr,
.alloc_pd = bnxt_re_alloc_pd,
.alloc_ucontext = bnxt_re_alloc_ucontext,
diff --git a/drivers/infiniband/hw/cxgb4/provider.c b/drivers/infiniband/hw/cxgb4/provider.c
index 3f1893e180ddf3..c0f01799f4a0b9 100644
--- a/drivers/infiniband/hw/cxgb4/provider.c
+++ b/drivers/infiniband/hw/cxgb4/provider.c
@@ -377,14 +377,11 @@ static const char * const names[] = {
[IP6OUTRSTS] = "ip6OutRsts"
};
-static struct rdma_hw_stats *c4iw_alloc_stats(struct ib_device *ibdev,
- u32 port_num)
+static struct rdma_hw_stats *c4iw_alloc_port_stats(struct ib_device *ibdev,
+ u32 port_num)
{
BUILD_BUG_ON(ARRAY_SIZE(names) != NR_COUNTERS);
- if (port_num != 0)
- return NULL;
-
return rdma_alloc_hw_stats_struct(names, NR_COUNTERS,
RDMA_HW_STATS_DEFAULT_LIFESPAN);
}
@@ -455,7 +452,7 @@ static const struct ib_device_ops c4iw_dev_ops = {
.driver_id = RDMA_DRIVER_CXGB4,
.uverbs_abi_ver = C4IW_UVERBS_ABI_VERSION,
- .alloc_hw_stats = c4iw_alloc_stats,
+ .alloc_hw_port_stats = c4iw_alloc_port_stats,
.alloc_mr = c4iw_alloc_mr,
.alloc_pd = c4iw_allocate_pd,
.alloc_ucontext = c4iw_alloc_ucontext,
diff --git a/drivers/infiniband/hw/efa/efa.h b/drivers/infiniband/hw/efa/efa.h
index ea322cec27d230..2b8ca099b38115 100644
--- a/drivers/infiniband/hw/efa/efa.h
+++ b/drivers/infiniband/hw/efa/efa.h
@@ -157,7 +157,8 @@ int efa_modify_qp(struct ib_qp *ibqp, struct ib_qp_attr *qp_attr,
int qp_attr_mask, struct ib_udata *udata);
enum rdma_link_layer efa_port_link_layer(struct ib_device *ibdev,
u32 port_num);
-struct rdma_hw_stats *efa_alloc_hw_stats(struct ib_device *ibdev, u32 port_num);
+struct rdma_hw_stats *efa_alloc_hw_port_stats(struct ib_device *ibdev, u32 port_num);
+struct rdma_hw_stats *efa_alloc_hw_device_stats(struct ib_device *ibdev);
int efa_get_hw_stats(struct ib_device *ibdev, struct rdma_hw_stats *stats,
u32 port_num, int index);
diff --git a/drivers/infiniband/hw/efa/efa_main.c b/drivers/infiniband/hw/efa/efa_main.c
index 816cfd65b7ac84..203e6ddcacbc9c 100644
--- a/drivers/infiniband/hw/efa/efa_main.c
+++ b/drivers/infiniband/hw/efa/efa_main.c
@@ -242,7 +242,8 @@ static const struct ib_device_ops efa_dev_ops = {
.driver_id = RDMA_DRIVER_EFA,
.uverbs_abi_ver = EFA_UVERBS_ABI_VERSION,
- .alloc_hw_stats = efa_alloc_hw_stats,
+ .alloc_hw_port_stats = efa_alloc_hw_port_stats,
+ .alloc_hw_device_stats = efa_alloc_hw_device_stats,
.alloc_pd = efa_alloc_pd,
.alloc_ucontext = efa_alloc_ucontext,
.create_cq = efa_create_cq,
diff --git a/drivers/infiniband/hw/efa/efa_verbs.c b/drivers/infiniband/hw/efa/efa_verbs.c
index 51572f1dc6111b..be6d3ff0f1be24 100644
--- a/drivers/infiniband/hw/efa/efa_verbs.c
+++ b/drivers/infiniband/hw/efa/efa_verbs.c
@@ -1904,13 +1904,22 @@ int efa_destroy_ah(struct ib_ah *ibah, u32 flags)
return 0;
}
-struct rdma_hw_stats *efa_alloc_hw_stats(struct ib_device *ibdev, u32 port_num)
+struct rdma_hw_stats *efa_alloc_hw_port_stats(struct ib_device *ibdev, u32 port_num)
{
return rdma_alloc_hw_stats_struct(efa_stats_names,
ARRAY_SIZE(efa_stats_names),
RDMA_HW_STATS_DEFAULT_LIFESPAN);
}
+struct rdma_hw_stats *efa_alloc_hw_device_stats(struct ib_device *ibdev)
+{
+ /*
+ * It is probably a bug that efa reports its port stats as device
+ * stats
+ */
+ return efa_alloc_hw_port_stats(ibdev, 0);
+}
+
int efa_get_hw_stats(struct ib_device *ibdev, struct rdma_hw_stats *stats,
u32 port_num, int index)
{
diff --git a/drivers/infiniband/hw/hfi1/verbs.c b/drivers/infiniband/hw/hfi1/verbs.c
index 554294340caa28..85deba07a6754b 100644
--- a/drivers/infiniband/hw/hfi1/verbs.c
+++ b/drivers/infiniband/hw/hfi1/verbs.c
@@ -1693,54 +1693,53 @@ static int init_cntr_names(const char *names_in,
return 0;
}
-static struct rdma_hw_stats *alloc_hw_stats(struct ib_device *ibdev,
- u32 port_num)
+static int init_counters(struct ib_device *ibdev)
{
- int i, err;
+ struct hfi1_devdata *dd = dd_from_ibdev(ibdev);
+ int i, err = 0;
mutex_lock(&cntr_names_lock);
- if (!cntr_names_initialized) {
- struct hfi1_devdata *dd = dd_from_ibdev(ibdev);
-
- err = init_cntr_names(dd->cntrnames,
- dd->cntrnameslen,
- num_driver_cntrs,
- &num_dev_cntrs,
- &dev_cntr_names);
- if (err) {
- mutex_unlock(&cntr_names_lock);
- return NULL;
- }
-
- for (i = 0; i < num_driver_cntrs; i++)
- dev_cntr_names[num_dev_cntrs + i] =
- driver_cntr_names[i];
-
- err = init_cntr_names(dd->portcntrnames,
- dd->portcntrnameslen,
- 0,
- &num_port_cntrs,
- &port_cntr_names);
- if (err) {
- kfree(dev_cntr_names);
- dev_cntr_names = NULL;
- mutex_unlock(&cntr_names_lock);
- return NULL;
- }
- cntr_names_initialized = 1;
+ if (cntr_names_initialized)
+ goto out_unlock;
+
+ err = init_cntr_names(dd->cntrnames, dd->cntrnameslen, num_driver_cntrs,
+ &num_dev_cntrs, &dev_cntr_names);
+ if (err)
+ goto out_unlock;
+
+ for (i = 0; i < num_driver_cntrs; i++)
+ dev_cntr_names[num_dev_cntrs + i] = driver_cntr_names[i];
+
+ err = init_cntr_names(dd->portcntrnames, dd->portcntrnameslen, 0,
+ &num_port_cntrs, &port_cntr_names);
+ if (err) {
+ kfree(dev_cntr_names);
+ dev_cntr_names = NULL;
+ goto out_unlock;
}
+ cntr_names_initialized = 1;
+
+out_unlock:
mutex_unlock(&cntr_names_lock);
+ return err;
+}
- if (!port_num)
- return rdma_alloc_hw_stats_struct(
- dev_cntr_names,
- num_dev_cntrs + num_driver_cntrs,
- RDMA_HW_STATS_DEFAULT_LIFESPAN);
- else
- return rdma_alloc_hw_stats_struct(
- port_cntr_names,
- num_port_cntrs,
- RDMA_HW_STATS_DEFAULT_LIFESPAN);
+static struct rdma_hw_stats *hfi1_alloc_hw_device_stats(struct ib_device *ibdev)
+{
+ if (init_counters(ibdev))
+ return NULL;
+ return rdma_alloc_hw_stats_struct(dev_cntr_names,
+ num_dev_cntrs + num_driver_cntrs,
+ RDMA_HW_STATS_DEFAULT_LIFESPAN);
+}
+
+static struct rdma_hw_stats *hfi_alloc_hw_port_stats(struct ib_device *ibdev,
+ u32 port_num)
+{
+ if (init_counters(ibdev))
+ return NULL;
+ return rdma_alloc_hw_stats_struct(port_cntr_names, num_port_cntrs,
+ RDMA_HW_STATS_DEFAULT_LIFESPAN);
}
static u64 hfi1_sps_ints(void)
@@ -1787,7 +1786,8 @@ static const struct ib_device_ops hfi1_dev_ops = {
.owner = THIS_MODULE,
.driver_id = RDMA_DRIVER_HFI1,
- .alloc_hw_stats = alloc_hw_stats,
+ .alloc_hw_device_stats = hfi1_alloc_hw_device_stats,
+ .alloc_hw_port_stats = hfi_alloc_hw_port_stats,
.alloc_rdma_netdev = hfi1_vnic_alloc_rn,
.get_dev_fw_str = hfi1_get_dev_fw_str,
.get_hw_stats = get_hw_stats,
diff --git a/drivers/infiniband/hw/i40iw/i40iw_verbs.c b/drivers/infiniband/hw/i40iw/i40iw_verbs.c
index b876d722fcc887..33f16d2b8e74fb 100644
--- a/drivers/infiniband/hw/i40iw/i40iw_verbs.c
+++ b/drivers/infiniband/hw/i40iw/i40iw_verbs.c
@@ -2441,12 +2441,12 @@ static void i40iw_get_dev_fw_str(struct ib_device *dev, char *str)
}
/**
- * i40iw_alloc_hw_stats - Allocate a hw stats structure
+ * i40iw_alloc_hw_port_stats - Allocate a hw stats structure
* @ibdev: device pointer from stack
* @port_num: port number
*/
-static struct rdma_hw_stats *i40iw_alloc_hw_stats(struct ib_device *ibdev,
- u32 port_num)
+static struct rdma_hw_stats *i40iw_alloc_hw_port_stats(struct ib_device *ibdev,
+ u32 port_num)
{
struct i40iw_device *iwdev = to_iwdev(ibdev);
struct i40iw_sc_dev *dev = &iwdev->sc_dev;
@@ -2468,6 +2468,16 @@ static struct rdma_hw_stats *i40iw_alloc_hw_stats(struct ib_device *ibdev,
lifespan);
}
+static struct rdma_hw_stats *
+i40iw_alloc_hw_device_stats(struct ib_device *ibdev)
+{
+ /*
+ * It is probably a bug that i40iw reports its port stats as device
+ * stats
+ */
+ return i40iw_alloc_hw_port_stats(ibdev, 0);
+}
+
/**
* i40iw_get_hw_stats - Populates the rdma_hw_stats structure
* @ibdev: device pointer from stack
@@ -2521,7 +2531,8 @@ static const struct ib_device_ops i40iw_dev_ops = {
/* NOTE: Older kernels wrongly use 0 for the uverbs_abi_ver */
.uverbs_abi_ver = I40IW_ABI_VER,
- .alloc_hw_stats = i40iw_alloc_hw_stats,
+ .alloc_hw_device_stats = i40iw_alloc_hw_device_stats,
+ .alloc_hw_port_stats = i40iw_alloc_hw_port_stats,
.alloc_mr = i40iw_alloc_mr,
.alloc_pd = i40iw_alloc_pd,
.alloc_ucontext = i40iw_alloc_ucontext,
diff --git a/drivers/infiniband/hw/mlx4/main.c b/drivers/infiniband/hw/mlx4/main.c
index 22898d97ecbdac..341162aa2175c4 100644
--- a/drivers/infiniband/hw/mlx4/main.c
+++ b/drivers/infiniband/hw/mlx4/main.c
@@ -2105,17 +2105,29 @@ static const struct diag_counter diag_device_only[] = {
DIAG_COUNTER(rq_num_udsdprd, 0x118),
};
-static struct rdma_hw_stats *mlx4_ib_alloc_hw_stats(struct ib_device *ibdev,
- u32 port_num)
+static struct rdma_hw_stats *
+mlx4_ib_alloc_hw_device_stats(struct ib_device *ibdev)
{
struct mlx4_ib_dev *dev = to_mdev(ibdev);
struct mlx4_ib_diag_counters *diag = dev->diag_counters;
- if (!diag[!!port_num].name)
+ if (!diag[0].name)
return NULL;
- return rdma_alloc_hw_stats_struct(diag[!!port_num].name,
- diag[!!port_num].num_counters,
+ return rdma_alloc_hw_stats_struct(diag[0].name, diag[0].num_counters,
+ RDMA_HW_STATS_DEFAULT_LIFESPAN);
+}
+
+static struct rdma_hw_stats *
+mlx4_ib_alloc_hw_port_stats(struct ib_device *ibdev, u32 port_num)
+{
+ struct mlx4_ib_dev *dev = to_mdev(ibdev);
+ struct mlx4_ib_diag_counters *diag = dev->diag_counters;
+
+ if (!diag[1].name)
+ return NULL;
+
+ return rdma_alloc_hw_stats_struct(diag[1].name, diag[1].num_counters,
RDMA_HW_STATS_DEFAULT_LIFESPAN);
}
@@ -2206,7 +2218,8 @@ static void mlx4_ib_fill_diag_counters(struct mlx4_ib_dev *ibdev,
}
static const struct ib_device_ops mlx4_ib_hw_stats_ops = {
- .alloc_hw_stats = mlx4_ib_alloc_hw_stats,
+ .alloc_hw_device_stats = mlx4_ib_alloc_hw_device_stats,
+ .alloc_hw_port_stats = mlx4_ib_alloc_hw_port_stats,
.get_hw_stats = mlx4_ib_get_hw_stats,
};
diff --git a/drivers/infiniband/hw/mlx5/counters.c b/drivers/infiniband/hw/mlx5/counters.c
index e365341057cbe2..224ba36f2946ca 100644
--- a/drivers/infiniband/hw/mlx5/counters.c
+++ b/drivers/infiniband/hw/mlx5/counters.c
@@ -161,22 +161,29 @@ u16 mlx5_ib_get_counters_id(struct mlx5_ib_dev *dev, u32 port_num)
return cnts->set_id;
}
-static struct rdma_hw_stats *mlx5_ib_alloc_hw_stats(struct ib_device *ibdev,
- u32 port_num)
+static struct rdma_hw_stats *
+mlx5_ib_alloc_hw_device_stats(struct ib_device *ibdev)
{
struct mlx5_ib_dev *dev = to_mdev(ibdev);
- const struct mlx5_ib_counters *cnts;
- bool is_switchdev = is_mdev_switchdev_mode(dev->mdev);
+ const struct mlx5_ib_counters *cnts = &dev->port[0].cnts;
- if ((is_switchdev && port_num) || (!is_switchdev && !port_num))
- return NULL;
+ return rdma_alloc_hw_stats_struct(cnts->names,
+ cnts->num_q_counters +
+ cnts->num_cong_counters +
+ cnts->num_ext_ppcnt_counters,
+ RDMA_HW_STATS_DEFAULT_LIFESPAN);
+}
- cnts = get_counters(dev, port_num - 1);
+static struct rdma_hw_stats *
+mlx5_ib_alloc_hw_port_stats(struct ib_device *ibdev, u32 port_num)
+{
+ struct mlx5_ib_dev *dev = to_mdev(ibdev);
+ const struct mlx5_ib_counters *cnts = &dev->port[port_num - 1].cnts;
return rdma_alloc_hw_stats_struct(cnts->names,
cnts->num_q_counters +
- cnts->num_cong_counters +
- cnts->num_ext_ppcnt_counters,
+ cnts->num_cong_counters +
+ cnts->num_ext_ppcnt_counters,
RDMA_HW_STATS_DEFAULT_LIFESPAN);
}
@@ -666,7 +673,17 @@ void mlx5_ib_counters_clear_description(struct ib_counters *counters)
}
static const struct ib_device_ops hw_stats_ops = {
- .alloc_hw_stats = mlx5_ib_alloc_hw_stats,
+ .alloc_hw_port_stats = mlx5_ib_alloc_hw_port_stats,
+ .get_hw_stats = mlx5_ib_get_hw_stats,
+ .counter_bind_qp = mlx5_ib_counter_bind_qp,
+ .counter_unbind_qp = mlx5_ib_counter_unbind_qp,
+ .counter_dealloc = mlx5_ib_counter_dealloc,
+ .counter_alloc_stats = mlx5_ib_counter_alloc_stats,
+ .counter_update_stats = mlx5_ib_counter_update_stats,
+};
+
+static const struct ib_device_ops hw_switchdev_stats_ops = {
+ .alloc_hw_device_stats = mlx5_ib_alloc_hw_device_stats,
.get_hw_stats = mlx5_ib_get_hw_stats,
.counter_bind_qp = mlx5_ib_counter_bind_qp,
.counter_unbind_qp = mlx5_ib_counter_unbind_qp,
@@ -690,7 +707,10 @@ int mlx5_ib_counters_init(struct mlx5_ib_dev *dev)
if (!MLX5_CAP_GEN(dev->mdev, max_qp_cnt))
return 0;
- ib_set_device_ops(&dev->ib_dev, &hw_stats_ops);
+ if (is_mdev_switchdev_mode(dev->mdev))
+ ib_set_device_ops(&dev->ib_dev, &hw_switchdev_stats_ops);
+ else
+ ib_set_device_ops(&dev->ib_dev, &hw_stats_ops);
return mlx5_ib_alloc_counters(dev);
}
diff --git a/drivers/infiniband/sw/rxe/rxe_hw_counters.c b/drivers/infiniband/sw/rxe/rxe_hw_counters.c
index f469fd1c753d99..d5ceb706d964f3 100644
--- a/drivers/infiniband/sw/rxe/rxe_hw_counters.c
+++ b/drivers/infiniband/sw/rxe/rxe_hw_counters.c
@@ -40,13 +40,10 @@ int rxe_ib_get_hw_stats(struct ib_device *ibdev,
return ARRAY_SIZE(rxe_counter_name);
}
-struct rdma_hw_stats *rxe_ib_alloc_hw_stats(struct ib_device *ibdev,
- u32 port_num)
+struct rdma_hw_stats *rxe_ib_alloc_hw_port_stats(struct ib_device *ibdev,
+ u32 port_num)
{
BUILD_BUG_ON(ARRAY_SIZE(rxe_counter_name) != RXE_NUM_OF_COUNTERS);
- /* We support only per port stats */
- if (!port_num)
- return NULL;
return rdma_alloc_hw_stats_struct(rxe_counter_name,
ARRAY_SIZE(rxe_counter_name),
diff --git a/drivers/infiniband/sw/rxe/rxe_hw_counters.h b/drivers/infiniband/sw/rxe/rxe_hw_counters.h
index 2f369acb46d79b..71f4d4fa9dc8b6 100644
--- a/drivers/infiniband/sw/rxe/rxe_hw_counters.h
+++ b/drivers/infiniband/sw/rxe/rxe_hw_counters.h
@@ -29,8 +29,8 @@ enum rxe_counters {
RXE_NUM_OF_COUNTERS
};
-struct rdma_hw_stats *rxe_ib_alloc_hw_stats(struct ib_device *ibdev,
- u32 port_num);
+struct rdma_hw_stats *rxe_ib_alloc_hw_port_stats(struct ib_device *ibdev,
+ u32 port_num);
int rxe_ib_get_hw_stats(struct ib_device *ibdev,
struct rdma_hw_stats *stats,
u32 port, int index);
diff --git a/drivers/infiniband/sw/rxe/rxe_verbs.c b/drivers/infiniband/sw/rxe/rxe_verbs.c
index aeb5e232c19575..80f3094b17b5c3 100644
--- a/drivers/infiniband/sw/rxe/rxe_verbs.c
+++ b/drivers/infiniband/sw/rxe/rxe_verbs.c
@@ -1058,7 +1058,7 @@ static const struct ib_device_ops rxe_dev_ops = {
.driver_id = RDMA_DRIVER_RXE,
.uverbs_abi_ver = RXE_UVERBS_ABI_VERSION,
- .alloc_hw_stats = rxe_ib_alloc_hw_stats,
+ .alloc_hw_port_stats = rxe_ib_alloc_hw_port_stats,
.alloc_mr = rxe_alloc_mr,
.alloc_pd = rxe_alloc_pd,
.alloc_ucontext = rxe_alloc_ucontext,
diff --git a/include/rdma/ib_verbs.h b/include/rdma/ib_verbs.h
index 7e2f3699b8987b..8fa7fd20e94aa7 100644
--- a/include/rdma/ib_verbs.h
+++ b/include/rdma/ib_verbs.h
@@ -2523,13 +2523,14 @@ struct ib_device_ops {
unsigned int *meta_sg_offset);
/**
- * alloc_hw_stats - Allocate a struct rdma_hw_stats and fill in the
- * driver initialized data. The struct is kfree()'ed by the sysfs
- * core when the device is removed. A lifespan of -1 in the return
- * struct tells the core to set a default lifespan.
+ * alloc_hw_[device,port]_stats - Allocate a struct rdma_hw_stats and
+ * fill in the driver initialized data. The struct is kfree()'ed by
+ * the sysfs core when the device is removed. A lifespan of -1 in the
+ * return struct tells the core to set a default lifespan.
*/
- struct rdma_hw_stats *(*alloc_hw_stats)(struct ib_device *device,
- u32 port_num);
+ struct rdma_hw_stats *(*alloc_hw_device_stats)(struct ib_device *device);
+ struct rdma_hw_stats *(*alloc_hw_port_stats)(struct ib_device *device,
+ u32 port_num);
/**
* get_hw_stats - Fill in the counter value(s) in the stats struct.
* @index - The index in the value array we wish to have updated, or
--
2.31.1
^ permalink raw reply related [flat|nested] 35+ messages in thread* RE: [PATCH 01/13] RDMA: Split the alloc_hw_stats() ops to port and device variants
2021-05-17 16:47 ` [PATCH 01/13] RDMA: Split the alloc_hw_stats() ops to port and device variants Jason Gunthorpe
@ 2021-05-17 23:06 ` Saleem, Shiraz
2021-05-17 23:10 ` Jason Gunthorpe
2021-05-18 3:49 ` Mark Zhang
2021-05-19 11:29 ` Gal Pressman
2 siblings, 1 reply; 35+ messages in thread
From: Saleem, Shiraz @ 2021-05-17 23:06 UTC (permalink / raw)
To: Jason Gunthorpe, Potnuri Bharat Teja, Dennis Dalessandro,
Devesh Sharma, Doug Ledford, Latif, Faisal, Gal Pressman,
Leon Romanovsky, linux-rdma, Mike Marciniszyn, Naresh Kumar PBS,
Selvin Xavier, Yossi Leybovich, Somnath Kotur,
Sriharsha Basavapatna, Yishai Hadas, Zhu Yanjun
Cc: Greg KH, Kees Cook, Nathan Chancellor
> Subject: [PATCH 01/13] RDMA: Split the alloc_hw_stats() ops to port and device
> variants
>
> This is being used to implement both the port and device global stats, which is
> causing some confusion in the drivers. For instance EFA and i40iw both seem to
> be misusing the device stats.
>
> Split it into two ops so drivers that don't support one or the other can leave the op
> NULL'd, making the calling code a little simpler to understand.
>
> Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>
> ---
> drivers/infiniband/core/counters.c | 4 +-
> drivers/infiniband/core/device.c | 3 +-
> drivers/infiniband/core/nldev.c | 2 +-
> drivers/infiniband/core/sysfs.c | 15 +++-
> drivers/infiniband/hw/bnxt_re/hw_counters.c | 7 +-
> drivers/infiniband/hw/bnxt_re/hw_counters.h | 4 +-
> drivers/infiniband/hw/bnxt_re/main.c | 2 +-
> drivers/infiniband/hw/cxgb4/provider.c | 9 +--
> drivers/infiniband/hw/efa/efa.h | 3 +-
> drivers/infiniband/hw/efa/efa_main.c | 3 +-
> drivers/infiniband/hw/efa/efa_verbs.c | 11 ++-
> drivers/infiniband/hw/hfi1/verbs.c | 86 ++++++++++-----------
> drivers/infiniband/hw/i40iw/i40iw_verbs.c | 19 ++++-
> drivers/infiniband/hw/mlx4/main.c | 25 ++++--
> drivers/infiniband/hw/mlx5/counters.c | 42 +++++++---
> drivers/infiniband/sw/rxe/rxe_hw_counters.c | 7 +-
> drivers/infiniband/sw/rxe/rxe_hw_counters.h | 4 +-
> drivers/infiniband/sw/rxe/rxe_verbs.c | 2 +-
> include/rdma/ib_verbs.h | 13 ++--
> 19 files changed, 158 insertions(+), 103 deletions(-)
>
[...]
> /**
> - * i40iw_alloc_hw_stats - Allocate a hw stats structure
> + * i40iw_alloc_hw_port_stats - Allocate a hw stats structure
> * @ibdev: device pointer from stack
> * @port_num: port number
> */
> -static struct rdma_hw_stats *i40iw_alloc_hw_stats(struct ib_device *ibdev,
> - u32 port_num)
> +static struct rdma_hw_stats *i40iw_alloc_hw_port_stats(struct ib_device *ibdev,
> + u32 port_num)
> {
> struct i40iw_device *iwdev = to_iwdev(ibdev);
> struct i40iw_sc_dev *dev = &iwdev->sc_dev; @@ -2468,6 +2468,16 @@
> static struct rdma_hw_stats *i40iw_alloc_hw_stats(struct ib_device *ibdev,
> lifespan);
> }
>
> +static struct rdma_hw_stats *
> +i40iw_alloc_hw_device_stats(struct ib_device *ibdev) {
> + /*
> + * It is probably a bug that i40iw reports its port stats as device
> + * stats
> + */
The number of physical ports per ib device is 1.
> + return i40iw_alloc_hw_port_stats(ibdev, 0); }
> +
> /**
^ permalink raw reply [flat|nested] 35+ messages in thread* Re: [PATCH 01/13] RDMA: Split the alloc_hw_stats() ops to port and device variants
2021-05-17 23:06 ` Saleem, Shiraz
@ 2021-05-17 23:10 ` Jason Gunthorpe
2021-05-18 0:18 ` Saleem, Shiraz
0 siblings, 1 reply; 35+ messages in thread
From: Jason Gunthorpe @ 2021-05-17 23:10 UTC (permalink / raw)
To: Saleem, Shiraz
Cc: Potnuri Bharat Teja, Dennis Dalessandro, Devesh Sharma,
Doug Ledford, Latif, Faisal, Gal Pressman, Leon Romanovsky,
linux-rdma, Mike Marciniszyn, Naresh Kumar PBS, Selvin Xavier,
Yossi Leybovich, Somnath Kotur, Sriharsha Basavapatna,
Yishai Hadas, Zhu Yanjun, Greg KH, Kees Cook, Nathan Chancellor
On Mon, May 17, 2021 at 11:06:29PM +0000, Saleem, Shiraz wrote:
> > Subject: [PATCH 01/13] RDMA: Split the alloc_hw_stats() ops to port and device
> > variants
> >
> > This is being used to implement both the port and device global stats, which is
> > causing some confusion in the drivers. For instance EFA and i40iw both seem to
> > be misusing the device stats.
> >
> > Split it into two ops so drivers that don't support one or the other can leave the op
> > NULL'd, making the calling code a little simpler to understand.
> >
> > Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>
> > drivers/infiniband/core/counters.c | 4 +-
> > drivers/infiniband/core/device.c | 3 +-
> > drivers/infiniband/core/nldev.c | 2 +-
> > drivers/infiniband/core/sysfs.c | 15 +++-
> > drivers/infiniband/hw/bnxt_re/hw_counters.c | 7 +-
> > drivers/infiniband/hw/bnxt_re/hw_counters.h | 4 +-
> > drivers/infiniband/hw/bnxt_re/main.c | 2 +-
> > drivers/infiniband/hw/cxgb4/provider.c | 9 +--
> > drivers/infiniband/hw/efa/efa.h | 3 +-
> > drivers/infiniband/hw/efa/efa_main.c | 3 +-
> > drivers/infiniband/hw/efa/efa_verbs.c | 11 ++-
> > drivers/infiniband/hw/hfi1/verbs.c | 86 ++++++++++-----------
> > drivers/infiniband/hw/i40iw/i40iw_verbs.c | 19 ++++-
> > drivers/infiniband/hw/mlx4/main.c | 25 ++++--
> > drivers/infiniband/hw/mlx5/counters.c | 42 +++++++---
> > drivers/infiniband/sw/rxe/rxe_hw_counters.c | 7 +-
> > drivers/infiniband/sw/rxe/rxe_hw_counters.h | 4 +-
> > drivers/infiniband/sw/rxe/rxe_verbs.c | 2 +-
> > include/rdma/ib_verbs.h | 13 ++--
> > 19 files changed, 158 insertions(+), 103 deletions(-)
> >
>
> [...]
>
> > /**
> > - * i40iw_alloc_hw_stats - Allocate a hw stats structure
> > + * i40iw_alloc_hw_port_stats - Allocate a hw stats structure
> > * @ibdev: device pointer from stack
> > * @port_num: port number
> > */
> > -static struct rdma_hw_stats *i40iw_alloc_hw_stats(struct ib_device *ibdev,
> > - u32 port_num)
> > +static struct rdma_hw_stats *i40iw_alloc_hw_port_stats(struct ib_device *ibdev,
> > + u32 port_num)
> > {
> > struct i40iw_device *iwdev = to_iwdev(ibdev);
> > struct i40iw_sc_dev *dev = &iwdev->sc_dev; @@ -2468,6 +2468,16 @@
> > static struct rdma_hw_stats *i40iw_alloc_hw_stats(struct ib_device *ibdev,
> > lifespan);
> > }
> >
> > +static struct rdma_hw_stats *
> > +i40iw_alloc_hw_device_stats(struct ib_device *ibdev) {
> > + /*
> > + * It is probably a bug that i40iw reports its port stats as device
> > + * stats
> > + */
>
> The number of physical ports per ib device is 1.
Does something skip the port stats in this case? I don't see anything
like that?
What does the sysfs look like? Aren't there duplicated HW stats?
Jason
^ permalink raw reply [flat|nested] 35+ messages in thread* RE: [PATCH 01/13] RDMA: Split the alloc_hw_stats() ops to port and device variants
2021-05-17 23:10 ` Jason Gunthorpe
@ 2021-05-18 0:18 ` Saleem, Shiraz
2021-05-18 0:20 ` Jason Gunthorpe
0 siblings, 1 reply; 35+ messages in thread
From: Saleem, Shiraz @ 2021-05-18 0:18 UTC (permalink / raw)
To: Jason Gunthorpe
Cc: Potnuri Bharat Teja, Dennis Dalessandro, Devesh Sharma,
Doug Ledford, Latif, Faisal, Gal Pressman, Leon Romanovsky,
linux-rdma, Mike Marciniszyn, Naresh Kumar PBS, Selvin Xavier,
Yossi Leybovich, Somnath Kotur, Sriharsha Basavapatna,
Yishai Hadas, Zhu Yanjun, Greg KH, Kees Cook, Nathan Chancellor
> Subject: Re: [PATCH 01/13] RDMA: Split the alloc_hw_stats() ops to port and
> device variants
>
> On Mon, May 17, 2021 at 11:06:29PM +0000, Saleem, Shiraz wrote:
> > > Subject: [PATCH 01/13] RDMA: Split the alloc_hw_stats() ops to port
> > > and device variants
> > >
> > > This is being used to implement both the port and device global
> > > stats, which is causing some confusion in the drivers. For instance
> > > EFA and i40iw both seem to be misusing the device stats.
> > >
> > > Split it into two ops so drivers that don't support one or the other
> > > can leave the op NULL'd, making the calling code a little simpler to
> understand.
> > >
> > > Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>
> > > drivers/infiniband/core/counters.c | 4 +-
> > > drivers/infiniband/core/device.c | 3 +-
> > > drivers/infiniband/core/nldev.c | 2 +-
> > > drivers/infiniband/core/sysfs.c | 15 +++-
> > > drivers/infiniband/hw/bnxt_re/hw_counters.c | 7 +-
> > > drivers/infiniband/hw/bnxt_re/hw_counters.h | 4 +-
> > > drivers/infiniband/hw/bnxt_re/main.c | 2 +-
> > > drivers/infiniband/hw/cxgb4/provider.c | 9 +--
> > > drivers/infiniband/hw/efa/efa.h | 3 +-
> > > drivers/infiniband/hw/efa/efa_main.c | 3 +-
> > > drivers/infiniband/hw/efa/efa_verbs.c | 11 ++-
> > > drivers/infiniband/hw/hfi1/verbs.c | 86 ++++++++++-----------
> > > drivers/infiniband/hw/i40iw/i40iw_verbs.c | 19 ++++-
> > > drivers/infiniband/hw/mlx4/main.c | 25 ++++--
> > > drivers/infiniband/hw/mlx5/counters.c | 42 +++++++---
> > > drivers/infiniband/sw/rxe/rxe_hw_counters.c | 7 +-
> > > drivers/infiniband/sw/rxe/rxe_hw_counters.h | 4 +-
> > > drivers/infiniband/sw/rxe/rxe_verbs.c | 2 +-
> > > include/rdma/ib_verbs.h | 13 ++--
> > > 19 files changed, 158 insertions(+), 103 deletions(-)
> > >
> >
> > [...]
> >
> > > /**
> > > - * i40iw_alloc_hw_stats - Allocate a hw stats structure
> > > + * i40iw_alloc_hw_port_stats - Allocate a hw stats structure
> > > * @ibdev: device pointer from stack
> > > * @port_num: port number
> > > */
> > > -static struct rdma_hw_stats *i40iw_alloc_hw_stats(struct ib_device *ibdev,
> > > - u32 port_num)
> > > +static struct rdma_hw_stats *i40iw_alloc_hw_port_stats(struct ib_device
> *ibdev,
> > > + u32 port_num)
> > > {
> > > struct i40iw_device *iwdev = to_iwdev(ibdev);
> > > struct i40iw_sc_dev *dev = &iwdev->sc_dev; @@ -2468,6 +2468,16 @@
> > > static struct rdma_hw_stats *i40iw_alloc_hw_stats(struct ib_device *ibdev,
> > > lifespan);
> > > }
> > >
> > > +static struct rdma_hw_stats *
> > > +i40iw_alloc_hw_device_stats(struct ib_device *ibdev) {
> > > + /*
> > > + * It is probably a bug that i40iw reports its port stats as device
> > > + * stats
> > > + */
> >
> > The number of physical ports per ib device is 1.
>
> Does something skip the port stats in this case? I don't see anything like that?
No I don't see any check perse to prevent port stats from getting initialized.
>
> What does the sysfs look like? Aren't there duplicated HW stats?
>
Yeah it is duplicated. So we are saying for phys_port_cnt = 1, we want the stats to show up in only place?
# cd /sys/class/infiniband/iwp3s0f0/hw_counters; for f in *; do echo -n "$f: "; cat "$f"; done; cd -
cnpHandled: 0
cnpIgnored: 0
cnpSent: 0
ip4InDiscards: 0
ip4InMcastOctets: 0
ip4InMcastPkts: 0
ip4InOctets: 5884668
ip4InPkts: 87399
ip4InReasmRqd: 0
ip4InTruncatedPkts: 0
ip4OutMcastOctets: 0
ip4OutMcastPkts: 0
ip4OutNoRoutes: 0
ip4OutOctets: 7224092
ip4OutPkts: 101959
ip4OutSegRqd: 0
ip6InDiscards: 0
ip6InMcastOctets: 0
ip6InMcastPkts: 0
ip6InOctets: 0
ip6InPkts: 0
ip6InReasmRqd: 0
ip6InTruncatedPkts: 0
ip6OutMcastOctets: 0
ip6OutMcastPkts: 0
ip6OutNoRoutes: 0
ip6OutOctets: 0
ip6OutPkts: 0
ip6OutSegRqd: 0
iwInRdmaReads: 3
iwInRdmaSends: 29127
iwInRdmaWrites: 0
iwOutRdmaReads: 14564
iwOutRdmaSends: 29126
iwOutRdmaWrites: 14562
iwRdmaBnd: 0
iwRdmaInv: 0
lifespan: 10
RxECNMrkd: 0
RxUDP: 9
rxVlanErrors: 0
tcpInOptErrors: 0
tcpInProtoErrors: 0
tcpInSegs: 87399
tcpOutSegs: 101959
tcpRetransSegs: 0
TxUDP: 0
/sys/class/infiniband/iwp3s0f0/hw_counters
# cd /sys/class/infiniband/iwp3s0f0/ports/1/hw_counters; for f in *; do echo -n "$f: "; cat "$f"; done; cd -
cnpHandled: 0
cnpIgnored: 0
cnpSent: 0
ip4InDiscards: 0
ip4InMcastOctets: 0
ip4InMcastPkts: 0
ip4InOctets: 5884668
ip4InPkts: 87399
ip4InReasmRqd: 0
ip4InTruncatedPkts: 0
ip4OutMcastOctets: 0
ip4OutMcastPkts: 0
ip4OutNoRoutes: 0
ip4OutOctets: 7224092
ip4OutPkts: 101959
ip4OutSegRqd: 0
ip6InDiscards: 0
ip6InMcastOctets: 0
ip6InMcastPkts: 0
ip6InOctets: 0
ip6InPkts: 0
ip6InReasmRqd: 0
ip6InTruncatedPkts: 0
ip6OutMcastOctets: 0
ip6OutMcastPkts: 0
ip6OutNoRoutes: 0
ip6OutOctets: 0
ip6OutPkts: 0
ip6OutSegRqd: 0
iwInRdmaReads: 3
iwInRdmaSends: 29127
iwInRdmaWrites: 0
iwOutRdmaReads: 14564
iwOutRdmaSends: 29126
iwOutRdmaWrites: 14562
iwRdmaBnd: 0
iwRdmaInv: 0
lifespan: 10
RxECNMrkd: 0
RxUDP: 9
rxVlanErrors: 0
tcpInOptErrors: 0
tcpInProtoErrors: 0
tcpInSegs: 87399
tcpOutSegs: 101959
tcpRetransSegs: 0
TxUDP: 0
/sys/class/infiniband/iwp3s0f0/hw_counters
^ permalink raw reply [flat|nested] 35+ messages in thread* Re: [PATCH 01/13] RDMA: Split the alloc_hw_stats() ops to port and device variants
2021-05-18 0:18 ` Saleem, Shiraz
@ 2021-05-18 0:20 ` Jason Gunthorpe
2021-05-18 21:58 ` Saleem, Shiraz
0 siblings, 1 reply; 35+ messages in thread
From: Jason Gunthorpe @ 2021-05-18 0:20 UTC (permalink / raw)
To: Saleem, Shiraz
Cc: Potnuri Bharat Teja, Dennis Dalessandro, Devesh Sharma,
Doug Ledford, Latif, Faisal, Gal Pressman, Leon Romanovsky,
linux-rdma, Mike Marciniszyn, Naresh Kumar PBS, Selvin Xavier,
Yossi Leybovich, Somnath Kotur, Sriharsha Basavapatna,
Yishai Hadas, Zhu Yanjun, Greg KH, Kees Cook, Nathan Chancellor
On Tue, May 18, 2021 at 12:18:13AM +0000, Saleem, Shiraz wrote:
> > What does the sysfs look like? Aren't there duplicated HW stats?
>
>
> Yeah it is duplicated. So we are saying for phys_port_cnt = 1, we
> want the stats to show up in only place?
Yes.
Imagine you had a multi port device, and assign the stats
appropriately.
I didn't see anything in the list that made me think "device stat" but
I don't know what several of these do
If you can confirm that these are all port stats I can delete the
device stats in this series.
Jason
^ permalink raw reply [flat|nested] 35+ messages in thread
* RE: [PATCH 01/13] RDMA: Split the alloc_hw_stats() ops to port and device variants
2021-05-18 0:20 ` Jason Gunthorpe
@ 2021-05-18 21:58 ` Saleem, Shiraz
2021-05-19 12:25 ` Jason Gunthorpe
0 siblings, 1 reply; 35+ messages in thread
From: Saleem, Shiraz @ 2021-05-18 21:58 UTC (permalink / raw)
To: Jason Gunthorpe
Cc: Potnuri Bharat Teja, Dennis Dalessandro, Devesh Sharma,
Doug Ledford, Latif, Faisal, Gal Pressman, Leon Romanovsky,
linux-rdma, Mike Marciniszyn, Naresh Kumar PBS, Selvin Xavier,
Yossi Leybovich, Somnath Kotur, Sriharsha Basavapatna,
Yishai Hadas, Zhu Yanjun, Greg KH, Kees Cook, Nathan Chancellor
> Subject: Re: [PATCH 01/13] RDMA: Split the alloc_hw_stats() ops to port and
> device variants
>
> On Tue, May 18, 2021 at 12:18:13AM +0000, Saleem, Shiraz wrote:
>
> > > What does the sysfs look like? Aren't there duplicated HW stats?
> >
> >
> > Yeah it is duplicated. So we are saying for phys_port_cnt = 1, we want
> > the stats to show up in only place?
>
> Yes.
>
> Imagine you had a multi port device, and assign the stats appropriately.
>
> I didn't see anything in the list that made me think "device stat" but I don't know
> what several of these do
Are we exporting port stat when it should be really be device stat in some of the drivers though?
Most vendor drivers do port stats allocation only. Like...
https://elixir.bootlin.com/linux/v5.13-rc2/source/drivers/infiniband/hw/cxgb4/provider.c#L392
https://elixir.bootlin.com/linux/v5.13-rc2/source/drivers/infiniband/sw/rxe/rxe_hw_counters.c#L27
However .get_hw_stats callback appears to extract from same set of registers for each port
>
> If you can confirm that these are all port stats I can delete the device stats in this
> series.
>
hfi1 seems to have this separation between device and port stat with a different set of counters
https://elixir.bootlin.com/linux/v5.13-rc2/source/drivers/infiniband/hw/hfi1/verbs.c#L1760
So your device and port stats op callback separation feels warranted.
^ permalink raw reply [flat|nested] 35+ messages in thread
* Re: [PATCH 01/13] RDMA: Split the alloc_hw_stats() ops to port and device variants
2021-05-18 21:58 ` Saleem, Shiraz
@ 2021-05-19 12:25 ` Jason Gunthorpe
2021-05-19 16:50 ` Saleem, Shiraz
0 siblings, 1 reply; 35+ messages in thread
From: Jason Gunthorpe @ 2021-05-19 12:25 UTC (permalink / raw)
To: Saleem, Shiraz
Cc: Potnuri Bharat Teja, Dennis Dalessandro, Devesh Sharma,
Doug Ledford, Latif, Faisal, Gal Pressman, Leon Romanovsky,
linux-rdma, Mike Marciniszyn, Naresh Kumar PBS, Selvin Xavier,
Yossi Leybovich, Somnath Kotur, Sriharsha Basavapatna,
Yishai Hadas, Zhu Yanjun, Greg KH, Kees Cook, Nathan Chancellor
On Tue, May 18, 2021 at 09:58:28PM +0000, Saleem, Shiraz wrote:
> > Subject: Re: [PATCH 01/13] RDMA: Split the alloc_hw_stats() ops to port and
> > device variants
> >
> > On Tue, May 18, 2021 at 12:18:13AM +0000, Saleem, Shiraz wrote:
> >
> > > > What does the sysfs look like? Aren't there duplicated HW stats?
> > >
> > >
> > > Yeah it is duplicated. So we are saying for phys_port_cnt = 1, we want
> > > the stats to show up in only place?
> >
> > Yes.
> >
> > Imagine you had a multi port device, and assign the stats appropriately.
> >
> > I didn't see anything in the list that made me think "device stat" but I don't know
> > what several of these do
>
> Are we exporting port stat when it should be really be device stat in some of the drivers though?
>
> Most vendor drivers do port stats allocation only. Like...
> https://elixir.bootlin.com/linux/v5.13-rc2/source/drivers/infiniband/hw/cxgb4/provider.c#L392
> https://elixir.bootlin.com/linux/v5.13-rc2/source/drivers/infiniband/sw/rxe/rxe_hw_counters.c#L27
Yes they don't have global device counters, presumably because they
only have 1 port.
> However .get_hw_stats callback appears to extract from same set of registers for each port
Most devices should only have port counters
> > If you can confirm that these are all port stats I can delete the device stats in this
> > series.
>
> So your device and port stats op callback separation feels warranted.
So should I delete the device counters here?
Jason
^ permalink raw reply [flat|nested] 35+ messages in thread
* RE: [PATCH 01/13] RDMA: Split the alloc_hw_stats() ops to port and device variants
2021-05-19 12:25 ` Jason Gunthorpe
@ 2021-05-19 16:50 ` Saleem, Shiraz
0 siblings, 0 replies; 35+ messages in thread
From: Saleem, Shiraz @ 2021-05-19 16:50 UTC (permalink / raw)
To: Jason Gunthorpe
Cc: Potnuri Bharat Teja, Dennis Dalessandro, Devesh Sharma,
Doug Ledford, Latif, Faisal, Gal Pressman, Leon Romanovsky,
linux-rdma, Mike Marciniszyn, Naresh Kumar PBS, Selvin Xavier,
Yossi Leybovich, Somnath Kotur, Sriharsha Basavapatna,
Yishai Hadas, Zhu Yanjun, Greg KH, Kees Cook, Nathan Chancellor
> Subject: Re: [PATCH 01/13] RDMA: Split the alloc_hw_stats() ops to port and
> device variants
>
> On Tue, May 18, 2021 at 09:58:28PM +0000, Saleem, Shiraz wrote:
> > > Subject: Re: [PATCH 01/13] RDMA: Split the alloc_hw_stats() ops to
> > > port and device variants
> > >
> > > On Tue, May 18, 2021 at 12:18:13AM +0000, Saleem, Shiraz wrote:
> > >
> > > > > What does the sysfs look like? Aren't there duplicated HW stats?
> > > >
> > > >
> > > > Yeah it is duplicated. So we are saying for phys_port_cnt = 1, we
> > > > want the stats to show up in only place?
> > >
> > > Yes.
> > >
> > > Imagine you had a multi port device, and assign the stats appropriately.
> > >
> > > I didn't see anything in the list that made me think "device stat"
> > > but I don't know what several of these do
> >
> > Are we exporting port stat when it should be really be device stat in some of the
> drivers though?
> >
> > Most vendor drivers do port stats allocation only. Like...
> > https://elixir.bootlin.com/linux/v5.13-rc2/source/drivers/infiniband/h
> > w/cxgb4/provider.c#L392
> > https://elixir.bootlin.com/linux/v5.13-rc2/source/drivers/infiniband/s
> > w/rxe/rxe_hw_counters.c#L27
>
> Yes they don't have global device counters, presumably because they only have 1
> port.
>
> > However .get_hw_stats callback appears to extract from same set of
> > registers for each port
>
> Most devices should only have port counters
>
> > > If you can confirm that these are all port stats I can delete the
> > > device stats in this series.
> >
> > So your device and port stats op callback separation feels warranted.
>
> So should I delete the device counters here?
For i40iw, yes. Just expose port stats and NULL out the device stat op.
^ permalink raw reply [flat|nested] 35+ messages in thread
* Re: [PATCH 01/13] RDMA: Split the alloc_hw_stats() ops to port and device variants
2021-05-17 16:47 ` [PATCH 01/13] RDMA: Split the alloc_hw_stats() ops to port and device variants Jason Gunthorpe
2021-05-17 23:06 ` Saleem, Shiraz
@ 2021-05-18 3:49 ` Mark Zhang
2021-05-19 1:10 ` Saleem, Shiraz
2021-05-20 16:29 ` Jason Gunthorpe
2021-05-19 11:29 ` Gal Pressman
2 siblings, 2 replies; 35+ messages in thread
From: Mark Zhang @ 2021-05-18 3:49 UTC (permalink / raw)
To: Jason Gunthorpe, Potnuri Bharat Teja, Dennis Dalessandro,
Devesh Sharma, Doug Ledford, Faisal Latif, Gal Pressman,
Leon Romanovsky, linux-rdma, Mike Marciniszyn, Naresh Kumar PBS,
Selvin Xavier, Shiraz Saleem, Yossi Leybovich, Somnath Kotur,
Sriharsha Basavapatna, Yishai Hadas, Zhu Yanjun
Cc: Greg KH, Kees Cook, Nathan Chancellor
On 5/18/2021 12:47 AM, Jason Gunthorpe wrote:
> External email: Use caution opening links or attachments
>
>
> This is being used to implement both the port and device global stats,
> which is causing some confusion in the drivers. For instance EFA and i40iw
> both seem to be misusing the device stats.
>
> Split it into two ops so drivers that don't support one or the other can
> leave the op NULL'd, making the calling code a little simpler to
> understand.
>
> Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>
> ---
> drivers/infiniband/core/counters.c | 4 +-
> drivers/infiniband/core/device.c | 3 +-
> drivers/infiniband/core/nldev.c | 2 +-
> drivers/infiniband/core/sysfs.c | 15 +++-
> drivers/infiniband/hw/bnxt_re/hw_counters.c | 7 +-
> drivers/infiniband/hw/bnxt_re/hw_counters.h | 4 +-
> drivers/infiniband/hw/bnxt_re/main.c | 2 +-
> drivers/infiniband/hw/cxgb4/provider.c | 9 +--
> drivers/infiniband/hw/efa/efa.h | 3 +-
> drivers/infiniband/hw/efa/efa_main.c | 3 +-
> drivers/infiniband/hw/efa/efa_verbs.c | 11 ++-
> drivers/infiniband/hw/hfi1/verbs.c | 86 ++++++++++-----------
> drivers/infiniband/hw/i40iw/i40iw_verbs.c | 19 ++++-
> drivers/infiniband/hw/mlx4/main.c | 25 ++++--
> drivers/infiniband/hw/mlx5/counters.c | 42 +++++++---
> drivers/infiniband/sw/rxe/rxe_hw_counters.c | 7 +-
> drivers/infiniband/sw/rxe/rxe_hw_counters.h | 4 +-
> drivers/infiniband/sw/rxe/rxe_verbs.c | 2 +-
> include/rdma/ib_verbs.h | 13 ++--
> 19 files changed, 158 insertions(+), 103 deletions(-)
>
[...]
> diff --git a/drivers/infiniband/core/sysfs.c b/drivers/infiniband/core/sysfs.c
> index 05b702de00e89b..29082d8d45fc4f 100644
> --- a/drivers/infiniband/core/sysfs.c
> +++ b/drivers/infiniband/core/sysfs.c
> @@ -981,8 +981,15 @@ static void setup_hw_stats(struct ib_device *device, struct ib_port *port,
> struct rdma_hw_stats *stats;
> int i, ret;
>
> - stats = device->ops.alloc_hw_stats(device, port_num);
> -
> + if (port_num) {
> + if (!device->ops.alloc_hw_port_stats)
> + return;
Do we need this "if (!device->ops.alloc_hw_port_stats)" here?
> + stats = device->ops.alloc_hw_port_stats(device, port_num);
> + } else {
> + if (!device->ops.alloc_hw_device_stats)
> + return;
And here?
> + stats = device->ops.alloc_hw_device_stats(device);
> + }
> if (!stats)
> return;
>
> @@ -1165,7 +1172,7 @@ static int add_port(struct ib_core_device *coredev, int port_num)
> * port, so holder should be device. Therefore skip per port conunter
> * initialization.
> */
> - if (device->ops.alloc_hw_stats && port_num && is_full_dev)
> + if (device->ops.alloc_hw_port_stats && port_num && is_full_dev)
> setup_hw_stats(device, p, port_num);
>
> list_add_tail(&p->kobj.entry, &coredev->port_list);
> @@ -1409,7 +1416,7 @@ int ib_device_register_sysfs(struct ib_device *device)
> if (ret)
> return ret;
>
> - if (device->ops.alloc_hw_stats)
> + if (device->ops.alloc_hw_device_stats)
> setup_hw_stats(device, NULL, 0);
>
> return 0;
[...]
> diff --git a/drivers/infiniband/hw/cxgb4/provider.c b/drivers/infiniband/hw/cxgb4/provider.c
> index 3f1893e180ddf3..c0f01799f4a0b9 100644
> --- a/drivers/infiniband/hw/cxgb4/provider.c
> +++ b/drivers/infiniband/hw/cxgb4/provider.c
> @@ -377,14 +377,11 @@ static const char * const names[] = {
> [IP6OUTRSTS] = "ip6OutRsts"
> };
>
> -static struct rdma_hw_stats *c4iw_alloc_stats(struct ib_device *ibdev,
> - u32 port_num)
> +static struct rdma_hw_stats *c4iw_alloc_port_stats(struct ib_device *ibdev,
> + u32 port_num)
> {
> BUILD_BUG_ON(ARRAY_SIZE(names) != NR_COUNTERS);
>
> - if (port_num != 0)
> - return NULL;
> -
I'm not familiar with this driver, but if port_num must be 0 here, does
it mean this is per-device not per-port?
^ permalink raw reply [flat|nested] 35+ messages in thread* RE: [PATCH 01/13] RDMA: Split the alloc_hw_stats() ops to port and device variants
2021-05-18 3:49 ` Mark Zhang
@ 2021-05-19 1:10 ` Saleem, Shiraz
2021-05-20 16:29 ` Jason Gunthorpe
1 sibling, 0 replies; 35+ messages in thread
From: Saleem, Shiraz @ 2021-05-19 1:10 UTC (permalink / raw)
To: Mark Zhang, Jason Gunthorpe, Potnuri Bharat Teja,
Dennis Dalessandro, Devesh Sharma, Doug Ledford, Latif, Faisal,
Gal Pressman, Leon Romanovsky, linux-rdma, Mike Marciniszyn,
Naresh Kumar PBS, Selvin Xavier, Yossi Leybovich, Somnath Kotur,
Sriharsha Basavapatna, Yishai Hadas, Zhu Yanjun
Cc: Greg KH, Kees Cook, Nathan Chancellor
> Subject: Re: [PATCH 01/13] RDMA: Split the alloc_hw_stats() ops to port and
> device variants
>
> On 5/18/2021 12:47 AM, Jason Gunthorpe wrote:
> > External email: Use caution opening links or attachments
> >
> >
> > This is being used to implement both the port and device global stats,
> > which is causing some confusion in the drivers. For instance EFA and
> > i40iw both seem to be misusing the device stats.
> >
> > Split it into two ops so drivers that don't support one or the other
> > can leave the op NULL'd, making the calling code a little simpler to
> > understand.
> >
> > Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>
> > ---
[...]
> > -static struct rdma_hw_stats *c4iw_alloc_stats(struct ib_device *ibdev,
> > - u32 port_num)
> > +static struct rdma_hw_stats *c4iw_alloc_port_stats(struct ib_device *ibdev,
> > + u32 port_num)
> > {
> > BUILD_BUG_ON(ARRAY_SIZE(names) != NR_COUNTERS);
> >
> > - if (port_num != 0)
> > - return NULL;
> > -
>
> I'm not familiar with this driver, but if port_num must be 0 here, does it mean this is
> per-device not per-port?
Yeah, additionally per-device seems to coincide with the behavior in get_hw_stats callback for cxgb4.
Shiraz
^ permalink raw reply [flat|nested] 35+ messages in thread* Re: [PATCH 01/13] RDMA: Split the alloc_hw_stats() ops to port and device variants
2021-05-18 3:49 ` Mark Zhang
2021-05-19 1:10 ` Saleem, Shiraz
@ 2021-05-20 16:29 ` Jason Gunthorpe
1 sibling, 0 replies; 35+ messages in thread
From: Jason Gunthorpe @ 2021-05-20 16:29 UTC (permalink / raw)
To: Mark Zhang
Cc: Potnuri Bharat Teja, Dennis Dalessandro, Devesh Sharma,
Doug Ledford, Faisal Latif, Gal Pressman, Leon Romanovsky,
linux-rdma, Mike Marciniszyn, Naresh Kumar PBS, Selvin Xavier,
Shiraz Saleem, Yossi Leybovich, Somnath Kotur,
Sriharsha Basavapatna, Yishai Hadas, Zhu Yanjun, Greg KH,
Kees Cook, Nathan Chancellor
On Tue, May 18, 2021 at 11:49:57AM +0800, Mark Zhang wrote:
> On 5/18/2021 12:47 AM, Jason Gunthorpe wrote:
> > diff --git a/drivers/infiniband/core/sysfs.c b/drivers/infiniband/core/sysfs.c
> > index 05b702de00e89b..29082d8d45fc4f 100644
> > +++ b/drivers/infiniband/core/sysfs.c
> > @@ -981,8 +981,15 @@ static void setup_hw_stats(struct ib_device *device, struct ib_port *port,
> > struct rdma_hw_stats *stats;
> > int i, ret;
> >
> > - stats = device->ops.alloc_hw_stats(device, port_num);
> > -
> > + if (port_num) {
> > + if (!device->ops.alloc_hw_port_stats)
> > + return;
>
> Do we need this "if (!device->ops.alloc_hw_port_stats)" here?
Not really in this patch, one of the next patches needs them near
here, but there isn't a good reason to leave them in this patch at
this point.
> > index 3f1893e180ddf3..c0f01799f4a0b9 100644
> > +++ b/drivers/infiniband/hw/cxgb4/provider.c
> > @@ -377,14 +377,11 @@ static const char * const names[] = {
> > [IP6OUTRSTS] = "ip6OutRsts"
> > };
> >
> > -static struct rdma_hw_stats *c4iw_alloc_stats(struct ib_device *ibdev,
> > - u32 port_num)
> > +static struct rdma_hw_stats *c4iw_alloc_port_stats(struct ib_device *ibdev,
> > + u32 port_num)
> > {
> > BUILD_BUG_ON(ARRAY_SIZE(names) != NR_COUNTERS);
> >
> > - if (port_num != 0)
> > - return NULL;
> > -
>
> I'm not familiar with this driver, but if port_num must be 0 here, does it
> mean this is per-device not per-port?
Yes, I switched it, though the actual counters look per port to me.
Thanks,
Jason
^ permalink raw reply [flat|nested] 35+ messages in thread
* Re: [PATCH 01/13] RDMA: Split the alloc_hw_stats() ops to port and device variants
2021-05-17 16:47 ` [PATCH 01/13] RDMA: Split the alloc_hw_stats() ops to port and device variants Jason Gunthorpe
2021-05-17 23:06 ` Saleem, Shiraz
2021-05-18 3:49 ` Mark Zhang
@ 2021-05-19 11:29 ` Gal Pressman
2021-05-19 13:44 ` Jason Gunthorpe
2 siblings, 1 reply; 35+ messages in thread
From: Gal Pressman @ 2021-05-19 11:29 UTC (permalink / raw)
To: Jason Gunthorpe, Potnuri Bharat Teja, Dennis Dalessandro,
Devesh Sharma, Doug Ledford, Faisal Latif, Leon Romanovsky,
linux-rdma, Mike Marciniszyn, Naresh Kumar PBS, Selvin Xavier,
Shiraz Saleem, Yossi Leybovich, Somnath Kotur,
Sriharsha Basavapatna, Yishai Hadas, Zhu Yanjun
Cc: Greg KH, Kees Cook, Nathan Chancellor
On 17/05/2021 19:47, Jason Gunthorpe wrote:
> +struct rdma_hw_stats *efa_alloc_hw_device_stats(struct ib_device *ibdev)
> +{
> + /*
> + * It is probably a bug that efa reports its port stats as device
> + * stats
> + */
Hmm, yea this needs some work.
Most of these stats are in fact port stats, but others (admin commands,
keep-alive, allocation/creation error) are device stats.
You can split them if you wish, or I can send a followup patch.
Thanks,
Tested-by: Gal Pressman <galpress@amazon.com>
^ permalink raw reply [flat|nested] 35+ messages in thread* Re: [PATCH 01/13] RDMA: Split the alloc_hw_stats() ops to port and device variants
2021-05-19 11:29 ` Gal Pressman
@ 2021-05-19 13:44 ` Jason Gunthorpe
0 siblings, 0 replies; 35+ messages in thread
From: Jason Gunthorpe @ 2021-05-19 13:44 UTC (permalink / raw)
To: Gal Pressman
Cc: Potnuri Bharat Teja, Dennis Dalessandro, Devesh Sharma,
Doug Ledford, Faisal Latif, Leon Romanovsky, linux-rdma,
Mike Marciniszyn, Naresh Kumar PBS, Selvin Xavier, Shiraz Saleem,
Yossi Leybovich, Somnath Kotur, Sriharsha Basavapatna,
Yishai Hadas, Zhu Yanjun, Greg KH, Kees Cook, Nathan Chancellor
On Wed, May 19, 2021 at 02:29:57PM +0300, Gal Pressman wrote:
> On 17/05/2021 19:47, Jason Gunthorpe wrote:
> > +struct rdma_hw_stats *efa_alloc_hw_device_stats(struct ib_device *ibdev)
> > +{
> > + /*
> > + * It is probably a bug that efa reports its port stats as device
> > + * stats
> > + */
>
> Hmm, yea this needs some work.
> Most of these stats are in fact port stats, but others (admin commands,
> keep-alive, allocation/creation error) are device stats.
>
> You can split them if you wish, or I can send a followup patch.
I'm not sure I'd guess right, you'd probably better take it
> Thanks,
> Tested-by: Gal Pressman <galpress@amazon.com>
Thanks,
Jason
^ permalink raw reply [flat|nested] 35+ messages in thread
* [PATCH 02/13] RDMA/core: Replace the ib_port_data hw_stats pointers with a ib_port pointer
2021-05-17 16:47 [PATCH 00/13] Reorganize sysfs file creation for struct ib_devices Jason Gunthorpe
2021-05-17 16:47 ` [PATCH 01/13] RDMA: Split the alloc_hw_stats() ops to port and device variants Jason Gunthorpe
@ 2021-05-17 16:47 ` Jason Gunthorpe
2021-05-17 16:47 ` [PATCH 03/13] RDMA/core: Split port and device counter sysfs attributes Jason Gunthorpe
` (11 subsequent siblings)
13 siblings, 0 replies; 35+ messages in thread
From: Jason Gunthorpe @ 2021-05-17 16:47 UTC (permalink / raw)
To: Doug Ledford, linux-rdma; +Cc: Greg KH, Kees Cook, Nathan Chancellor
It is much saner to store a pointer to the kobject structure that contains
the cannonical stats pointer than to copy the stats pointers into a public
structure.
Future patches will require the sysfs pointer for other purposes.
Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>
---
drivers/infiniband/core/core_priv.h | 1 +
drivers/infiniband/core/nldev.c | 8 ++------
drivers/infiniband/core/sysfs.c | 14 +++++++++++---
include/rdma/ib_verbs.h | 3 ++-
4 files changed, 16 insertions(+), 10 deletions(-)
diff --git a/drivers/infiniband/core/core_priv.h b/drivers/infiniband/core/core_priv.h
index 29809dd300419f..ec5c2c3db42303 100644
--- a/drivers/infiniband/core/core_priv.h
+++ b/drivers/infiniband/core/core_priv.h
@@ -378,6 +378,7 @@ struct net_device *rdma_read_gid_attr_ndev_rcu(const struct ib_gid_attr *attr);
void ib_free_port_attrs(struct ib_core_device *coredev);
int ib_setup_port_attrs(struct ib_core_device *coredev);
+struct rdma_hw_stats *ib_get_hw_stats_port(struct ib_device *ibdev, u32 port_num);
int rdma_compatdev_set(u8 enable);
diff --git a/drivers/infiniband/core/nldev.c b/drivers/infiniband/core/nldev.c
index 01316926cef63d..e9b4b2cccaa0ff 100644
--- a/drivers/infiniband/core/nldev.c
+++ b/drivers/infiniband/core/nldev.c
@@ -2066,7 +2066,8 @@ static int stat_get_doit_default_counter(struct sk_buff *skb,
}
port = nla_get_u32(tb[RDMA_NLDEV_ATTR_PORT_INDEX]);
- if (!rdma_is_port_valid(device, port)) {
+ stats = ib_get_hw_stats_port(device, port);
+ if (!stats) {
ret = -EINVAL;
goto err;
}
@@ -2088,11 +2089,6 @@ static int stat_get_doit_default_counter(struct sk_buff *skb,
goto err_msg;
}
- stats = device->port_data ? device->port_data[port].hw_stats : NULL;
- if (stats == NULL) {
- ret = -EINVAL;
- goto err_msg;
- }
mutex_lock(&stats->lock);
num_cnts = device->ops.get_hw_stats(device, stats, port, 0);
diff --git a/drivers/infiniband/core/sysfs.c b/drivers/infiniband/core/sysfs.c
index 29082d8d45fc4f..5f8b1677e1237b 100644
--- a/drivers/infiniband/core/sysfs.c
+++ b/drivers/infiniband/core/sysfs.c
@@ -1036,8 +1036,6 @@ static void setup_hw_stats(struct ib_device *device, struct ib_port *port,
goto err;
port->hw_stats_ag = hsag;
port->hw_stats = stats;
- if (device->port_data)
- device->port_data[port_num].hw_stats = stats;
} else {
struct kobject *kobj = &device->dev.kobj;
ret = sysfs_create_group(kobj, hsag);
@@ -1058,6 +1056,14 @@ static void setup_hw_stats(struct ib_device *device, struct ib_port *port,
kfree(stats);
}
+struct rdma_hw_stats *ib_get_hw_stats_port(struct ib_device *ibdev,
+ u32 port_num)
+{
+ if (!ibdev->port_data || !rdma_is_port_valid(ibdev, port_num))
+ return NULL;
+ return ibdev->port_data[port_num].sysfs->hw_stats;
+}
+
static int add_port(struct ib_core_device *coredev, int port_num)
{
struct ib_device *device = rdma_device_to_ibdev(&coredev->dev);
@@ -1176,6 +1182,8 @@ static int add_port(struct ib_core_device *coredev, int port_num)
setup_hw_stats(device, p, port_num);
list_add_tail(&p->kobj.entry, &coredev->port_list);
+ if (device->port_data && is_full_dev)
+ device->port_data[port_num].sysfs = p;
kobject_uevent(&p->kobj, KOBJ_ADD);
return 0;
@@ -1366,7 +1374,7 @@ void ib_free_port_attrs(struct ib_core_device *coredev)
free_hsag(&port->kobj, port->hw_stats_ag);
kfree(port->hw_stats);
if (device->port_data && is_full_dev)
- device->port_data[port->port_num].hw_stats = NULL;
+ device->port_data[port->port_num].sysfs = NULL;
if (port->pma_table)
sysfs_remove_group(p, port->pma_table);
diff --git a/include/rdma/ib_verbs.h b/include/rdma/ib_verbs.h
index 8fa7fd20e94aa7..e3be93e7096616 100644
--- a/include/rdma/ib_verbs.h
+++ b/include/rdma/ib_verbs.h
@@ -50,6 +50,7 @@ struct ib_uqp_object;
struct ib_usrq_object;
struct ib_uwq_object;
struct rdma_cm_id;
+struct ib_port;
extern struct workqueue_struct *ib_wq;
extern struct workqueue_struct *ib_comp_wq;
@@ -2183,7 +2184,7 @@ struct ib_port_data {
struct net_device __rcu *netdev;
struct hlist_node ndev_hash_link;
struct rdma_port_counter port_counter;
- struct rdma_hw_stats *hw_stats;
+ struct ib_port *sysfs;
};
/* rdma netdev type - specifies protocol type */
--
2.31.1
^ permalink raw reply related [flat|nested] 35+ messages in thread* [PATCH 03/13] RDMA/core: Split port and device counter sysfs attributes
2021-05-17 16:47 [PATCH 00/13] Reorganize sysfs file creation for struct ib_devices Jason Gunthorpe
2021-05-17 16:47 ` [PATCH 01/13] RDMA: Split the alloc_hw_stats() ops to port and device variants Jason Gunthorpe
2021-05-17 16:47 ` [PATCH 02/13] RDMA/core: Replace the ib_port_data hw_stats pointers with a ib_port pointer Jason Gunthorpe
@ 2021-05-17 16:47 ` Jason Gunthorpe
2021-05-17 16:47 ` [PATCH 04/13] RDMA/core: Split gid_attrs related sysfs from add_port() Jason Gunthorpe
` (10 subsequent siblings)
13 siblings, 0 replies; 35+ messages in thread
From: Jason Gunthorpe @ 2021-05-17 16:47 UTC (permalink / raw)
To: clang-built-linux, Doug Ledford, linux-rdma, Nick Desaulniers
Cc: Greg KH, Kees Cook, Nathan Chancellor
This code creates a 'struct hw_stats_attribute' for each sysfs entry that
contains a naked 'struct attribute' inside.
It then proceeds to attach this same structure to a 'struct device' kobj
and a 'struct ib_port' kobj. However, this violates the typing
requirements. 'struct device' requires the attribute to be a 'struct
device_attribute' and 'struct ib_port' requires the attribute to be
'struct port_attribute'.
This happens to work because the show/store function pointers in all three
structures happen to be at the same offset and happen to be nearly the
same signature. This means when container_of() was used to go between the
wrong two types it still managed to work.
However clang CFI detection notices that the function pointers have a
slightly different signature. As with show/store this was only working
because the device and port struct layouts happened to have the kobj at
the front.
Correct this by have two independent sets of data structures for the port
and device case. The two different attributes correctly include the
port/device_attribute struct and everything from there up is kept
split. The show/store function call chains start with device/port unique
functions that invoke a common show/store function pointer.
Reported-by: Nathan Chancellor <nathan@kernel.org>
Cc: Kees Cook <keescook@chromium.org>
Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>
---
drivers/infiniband/core/sysfs.c | 463 ++++++++++++++++++++------------
include/rdma/ib_verbs.h | 4 +-
2 files changed, 292 insertions(+), 175 deletions(-)
diff --git a/drivers/infiniband/core/sysfs.c b/drivers/infiniband/core/sysfs.c
index 5f8b1677e1237b..114fecda97648e 100644
--- a/drivers/infiniband/core/sysfs.c
+++ b/drivers/infiniband/core/sysfs.c
@@ -60,8 +60,7 @@ struct ib_port {
struct attribute_group gid_group;
struct attribute_group *pkey_group;
const struct attribute_group *pma_table;
- struct attribute_group *hw_stats_ag;
- struct rdma_hw_stats *hw_stats;
+ struct hw_stats_port_data *hw_stats_data;
u32 port_num;
};
@@ -85,16 +84,35 @@ struct port_table_attribute {
__be16 attr_id;
};
-struct hw_stats_attribute {
- struct attribute attr;
- ssize_t (*show)(struct kobject *kobj,
- struct attribute *attr, char *buf);
- ssize_t (*store)(struct kobject *kobj,
- struct attribute *attr,
- const char *buf,
- size_t count);
- int index;
- u32 port_num;
+struct hw_stats_device_attribute {
+ struct device_attribute attr;
+ ssize_t (*show)(struct ib_device *ibdev, struct rdma_hw_stats *stats,
+ unsigned int index, unsigned int port_num, char *buf);
+ ssize_t (*store)(struct ib_device *ibdev, struct rdma_hw_stats *stats,
+ unsigned int index, unsigned int port_num,
+ const char *buf, size_t count);
+};
+
+struct hw_stats_port_attribute {
+ struct port_attribute attr;
+ ssize_t (*show)(struct ib_device *ibdev, struct rdma_hw_stats *stats,
+ unsigned int index, unsigned int port_num, char *buf);
+ ssize_t (*store)(struct ib_device *ibdev, struct rdma_hw_stats *stats,
+ unsigned int index, unsigned int port_num,
+ const char *buf, size_t count);
+};
+
+struct hw_stats_device_data {
+ struct attribute_group group;
+ const struct attribute_group *groups[2];
+ struct rdma_hw_stats *stats;
+ struct hw_stats_device_attribute attrs[];
+};
+
+struct hw_stats_port_data {
+ struct attribute_group group;
+ struct rdma_hw_stats *stats;
+ struct hw_stats_port_attribute attrs[];
};
static ssize_t port_attr_show(struct kobject *kobj,
@@ -128,6 +146,53 @@ static const struct sysfs_ops port_sysfs_ops = {
.store = port_attr_store
};
+static ssize_t hw_stat_device_show(struct device *dev,
+ struct device_attribute *attr, char *buf)
+{
+ struct hw_stats_device_attribute *stat_attr =
+ container_of(attr, struct hw_stats_device_attribute, attr);
+ struct ib_device *ibdev = container_of(dev, struct ib_device, dev);
+
+ return stat_attr->show(ibdev, ibdev->hw_stats_data->stats,
+ stat_attr - ibdev->hw_stats_data->attrs, 0, buf);
+}
+
+static ssize_t hw_stat_device_store(struct device *dev,
+ struct device_attribute *attr,
+ const char *buf, size_t count)
+{
+ struct hw_stats_device_attribute *stat_attr =
+ container_of(attr, struct hw_stats_device_attribute, attr);
+ struct ib_device *ibdev = container_of(dev, struct ib_device, dev);
+
+ return stat_attr->store(ibdev, ibdev->hw_stats_data->stats,
+ stat_attr - ibdev->hw_stats_data->attrs, 0, buf,
+ count);
+}
+
+static ssize_t hw_stat_port_show(struct ib_port *port,
+ struct port_attribute *attr, char *buf)
+{
+ struct hw_stats_port_attribute *stat_attr =
+ container_of(attr, struct hw_stats_port_attribute, attr);
+
+ return stat_attr->show(port->ibdev, port->hw_stats_data->stats,
+ stat_attr - port->hw_stats_data->attrs,
+ port->port_num, buf);
+}
+
+static ssize_t hw_stat_port_store(struct ib_port *port,
+ struct port_attribute *attr, const char *buf,
+ size_t count)
+{
+ struct hw_stats_port_attribute *stat_attr =
+ container_of(attr, struct hw_stats_port_attribute, attr);
+
+ return stat_attr->store(port->ibdev, port->hw_stats_data->stats,
+ stat_attr - port->hw_stats_data->attrs,
+ port->port_num, buf, count);
+}
+
static ssize_t gid_attr_show(struct kobject *kobj,
struct attribute *attr, char *buf)
{
@@ -835,56 +900,30 @@ static int print_hw_stat(struct ib_device *dev, int port_num,
return sysfs_emit(buf, "%llu\n", stats->value[index] + v);
}
-static ssize_t show_hw_stats(struct kobject *kobj, struct attribute *attr,
- char *buf)
+static ssize_t show_hw_stats(struct ib_device *ibdev,
+ struct rdma_hw_stats *stats, unsigned int index,
+ unsigned int port_num, char *buf)
{
- struct ib_device *dev;
- struct ib_port *port;
- struct hw_stats_attribute *hsa;
- struct rdma_hw_stats *stats;
int ret;
- hsa = container_of(attr, struct hw_stats_attribute, attr);
- if (!hsa->port_num) {
- dev = container_of((struct device *)kobj,
- struct ib_device, dev);
- stats = dev->hw_stats;
- } else {
- port = container_of(kobj, struct ib_port, kobj);
- dev = port->ibdev;
- stats = port->hw_stats;
- }
mutex_lock(&stats->lock);
- ret = update_hw_stats(dev, stats, hsa->port_num, hsa->index);
+ ret = update_hw_stats(ibdev, stats, port_num, index);
if (ret)
goto unlock;
- ret = print_hw_stat(dev, hsa->port_num, stats, hsa->index, buf);
+ ret = print_hw_stat(ibdev, port_num, stats, index, buf);
unlock:
mutex_unlock(&stats->lock);
return ret;
}
-static ssize_t show_stats_lifespan(struct kobject *kobj,
- struct attribute *attr,
+static ssize_t show_stats_lifespan(struct ib_device *ibdev,
+ struct rdma_hw_stats *stats,
+ unsigned int index, unsigned int port_num,
char *buf)
{
- struct hw_stats_attribute *hsa;
- struct rdma_hw_stats *stats;
int msecs;
- hsa = container_of(attr, struct hw_stats_attribute, attr);
- if (!hsa->port_num) {
- struct ib_device *dev = container_of((struct device *)kobj,
- struct ib_device, dev);
-
- stats = dev->hw_stats;
- } else {
- struct ib_port *p = container_of(kobj, struct ib_port, kobj);
-
- stats = p->hw_stats;
- }
-
mutex_lock(&stats->lock);
msecs = jiffies_to_msecs(stats->lifespan);
mutex_unlock(&stats->lock);
@@ -892,12 +931,11 @@ static ssize_t show_stats_lifespan(struct kobject *kobj,
return sysfs_emit(buf, "%d\n", msecs);
}
-static ssize_t set_stats_lifespan(struct kobject *kobj,
- struct attribute *attr,
- const char *buf, size_t count)
+static ssize_t set_stats_lifespan(struct ib_device *ibdev,
+ struct rdma_hw_stats *stats,
+ unsigned int index, unsigned int port_num,
+ const char *buf, size_t count)
{
- struct hw_stats_attribute *hsa;
- struct rdma_hw_stats *stats;
int msecs;
int jiffies;
int ret;
@@ -908,17 +946,6 @@ static ssize_t set_stats_lifespan(struct kobject *kobj,
if (msecs < 0 || msecs > 10000)
return -EINVAL;
jiffies = msecs_to_jiffies(msecs);
- hsa = container_of(attr, struct hw_stats_attribute, attr);
- if (!hsa->port_num) {
- struct ib_device *dev = container_of((struct device *)kobj,
- struct ib_device, dev);
-
- stats = dev->hw_stats;
- } else {
- struct ib_port *p = container_of(kobj, struct ib_port, kobj);
-
- stats = p->hw_stats;
- }
mutex_lock(&stats->lock);
stats->lifespan = jiffies;
@@ -927,72 +954,125 @@ static ssize_t set_stats_lifespan(struct kobject *kobj,
return count;
}
-static void free_hsag(struct kobject *kobj, struct attribute_group *attr_group)
+static struct hw_stats_device_data *
+alloc_hw_stats_device(struct ib_device *ibdev)
{
- struct attribute **attr;
+ struct hw_stats_device_data *data;
+ struct rdma_hw_stats *stats;
+
+ if (!ibdev->ops.alloc_hw_device_stats)
+ return ERR_PTR(-EOPNOTSUPP);
+ stats = ibdev->ops.alloc_hw_device_stats(ibdev);
+ if (!stats)
+ return ERR_PTR(-ENOMEM);
+ if (!stats->names || stats->num_counters <= 0)
+ goto err_free_stats;
+
+ /*
+ * Two extra attribue elements here, one for the lifespan entry and
+ * one to NULL terminate the list for the sysfs core code
+ */
+ data = kzalloc(struct_size(data, attrs, stats->num_counters + 1),
+ GFP_KERNEL);
+ if (!data)
+ goto err_free_stats;
+ data->group.attrs = kcalloc(stats->num_counters + 2,
+ sizeof(*data->group.attrs), GFP_KERNEL);
+ if (!data->group.attrs)
+ goto err_free_data;
- sysfs_remove_group(kobj, attr_group);
+ mutex_init(&stats->lock);
+ data->group.name = "hw_counters";
+ data->stats = stats;
+ data->groups[0] = &data->group;
+ return data;
- for (attr = attr_group->attrs; *attr; attr++)
- kfree(*attr);
- kfree(attr_group);
+err_free_data:
+ kfree(data);
+err_free_stats:
+ kfree(stats);
+ return ERR_PTR(-ENOMEM);
}
-static struct attribute *alloc_hsa(int index, u32 port_num, const char *name)
+static void free_hw_stats_device(struct hw_stats_device_data *data)
{
- struct hw_stats_attribute *hsa;
+ kfree(data->group.attrs);
+ kfree(data->stats);
+ kfree(data);
+}
- hsa = kmalloc(sizeof(*hsa), GFP_KERNEL);
- if (!hsa)
- return NULL;
+static int setup_hw_device_stats(struct ib_device *ibdev)
+{
+ struct hw_stats_device_attribute *attr;
+ struct hw_stats_device_data *data;
+ int i, ret;
- hsa->attr.name = (char *)name;
- hsa->attr.mode = S_IRUGO;
- hsa->show = show_hw_stats;
- hsa->store = NULL;
- hsa->index = index;
- hsa->port_num = port_num;
+ data = alloc_hw_stats_device(ibdev);
+ if (IS_ERR(data))
+ return PTR_ERR(data);
- return &hsa->attr;
-}
+ ret = ibdev->ops.get_hw_stats(ibdev, data->stats, 0,
+ data->stats->num_counters);
+ if (ret != data->stats->num_counters) {
+ if (WARN_ON(ret >= 0))
+ ret = -EINVAL;
+ goto err_free;
+ }
-static struct attribute *alloc_hsa_lifespan(char *name, u32 port_num)
-{
- struct hw_stats_attribute *hsa;
+ data->stats->timestamp = jiffies;
- hsa = kmalloc(sizeof(*hsa), GFP_KERNEL);
- if (!hsa)
- return NULL;
+ for (i = 0; i < data->stats->num_counters; i++) {
+ attr = &data->attrs[i];
+ sysfs_attr_init(&attr->attr.attr);
+ attr->attr.attr.name = data->stats->names[i];
+ attr->attr.attr.mode = 0444;
+ attr->attr.show = hw_stat_device_show;
+ attr->show = show_hw_stats;
+ data->group.attrs[i] = &attr->attr.attr;
+ }
- hsa->attr.name = name;
- hsa->attr.mode = S_IWUSR | S_IRUGO;
- hsa->show = show_stats_lifespan;
- hsa->store = set_stats_lifespan;
- hsa->index = 0;
- hsa->port_num = port_num;
+ attr = &data->attrs[i];
+ sysfs_attr_init(&attr->attr.attr);
+ attr->attr.attr.name = "lifespan";
+ attr->attr.attr.mode = 0644;
+ attr->attr.show = hw_stat_device_show;
+ attr->show = show_stats_lifespan;
+ attr->attr.store = hw_stat_device_store;
+ attr->store = set_stats_lifespan;
+ data->group.attrs[i] = &attr->attr.attr;
+
+ ibdev->hw_stats_data = data;
+ ret = device_add_groups(&ibdev->dev, data->groups);
+ if (ret)
+ goto err_free;
+ return 0;
- return &hsa->attr;
+err_free:
+ free_hw_stats_device(data);
+ ibdev->hw_stats_data = NULL;
+ return ret;
}
-static void setup_hw_stats(struct ib_device *device, struct ib_port *port,
- u32 port_num)
+static void destroy_hw_device_stats(struct ib_device *ibdev)
{
- struct attribute_group *hsag;
+ if (!ibdev->hw_stats_data)
+ return;
+ device_remove_groups(&ibdev->dev, ibdev->hw_stats_data->groups);
+ free_hw_stats_device(ibdev->hw_stats_data);
+ ibdev->hw_stats_data = NULL;
+}
+
+static struct hw_stats_port_data *alloc_hw_stats_port(struct ib_port *port)
+{
+ struct ib_device *ibdev = port->ibdev;
+ struct hw_stats_port_data *data;
struct rdma_hw_stats *stats;
- int i, ret;
- if (port_num) {
- if (!device->ops.alloc_hw_port_stats)
- return;
- stats = device->ops.alloc_hw_port_stats(device, port_num);
- } else {
- if (!device->ops.alloc_hw_device_stats)
- return;
- stats = device->ops.alloc_hw_device_stats(device);
- }
+ if (!ibdev->ops.alloc_hw_port_stats)
+ return ERR_PTR(-EOPNOTSUPP);
+ stats = ibdev->ops.alloc_hw_port_stats(port->ibdev, port->port_num);
if (!stats)
- return;
-
+ return ERR_PTR(-ENOMEM);
if (!stats->names || stats->num_counters <= 0)
goto err_free_stats;
@@ -1000,68 +1080,102 @@ static void setup_hw_stats(struct ib_device *device, struct ib_port *port,
* Two extra attribue elements here, one for the lifespan entry and
* one to NULL terminate the list for the sysfs core code
*/
- hsag = kzalloc(sizeof(*hsag) +
- sizeof(void *) * (stats->num_counters + 2),
+ data = kzalloc(struct_size(data, attrs, stats->num_counters + 1),
GFP_KERNEL);
- if (!hsag)
+ if (!data)
goto err_free_stats;
+ data->group.attrs = kcalloc(stats->num_counters + 2,
+ sizeof(*data->group.attrs), GFP_KERNEL);
+ if (!data->group.attrs)
+ goto err_free_data;
- ret = device->ops.get_hw_stats(device, stats, port_num,
- stats->num_counters);
- if (ret != stats->num_counters)
- goto err_free_hsag;
+ mutex_init(&stats->lock);
+ data->group.name = "hw_counters";
+ data->stats = stats;
+ return data;
- stats->timestamp = jiffies;
+err_free_data:
+ kfree(data);
+err_free_stats:
+ kfree(stats);
+ return ERR_PTR(-ENOMEM);
+}
- hsag->name = "hw_counters";
- hsag->attrs = (void *)hsag + sizeof(*hsag);
+static void free_hw_stats_port(struct hw_stats_port_data *data)
+{
+ kfree(data->group.attrs);
+ kfree(data->stats);
+ kfree(data);
+}
- for (i = 0; i < stats->num_counters; i++) {
- hsag->attrs[i] = alloc_hsa(i, port_num, stats->names[i]);
- if (!hsag->attrs[i])
- goto err;
- sysfs_attr_init(hsag->attrs[i]);
- }
+static int setup_hw_port_stats(struct ib_port *port)
+{
+ struct hw_stats_port_attribute *attr;
+ struct hw_stats_port_data *data;
+ int i, ret;
- mutex_init(&stats->lock);
- /* treat an error here as non-fatal */
- hsag->attrs[i] = alloc_hsa_lifespan("lifespan", port_num);
- if (hsag->attrs[i])
- sysfs_attr_init(hsag->attrs[i]);
-
- if (port) {
- struct kobject *kobj = &port->kobj;
- ret = sysfs_create_group(kobj, hsag);
- if (ret)
- goto err;
- port->hw_stats_ag = hsag;
- port->hw_stats = stats;
- } else {
- struct kobject *kobj = &device->dev.kobj;
- ret = sysfs_create_group(kobj, hsag);
- if (ret)
- goto err;
- device->hw_stats_ag = hsag;
- device->hw_stats = stats;
+ data = alloc_hw_stats_port(port);
+ if (IS_ERR(data))
+ return PTR_ERR(data);
+
+ ret = port->ibdev->ops.get_hw_stats(port->ibdev, data->stats,
+ port->port_num,
+ data->stats->num_counters);
+ if (ret != data->stats->num_counters) {
+ if (WARN_ON(ret >= 0))
+ ret = -EINVAL;
+ goto err_free;
+ }
+ data->stats->timestamp = jiffies;
+
+ for (i = 0; i < data->stats->num_counters; i++) {
+ attr = &data->attrs[i];
+ sysfs_attr_init(&attr->attr.attr);
+ attr->attr.attr.name = data->stats->names[i];
+ attr->attr.attr.mode = 0444;
+ attr->attr.show = hw_stat_port_show;
+ attr->show = show_hw_stats;
+ data->group.attrs[i] = &attr->attr.attr;
}
- return;
+ attr = &data->attrs[i];
+ sysfs_attr_init(&attr->attr.attr);
+ attr->attr.attr.name = "lifespan";
+ attr->attr.attr.mode = 0644;
+ attr->attr.show = hw_stat_port_show;
+ attr->show = show_stats_lifespan;
+ attr->attr.store = hw_stat_port_store;
+ attr->store = set_stats_lifespan;
+ data->group.attrs[i] = &attr->attr.attr;
+
+ port->hw_stats_data = data;
+ ret = sysfs_create_group(&port->kobj, &data->group);
+ if (ret)
+ goto err_free;
+ return 0;
-err:
- for (; i >= 0; i--)
- kfree(hsag->attrs[i]);
-err_free_hsag:
- kfree(hsag);
-err_free_stats:
- kfree(stats);
+err_free:
+ free_hw_stats_port(data);
+ port->hw_stats_data = NULL;
+ return ret;
+}
+
+static void destroy_hw_port_stats(struct ib_port *port)
+{
+ if (!port->hw_stats_data)
+ return;
+ sysfs_remove_group(&port->kobj, &port->hw_stats_data->group);
+ free_hw_stats_port(port->hw_stats_data);
+ port->hw_stats_data = NULL;
}
struct rdma_hw_stats *ib_get_hw_stats_port(struct ib_device *ibdev,
u32 port_num)
{
- if (!ibdev->port_data || !rdma_is_port_valid(ibdev, port_num))
+ if (!ibdev->port_data || !rdma_is_port_valid(ibdev, port_num) ||
+ !ibdev->port_data[port_num].sysfs->hw_stats_data)
return NULL;
- return ibdev->port_data[port_num].sysfs->hw_stats;
+ return ibdev->port_data[port_num].sysfs->hw_stats_data->stats;
}
static int add_port(struct ib_core_device *coredev, int port_num)
@@ -1166,21 +1280,23 @@ static int add_port(struct ib_core_device *coredev, int port_num)
goto err_free_pkey;
}
+ /*
+ * If port == 0, it means hw_counters are per device and not per
+ * port, so holder should be device. Therefore skip per port
+ * counter initialization.
+ */
+ if (port_num && is_full_dev) {
+ ret = setup_hw_port_stats(p);
+ if (ret && ret != -EOPNOTSUPP)
+ goto err_remove_pkey;
+ }
if (device->ops.init_port && is_full_dev) {
ret = device->ops.init_port(device, port_num, &p->kobj);
if (ret)
- goto err_remove_pkey;
+ goto err_remove_stats;
}
- /*
- * If port == 0, it means hw_counters are per device and not per
- * port, so holder should be device. Therefore skip per port conunter
- * initialization.
- */
- if (device->ops.alloc_hw_port_stats && port_num && is_full_dev)
- setup_hw_stats(device, p, port_num);
-
list_add_tail(&p->kobj.entry, &coredev->port_list);
if (device->port_data && is_full_dev)
device->port_data[port_num].sysfs = p;
@@ -1188,6 +1304,9 @@ static int add_port(struct ib_core_device *coredev, int port_num)
kobject_uevent(&p->kobj, KOBJ_ADD);
return 0;
+err_remove_stats:
+ destroy_hw_port_stats(p);
+
err_remove_pkey:
if (p->pkey_group)
sysfs_remove_group(&p->kobj, p->pkey_group);
@@ -1370,9 +1489,7 @@ void ib_free_port_attrs(struct ib_core_device *coredev)
struct ib_port *port = container_of(p, struct ib_port, kobj);
list_del(&p->entry);
- if (port->hw_stats_ag)
- free_hsag(&port->kobj, port->hw_stats_ag);
- kfree(port->hw_stats);
+ destroy_hw_port_stats(port);
if (device->port_data && is_full_dev)
device->port_data[port->port_num].sysfs = NULL;
@@ -1424,18 +1541,18 @@ int ib_device_register_sysfs(struct ib_device *device)
if (ret)
return ret;
- if (device->ops.alloc_hw_device_stats)
- setup_hw_stats(device, NULL, 0);
+ ret = setup_hw_device_stats(device);
+ if (ret && ret != -EOPNOTSUPP) {
+ ib_free_port_attrs(&device->coredev);
+ return ret;
+ }
return 0;
}
void ib_device_unregister_sysfs(struct ib_device *device)
{
- if (device->hw_stats_ag)
- free_hsag(&device->dev.kobj, device->hw_stats_ag);
- kfree(device->hw_stats);
-
+ destroy_hw_device_stats(device);
ib_free_port_attrs(&device->coredev);
}
diff --git a/include/rdma/ib_verbs.h b/include/rdma/ib_verbs.h
index e3be93e7096616..53eba744ad8fb6 100644
--- a/include/rdma/ib_verbs.h
+++ b/include/rdma/ib_verbs.h
@@ -51,6 +51,7 @@ struct ib_usrq_object;
struct ib_uwq_object;
struct rdma_cm_id;
struct ib_port;
+struct hw_stats_device_data;
extern struct workqueue_struct *ib_wq;
extern struct workqueue_struct *ib_comp_wq;
@@ -2696,8 +2697,7 @@ struct ib_device {
u8 node_type;
u32 phys_port_cnt;
struct ib_device_attr attrs;
- struct attribute_group *hw_stats_ag;
- struct rdma_hw_stats *hw_stats;
+ struct hw_stats_device_data *hw_stats_data;
#ifdef CONFIG_CGROUP_RDMA
struct rdmacg_device cg_device;
--
2.31.1
^ permalink raw reply related [flat|nested] 35+ messages in thread* [PATCH 04/13] RDMA/core: Split gid_attrs related sysfs from add_port()
2021-05-17 16:47 [PATCH 00/13] Reorganize sysfs file creation for struct ib_devices Jason Gunthorpe
` (2 preceding siblings ...)
2021-05-17 16:47 ` [PATCH 03/13] RDMA/core: Split port and device counter sysfs attributes Jason Gunthorpe
@ 2021-05-17 16:47 ` Jason Gunthorpe
2021-05-17 16:47 ` [PATCH 05/13] RDMA/core: Simplify how the gid_attrs sysfs is created Jason Gunthorpe
` (9 subsequent siblings)
13 siblings, 0 replies; 35+ messages in thread
From: Jason Gunthorpe @ 2021-05-17 16:47 UTC (permalink / raw)
To: Doug Ledford, linux-rdma; +Cc: Greg KH, Kees Cook, Nathan Chancellor
The gid_attrs directory is a dedicated kobj nested under the port,
construct/destruct it with its own pair of functions for
understandability. This is much more readable than having it weirdly
inlined out of order into the add_port() function.
Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>
---
drivers/infiniband/core/sysfs.c | 160 ++++++++++++++++++--------------
1 file changed, 89 insertions(+), 71 deletions(-)
diff --git a/drivers/infiniband/core/sysfs.c b/drivers/infiniband/core/sysfs.c
index 114fecda97648e..d2a089a6f66639 100644
--- a/drivers/infiniband/core/sysfs.c
+++ b/drivers/infiniband/core/sysfs.c
@@ -1178,6 +1178,85 @@ struct rdma_hw_stats *ib_get_hw_stats_port(struct ib_device *ibdev,
return ibdev->port_data[port_num].sysfs->hw_stats_data->stats;
}
+/*
+ * Create the sysfs:
+ * ibp0s9/ports/XX/gid_attrs/{ndevs,types}/YYY
+ * YYY is the gid table index in decimal
+ */
+static int setup_gid_attrs(struct ib_port *port,
+ const struct ib_port_attr *attr)
+{
+ struct gid_attr_group *gid_attr_group;
+ int ret;
+ int i;
+
+ gid_attr_group = kzalloc(sizeof(*gid_attr_group), GFP_KERNEL);
+ if (!gid_attr_group)
+ return -ENOMEM;
+
+ gid_attr_group->port = port;
+ ret = kobject_init_and_add(&gid_attr_group->kobj, &gid_attr_type,
+ &port->kobj, "gid_attrs");
+ if (ret)
+ goto err_put_gid_attrs;
+
+ gid_attr_group->ndev.name = "ndevs";
+ gid_attr_group->ndev.attrs =
+ alloc_group_attrs(show_port_gid_attr_ndev, attr->gid_tbl_len);
+ if (!gid_attr_group->ndev.attrs) {
+ ret = -ENOMEM;
+ goto err_put_gid_attrs;
+ }
+
+ ret = sysfs_create_group(&gid_attr_group->kobj, &gid_attr_group->ndev);
+ if (ret)
+ goto err_free_gid_ndev;
+
+ gid_attr_group->type.name = "types";
+ gid_attr_group->type.attrs = alloc_group_attrs(
+ show_port_gid_attr_gid_type, attr->gid_tbl_len);
+ if (!gid_attr_group->type.attrs) {
+ ret = -ENOMEM;
+ goto err_remove_gid_ndev;
+ }
+
+ ret = sysfs_create_group(&gid_attr_group->kobj, &gid_attr_group->type);
+ if (ret)
+ goto err_free_gid_type;
+
+ port->gid_attr_group = gid_attr_group;
+ return 0;
+
+err_free_gid_type:
+ for (i = 0; i < attr->gid_tbl_len; ++i)
+ kfree(gid_attr_group->type.attrs[i]);
+
+ kfree(gid_attr_group->type.attrs);
+ gid_attr_group->type.attrs = NULL;
+err_remove_gid_ndev:
+ sysfs_remove_group(&gid_attr_group->kobj, &gid_attr_group->ndev);
+err_free_gid_ndev:
+ for (i = 0; i < attr->gid_tbl_len; ++i)
+ kfree(gid_attr_group->ndev.attrs[i]);
+
+ kfree(gid_attr_group->ndev.attrs);
+ gid_attr_group->ndev.attrs = NULL;
+err_put_gid_attrs:
+ kobject_put(&gid_attr_group->kobj);
+ return ret;
+}
+
+static void destroy_gid_attrs(struct ib_port *port)
+{
+ struct gid_attr_group *gid_attr_group = port->gid_attr_group;
+
+ sysfs_remove_group(&gid_attr_group->kobj,
+ &gid_attr_group->ndev);
+ sysfs_remove_group(&gid_attr_group->kobj,
+ &gid_attr_group->type);
+ kobject_put(&gid_attr_group->kobj);
+}
+
static int add_port(struct ib_core_device *coredev, int port_num)
{
struct ib_device *device = rdma_device_to_ibdev(&coredev->dev);
@@ -1204,23 +1283,11 @@ static int add_port(struct ib_core_device *coredev, int port_num)
if (ret)
goto err_put;
- p->gid_attr_group = kzalloc(sizeof(*p->gid_attr_group), GFP_KERNEL);
- if (!p->gid_attr_group) {
- ret = -ENOMEM;
- goto err_put;
- }
-
- p->gid_attr_group->port = p;
- ret = kobject_init_and_add(&p->gid_attr_group->kobj, &gid_attr_type,
- &p->kobj, "gid_attrs");
- if (ret)
- goto err_put_gid_attrs;
-
if (device->ops.process_mad && is_full_dev) {
p->pma_table = get_counter_table(device, port_num);
ret = sysfs_create_group(&p->kobj, p->pma_table);
if (ret)
- goto err_put_gid_attrs;
+ goto err_put;
}
p->gid_group.name = "gids";
@@ -1234,37 +1301,11 @@ static int add_port(struct ib_core_device *coredev, int port_num)
if (ret)
goto err_free_gid;
- p->gid_attr_group->ndev.name = "ndevs";
- p->gid_attr_group->ndev.attrs = alloc_group_attrs(show_port_gid_attr_ndev,
- attr.gid_tbl_len);
- if (!p->gid_attr_group->ndev.attrs) {
- ret = -ENOMEM;
- goto err_remove_gid;
- }
-
- ret = sysfs_create_group(&p->gid_attr_group->kobj,
- &p->gid_attr_group->ndev);
- if (ret)
- goto err_free_gid_ndev;
-
- p->gid_attr_group->type.name = "types";
- p->gid_attr_group->type.attrs = alloc_group_attrs(show_port_gid_attr_gid_type,
- attr.gid_tbl_len);
- if (!p->gid_attr_group->type.attrs) {
- ret = -ENOMEM;
- goto err_remove_gid_ndev;
- }
-
- ret = sysfs_create_group(&p->gid_attr_group->kobj,
- &p->gid_attr_group->type);
- if (ret)
- goto err_free_gid_type;
-
if (attr.pkey_tbl_len) {
p->pkey_group = kzalloc(sizeof(*p->pkey_group), GFP_KERNEL);
if (!p->pkey_group) {
ret = -ENOMEM;
- goto err_remove_gid_type;
+ goto err_remove_gid;
}
p->pkey_group->name = "pkeys";
@@ -1290,11 +1331,14 @@ static int add_port(struct ib_core_device *coredev, int port_num)
if (ret && ret != -EOPNOTSUPP)
goto err_remove_pkey;
}
+ ret = setup_gid_attrs(p, &attr);
+ if (ret)
+ goto err_remove_stats;
if (device->ops.init_port && is_full_dev) {
ret = device->ops.init_port(device, port_num, &p->kobj);
if (ret)
- goto err_remove_stats;
+ goto err_remove_gid_attrs;
}
list_add_tail(&p->kobj.entry, &coredev->port_list);
@@ -1304,6 +1348,9 @@ static int add_port(struct ib_core_device *coredev, int port_num)
kobject_uevent(&p->kobj, KOBJ_ADD);
return 0;
+err_remove_gid_attrs:
+ destroy_gid_attrs(p);
+
err_remove_stats:
destroy_hw_port_stats(p);
@@ -1323,28 +1370,6 @@ static int add_port(struct ib_core_device *coredev, int port_num)
err_free_pkey_group:
kfree(p->pkey_group);
-err_remove_gid_type:
- sysfs_remove_group(&p->gid_attr_group->kobj,
- &p->gid_attr_group->type);
-
-err_free_gid_type:
- for (i = 0; i < attr.gid_tbl_len; ++i)
- kfree(p->gid_attr_group->type.attrs[i]);
-
- kfree(p->gid_attr_group->type.attrs);
- p->gid_attr_group->type.attrs = NULL;
-
-err_remove_gid_ndev:
- sysfs_remove_group(&p->gid_attr_group->kobj,
- &p->gid_attr_group->ndev);
-
-err_free_gid_ndev:
- for (i = 0; i < attr.gid_tbl_len; ++i)
- kfree(p->gid_attr_group->ndev.attrs[i]);
-
- kfree(p->gid_attr_group->ndev.attrs);
- p->gid_attr_group->ndev.attrs = NULL;
-
err_remove_gid:
sysfs_remove_group(&p->kobj, &p->gid_group);
@@ -1359,9 +1384,6 @@ static int add_port(struct ib_core_device *coredev, int port_num)
if (p->pma_table)
sysfs_remove_group(&p->kobj, p->pma_table);
-err_put_gid_attrs:
- kobject_put(&p->gid_attr_group->kobj);
-
err_put:
kobject_put(&p->kobj);
return ret;
@@ -1498,11 +1520,7 @@ void ib_free_port_attrs(struct ib_core_device *coredev)
if (port->pkey_group)
sysfs_remove_group(p, port->pkey_group);
sysfs_remove_group(p, &port->gid_group);
- sysfs_remove_group(&port->gid_attr_group->kobj,
- &port->gid_attr_group->ndev);
- sysfs_remove_group(&port->gid_attr_group->kobj,
- &port->gid_attr_group->type);
- kobject_put(&port->gid_attr_group->kobj);
+ destroy_gid_attrs(port);
kobject_put(p);
}
--
2.31.1
^ permalink raw reply related [flat|nested] 35+ messages in thread* [PATCH 05/13] RDMA/core: Simplify how the gid_attrs sysfs is created
2021-05-17 16:47 [PATCH 00/13] Reorganize sysfs file creation for struct ib_devices Jason Gunthorpe
` (3 preceding siblings ...)
2021-05-17 16:47 ` [PATCH 04/13] RDMA/core: Split gid_attrs related sysfs from add_port() Jason Gunthorpe
@ 2021-05-17 16:47 ` Jason Gunthorpe
2021-05-17 16:47 ` [PATCH 06/13] RDMA/core: Simplify how the port " Jason Gunthorpe
` (8 subsequent siblings)
13 siblings, 0 replies; 35+ messages in thread
From: Jason Gunthorpe @ 2021-05-17 16:47 UTC (permalink / raw)
To: Doug Ledford, linux-rdma; +Cc: Greg KH, Kees Cook, Nathan Chancellor
Instead of having an whole bunch of different allocations to create the
gid_attr kobjects reduce it to three, one for the kobj struct plus the
attributes, and one for the attribute list for each of the two
groups.
Move the freeing of all allocations to the release function.
Reorder the operations so all the allocations happen first then the
kobject & sysfs operations are last.
This removes the majority of the complicated error unwind since the
release function will always undo all the memory allocations. Freeing the
memory is also much simpler since there is a lot less of it.
Consolidate creating the "group of array indexes" pattern into one helper
function. Ensure kobject_del is used.
Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>
---
drivers/infiniband/core/sysfs.c | 171 +++++++++++++++++---------------
1 file changed, 90 insertions(+), 81 deletions(-)
diff --git a/drivers/infiniband/core/sysfs.c b/drivers/infiniband/core/sysfs.c
index d2a089a6f66639..569e847757b944 100644
--- a/drivers/infiniband/core/sysfs.c
+++ b/drivers/infiniband/core/sysfs.c
@@ -47,23 +47,6 @@
struct ib_port;
-struct gid_attr_group {
- struct ib_port *port;
- struct kobject kobj;
- struct attribute_group ndev;
- struct attribute_group type;
-};
-struct ib_port {
- struct kobject kobj;
- struct ib_device *ibdev;
- struct gid_attr_group *gid_attr_group;
- struct attribute_group gid_group;
- struct attribute_group *pkey_group;
- const struct attribute_group *pma_table;
- struct hw_stats_port_data *hw_stats_data;
- u32 port_num;
-};
-
struct port_attribute {
struct attribute attr;
ssize_t (*show)(struct ib_port *, struct port_attribute *, char *buf);
@@ -84,6 +67,25 @@ struct port_table_attribute {
__be16 attr_id;
};
+struct gid_attr_group {
+ struct ib_port *port;
+ struct kobject kobj;
+ struct attribute_group groups[2];
+ const struct attribute_group *groups_list[3];
+ struct port_table_attribute attrs_list[];
+};
+
+struct ib_port {
+ struct kobject kobj;
+ struct ib_device *ibdev;
+ struct gid_attr_group *gid_attr_group;
+ struct attribute_group gid_group;
+ struct attribute_group *pkey_group;
+ const struct attribute_group *pma_table;
+ struct hw_stats_port_data *hw_stats_data;
+ u32 port_num;
+};
+
struct hw_stats_device_attribute {
struct device_attribute attr;
ssize_t (*show)(struct ib_device *ibdev, struct rdma_hw_stats *stats,
@@ -776,26 +778,13 @@ static void ib_port_release(struct kobject *kobj)
static void ib_port_gid_attr_release(struct kobject *kobj)
{
- struct gid_attr_group *g = container_of(kobj, struct gid_attr_group,
- kobj);
- struct attribute *a;
+ struct gid_attr_group *gid_attr_group =
+ container_of(kobj, struct gid_attr_group, kobj);
int i;
- if (g->ndev.attrs) {
- for (i = 0; (a = g->ndev.attrs[i]); ++i)
- kfree(a);
-
- kfree(g->ndev.attrs);
- }
-
- if (g->type.attrs) {
- for (i = 0; (a = g->type.attrs[i]); ++i)
- kfree(a);
-
- kfree(g->type.attrs);
- }
-
- kfree(g);
+ for (i = 0; i != ARRAY_SIZE(gid_attr_group->groups); i++)
+ kfree(gid_attr_group->groups[i].attrs);
+ kfree(gid_attr_group);
}
static struct kobj_type port_type = {
@@ -1178,6 +1167,42 @@ struct rdma_hw_stats *ib_get_hw_stats_port(struct ib_device *ibdev,
return ibdev->port_data[port_num].sysfs->hw_stats_data->stats;
}
+static int alloc_port_table_group(
+ const char *name, struct attribute_group *group,
+ struct port_table_attribute *attrs, size_t num,
+ ssize_t (*show)(struct ib_port *, struct port_attribute *, char *buf))
+{
+ struct attribute **attr_list;
+ int i;
+
+ attr_list = kcalloc(num + 1, sizeof(*attr_list), GFP_KERNEL);
+ if (!attr_list)
+ return -ENOMEM;
+
+ for (i = 0; i < num; i++) {
+ struct port_table_attribute *element = &attrs[i];
+
+ if (snprintf(element->name, sizeof(element->name), "%d", i) >=
+ sizeof(element->name)) {
+ goto err;
+ }
+
+ sysfs_attr_init(&element->attr.attr);
+ element->attr.attr.name = element->name;
+ element->attr.attr.mode = 0444;
+ element->attr.show = show;
+ element->index = i;
+
+ attr_list[i] = &element->attr.attr;
+ }
+ group->name = name;
+ group->attrs = attr_list;
+ return 0;
+err:
+ kfree(attr_list);
+ return -EINVAL;
+}
+
/*
* Create the sysfs:
* ibp0s9/ports/XX/gid_attrs/{ndevs,types}/YYY
@@ -1188,60 +1213,44 @@ static int setup_gid_attrs(struct ib_port *port,
{
struct gid_attr_group *gid_attr_group;
int ret;
- int i;
- gid_attr_group = kzalloc(sizeof(*gid_attr_group), GFP_KERNEL);
+ gid_attr_group = kzalloc(struct_size(gid_attr_group, attrs_list,
+ attr->gid_tbl_len * 2),
+ GFP_KERNEL);
if (!gid_attr_group)
return -ENOMEM;
-
gid_attr_group->port = port;
- ret = kobject_init_and_add(&gid_attr_group->kobj, &gid_attr_type,
- &port->kobj, "gid_attrs");
- if (ret)
- goto err_put_gid_attrs;
-
- gid_attr_group->ndev.name = "ndevs";
- gid_attr_group->ndev.attrs =
- alloc_group_attrs(show_port_gid_attr_ndev, attr->gid_tbl_len);
- if (!gid_attr_group->ndev.attrs) {
- ret = -ENOMEM;
- goto err_put_gid_attrs;
- }
+ kobject_init(&gid_attr_group->kobj, &gid_attr_type);
- ret = sysfs_create_group(&gid_attr_group->kobj, &gid_attr_group->ndev);
+ ret = alloc_port_table_group("ndevs", &gid_attr_group->groups[0],
+ gid_attr_group->attrs_list,
+ attr->gid_tbl_len,
+ show_port_gid_attr_ndev);
if (ret)
- goto err_free_gid_ndev;
-
- gid_attr_group->type.name = "types";
- gid_attr_group->type.attrs = alloc_group_attrs(
- show_port_gid_attr_gid_type, attr->gid_tbl_len);
- if (!gid_attr_group->type.attrs) {
- ret = -ENOMEM;
- goto err_remove_gid_ndev;
- }
+ goto err_put;
+ gid_attr_group->groups_list[0] = &gid_attr_group->groups[0];
- ret = sysfs_create_group(&gid_attr_group->kobj, &gid_attr_group->type);
+ ret = alloc_port_table_group(
+ "types", &gid_attr_group->groups[1],
+ gid_attr_group->attrs_list + attr->gid_tbl_len,
+ attr->gid_tbl_len, show_port_gid_attr_gid_type);
if (ret)
- goto err_free_gid_type;
+ goto err_put;
+ gid_attr_group->groups_list[1] = &gid_attr_group->groups[1];
+ ret = kobject_add(&gid_attr_group->kobj, &port->kobj, "gid_attrs");
+ if (ret)
+ goto err_put;
+ ret = sysfs_create_groups(&gid_attr_group->kobj,
+ gid_attr_group->groups_list);
+ if (ret)
+ goto err_del;
port->gid_attr_group = gid_attr_group;
return 0;
-err_free_gid_type:
- for (i = 0; i < attr->gid_tbl_len; ++i)
- kfree(gid_attr_group->type.attrs[i]);
-
- kfree(gid_attr_group->type.attrs);
- gid_attr_group->type.attrs = NULL;
-err_remove_gid_ndev:
- sysfs_remove_group(&gid_attr_group->kobj, &gid_attr_group->ndev);
-err_free_gid_ndev:
- for (i = 0; i < attr->gid_tbl_len; ++i)
- kfree(gid_attr_group->ndev.attrs[i]);
-
- kfree(gid_attr_group->ndev.attrs);
- gid_attr_group->ndev.attrs = NULL;
-err_put_gid_attrs:
+err_del:
+ kobject_del(&gid_attr_group->kobj);
+err_put:
kobject_put(&gid_attr_group->kobj);
return ret;
}
@@ -1250,10 +1259,10 @@ static void destroy_gid_attrs(struct ib_port *port)
{
struct gid_attr_group *gid_attr_group = port->gid_attr_group;
- sysfs_remove_group(&gid_attr_group->kobj,
- &gid_attr_group->ndev);
- sysfs_remove_group(&gid_attr_group->kobj,
- &gid_attr_group->type);
+ if (!gid_attr_group)
+ return;
+ sysfs_remove_groups(&gid_attr_group->kobj, gid_attr_group->groups_list);
+ kobject_del(&gid_attr_group->kobj);
kobject_put(&gid_attr_group->kobj);
}
--
2.31.1
^ permalink raw reply related [flat|nested] 35+ messages in thread* [PATCH 06/13] RDMA/core: Simplify how the port sysfs is created
2021-05-17 16:47 [PATCH 00/13] Reorganize sysfs file creation for struct ib_devices Jason Gunthorpe
` (4 preceding siblings ...)
2021-05-17 16:47 ` [PATCH 05/13] RDMA/core: Simplify how the gid_attrs sysfs is created Jason Gunthorpe
@ 2021-05-17 16:47 ` Jason Gunthorpe
2021-05-17 16:47 ` [PATCH 07/13] RDMA/core: Create the device hw_counters through the normal groups mechanism Jason Gunthorpe
` (7 subsequent siblings)
13 siblings, 0 replies; 35+ messages in thread
From: Jason Gunthorpe @ 2021-05-17 16:47 UTC (permalink / raw)
To: Doug Ledford, linux-rdma; +Cc: Greg KH, Kees Cook, Nathan Chancellor
Use the same technique as gid_attrs now uses to manage the port
sysfs. Bundle everything into three allocations and use a single
sysfs_create_groups() to build everything in one shot.
All the memory is always freed in the kobj release function, removing most
of the error unwinding.
The gid_attr technique and the hw_counters are very similar, merge the two
together and combine the sysfs_create_group() call for hw_counters with
the single sysfs group setup.
Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>
---
drivers/infiniband/core/sysfs.c | 321 ++++++++++----------------------
1 file changed, 103 insertions(+), 218 deletions(-)
diff --git a/drivers/infiniband/core/sysfs.c b/drivers/infiniband/core/sysfs.c
index 569e847757b944..53838bce574264 100644
--- a/drivers/infiniband/core/sysfs.c
+++ b/drivers/infiniband/core/sysfs.c
@@ -79,11 +79,12 @@ struct ib_port {
struct kobject kobj;
struct ib_device *ibdev;
struct gid_attr_group *gid_attr_group;
- struct attribute_group gid_group;
- struct attribute_group *pkey_group;
- const struct attribute_group *pma_table;
struct hw_stats_port_data *hw_stats_data;
+
+ struct attribute_group groups[3];
+ const struct attribute_group *groups_list[5];
u32 port_num;
+ struct port_table_attribute attrs_list[];
};
struct hw_stats_device_attribute {
@@ -112,7 +113,6 @@ struct hw_stats_device_data {
};
struct hw_stats_port_data {
- struct attribute_group group;
struct rdma_hw_stats *stats;
struct hw_stats_port_attribute attrs[];
};
@@ -750,30 +750,16 @@ static const struct attribute_group pma_group_noietf = {
static void ib_port_release(struct kobject *kobj)
{
- struct ib_port *p = container_of(kobj, struct ib_port, kobj);
- struct attribute *a;
+ struct ib_port *port = container_of(kobj, struct ib_port, kobj);
int i;
- if (p->gid_group.attrs) {
- for (i = 0; (a = p->gid_group.attrs[i]); ++i)
- kfree(a);
-
- kfree(p->gid_group.attrs);
+ for (i = 0; i != ARRAY_SIZE(port->groups); i++)
+ kfree(port->groups[i].attrs);
+ if (port->hw_stats_data) {
+ kfree(port->hw_stats_data->stats);
+ kfree(port->hw_stats_data);
}
-
- if (p->pkey_group) {
- if (p->pkey_group->attrs) {
- for (i = 0; (a = p->pkey_group->attrs[i]); ++i)
- kfree(a);
-
- kfree(p->pkey_group->attrs);
- }
-
- kfree(p->pkey_group);
- p->pkey_group = NULL;
- }
-
- kfree(p);
+ kfree(port);
}
static void ib_port_gid_attr_release(struct kobject *kobj)
@@ -798,49 +784,6 @@ static struct kobj_type gid_attr_type = {
.release = ib_port_gid_attr_release
};
-static struct attribute **
-alloc_group_attrs(ssize_t (*show)(struct ib_port *,
- struct port_attribute *, char *buf),
- int len)
-{
- struct attribute **tab_attr;
- struct port_table_attribute *element;
- int i;
-
- tab_attr = kcalloc(1 + len, sizeof(struct attribute *), GFP_KERNEL);
- if (!tab_attr)
- return NULL;
-
- for (i = 0; i < len; i++) {
- element = kzalloc(sizeof(struct port_table_attribute),
- GFP_KERNEL);
- if (!element)
- goto err;
-
- if (snprintf(element->name, sizeof(element->name),
- "%d", i) >= sizeof(element->name)) {
- kfree(element);
- goto err;
- }
-
- element->attr.attr.name = element->name;
- element->attr.attr.mode = S_IRUGO;
- element->attr.show = show;
- element->index = i;
- sysfs_attr_init(&element->attr.attr);
-
- tab_attr[i] = &element->attr.attr;
- }
-
- return tab_attr;
-
-err:
- while (--i >= 0)
- kfree(tab_attr[i]);
- kfree(tab_attr);
- return NULL;
-}
-
/*
* Figure out which counter table to use depending on
* the device capabilities.
@@ -1051,7 +994,8 @@ static void destroy_hw_device_stats(struct ib_device *ibdev)
ibdev->hw_stats_data = NULL;
}
-static struct hw_stats_port_data *alloc_hw_stats_port(struct ib_port *port)
+static struct hw_stats_port_data *
+alloc_hw_stats_port(struct ib_port *port, struct attribute_group *group)
{
struct ib_device *ibdev = port->ibdev;
struct hw_stats_port_data *data;
@@ -1073,13 +1017,13 @@ static struct hw_stats_port_data *alloc_hw_stats_port(struct ib_port *port)
GFP_KERNEL);
if (!data)
goto err_free_stats;
- data->group.attrs = kcalloc(stats->num_counters + 2,
- sizeof(*data->group.attrs), GFP_KERNEL);
- if (!data->group.attrs)
+ group->attrs = kcalloc(stats->num_counters + 2,
+ sizeof(*group->attrs), GFP_KERNEL);
+ if (!group->attrs)
goto err_free_data;
mutex_init(&stats->lock);
- data->group.name = "hw_counters";
+ group->name = "hw_counters";
data->stats = stats;
return data;
@@ -1090,20 +1034,14 @@ static struct hw_stats_port_data *alloc_hw_stats_port(struct ib_port *port)
return ERR_PTR(-ENOMEM);
}
-static void free_hw_stats_port(struct hw_stats_port_data *data)
-{
- kfree(data->group.attrs);
- kfree(data->stats);
- kfree(data);
-}
-
-static int setup_hw_port_stats(struct ib_port *port)
+static int setup_hw_port_stats(struct ib_port *port,
+ struct attribute_group *group)
{
struct hw_stats_port_attribute *attr;
struct hw_stats_port_data *data;
int i, ret;
- data = alloc_hw_stats_port(port);
+ data = alloc_hw_stats_port(port, group);
if (IS_ERR(data))
return PTR_ERR(data);
@@ -1112,9 +1050,10 @@ static int setup_hw_port_stats(struct ib_port *port)
data->stats->num_counters);
if (ret != data->stats->num_counters) {
if (WARN_ON(ret >= 0))
- ret = -EINVAL;
- goto err_free;
+ return -EINVAL;
+ return ret;
}
+
data->stats->timestamp = jiffies;
for (i = 0; i < data->stats->num_counters; i++) {
@@ -1124,7 +1063,7 @@ static int setup_hw_port_stats(struct ib_port *port)
attr->attr.attr.mode = 0444;
attr->attr.show = hw_stat_port_show;
attr->show = show_hw_stats;
- data->group.attrs[i] = &attr->attr.attr;
+ group->attrs[i] = &attr->attr.attr;
}
attr = &data->attrs[i];
@@ -1135,27 +1074,10 @@ static int setup_hw_port_stats(struct ib_port *port)
attr->show = show_stats_lifespan;
attr->attr.store = hw_stat_port_store;
attr->store = set_stats_lifespan;
- data->group.attrs[i] = &attr->attr.attr;
+ group->attrs[i] = &attr->attr.attr;
port->hw_stats_data = data;
- ret = sysfs_create_group(&port->kobj, &data->group);
- if (ret)
- goto err_free;
return 0;
-
-err_free:
- free_hw_stats_port(data);
- port->hw_stats_data = NULL;
- return ret;
-}
-
-static void destroy_hw_port_stats(struct ib_port *port)
-{
- if (!port->hw_stats_data)
- return;
- sysfs_remove_group(&port->kobj, &port->hw_stats_data->group);
- free_hw_stats_port(port->hw_stats_data);
- port->hw_stats_data = NULL;
}
struct rdma_hw_stats *ib_get_hw_stats_port(struct ib_device *ibdev,
@@ -1266,68 +1188,42 @@ static void destroy_gid_attrs(struct ib_port *port)
kobject_put(&gid_attr_group->kobj);
}
-static int add_port(struct ib_core_device *coredev, int port_num)
+/*
+ * Create the sysfs:
+ * ibp0s9/ports/XX/{gids,pkeys,counters}/YYY
+ */
+static struct ib_port *setup_port(struct ib_core_device *coredev, int port_num,
+ const struct ib_port_attr *attr)
{
struct ib_device *device = rdma_device_to_ibdev(&coredev->dev);
bool is_full_dev = &device->coredev == coredev;
+ const struct attribute_group **cur_group;
struct ib_port *p;
- struct ib_port_attr attr;
- int i;
int ret;
- ret = ib_query_port(device, port_num, &attr);
- if (ret)
- return ret;
-
- p = kzalloc(sizeof *p, GFP_KERNEL);
+ p = kzalloc(struct_size(p, attrs_list,
+ attr->gid_tbl_len + attr->pkey_tbl_len),
+ GFP_KERNEL);
if (!p)
- return -ENOMEM;
-
- p->ibdev = device;
- p->port_num = port_num;
+ return ERR_PTR(-ENOMEM);
+ p->ibdev = device;
+ p->port_num = port_num;
+ kobject_init(&p->kobj, &port_type);
- ret = kobject_init_and_add(&p->kobj, &port_type,
- coredev->ports_kobj,
- "%d", port_num);
+ cur_group = p->groups_list;
+ ret = alloc_port_table_group("gids", &p->groups[0], p->attrs_list,
+ attr->gid_tbl_len, show_port_gid);
if (ret)
goto err_put;
+ *cur_group++ = &p->groups[0];
- if (device->ops.process_mad && is_full_dev) {
- p->pma_table = get_counter_table(device, port_num);
- ret = sysfs_create_group(&p->kobj, p->pma_table);
+ if (attr->pkey_tbl_len) {
+ ret = alloc_port_table_group("pkeys", &p->groups[1],
+ p->attrs_list + attr->gid_tbl_len,
+ attr->pkey_tbl_len, show_port_pkey);
if (ret)
goto err_put;
- }
-
- p->gid_group.name = "gids";
- p->gid_group.attrs = alloc_group_attrs(show_port_gid, attr.gid_tbl_len);
- if (!p->gid_group.attrs) {
- ret = -ENOMEM;
- goto err_remove_pma;
- }
-
- ret = sysfs_create_group(&p->kobj, &p->gid_group);
- if (ret)
- goto err_free_gid;
-
- if (attr.pkey_tbl_len) {
- p->pkey_group = kzalloc(sizeof(*p->pkey_group), GFP_KERNEL);
- if (!p->pkey_group) {
- ret = -ENOMEM;
- goto err_remove_gid;
- }
-
- p->pkey_group->name = "pkeys";
- p->pkey_group->attrs = alloc_group_attrs(show_port_pkey,
- attr.pkey_tbl_len);
- if (!p->pkey_group->attrs) {
- ret = -ENOMEM;
- goto err_free_pkey_group;
- }
-
- ret = sysfs_create_group(&p->kobj, p->pkey_group);
- if (ret)
- goto err_free_pkey;
+ *cur_group++ = &p->groups[1];
}
/*
@@ -1336,66 +1232,45 @@ static int add_port(struct ib_core_device *coredev, int port_num)
* counter initialization.
*/
if (port_num && is_full_dev) {
- ret = setup_hw_port_stats(p);
+ ret = setup_hw_port_stats(p, &p->groups[2]);
if (ret && ret != -EOPNOTSUPP)
- goto err_remove_pkey;
+ goto err_put;
+ if (!ret)
+ *cur_group++ = &p->groups[2];
}
- ret = setup_gid_attrs(p, &attr);
- if (ret)
- goto err_remove_stats;
- if (device->ops.init_port && is_full_dev) {
- ret = device->ops.init_port(device, port_num, &p->kobj);
- if (ret)
- goto err_remove_gid_attrs;
- }
+ if (device->ops.process_mad && is_full_dev)
+ *cur_group++ = get_counter_table(device, port_num);
+
+ ret = kobject_add(&p->kobj, coredev->ports_kobj, "%d", port_num);
+ if (ret)
+ goto err_put;
+ ret = sysfs_create_groups(&p->kobj, p->groups_list);
+ if (ret)
+ goto err_del;
list_add_tail(&p->kobj.entry, &coredev->port_list);
if (device->port_data && is_full_dev)
device->port_data[port_num].sysfs = p;
- kobject_uevent(&p->kobj, KOBJ_ADD);
- return 0;
-
-err_remove_gid_attrs:
- destroy_gid_attrs(p);
-
-err_remove_stats:
- destroy_hw_port_stats(p);
-
-err_remove_pkey:
- if (p->pkey_group)
- sysfs_remove_group(&p->kobj, p->pkey_group);
-
-err_free_pkey:
- if (p->pkey_group) {
- for (i = 0; i < attr.pkey_tbl_len; ++i)
- kfree(p->pkey_group->attrs[i]);
-
- kfree(p->pkey_group->attrs);
- p->pkey_group->attrs = NULL;
- }
-
-err_free_pkey_group:
- kfree(p->pkey_group);
-
-err_remove_gid:
- sysfs_remove_group(&p->kobj, &p->gid_group);
-
-err_free_gid:
- for (i = 0; i < attr.gid_tbl_len; ++i)
- kfree(p->gid_group.attrs[i]);
-
- kfree(p->gid_group.attrs);
- p->gid_group.attrs = NULL;
-
-err_remove_pma:
- if (p->pma_table)
- sysfs_remove_group(&p->kobj, p->pma_table);
+ return p;
+err_del:
+ kobject_del(&p->kobj);
err_put:
kobject_put(&p->kobj);
- return ret;
+ return ERR_PTR(ret);
+}
+
+static void destroy_port(struct ib_port *port)
+{
+ if (port->ibdev->port_data &&
+ port->ibdev->port_data[port->port_num].sysfs == port)
+ port->ibdev->port_data[port->port_num].sysfs = NULL;
+ list_del(&port->kobj.entry);
+ sysfs_remove_groups(&port->kobj, port->groups_list);
+ kobject_del(&port->kobj);
+ kobject_put(&port->kobj);
}
static const char *node_type_string(int node_type)
@@ -1512,25 +1387,13 @@ const struct attribute_group ib_dev_attr_group = {
void ib_free_port_attrs(struct ib_core_device *coredev)
{
- struct ib_device *device = rdma_device_to_ibdev(&coredev->dev);
- bool is_full_dev = &device->coredev == coredev;
struct kobject *p, *t;
list_for_each_entry_safe(p, t, &coredev->port_list, entry) {
struct ib_port *port = container_of(p, struct ib_port, kobj);
- list_del(&p->entry);
- destroy_hw_port_stats(port);
- if (device->port_data && is_full_dev)
- device->port_data[port->port_num].sysfs = NULL;
-
- if (port->pma_table)
- sysfs_remove_group(p, port->pma_table);
- if (port->pkey_group)
- sysfs_remove_group(p, port->pkey_group);
- sysfs_remove_group(p, &port->gid_group);
destroy_gid_attrs(port);
- kobject_put(p);
+ destroy_port(port);
}
kobject_put(coredev->ports_kobj);
@@ -1539,7 +1402,8 @@ void ib_free_port_attrs(struct ib_core_device *coredev)
int ib_setup_port_attrs(struct ib_core_device *coredev)
{
struct ib_device *device = rdma_device_to_ibdev(&coredev->dev);
- u32 port;
+ bool is_full_dev = &device->coredev == coredev;
+ u32 port_num;
int ret;
coredev->ports_kobj = kobject_create_and_add("ports",
@@ -1547,12 +1411,33 @@ int ib_setup_port_attrs(struct ib_core_device *coredev)
if (!coredev->ports_kobj)
return -ENOMEM;
- rdma_for_each_port (device, port) {
- ret = add_port(coredev, port);
+ rdma_for_each_port (device, port_num) {
+ struct ib_port_attr attr;
+ struct ib_port *port;
+
+ ret = ib_query_port(device, port_num, &attr);
+ if (ret)
+ goto err_put;
+
+ port = setup_port(coredev, port_num, &attr);
+ if (IS_ERR(port)) {
+ ret = PTR_ERR(port);
+ goto err_put;
+ }
+
+ ret = setup_gid_attrs(port, &attr);
if (ret)
goto err_put;
- }
+ if (device->ops.init_port && is_full_dev) {
+ ret = device->ops.init_port(device, port_num,
+ &port->kobj);
+ if (ret)
+ goto err_put;
+ }
+
+ kobject_uevent(&port->kobj, KOBJ_ADD);
+ }
return 0;
err_put:
--
2.31.1
^ permalink raw reply related [flat|nested] 35+ messages in thread* [PATCH 07/13] RDMA/core: Create the device hw_counters through the normal groups mechanism
2021-05-17 16:47 [PATCH 00/13] Reorganize sysfs file creation for struct ib_devices Jason Gunthorpe
` (5 preceding siblings ...)
2021-05-17 16:47 ` [PATCH 06/13] RDMA/core: Simplify how the port " Jason Gunthorpe
@ 2021-05-17 16:47 ` Jason Gunthorpe
2021-05-17 16:47 ` [PATCH 08/13] RDMA/core: Remove the kobject_uevent() NOP Jason Gunthorpe
` (6 subsequent siblings)
13 siblings, 0 replies; 35+ messages in thread
From: Jason Gunthorpe @ 2021-05-17 16:47 UTC (permalink / raw)
To: Doug Ledford, linux-rdma; +Cc: Greg KH, Kees Cook, Nathan Chancellor
Instead of calling device_add_groups() add the group to the existing
groups array which is managed through device_add().
This requires setting up the hw_counters before device_add(), so it gets
split up from the already split port sysfs flow.
Move all the memory freeing to the release function.
Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>
---
drivers/infiniband/core/core_priv.h | 4 +-
drivers/infiniband/core/device.c | 11 ++++-
drivers/infiniband/core/sysfs.c | 65 +++++++----------------------
include/rdma/ib_verbs.h | 9 ++--
4 files changed, 31 insertions(+), 58 deletions(-)
diff --git a/drivers/infiniband/core/core_priv.h b/drivers/infiniband/core/core_priv.h
index ec5c2c3db42303..6066c4b39876d6 100644
--- a/drivers/infiniband/core/core_priv.h
+++ b/drivers/infiniband/core/core_priv.h
@@ -78,8 +78,6 @@ static inline struct rdma_dev_net *rdma_net_to_dev_net(struct net *net)
return net_generic(net, rdma_dev_net_id);
}
-int ib_device_register_sysfs(struct ib_device *device);
-void ib_device_unregister_sysfs(struct ib_device *device);
int ib_device_rename(struct ib_device *ibdev, const char *name);
int ib_device_set_dim(struct ib_device *ibdev, u8 use_dim);
@@ -379,6 +377,8 @@ struct net_device *rdma_read_gid_attr_ndev_rcu(const struct ib_gid_attr *attr);
void ib_free_port_attrs(struct ib_core_device *coredev);
int ib_setup_port_attrs(struct ib_core_device *coredev);
struct rdma_hw_stats *ib_get_hw_stats_port(struct ib_device *ibdev, u32 port_num);
+void ib_device_release_hw_stats(struct hw_stats_device_data *data);
+int ib_setup_device_attrs(struct ib_device *ibdev);
int rdma_compatdev_set(u8 enable);
diff --git a/drivers/infiniband/core/device.c b/drivers/infiniband/core/device.c
index 86a16cd7d7fdb2..030a4041b2e03b 100644
--- a/drivers/infiniband/core/device.c
+++ b/drivers/infiniband/core/device.c
@@ -491,6 +491,8 @@ static void ib_device_release(struct device *device)
free_netdevs(dev);
WARN_ON(refcount_read(&dev->refcount));
+ if (dev->hw_stats_data)
+ ib_device_release_hw_stats(dev->hw_stats_data);
if (dev->port_data) {
ib_cache_release_one(dev);
ib_security_release_port_pkey_list(dev);
@@ -1394,6 +1396,10 @@ int ib_register_device(struct ib_device *device, const char *name,
return ret;
}
+ ret = ib_setup_device_attrs(device);
+ if (ret)
+ goto cache_cleanup;
+
ib_device_register_rdmacg(device);
rdma_counter_init(device);
@@ -1407,7 +1413,7 @@ int ib_register_device(struct ib_device *device, const char *name,
if (ret)
goto cg_cleanup;
- ret = ib_device_register_sysfs(device);
+ ret = ib_setup_port_attrs(&device->coredev);
if (ret) {
dev_warn(&device->dev,
"Couldn't register device with driver model\n");
@@ -1449,6 +1455,7 @@ int ib_register_device(struct ib_device *device, const char *name,
cg_cleanup:
dev_set_uevent_suppress(&device->dev, false);
ib_device_unregister_rdmacg(device);
+cache_cleanup:
ib_cache_cleanup_one(device);
return ret;
}
@@ -1473,7 +1480,7 @@ static void __ib_unregister_device(struct ib_device *ib_dev)
/* Expedite removing unregistered pointers from the hash table */
free_netdevs(ib_dev);
- ib_device_unregister_sysfs(ib_dev);
+ ib_free_port_attrs(&ib_dev->coredev);
device_del(&ib_dev->dev);
ib_device_unregister_rdmacg(ib_dev);
ib_cache_cleanup_one(ib_dev);
diff --git a/drivers/infiniband/core/sysfs.c b/drivers/infiniband/core/sysfs.c
index 53838bce574264..168c43a644d76c 100644
--- a/drivers/infiniband/core/sysfs.c
+++ b/drivers/infiniband/core/sysfs.c
@@ -107,7 +107,6 @@ struct hw_stats_port_attribute {
struct hw_stats_device_data {
struct attribute_group group;
- const struct attribute_group *groups[2];
struct rdma_hw_stats *stats;
struct hw_stats_device_attribute attrs[];
};
@@ -916,7 +915,6 @@ alloc_hw_stats_device(struct ib_device *ibdev)
mutex_init(&stats->lock);
data->group.name = "hw_counters";
data->stats = stats;
- data->groups[0] = &data->group;
return data;
err_free_data:
@@ -926,29 +924,33 @@ alloc_hw_stats_device(struct ib_device *ibdev)
return ERR_PTR(-ENOMEM);
}
-static void free_hw_stats_device(struct hw_stats_device_data *data)
+void ib_device_release_hw_stats(struct hw_stats_device_data *data)
{
kfree(data->group.attrs);
kfree(data->stats);
kfree(data);
}
-static int setup_hw_device_stats(struct ib_device *ibdev)
+int ib_setup_device_attrs(struct ib_device *ibdev)
{
struct hw_stats_device_attribute *attr;
struct hw_stats_device_data *data;
int i, ret;
data = alloc_hw_stats_device(ibdev);
- if (IS_ERR(data))
+ if (IS_ERR(data)) {
+ if (PTR_ERR(data) == -EOPNOTSUPP)
+ return 0;
return PTR_ERR(data);
+ }
+ ibdev->hw_stats_data = data;
ret = ibdev->ops.get_hw_stats(ibdev, data->stats, 0,
data->stats->num_counters);
if (ret != data->stats->num_counters) {
if (WARN_ON(ret >= 0))
- ret = -EINVAL;
- goto err_free;
+ return -EINVAL;
+ return ret;
}
data->stats->timestamp = jiffies;
@@ -972,26 +974,12 @@ static int setup_hw_device_stats(struct ib_device *ibdev)
attr->attr.store = hw_stat_device_store;
attr->store = set_stats_lifespan;
data->group.attrs[i] = &attr->attr.attr;
-
- ibdev->hw_stats_data = data;
- ret = device_add_groups(&ibdev->dev, data->groups);
- if (ret)
- goto err_free;
- return 0;
-
-err_free:
- free_hw_stats_device(data);
- ibdev->hw_stats_data = NULL;
- return ret;
-}
-
-static void destroy_hw_device_stats(struct ib_device *ibdev)
-{
- if (!ibdev->hw_stats_data)
- return;
- device_remove_groups(&ibdev->dev, ibdev->hw_stats_data->groups);
- free_hw_stats_device(ibdev->hw_stats_data);
- ibdev->hw_stats_data = NULL;
+ for (i = 0; i != ARRAY_SIZE(ibdev->groups); i++)
+ if (!ibdev->groups[i]) {
+ ibdev->groups[i] = &data->group;
+ return 0;
+ }
+ return WARN_ON(-EINVAL);
}
static struct hw_stats_port_data *
@@ -1445,29 +1433,6 @@ int ib_setup_port_attrs(struct ib_core_device *coredev)
return ret;
}
-int ib_device_register_sysfs(struct ib_device *device)
-{
- int ret;
-
- ret = ib_setup_port_attrs(&device->coredev);
- if (ret)
- return ret;
-
- ret = setup_hw_device_stats(device);
- if (ret && ret != -EOPNOTSUPP) {
- ib_free_port_attrs(&device->coredev);
- return ret;
- }
-
- return 0;
-}
-
-void ib_device_unregister_sysfs(struct ib_device *device)
-{
- destroy_hw_device_stats(device);
- ib_free_port_attrs(&device->coredev);
-}
-
/**
* ib_port_register_module_stat - add module counters under relevant port
* of IB device.
diff --git a/include/rdma/ib_verbs.h b/include/rdma/ib_verbs.h
index 53eba744ad8fb6..5f70aab1f8663b 100644
--- a/include/rdma/ib_verbs.h
+++ b/include/rdma/ib_verbs.h
@@ -2678,11 +2678,12 @@ struct ib_device {
struct ib_core_device coredev;
};
- /* First group for device attributes,
- * Second group for driver provided attributes (optional).
- * It is NULL terminated array.
+ /* First group is for device attributes,
+ * Second group is for driver provided attributes (optional).
+ * Thrid group is for the hw_stats
+ * It is a NULL terminated array.
*/
- const struct attribute_group *groups[3];
+ const struct attribute_group *groups[4];
u64 uverbs_cmd_mask;
--
2.31.1
^ permalink raw reply related [flat|nested] 35+ messages in thread* [PATCH 08/13] RDMA/core: Remove the kobject_uevent() NOP
2021-05-17 16:47 [PATCH 00/13] Reorganize sysfs file creation for struct ib_devices Jason Gunthorpe
` (6 preceding siblings ...)
2021-05-17 16:47 ` [PATCH 07/13] RDMA/core: Create the device hw_counters through the normal groups mechanism Jason Gunthorpe
@ 2021-05-17 16:47 ` Jason Gunthorpe
2021-05-17 16:47 ` [PATCH 09/13] RDMA/core: Expose the ib port sysfs attribute machinery Jason Gunthorpe
` (5 subsequent siblings)
13 siblings, 0 replies; 35+ messages in thread
From: Jason Gunthorpe @ 2021-05-17 16:47 UTC (permalink / raw)
To: Doug Ledford, linux-rdma; +Cc: Greg KH, Kees Cook, Nathan Chancellor
This call does nothing because the ib_port kobj is nested under a struct
device kobject and the dev_uevent_filter() function of the struct device
blocks uevents for any children kobj's that are not also struct devices.
A uevent for the struct device will be triggered after
ib_setup_port_attrs() returns which causes udev to pick up all the deep
"attributes" which are implemented as kobjects nested under a struct
device and assign them to the udev object for the struct device:
$ udevadm info -a /sys/class/infiniband/ibp0s9
ATTR{ports/1/counters/excessive_buffer_overrun_errors}=="0"
Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>
---
drivers/infiniband/core/sysfs.c | 2 --
1 file changed, 2 deletions(-)
diff --git a/drivers/infiniband/core/sysfs.c b/drivers/infiniband/core/sysfs.c
index 168c43a644d76c..ce6aecbb5a7616 100644
--- a/drivers/infiniband/core/sysfs.c
+++ b/drivers/infiniband/core/sysfs.c
@@ -1423,8 +1423,6 @@ int ib_setup_port_attrs(struct ib_core_device *coredev)
if (ret)
goto err_put;
}
-
- kobject_uevent(&port->kobj, KOBJ_ADD);
}
return 0;
--
2.31.1
^ permalink raw reply related [flat|nested] 35+ messages in thread* [PATCH 09/13] RDMA/core: Expose the ib port sysfs attribute machinery
2021-05-17 16:47 [PATCH 00/13] Reorganize sysfs file creation for struct ib_devices Jason Gunthorpe
` (7 preceding siblings ...)
2021-05-17 16:47 ` [PATCH 08/13] RDMA/core: Remove the kobject_uevent() NOP Jason Gunthorpe
@ 2021-05-17 16:47 ` Jason Gunthorpe
2021-05-17 17:12 ` Greg KH
2021-05-17 16:47 ` [PATCH 10/13] RDMA/cm: Use an attribute_group on the ib_port_attribute intead of kobj's Jason Gunthorpe
` (4 subsequent siblings)
13 siblings, 1 reply; 35+ messages in thread
From: Jason Gunthorpe @ 2021-05-17 16:47 UTC (permalink / raw)
To: Doug Ledford, linux-rdma; +Cc: Greg KH, Kees Cook, Nathan Chancellor
Other things outside the core code are creating attributes against the
port. This patch exposes the basic machinery to do this.
The ib_port_attribute type allows creating groups of attributes attatched
to the port and comes with the usual machinery to do this.
Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>
---
drivers/infiniband/core/sysfs.c | 217 +++++++++++++++++---------------
include/rdma/ib_sysfs.h | 41 ++++++
2 files changed, 158 insertions(+), 100 deletions(-)
create mode 100644 include/rdma/ib_sysfs.h
diff --git a/drivers/infiniband/core/sysfs.c b/drivers/infiniband/core/sysfs.c
index ce6aecbb5a7616..58a45548bf1568 100644
--- a/drivers/infiniband/core/sysfs.c
+++ b/drivers/infiniband/core/sysfs.c
@@ -44,24 +44,10 @@
#include <rdma/ib_pma.h>
#include <rdma/ib_cache.h>
#include <rdma/rdma_counter.h>
-
-struct ib_port;
-
-struct port_attribute {
- struct attribute attr;
- ssize_t (*show)(struct ib_port *, struct port_attribute *, char *buf);
- ssize_t (*store)(struct ib_port *, struct port_attribute *,
- const char *buf, size_t count);
-};
-
-#define PORT_ATTR(_name, _mode, _show, _store) \
-struct port_attribute port_attr_##_name = __ATTR(_name, _mode, _show, _store)
-
-#define PORT_ATTR_RO(_name) \
-struct port_attribute port_attr_##_name = __ATTR_RO(_name)
+#include <rdma/ib_sysfs.h>
struct port_table_attribute {
- struct port_attribute attr;
+ struct ib_port_attribute attr;
char name[8];
int index;
__be16 attr_id;
@@ -97,7 +83,7 @@ struct hw_stats_device_attribute {
};
struct hw_stats_port_attribute {
- struct port_attribute attr;
+ struct ib_port_attribute attr;
ssize_t (*show)(struct ib_device *ibdev, struct rdma_hw_stats *stats,
unsigned int index, unsigned int port_num, char *buf);
ssize_t (*store)(struct ib_device *ibdev, struct rdma_hw_stats *stats,
@@ -119,29 +105,55 @@ struct hw_stats_port_data {
static ssize_t port_attr_show(struct kobject *kobj,
struct attribute *attr, char *buf)
{
- struct port_attribute *port_attr =
- container_of(attr, struct port_attribute, attr);
+ struct ib_port_attribute *port_attr =
+ container_of(attr, struct ib_port_attribute, attr);
struct ib_port *p = container_of(kobj, struct ib_port, kobj);
if (!port_attr->show)
return -EIO;
- return port_attr->show(p, port_attr, buf);
+ return port_attr->show(p->ibdev, p->port_num, port_attr, buf);
}
static ssize_t port_attr_store(struct kobject *kobj,
struct attribute *attr,
const char *buf, size_t count)
{
- struct port_attribute *port_attr =
- container_of(attr, struct port_attribute, attr);
+ struct ib_port_attribute *port_attr =
+ container_of(attr, struct ib_port_attribute, attr);
struct ib_port *p = container_of(kobj, struct ib_port, kobj);
if (!port_attr->store)
return -EIO;
- return port_attr->store(p, port_attr, buf, count);
+ return port_attr->store(p->ibdev, p->port_num, port_attr, buf, count);
}
+int ib_port_sysfs_create_groups(struct ib_device *ibdev, u32 port_num,
+ const struct attribute_group **groups)
+{
+ return sysfs_create_groups(&ibdev->port_data[port_num].sysfs->kobj,
+ groups);
+}
+EXPORT_SYMBOL(ib_port_sysfs_create_groups);
+
+void ib_port_sysfs_remove_groups(struct ib_device *ibdev, u32 port_num,
+ const struct attribute_group **groups)
+{
+ return sysfs_remove_groups(&ibdev->port_data[port_num].sysfs->kobj,
+ groups);
+}
+EXPORT_SYMBOL(ib_port_sysfs_remove_groups);
+
+struct ib_device *ib_port_sysfs_get_ibdev_kobj(struct kobject *kobj,
+ u32 *port_num)
+{
+ struct ib_port *port = container_of(kobj, struct ib_port, kobj);
+
+ *port_num = port->port_num;
+ return port->ibdev;
+}
+EXPORT_SYMBOL(ib_port_sysfs_get_ibdev_kobj);
+
static const struct sysfs_ops port_sysfs_ops = {
.show = port_attr_show,
.store = port_attr_store
@@ -171,25 +183,27 @@ static ssize_t hw_stat_device_store(struct device *dev,
count);
}
-static ssize_t hw_stat_port_show(struct ib_port *port,
- struct port_attribute *attr, char *buf)
+static ssize_t hw_stat_port_show(struct ib_device *ibdev, u32 port_num,
+ struct ib_port_attribute *attr, char *buf)
{
struct hw_stats_port_attribute *stat_attr =
container_of(attr, struct hw_stats_port_attribute, attr);
+ struct ib_port *port = ibdev->port_data[port_num].sysfs;
- return stat_attr->show(port->ibdev, port->hw_stats_data->stats,
+ return stat_attr->show(ibdev, port->hw_stats_data->stats,
stat_attr - port->hw_stats_data->attrs,
port->port_num, buf);
}
-static ssize_t hw_stat_port_store(struct ib_port *port,
- struct port_attribute *attr, const char *buf,
- size_t count)
+static ssize_t hw_stat_port_store(struct ib_device *ibdev, u32 port_num,
+ struct ib_port_attribute *attr,
+ const char *buf, size_t count)
{
struct hw_stats_port_attribute *stat_attr =
container_of(attr, struct hw_stats_port_attribute, attr);
+ struct ib_port *port = ibdev->port_data[port_num].sysfs;
- return stat_attr->store(port->ibdev, port->hw_stats_data->stats,
+ return stat_attr->store(ibdev, port->hw_stats_data->stats,
stat_attr - port->hw_stats_data->attrs,
port->port_num, buf, count);
}
@@ -197,23 +211,23 @@ static ssize_t hw_stat_port_store(struct ib_port *port,
static ssize_t gid_attr_show(struct kobject *kobj,
struct attribute *attr, char *buf)
{
- struct port_attribute *port_attr =
- container_of(attr, struct port_attribute, attr);
+ struct ib_port_attribute *port_attr =
+ container_of(attr, struct ib_port_attribute, attr);
struct ib_port *p = container_of(kobj, struct gid_attr_group,
kobj)->port;
if (!port_attr->show)
return -EIO;
- return port_attr->show(p, port_attr, buf);
+ return port_attr->show(p->ibdev, p->port_num, port_attr, buf);
}
static const struct sysfs_ops gid_attr_sysfs_ops = {
.show = gid_attr_show
};
-static ssize_t state_show(struct ib_port *p, struct port_attribute *unused,
- char *buf)
+static ssize_t state_show(struct ib_device *ibdev, u32 port_num,
+ struct ib_port_attribute *unused, char *buf)
{
struct ib_port_attr attr;
ssize_t ret;
@@ -227,7 +241,7 @@ static ssize_t state_show(struct ib_port *p, struct port_attribute *unused,
[IB_PORT_ACTIVE_DEFER] = "ACTIVE_DEFER"
};
- ret = ib_query_port(p->ibdev, p->port_num, &attr);
+ ret = ib_query_port(ibdev, port_num, &attr);
if (ret)
return ret;
@@ -238,81 +252,80 @@ static ssize_t state_show(struct ib_port *p, struct port_attribute *unused,
"UNKNOWN");
}
-static ssize_t lid_show(struct ib_port *p, struct port_attribute *unused,
- char *buf)
+static ssize_t lid_show(struct ib_device *ibdev, u32 port_num,
+ struct ib_port_attribute *unused, char *buf)
{
struct ib_port_attr attr;
ssize_t ret;
- ret = ib_query_port(p->ibdev, p->port_num, &attr);
+ ret = ib_query_port(ibdev, port_num, &attr);
if (ret)
return ret;
return sysfs_emit(buf, "0x%x\n", attr.lid);
}
-static ssize_t lid_mask_count_show(struct ib_port *p,
- struct port_attribute *unused,
- char *buf)
+static ssize_t lid_mask_count_show(struct ib_device *ibdev, u32 port_num,
+ struct ib_port_attribute *unused, char *buf)
{
struct ib_port_attr attr;
ssize_t ret;
- ret = ib_query_port(p->ibdev, p->port_num, &attr);
+ ret = ib_query_port(ibdev, port_num, &attr);
if (ret)
return ret;
return sysfs_emit(buf, "%d\n", attr.lmc);
}
-static ssize_t sm_lid_show(struct ib_port *p, struct port_attribute *unused,
- char *buf)
+static ssize_t sm_lid_show(struct ib_device *ibdev, u32 port_num,
+ struct ib_port_attribute *unused, char *buf)
{
struct ib_port_attr attr;
ssize_t ret;
- ret = ib_query_port(p->ibdev, p->port_num, &attr);
+ ret = ib_query_port(ibdev, port_num, &attr);
if (ret)
return ret;
return sysfs_emit(buf, "0x%x\n", attr.sm_lid);
}
-static ssize_t sm_sl_show(struct ib_port *p, struct port_attribute *unused,
- char *buf)
+static ssize_t sm_sl_show(struct ib_device *ibdev, u32 port_num,
+ struct ib_port_attribute *unused, char *buf)
{
struct ib_port_attr attr;
ssize_t ret;
- ret = ib_query_port(p->ibdev, p->port_num, &attr);
+ ret = ib_query_port(ibdev, port_num, &attr);
if (ret)
return ret;
return sysfs_emit(buf, "%d\n", attr.sm_sl);
}
-static ssize_t cap_mask_show(struct ib_port *p, struct port_attribute *unused,
- char *buf)
+static ssize_t cap_mask_show(struct ib_device *ibdev, u32 port_num,
+ struct ib_port_attribute *unused, char *buf)
{
struct ib_port_attr attr;
ssize_t ret;
- ret = ib_query_port(p->ibdev, p->port_num, &attr);
+ ret = ib_query_port(ibdev, port_num, &attr);
if (ret)
return ret;
return sysfs_emit(buf, "0x%08x\n", attr.port_cap_flags);
}
-static ssize_t rate_show(struct ib_port *p, struct port_attribute *unused,
- char *buf)
+static ssize_t rate_show(struct ib_device *ibdev, u32 port_num,
+ struct ib_port_attribute *unused, char *buf)
{
struct ib_port_attr attr;
char *speed = "";
int rate; /* in deci-Gb/sec */
ssize_t ret;
- ret = ib_query_port(p->ibdev, p->port_num, &attr);
+ ret = ib_query_port(ibdev, port_num, &attr);
if (ret)
return ret;
@@ -379,14 +392,14 @@ static const char *phys_state_to_str(enum ib_port_phys_state phys_state)
return "<unknown>";
}
-static ssize_t phys_state_show(struct ib_port *p, struct port_attribute *unused,
- char *buf)
+static ssize_t phys_state_show(struct ib_device *ibdev, u32 port_num,
+ struct ib_port_attribute *unused, char *buf)
{
struct ib_port_attr attr;
ssize_t ret;
- ret = ib_query_port(p->ibdev, p->port_num, &attr);
+ ret = ib_query_port(ibdev, port_num, &attr);
if (ret)
return ret;
@@ -394,12 +407,12 @@ static ssize_t phys_state_show(struct ib_port *p, struct port_attribute *unused,
phys_state_to_str(attr.phys_state));
}
-static ssize_t link_layer_show(struct ib_port *p, struct port_attribute *unused,
- char *buf)
+static ssize_t link_layer_show(struct ib_device *ibdev, u32 port_num,
+ struct ib_port_attribute *unused, char *buf)
{
const char *output;
- switch (rdma_port_get_link_layer(p->ibdev, p->port_num)) {
+ switch (rdma_port_get_link_layer(ibdev, port_num)) {
case IB_LINK_LAYER_INFINIBAND:
output = "InfiniBand";
break;
@@ -414,26 +427,26 @@ static ssize_t link_layer_show(struct ib_port *p, struct port_attribute *unused,
return sysfs_emit(buf, "%s\n", output);
}
-static PORT_ATTR_RO(state);
-static PORT_ATTR_RO(lid);
-static PORT_ATTR_RO(lid_mask_count);
-static PORT_ATTR_RO(sm_lid);
-static PORT_ATTR_RO(sm_sl);
-static PORT_ATTR_RO(cap_mask);
-static PORT_ATTR_RO(rate);
-static PORT_ATTR_RO(phys_state);
-static PORT_ATTR_RO(link_layer);
+static IB_PORT_ATTR_RO(state);
+static IB_PORT_ATTR_RO(lid);
+static IB_PORT_ATTR_RO(lid_mask_count);
+static IB_PORT_ATTR_RO(sm_lid);
+static IB_PORT_ATTR_RO(sm_sl);
+static IB_PORT_ATTR_RO(cap_mask);
+static IB_PORT_ATTR_RO(rate);
+static IB_PORT_ATTR_RO(phys_state);
+static IB_PORT_ATTR_RO(link_layer);
static struct attribute *port_default_attrs[] = {
- &port_attr_state.attr,
- &port_attr_lid.attr,
- &port_attr_lid_mask_count.attr,
- &port_attr_sm_lid.attr,
- &port_attr_sm_sl.attr,
- &port_attr_cap_mask.attr,
- &port_attr_rate.attr,
- &port_attr_phys_state.attr,
- &port_attr_link_layer.attr,
+ &ib_port_attr_state.attr,
+ &ib_port_attr_lid.attr,
+ &ib_port_attr_lid_mask_count.attr,
+ &ib_port_attr_sm_lid.attr,
+ &ib_port_attr_sm_sl.attr,
+ &ib_port_attr_cap_mask.attr,
+ &ib_port_attr_rate.attr,
+ &ib_port_attr_phys_state.attr,
+ &ib_port_attr_link_layer.attr,
NULL
};
@@ -457,7 +470,8 @@ static ssize_t print_gid_type(const struct ib_gid_attr *gid_attr, char *buf)
}
static ssize_t _show_port_gid_attr(
- struct ib_port *p, struct port_attribute *attr, char *buf,
+ struct ib_device *ibdev, u32 port_num, struct ib_port_attribute *attr,
+ char *buf,
ssize_t (*print)(const struct ib_gid_attr *gid_attr, char *buf))
{
struct port_table_attribute *tab_attr =
@@ -465,7 +479,7 @@ static ssize_t _show_port_gid_attr(
const struct ib_gid_attr *gid_attr;
ssize_t ret;
- gid_attr = rdma_get_gid_attr(p->ibdev, p->port_num, tab_attr->index);
+ gid_attr = rdma_get_gid_attr(ibdev, port_num, tab_attr->index);
if (IS_ERR(gid_attr))
/* -EINVAL is returned for user space compatibility reasons. */
return -EINVAL;
@@ -475,15 +489,15 @@ static ssize_t _show_port_gid_attr(
return ret;
}
-static ssize_t show_port_gid(struct ib_port *p, struct port_attribute *attr,
- char *buf)
+static ssize_t show_port_gid(struct ib_device *ibdev, u32 port_num,
+ struct ib_port_attribute *attr, char *buf)
{
struct port_table_attribute *tab_attr =
container_of(attr, struct port_table_attribute, attr);
const struct ib_gid_attr *gid_attr;
int len;
- gid_attr = rdma_get_gid_attr(p->ibdev, p->port_num, tab_attr->index);
+ gid_attr = rdma_get_gid_attr(ibdev, port_num, tab_attr->index);
if (IS_ERR(gid_attr)) {
const union ib_gid zgid = {};
@@ -504,28 +518,30 @@ static ssize_t show_port_gid(struct ib_port *p, struct port_attribute *attr,
return len;
}
-static ssize_t show_port_gid_attr_ndev(struct ib_port *p,
- struct port_attribute *attr, char *buf)
+static ssize_t show_port_gid_attr_ndev(struct ib_device *ibdev, u32 port_num,
+ struct ib_port_attribute *attr,
+ char *buf)
{
- return _show_port_gid_attr(p, attr, buf, print_ndev);
+ return _show_port_gid_attr(ibdev, port_num, attr, buf, print_ndev);
}
-static ssize_t show_port_gid_attr_gid_type(struct ib_port *p,
- struct port_attribute *attr,
+static ssize_t show_port_gid_attr_gid_type(struct ib_device *ibdev,
+ u32 port_num,
+ struct ib_port_attribute *attr,
char *buf)
{
- return _show_port_gid_attr(p, attr, buf, print_gid_type);
+ return _show_port_gid_attr(ibdev, port_num, attr, buf, print_gid_type);
}
-static ssize_t show_port_pkey(struct ib_port *p, struct port_attribute *attr,
- char *buf)
+static ssize_t show_port_pkey(struct ib_device *ibdev, u32 port_num,
+ struct ib_port_attribute *attr, char *buf)
{
struct port_table_attribute *tab_attr =
container_of(attr, struct port_table_attribute, attr);
u16 pkey;
int ret;
- ret = ib_query_pkey(p->ibdev, p->port_num, tab_attr->index, &pkey);
+ ret = ib_query_pkey(ibdev, port_num, tab_attr->index, &pkey);
if (ret)
return ret;
@@ -594,8 +610,8 @@ static int get_perf_mad(struct ib_device *dev, int port_num, __be16 attr,
return ret;
}
-static ssize_t show_pma_counter(struct ib_port *p, struct port_attribute *attr,
- char *buf)
+static ssize_t show_pma_counter(struct ib_device *ibdev, u32 port_num,
+ struct ib_port_attribute *attr, char *buf)
{
struct port_table_attribute *tab_attr =
container_of(attr, struct port_table_attribute, attr);
@@ -605,7 +621,7 @@ static ssize_t show_pma_counter(struct ib_port *p, struct port_attribute *attr,
u8 data[8];
int len;
- ret = get_perf_mad(p->ibdev, p->port_num, tab_attr->attr_id, &data,
+ ret = get_perf_mad(ibdev, port_num, tab_attr->attr_id, &data,
40 + offset / 8, sizeof(data));
if (ret < 0)
return ret;
@@ -1077,10 +1093,11 @@ struct rdma_hw_stats *ib_get_hw_stats_port(struct ib_device *ibdev,
return ibdev->port_data[port_num].sysfs->hw_stats_data->stats;
}
-static int alloc_port_table_group(
- const char *name, struct attribute_group *group,
- struct port_table_attribute *attrs, size_t num,
- ssize_t (*show)(struct ib_port *, struct port_attribute *, char *buf))
+static int
+alloc_port_table_group(const char *name, struct attribute_group *group,
+ struct port_table_attribute *attrs, size_t num,
+ ssize_t (*show)(struct ib_device *ibdev, u32 port_num,
+ struct ib_port_attribute *, char *buf))
{
struct attribute **attr_list;
int i;
diff --git a/include/rdma/ib_sysfs.h b/include/rdma/ib_sysfs.h
new file mode 100644
index 00000000000000..f869d0e4fd3030
--- /dev/null
+++ b/include/rdma/ib_sysfs.h
@@ -0,0 +1,41 @@
+/* SPDX-License-Identifier: GPL-2.0 OR BSD-3-Clause */
+/*
+ * Copyright (c) 2021 Mellanox Technologies Ltd. All rights reserved.
+ */
+#ifndef DEF_RDMA_IB_SYSFS_H
+#define DEF_RDMA_IB_SYSFS_H
+
+#include <linux/sysfs.h>
+
+struct ib_device;
+
+struct ib_port_attribute {
+ struct attribute attr;
+ ssize_t (*show)(struct ib_device *ibdev, u32 port_num,
+ struct ib_port_attribute *attr, char *buf);
+ ssize_t (*store)(struct ib_device *ibdev, u32 port_num,
+ struct ib_port_attribute *attr, const char *buf,
+ size_t count);
+};
+
+#define IB_PORT_ATTR_RW(_name) \
+ struct ib_port_attribute ib_port_attr_##_name = __ATTR_RW(_name)
+
+#define IB_PORT_ATTR_ADMIN_RW(_name) \
+ struct ib_port_attribute ib_port_attr_##_name = \
+ __ATTR_RW_MODE(_name, 0600)
+
+#define IB_PORT_ATTR_RO(_name) \
+ struct ib_port_attribute ib_port_attr_##_name = __ATTR_RO(_name)
+
+#define IB_PORT_ATTR_WO(_name) \
+ struct ib_port_attribute ib_port_attr_##_name = __ATTR_WO(_name)
+
+int ib_port_sysfs_create_groups(struct ib_device *ibdev, u32 port_num,
+ const struct attribute_group **groups);
+void ib_port_sysfs_remove_groups(struct ib_device *ibdev, u32 port_num,
+ const struct attribute_group **groups);
+struct ib_device *ib_port_sysfs_get_ibdev_kobj(struct kobject *kobj,
+ u32 *port_num);
+
+#endif
--
2.31.1
^ permalink raw reply related [flat|nested] 35+ messages in thread* Re: [PATCH 09/13] RDMA/core: Expose the ib port sysfs attribute machinery
2021-05-17 16:47 ` [PATCH 09/13] RDMA/core: Expose the ib port sysfs attribute machinery Jason Gunthorpe
@ 2021-05-17 17:12 ` Greg KH
2021-05-17 17:31 ` Jason Gunthorpe
0 siblings, 1 reply; 35+ messages in thread
From: Greg KH @ 2021-05-17 17:12 UTC (permalink / raw)
To: Jason Gunthorpe; +Cc: Doug Ledford, linux-rdma, Kees Cook, Nathan Chancellor
On Mon, May 17, 2021 at 01:47:37PM -0300, Jason Gunthorpe wrote:
> Other things outside the core code are creating attributes against the
> port. This patch exposes the basic machinery to do this.
>
> The ib_port_attribute type allows creating groups of attributes attatched
> to the port and comes with the usual machinery to do this.
>
> Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>
> ---
> drivers/infiniband/core/sysfs.c | 217 +++++++++++++++++---------------
> include/rdma/ib_sysfs.h | 41 ++++++
> 2 files changed, 158 insertions(+), 100 deletions(-)
> create mode 100644 include/rdma/ib_sysfs.h
>
> diff --git a/drivers/infiniband/core/sysfs.c b/drivers/infiniband/core/sysfs.c
> index ce6aecbb5a7616..58a45548bf1568 100644
> --- a/drivers/infiniband/core/sysfs.c
> +++ b/drivers/infiniband/core/sysfs.c
> @@ -44,24 +44,10 @@
> #include <rdma/ib_pma.h>
> #include <rdma/ib_cache.h>
> #include <rdma/rdma_counter.h>
> -
> -struct ib_port;
> -
> -struct port_attribute {
> - struct attribute attr;
> - ssize_t (*show)(struct ib_port *, struct port_attribute *, char *buf);
> - ssize_t (*store)(struct ib_port *, struct port_attribute *,
> - const char *buf, size_t count);
> -};
> -
> -#define PORT_ATTR(_name, _mode, _show, _store) \
> -struct port_attribute port_attr_##_name = __ATTR(_name, _mode, _show, _store)
> -
> -#define PORT_ATTR_RO(_name) \
> -struct port_attribute port_attr_##_name = __ATTR_RO(_name)
> +#include <rdma/ib_sysfs.h>
>
> struct port_table_attribute {
> - struct port_attribute attr;
> + struct ib_port_attribute attr;
> char name[8];
> int index;
> __be16 attr_id;
> @@ -97,7 +83,7 @@ struct hw_stats_device_attribute {
> };
>
> struct hw_stats_port_attribute {
> - struct port_attribute attr;
> + struct ib_port_attribute attr;
> ssize_t (*show)(struct ib_device *ibdev, struct rdma_hw_stats *stats,
> unsigned int index, unsigned int port_num, char *buf);
> ssize_t (*store)(struct ib_device *ibdev, struct rdma_hw_stats *stats,
> @@ -119,29 +105,55 @@ struct hw_stats_port_data {
> static ssize_t port_attr_show(struct kobject *kobj,
> struct attribute *attr, char *buf)
> {
> - struct port_attribute *port_attr =
> - container_of(attr, struct port_attribute, attr);
> + struct ib_port_attribute *port_attr =
> + container_of(attr, struct ib_port_attribute, attr);
> struct ib_port *p = container_of(kobj, struct ib_port, kobj);
>
> if (!port_attr->show)
> return -EIO;
>
> - return port_attr->show(p, port_attr, buf);
> + return port_attr->show(p->ibdev, p->port_num, port_attr, buf);
> }
>
> static ssize_t port_attr_store(struct kobject *kobj,
> struct attribute *attr,
> const char *buf, size_t count)
> {
> - struct port_attribute *port_attr =
> - container_of(attr, struct port_attribute, attr);
> + struct ib_port_attribute *port_attr =
> + container_of(attr, struct ib_port_attribute, attr);
> struct ib_port *p = container_of(kobj, struct ib_port, kobj);
>
> if (!port_attr->store)
> return -EIO;
> - return port_attr->store(p, port_attr, buf, count);
> + return port_attr->store(p->ibdev, p->port_num, port_attr, buf, count);
> }
>
> +int ib_port_sysfs_create_groups(struct ib_device *ibdev, u32 port_num,
> + const struct attribute_group **groups)
> +{
> + return sysfs_create_groups(&ibdev->port_data[port_num].sysfs->kobj,
> + groups);
> +}
> +EXPORT_SYMBOL(ib_port_sysfs_create_groups);
You are wrapping _GPL symbols here with a "convenience" function, please
make these all EXPORT_SYMBOL_GPL() so I don't get nervous.
thanks,
greg k-h
^ permalink raw reply [flat|nested] 35+ messages in thread* Re: [PATCH 09/13] RDMA/core: Expose the ib port sysfs attribute machinery
2021-05-17 17:12 ` Greg KH
@ 2021-05-17 17:31 ` Jason Gunthorpe
2021-05-17 17:37 ` Greg KH
0 siblings, 1 reply; 35+ messages in thread
From: Jason Gunthorpe @ 2021-05-17 17:31 UTC (permalink / raw)
To: Greg KH; +Cc: Doug Ledford, linux-rdma, Kees Cook, Nathan Chancellor
On Mon, May 17, 2021 at 07:12:55PM +0200, Greg KH wrote:
> > +int ib_port_sysfs_create_groups(struct ib_device *ibdev, u32 port_num,
> > + const struct attribute_group **groups)
> > +{
> > + return sysfs_create_groups(&ibdev->port_data[port_num].sysfs->kobj,
> > + groups);
> > +}
> > +EXPORT_SYMBOL(ib_port_sysfs_create_groups);
>
> You are wrapping _GPL symbols here with a "convenience" function, please
> make these all EXPORT_SYMBOL_GPL() so I don't get nervous.
These functions get deleted in a following patch once everything can
be switched to ops->get_port_groups(), which provides even less
flexability for the driver to do things wrong.
The whole subsystem already uses !GPL export so it is very strange to
see a GPL symbol at all:
$ git grep EXPORT_SYMBOL\( drivers/infiniband/core/ | wc -l
310
$ git grep EXPORT_SYMBOL_GPL\( drivers/infiniband/core/ | wc -l
1
Anyhow, if it makes you happy I'll change it.
Jason
^ permalink raw reply [flat|nested] 35+ messages in thread* Re: [PATCH 09/13] RDMA/core: Expose the ib port sysfs attribute machinery
2021-05-17 17:31 ` Jason Gunthorpe
@ 2021-05-17 17:37 ` Greg KH
0 siblings, 0 replies; 35+ messages in thread
From: Greg KH @ 2021-05-17 17:37 UTC (permalink / raw)
To: Jason Gunthorpe; +Cc: Doug Ledford, linux-rdma, Kees Cook, Nathan Chancellor
On Mon, May 17, 2021 at 02:31:00PM -0300, Jason Gunthorpe wrote:
> On Mon, May 17, 2021 at 07:12:55PM +0200, Greg KH wrote:
>
> > > +int ib_port_sysfs_create_groups(struct ib_device *ibdev, u32 port_num,
> > > + const struct attribute_group **groups)
> > > +{
> > > + return sysfs_create_groups(&ibdev->port_data[port_num].sysfs->kobj,
> > > + groups);
> > > +}
> > > +EXPORT_SYMBOL(ib_port_sysfs_create_groups);
> >
> > You are wrapping _GPL symbols here with a "convenience" function, please
> > make these all EXPORT_SYMBOL_GPL() so I don't get nervous.
>
> These functions get deleted in a following patch once everything can
> be switched to ops->get_port_groups(), which provides even less
> flexability for the driver to do things wrong.
>
> The whole subsystem already uses !GPL export so it is very strange to
> see a GPL symbol at all:
>
> $ git grep EXPORT_SYMBOL\( drivers/infiniband/core/ | wc -l
> 310
> $ git grep EXPORT_SYMBOL_GPL\( drivers/infiniband/core/ | wc -l
> 1
>
> Anyhow, if it makes you happy I'll change it.
Please do, it's a simple wrapper function.
thanks,
greg k-h
^ permalink raw reply [flat|nested] 35+ messages in thread
* [PATCH 10/13] RDMA/cm: Use an attribute_group on the ib_port_attribute intead of kobj's
2021-05-17 16:47 [PATCH 00/13] Reorganize sysfs file creation for struct ib_devices Jason Gunthorpe
` (8 preceding siblings ...)
2021-05-17 16:47 ` [PATCH 09/13] RDMA/core: Expose the ib port sysfs attribute machinery Jason Gunthorpe
@ 2021-05-17 16:47 ` Jason Gunthorpe
2021-05-17 16:47 ` [PATCH 11/13] RDMA/qib: Use attributes for the port sysfs Jason Gunthorpe
` (3 subsequent siblings)
13 siblings, 0 replies; 35+ messages in thread
From: Jason Gunthorpe @ 2021-05-17 16:47 UTC (permalink / raw)
To: Doug Ledford, linux-rdma; +Cc: Greg KH, Kees Cook, Nathan Chancellor
This code is trying to attach a list of counters grouped into 4 groups to
the ib_port sysfs. Instead of creating a bunch of kobjects simply express
everything naturally as an ib_port_attribute and add a single
attribute_groups list.
Remove all the naked kobject manipulations.
Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>
---
drivers/infiniband/core/cm.c | 227 ++++++++++++----------------
drivers/infiniband/core/core_priv.h | 8 +-
drivers/infiniband/core/sysfs.c | 50 ++----
3 files changed, 119 insertions(+), 166 deletions(-)
diff --git a/drivers/infiniband/core/cm.c b/drivers/infiniband/core/cm.c
index 0ead0d22315401..fc8fcb502eb479 100644
--- a/drivers/infiniband/core/cm.c
+++ b/drivers/infiniband/core/cm.c
@@ -25,6 +25,7 @@
#include <rdma/ib_cache.h>
#include <rdma/ib_cm.h>
+#include <rdma/ib_sysfs.h>
#include "cm_msgs.h"
#include "core_priv.h"
#include "cm_trace.h"
@@ -150,53 +151,10 @@ enum {
CM_COUNTER_GROUPS
};
-static char const counter_group_names[CM_COUNTER_GROUPS]
- [sizeof("cm_rx_duplicates")] = {
- "cm_tx_msgs", "cm_tx_retries",
- "cm_rx_msgs", "cm_rx_duplicates"
-};
-
-struct cm_counter_group {
- struct kobject obj;
- atomic_long_t counter[CM_ATTR_COUNT];
-};
-
struct cm_counter_attribute {
- struct attribute attr;
- int index;
-};
-
-#define CM_COUNTER_ATTR(_name, _index) \
-struct cm_counter_attribute cm_##_name##_counter_attr = { \
- .attr = { .name = __stringify(_name), .mode = 0444 }, \
- .index = _index \
-}
-
-static CM_COUNTER_ATTR(req, CM_REQ_COUNTER);
-static CM_COUNTER_ATTR(mra, CM_MRA_COUNTER);
-static CM_COUNTER_ATTR(rej, CM_REJ_COUNTER);
-static CM_COUNTER_ATTR(rep, CM_REP_COUNTER);
-static CM_COUNTER_ATTR(rtu, CM_RTU_COUNTER);
-static CM_COUNTER_ATTR(dreq, CM_DREQ_COUNTER);
-static CM_COUNTER_ATTR(drep, CM_DREP_COUNTER);
-static CM_COUNTER_ATTR(sidr_req, CM_SIDR_REQ_COUNTER);
-static CM_COUNTER_ATTR(sidr_rep, CM_SIDR_REP_COUNTER);
-static CM_COUNTER_ATTR(lap, CM_LAP_COUNTER);
-static CM_COUNTER_ATTR(apr, CM_APR_COUNTER);
-
-static struct attribute *cm_counter_default_attrs[] = {
- &cm_req_counter_attr.attr,
- &cm_mra_counter_attr.attr,
- &cm_rej_counter_attr.attr,
- &cm_rep_counter_attr.attr,
- &cm_rtu_counter_attr.attr,
- &cm_dreq_counter_attr.attr,
- &cm_drep_counter_attr.attr,
- &cm_sidr_req_counter_attr.attr,
- &cm_sidr_rep_counter_attr.attr,
- &cm_lap_counter_attr.attr,
- &cm_apr_counter_attr.attr,
- NULL
+ struct ib_port_attribute attr;
+ unsigned short group;
+ unsigned short index;
};
struct cm_port {
@@ -205,7 +163,7 @@ struct cm_port {
u32 port_num;
struct list_head cm_priv_prim_list;
struct list_head cm_priv_altr_list;
- struct cm_counter_group counter_group[CM_COUNTER_GROUPS];
+ atomic_long_t counters[CM_COUNTER_GROUPS][CM_ATTR_COUNT];
};
struct cm_device {
@@ -1934,8 +1892,8 @@ static void cm_dup_req_handler(struct cm_work *work,
struct ib_mad_send_buf *msg = NULL;
int ret;
- atomic_long_inc(&work->port->counter_group[CM_RECV_DUPLICATES].
- counter[CM_REQ_COUNTER]);
+ atomic_long_inc(
+ &work->port->counters[CM_RECV_DUPLICATES][CM_REQ_COUNTER]);
/* Quick state check to discard duplicate REQs. */
spin_lock_irq(&cm_id_priv->lock);
@@ -2426,8 +2384,8 @@ static void cm_dup_rep_handler(struct cm_work *work)
if (!cm_id_priv)
return;
- atomic_long_inc(&work->port->counter_group[CM_RECV_DUPLICATES].
- counter[CM_REP_COUNTER]);
+ atomic_long_inc(
+ &work->port->counters[CM_RECV_DUPLICATES][CM_REP_COUNTER]);
ret = cm_alloc_response_msg(work->port, work->mad_recv_wc, &msg);
if (ret)
goto deref;
@@ -2604,8 +2562,8 @@ static int cm_rtu_handler(struct cm_work *work)
if (cm_id_priv->id.state != IB_CM_REP_SENT &&
cm_id_priv->id.state != IB_CM_MRA_REP_RCVD) {
spin_unlock_irq(&cm_id_priv->lock);
- atomic_long_inc(&work->port->counter_group[CM_RECV_DUPLICATES].
- counter[CM_RTU_COUNTER]);
+ atomic_long_inc(&work->port->counters[CM_RECV_DUPLICATES]
+ [CM_RTU_COUNTER]);
goto out;
}
cm_id_priv->id.state = IB_CM_ESTABLISHED;
@@ -2810,8 +2768,8 @@ static int cm_dreq_handler(struct cm_work *work)
cpu_to_be32(IBA_GET(CM_DREQ_REMOTE_COMM_ID, dreq_msg)),
cpu_to_be32(IBA_GET(CM_DREQ_LOCAL_COMM_ID, dreq_msg)));
if (!cm_id_priv) {
- atomic_long_inc(&work->port->counter_group[CM_RECV_DUPLICATES].
- counter[CM_DREQ_COUNTER]);
+ atomic_long_inc(&work->port->counters[CM_RECV_DUPLICATES]
+ [CM_DREQ_COUNTER]);
cm_issue_drep(work->port, work->mad_recv_wc);
trace_icm_no_priv_err(
IBA_GET(CM_DREQ_LOCAL_COMM_ID, dreq_msg),
@@ -2840,8 +2798,8 @@ static int cm_dreq_handler(struct cm_work *work)
case IB_CM_MRA_REP_RCVD:
break;
case IB_CM_TIMEWAIT:
- atomic_long_inc(&work->port->counter_group[CM_RECV_DUPLICATES].
- counter[CM_DREQ_COUNTER]);
+ atomic_long_inc(&work->port->counters[CM_RECV_DUPLICATES]
+ [CM_DREQ_COUNTER]);
msg = cm_alloc_response_msg_no_ah(work->port, work->mad_recv_wc);
if (IS_ERR(msg))
goto unlock;
@@ -2856,8 +2814,8 @@ static int cm_dreq_handler(struct cm_work *work)
cm_free_msg(msg);
goto deref;
case IB_CM_DREQ_RCVD:
- atomic_long_inc(&work->port->counter_group[CM_RECV_DUPLICATES].
- counter[CM_DREQ_COUNTER]);
+ atomic_long_inc(&work->port->counters[CM_RECV_DUPLICATES]
+ [CM_DREQ_COUNTER]);
goto unlock;
default:
trace_icm_dreq_unknown_err(&cm_id_priv->id);
@@ -3212,17 +3170,17 @@ static int cm_mra_handler(struct cm_work *work)
ib_modify_mad(cm_id_priv->av.port->mad_agent,
cm_id_priv->msg, timeout)) {
if (cm_id_priv->id.lap_state == IB_CM_MRA_LAP_RCVD)
- atomic_long_inc(&work->port->
- counter_group[CM_RECV_DUPLICATES].
- counter[CM_MRA_COUNTER]);
+ atomic_long_inc(
+ &work->port->counters[CM_RECV_DUPLICATES]
+ [CM_MRA_COUNTER]);
goto out;
}
cm_id_priv->id.lap_state = IB_CM_MRA_LAP_RCVD;
break;
case IB_CM_MRA_REQ_RCVD:
case IB_CM_MRA_REP_RCVD:
- atomic_long_inc(&work->port->counter_group[CM_RECV_DUPLICATES].
- counter[CM_MRA_COUNTER]);
+ atomic_long_inc(&work->port->counters[CM_RECV_DUPLICATES]
+ [CM_MRA_COUNTER]);
fallthrough;
default:
trace_icm_mra_unknown_err(&cm_id_priv->id);
@@ -3328,8 +3286,8 @@ static int cm_lap_handler(struct cm_work *work)
case IB_CM_LAP_IDLE:
break;
case IB_CM_MRA_LAP_SENT:
- atomic_long_inc(&work->port->counter_group[CM_RECV_DUPLICATES].
- counter[CM_LAP_COUNTER]);
+ atomic_long_inc(&work->port->counters[CM_RECV_DUPLICATES]
+ [CM_LAP_COUNTER]);
msg = cm_alloc_response_msg_no_ah(work->port, work->mad_recv_wc);
if (IS_ERR(msg))
goto unlock;
@@ -3346,8 +3304,8 @@ static int cm_lap_handler(struct cm_work *work)
cm_free_msg(msg);
goto deref;
case IB_CM_LAP_RCVD:
- atomic_long_inc(&work->port->counter_group[CM_RECV_DUPLICATES].
- counter[CM_LAP_COUNTER]);
+ atomic_long_inc(&work->port->counters[CM_RECV_DUPLICATES]
+ [CM_LAP_COUNTER]);
goto unlock;
default:
goto unlock;
@@ -3576,8 +3534,8 @@ static int cm_sidr_req_handler(struct cm_work *work)
listen_cm_id_priv = cm_insert_remote_sidr(cm_id_priv);
if (listen_cm_id_priv) {
spin_unlock_irq(&cm.lock);
- atomic_long_inc(&work->port->counter_group[CM_RECV_DUPLICATES].
- counter[CM_SIDR_REQ_COUNTER]);
+ atomic_long_inc(&work->port->counters[CM_RECV_DUPLICATES]
+ [CM_SIDR_REQ_COUNTER]);
goto out; /* Duplicate message. */
}
cm_id_priv->id.state = IB_CM_SIDR_REQ_RCVD;
@@ -3821,12 +3779,10 @@ static void cm_send_handler(struct ib_mad_agent *mad_agent,
if (!msg->context[0] && (attr_index != CM_REJ_COUNTER))
msg->retries = 1;
- atomic_long_add(1 + msg->retries,
- &port->counter_group[CM_XMIT].counter[attr_index]);
+ atomic_long_add(1 + msg->retries, &port->counters[CM_XMIT][attr_index]);
if (msg->retries)
atomic_long_add(msg->retries,
- &port->counter_group[CM_XMIT_RETRIES].
- counter[attr_index]);
+ &port->counters[CM_XMIT_RETRIES][attr_index]);
switch (mad_send_wc->status) {
case IB_WC_SUCCESS:
@@ -4063,8 +4019,7 @@ static void cm_recv_handler(struct ib_mad_agent *mad_agent,
}
attr_id = be16_to_cpu(mad_recv_wc->recv_buf.mad->mad_hdr.attr_id);
- atomic_long_inc(&port->counter_group[CM_RECV].
- counter[attr_id - CM_ATTR_ID_OFFSET]);
+ atomic_long_inc(&port->counters[CM_RECV][attr_id - CM_ATTR_ID_OFFSET]);
work = kmalloc(struct_size(work, path, paths), GFP_KERNEL);
if (!work) {
@@ -4262,59 +4217,74 @@ int ib_cm_init_qp_attr(struct ib_cm_id *cm_id,
}
EXPORT_SYMBOL(ib_cm_init_qp_attr);
-static ssize_t cm_show_counter(struct kobject *obj, struct attribute *attr,
- char *buf)
+static ssize_t cm_show_counter(struct ib_device *ibdev, u32 port_num,
+ struct ib_port_attribute *attr, char *buf)
{
- struct cm_counter_group *group;
- struct cm_counter_attribute *cm_attr;
+ struct cm_counter_attribute *cm_attr =
+ container_of(attr, struct cm_counter_attribute, attr);
+ struct cm_device *cm_dev = ib_get_client_data(ibdev, &cm_client);
- group = container_of(obj, struct cm_counter_group, obj);
- cm_attr = container_of(attr, struct cm_counter_attribute, attr);
+ if (WARN_ON(!cm_dev))
+ return -EINVAL;
- return sysfs_emit(buf, "%ld\n",
- atomic_long_read(&group->counter[cm_attr->index]));
+ return sysfs_emit(
+ buf, "%ld\n",
+ atomic_long_read(
+ &cm_dev->port[port_num - 1]
+ ->counters[cm_attr->group][cm_attr->index]));
}
-static const struct sysfs_ops cm_counter_ops = {
- .show = cm_show_counter
-};
-
-static struct kobj_type cm_counter_obj_type = {
- .sysfs_ops = &cm_counter_ops,
- .default_attrs = cm_counter_default_attrs
-};
-
-static int cm_create_port_fs(struct cm_port *port)
-{
- int i, ret;
-
- for (i = 0; i < CM_COUNTER_GROUPS; i++) {
- ret = ib_port_register_module_stat(port->cm_dev->ib_device,
- port->port_num,
- &port->counter_group[i].obj,
- &cm_counter_obj_type,
- counter_group_names[i]);
- if (ret)
- goto error;
+#define CM_COUNTER_ATTR(_name, _group, _index) \
+ { \
+ .attr = __ATTR(_name, 0444, cm_show_counter, NULL), \
+ .group = _group, .index = _index \
}
- return 0;
-
-error:
- while (i--)
- ib_port_unregister_module_stat(&port->counter_group[i].obj);
- return ret;
-
-}
-
-static void cm_remove_port_fs(struct cm_port *port)
-{
- int i;
-
- for (i = 0; i < CM_COUNTER_GROUPS; i++)
- ib_port_unregister_module_stat(&port->counter_group[i].obj);
+#define CM_COUNTER_GROUP(_group, _name) \
+ static struct cm_counter_attribute cm_counter_attr_##_group[] = { \
+ CM_COUNTER_ATTR(req, _group, CM_REQ_COUNTER), \
+ CM_COUNTER_ATTR(mra, _group, CM_MRA_COUNTER), \
+ CM_COUNTER_ATTR(rej, _group, CM_REJ_COUNTER), \
+ CM_COUNTER_ATTR(rep, _group, CM_REP_COUNTER), \
+ CM_COUNTER_ATTR(rtu, _group, CM_RTU_COUNTER), \
+ CM_COUNTER_ATTR(dreq, _group, CM_DREQ_COUNTER), \
+ CM_COUNTER_ATTR(drep, _group, CM_DREP_COUNTER), \
+ CM_COUNTER_ATTR(sidr_req, _group, CM_SIDR_REQ_COUNTER), \
+ CM_COUNTER_ATTR(sidr_rep, _group, CM_SIDR_REP_COUNTER), \
+ CM_COUNTER_ATTR(lap, _group, CM_LAP_COUNTER), \
+ CM_COUNTER_ATTR(apr, _group, CM_APR_COUNTER), \
+ }; \
+ static struct attribute *cm_counter_attrs_##_group[] = { \
+ &cm_counter_attr_##_group[0].attr.attr, \
+ &cm_counter_attr_##_group[1].attr.attr, \
+ &cm_counter_attr_##_group[2].attr.attr, \
+ &cm_counter_attr_##_group[3].attr.attr, \
+ &cm_counter_attr_##_group[4].attr.attr, \
+ &cm_counter_attr_##_group[5].attr.attr, \
+ &cm_counter_attr_##_group[6].attr.attr, \
+ &cm_counter_attr_##_group[7].attr.attr, \
+ &cm_counter_attr_##_group[8].attr.attr, \
+ &cm_counter_attr_##_group[9].attr.attr, \
+ &cm_counter_attr_##_group[10].attr.attr, \
+ NULL, \
+ }; \
+ static const struct attribute_group cm_counter_group_##_group = { \
+ .name = _name, \
+ .attrs = cm_counter_attrs_##_group, \
+ };
-}
+CM_COUNTER_GROUP(CM_XMIT, "cm_tx_msgs")
+CM_COUNTER_GROUP(CM_XMIT_RETRIES, "cm_tx_retries")
+CM_COUNTER_GROUP(CM_RECV, "cm_rx_msgs")
+CM_COUNTER_GROUP(CM_RECV_DUPLICATES, "cm_rx_duplicates")
+
+static const struct attribute_group *cm_counter_groups[] = {
+ &cm_counter_group_CM_XMIT,
+ &cm_counter_group_CM_XMIT_RETRIES,
+ &cm_counter_group_CM_RECV,
+ &cm_counter_group_CM_RECV_DUPLICATES,
+ NULL,
+};
static int cm_add_one(struct ib_device *ib_device)
{
@@ -4341,6 +4311,8 @@ static int cm_add_one(struct ib_device *ib_device)
cm_dev->ack_delay = ib_device->attrs.local_ca_ack_delay;
cm_dev->going_down = 0;
+ ib_set_client_data(ib_device, &cm_client, cm_dev);
+
set_bit(IB_MGMT_METHOD_SEND, reg_req.method_mask);
rdma_for_each_port (ib_device, i) {
if (!rdma_cap_ib_cm(ib_device, i))
@@ -4359,7 +4331,8 @@ static int cm_add_one(struct ib_device *ib_device)
INIT_LIST_HEAD(&port->cm_priv_prim_list);
INIT_LIST_HEAD(&port->cm_priv_altr_list);
- ret = cm_create_port_fs(port);
+ ret = ib_port_register_client_groups(ib_device, i,
+ cm_counter_groups);
if (ret)
goto error1;
@@ -4388,8 +4361,6 @@ static int cm_add_one(struct ib_device *ib_device)
goto free;
}
- ib_set_client_data(ib_device, &cm_client, cm_dev);
-
write_lock_irqsave(&cm.device_lock, flags);
list_add_tail(&cm_dev->list, &cm.device_list);
write_unlock_irqrestore(&cm.device_lock, flags);
@@ -4398,7 +4369,7 @@ static int cm_add_one(struct ib_device *ib_device)
error3:
ib_unregister_mad_agent(port->mad_agent);
error2:
- cm_remove_port_fs(port);
+ ib_port_unregister_client_groups(ib_device, i, cm_counter_groups);
error1:
port_modify.set_port_cap_mask = 0;
port_modify.clr_port_cap_mask = IB_PORT_CM_SUP;
@@ -4410,7 +4381,8 @@ static int cm_add_one(struct ib_device *ib_device)
port = cm_dev->port[i-1];
ib_modify_port(ib_device, port->port_num, 0, &port_modify);
ib_unregister_mad_agent(port->mad_agent);
- cm_remove_port_fs(port);
+ ib_port_unregister_client_groups(ib_device, i,
+ cm_counter_groups);
kfree(port);
}
free:
@@ -4462,7 +4434,8 @@ static void cm_remove_one(struct ib_device *ib_device, void *client_data)
port->mad_agent = NULL;
spin_unlock_irq(&cm.state_lock);
ib_unregister_mad_agent(cur_mad_agent);
- cm_remove_port_fs(port);
+ ib_port_unregister_client_groups(ib_device, i,
+ cm_counter_groups);
kfree(port);
}
diff --git a/drivers/infiniband/core/core_priv.h b/drivers/infiniband/core/core_priv.h
index 6066c4b39876d6..78782cce47a19f 100644
--- a/drivers/infiniband/core/core_priv.h
+++ b/drivers/infiniband/core/core_priv.h
@@ -382,10 +382,10 @@ int ib_setup_device_attrs(struct ib_device *ibdev);
int rdma_compatdev_set(u8 enable);
-int ib_port_register_module_stat(struct ib_device *device, u32 port_num,
- struct kobject *kobj, struct kobj_type *ktype,
- const char *name);
-void ib_port_unregister_module_stat(struct kobject *kobj);
+int ib_port_register_client_groups(struct ib_device *ibdev, u32 port_num,
+ const struct attribute_group **groups);
+void ib_port_unregister_client_groups(struct ib_device *ibdev, u32 port_num,
+ const struct attribute_group **groups);
int ib_device_set_netns_put(struct sk_buff *skb,
struct ib_device *dev, u32 ns_fd);
diff --git a/drivers/infiniband/core/sysfs.c b/drivers/infiniband/core/sysfs.c
index 58a45548bf1568..5d9c8bfc280d8f 100644
--- a/drivers/infiniband/core/sysfs.c
+++ b/drivers/infiniband/core/sysfs.c
@@ -1449,46 +1449,26 @@ int ib_setup_port_attrs(struct ib_core_device *coredev)
}
/**
- * ib_port_register_module_stat - add module counters under relevant port
- * of IB device.
+ * ib_port_register_client_group - Add an ib_client's attributes to the port
*
- * @device: IB device to add counters
+ * @ibdev: IB device to add counters
* @port_num: valid port number
- * @kobj: pointer to the kobject to initialize
- * @ktype: pointer to the ktype for this kobject.
- * @name: the name of the kobject
+ * @groups: Group list of attributes
+ *
+ * Do not use. Only for legacy sysfs compatibility.
*/
-int ib_port_register_module_stat(struct ib_device *device, u32 port_num,
- struct kobject *kobj, struct kobj_type *ktype,
- const char *name)
+int ib_port_register_client_groups(struct ib_device *ibdev, u32 port_num,
+ const struct attribute_group **groups)
{
- struct kobject *p, *t;
- int ret;
-
- list_for_each_entry_safe(p, t, &device->coredev.port_list, entry) {
- struct ib_port *port = container_of(p, struct ib_port, kobj);
-
- if (port->port_num != port_num)
- continue;
-
- ret = kobject_init_and_add(kobj, ktype, &port->kobj, "%s",
- name);
- if (ret) {
- kobject_put(kobj);
- return ret;
- }
- }
-
- return 0;
+ return sysfs_create_groups(&ibdev->port_data[port_num].sysfs->kobj,
+ groups);
}
-EXPORT_SYMBOL(ib_port_register_module_stat);
+EXPORT_SYMBOL(ib_port_register_client_groups);
-/**
- * ib_port_unregister_module_stat - release module counters
- * @kobj: pointer to the kobject to release
- */
-void ib_port_unregister_module_stat(struct kobject *kobj)
+void ib_port_unregister_client_groups(struct ib_device *ibdev, u32 port_num,
+ const struct attribute_group **groups)
{
- kobject_put(kobj);
+ return sysfs_remove_groups(&ibdev->port_data[port_num].sysfs->kobj,
+ groups);
}
-EXPORT_SYMBOL(ib_port_unregister_module_stat);
+EXPORT_SYMBOL(ib_port_unregister_client_groups);
--
2.31.1
^ permalink raw reply related [flat|nested] 35+ messages in thread* [PATCH 11/13] RDMA/qib: Use attributes for the port sysfs
2021-05-17 16:47 [PATCH 00/13] Reorganize sysfs file creation for struct ib_devices Jason Gunthorpe
` (9 preceding siblings ...)
2021-05-17 16:47 ` [PATCH 10/13] RDMA/cm: Use an attribute_group on the ib_port_attribute intead of kobj's Jason Gunthorpe
@ 2021-05-17 16:47 ` Jason Gunthorpe
2021-05-17 17:11 ` Greg KH
2021-05-17 16:47 ` [PATCH 12/13] RDMA/hfi1: " Jason Gunthorpe
` (2 subsequent siblings)
13 siblings, 1 reply; 35+ messages in thread
From: Jason Gunthorpe @ 2021-05-17 16:47 UTC (permalink / raw)
To: Dennis Dalessandro, Doug Ledford, linux-rdma, Mike Marciniszyn
Cc: Greg KH, Kees Cook, Nathan Chancellor
qib should not be creating a mess of kobjects to attach to the port
kobject - this is all attributes. The proper API is to create an
attribute_group list and create it against the port's kobject.
Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>
---
drivers/infiniband/hw/qib/qib.h | 5 +-
drivers/infiniband/hw/qib/qib_sysfs.c | 596 +++++++++++---------------
2 files changed, 248 insertions(+), 353 deletions(-)
diff --git a/drivers/infiniband/hw/qib/qib.h b/drivers/infiniband/hw/qib/qib.h
index 88497739029e02..3decd6d0843172 100644
--- a/drivers/infiniband/hw/qib/qib.h
+++ b/drivers/infiniband/hw/qib/qib.h
@@ -521,10 +521,7 @@ struct qib_pportdata {
struct qib_devdata *dd;
struct qib_chippport_specific *cpspec; /* chip-specific per-port */
- struct kobject pport_kobj;
- struct kobject pport_cc_kobj;
- struct kobject sl2vl_kobj;
- struct kobject diagc_kobj;
+ const struct attribute_group *groups[5];
/* GUID for this interface, in network order */
__be64 guid;
diff --git a/drivers/infiniband/hw/qib/qib_sysfs.c b/drivers/infiniband/hw/qib/qib_sysfs.c
index 5e9e66f2706479..2c81285d245fa7 100644
--- a/drivers/infiniband/hw/qib/qib_sysfs.c
+++ b/drivers/infiniband/hw/qib/qib_sysfs.c
@@ -32,25 +32,38 @@
* SOFTWARE.
*/
#include <linux/ctype.h>
+#include <rdma/ib_sysfs.h>
#include "qib.h"
#include "qib_mad.h"
-/* start of per-port functions */
+static struct qib_pportdata *qib_get_pportdata_kobj(struct kobject *kobj)
+{
+ u32 port_num;
+ struct ib_device *ibdev = ib_port_sysfs_get_ibdev_kobj(kobj, &port_num);
+ struct qib_devdata *dd = dd_from_ibdev(ibdev);
+
+ return &dd->pport[port_num - 1];
+}
+
/*
* Get/Set heartbeat enable. OR of 1=enabled, 2=auto
*/
-static ssize_t show_hrtbt_enb(struct qib_pportdata *ppd, char *buf)
+static ssize_t hrtbt_enable_show(struct ib_device *ibdev, u32 port_num,
+ struct ib_port_attribute *attr, char *buf)
{
- struct qib_devdata *dd = ppd->dd;
+ struct qib_devdata *dd = dd_from_ibdev(ibdev);
+ struct qib_pportdata *ppd = &dd->pport[port_num - 1];
return sysfs_emit(buf, "%d\n", dd->f_get_ib_cfg(ppd, QIB_IB_CFG_HRTBT));
}
-static ssize_t store_hrtbt_enb(struct qib_pportdata *ppd, const char *buf,
- size_t count)
+static ssize_t hrtbt_enable_store(struct ib_device *ibdev, u32 port_num,
+ struct ib_port_attribute *attr,
+ const char *buf, size_t count)
{
- struct qib_devdata *dd = ppd->dd;
+ struct qib_devdata *dd = dd_from_ibdev(ibdev);
+ struct qib_pportdata *ppd = &dd->pport[port_num - 1];
int ret;
u16 val;
@@ -70,11 +83,14 @@ static ssize_t store_hrtbt_enb(struct qib_pportdata *ppd, const char *buf,
ret = dd->f_set_ib_cfg(ppd, QIB_IB_CFG_HRTBT, val);
return ret < 0 ? ret : count;
}
+static IB_PORT_ATTR_RW(hrtbt_enable);
-static ssize_t store_loopback(struct qib_pportdata *ppd, const char *buf,
+static ssize_t loopback_store(struct ib_device *ibdev, u32 port_num,
+ struct ib_port_attribute *attr, const char *buf,
size_t count)
{
- struct qib_devdata *dd = ppd->dd;
+ struct qib_devdata *dd = dd_from_ibdev(ibdev);
+ struct qib_pportdata *ppd = &dd->pport[port_num - 1];
int ret = count, r;
r = dd->f_set_ib_loopback(ppd, buf);
@@ -83,11 +99,14 @@ static ssize_t store_loopback(struct qib_pportdata *ppd, const char *buf,
return ret;
}
+static IB_PORT_ATTR_WO(loopback);
-static ssize_t store_led_override(struct qib_pportdata *ppd, const char *buf,
- size_t count)
+static ssize_t led_override_store(struct ib_device *ibdev, u32 port_num,
+ struct ib_port_attribute *attr,
+ const char *buf, size_t count)
{
- struct qib_devdata *dd = ppd->dd;
+ struct qib_devdata *dd = dd_from_ibdev(ibdev);
+ struct qib_pportdata *ppd = &dd->pport[port_num - 1];
int ret;
u16 val;
@@ -100,14 +119,20 @@ static ssize_t store_led_override(struct qib_pportdata *ppd, const char *buf,
qib_set_led_override(ppd, val);
return count;
}
+static IB_PORT_ATTR_WO(led_override);
-static ssize_t show_status(struct qib_pportdata *ppd, char *buf)
+static ssize_t status_show(struct ib_device *ibdev, u32 port_num,
+ struct ib_port_attribute *attr, char *buf)
{
+ struct qib_devdata *dd = dd_from_ibdev(ibdev);
+ struct qib_pportdata *ppd = &dd->pport[port_num - 1];
+
if (!ppd->statusp)
return -EINVAL;
return sysfs_emit(buf, "0x%llx\n", (unsigned long long)*(ppd->statusp));
}
+static IB_PORT_ATTR_RO(status);
/*
* For userland compatibility, these offsets must remain fixed.
@@ -127,8 +152,11 @@ static const char * const qib_status_str[] = {
NULL,
};
-static ssize_t show_status_str(struct qib_pportdata *ppd, char *buf)
+static ssize_t status_str_show(struct ib_device *ibdev, u32 port_num,
+ struct ib_port_attribute *attr, char *buf)
{
+ struct qib_devdata *dd = dd_from_ibdev(ibdev);
+ struct qib_pportdata *ppd = &dd->pport[port_num - 1];
int i, any;
u64 s;
ssize_t ret;
@@ -160,38 +188,22 @@ static ssize_t show_status_str(struct qib_pportdata *ppd, char *buf)
bail:
return ret;
}
+static IB_PORT_ATTR_RO(status_str);
/* end of per-port functions */
-/*
- * Start of per-port file structures and support code
- * Because we are fitting into other infrastructure, we have to supply the
- * full set of kobject/sysfs_ops structures and routines.
- */
-#define QIB_PORT_ATTR(name, mode, show, store) \
- static struct qib_port_attr qib_port_attr_##name = \
- __ATTR(name, mode, show, store)
-
-struct qib_port_attr {
- struct attribute attr;
- ssize_t (*show)(struct qib_pportdata *, char *);
- ssize_t (*store)(struct qib_pportdata *, const char *, size_t);
+static struct attribute *port_linkcontrol_attributes[] = {
+ &ib_port_attr_loopback.attr,
+ &ib_port_attr_led_override.attr,
+ &ib_port_attr_hrtbt_enable.attr,
+ &ib_port_attr_status.attr,
+ &ib_port_attr_status_str.attr,
+ NULL
};
-QIB_PORT_ATTR(loopback, S_IWUSR, NULL, store_loopback);
-QIB_PORT_ATTR(led_override, S_IWUSR, NULL, store_led_override);
-QIB_PORT_ATTR(hrtbt_enable, S_IWUSR | S_IRUGO, show_hrtbt_enb,
- store_hrtbt_enb);
-QIB_PORT_ATTR(status, S_IRUGO, show_status, NULL);
-QIB_PORT_ATTR(status_str, S_IRUGO, show_status_str, NULL);
-
-static struct attribute *port_default_attributes[] = {
- &qib_port_attr_loopback.attr,
- &qib_port_attr_led_override.attr,
- &qib_port_attr_hrtbt_enable.attr,
- &qib_port_attr_status.attr,
- &qib_port_attr_status_str.attr,
- NULL
+static const struct attribute_group port_linkcontrol_group = {
+ .name = "linkcontrol",
+ .attrs = port_linkcontrol_attributes,
};
/*
@@ -201,13 +213,12 @@ static struct attribute *port_default_attributes[] = {
/*
* Congestion control table size followed by table entries
*/
-static ssize_t read_cc_table_bin(struct file *filp, struct kobject *kobj,
- struct bin_attribute *bin_attr,
- char *buf, loff_t pos, size_t count)
+static ssize_t cc_table_bin_read(struct file *filp, struct kobject *kobj,
+ struct bin_attribute *bin_attr, char *buf,
+ loff_t pos, size_t count)
{
+ struct qib_pportdata *ppd = qib_get_pportdata_kobj(kobj);
int ret;
- struct qib_pportdata *ppd =
- container_of(kobj, struct qib_pportdata, pport_cc_kobj);
if (!qib_cc_table_size || !ppd->ccti_entries_shadow)
return -EINVAL;
@@ -230,34 +241,19 @@ static ssize_t read_cc_table_bin(struct file *filp, struct kobject *kobj,
return count;
}
-
-static void qib_port_release(struct kobject *kobj)
-{
- /* nothing to do since memory is freed by qib_free_devdata() */
-}
-
-static struct kobj_type qib_port_cc_ktype = {
- .release = qib_port_release,
-};
-
-static const struct bin_attribute cc_table_bin_attr = {
- .attr = {.name = "cc_table_bin", .mode = 0444},
- .read = read_cc_table_bin,
- .size = PAGE_SIZE,
-};
+static BIN_ATTR_RO(cc_table_bin, PAGE_SIZE);
/*
* Congestion settings: port control, control map and an array of 16
* entries for the congestion entries - increase, timer, event log
* trigger threshold and the minimum injection rate delay.
*/
-static ssize_t read_cc_setting_bin(struct file *filp, struct kobject *kobj,
- struct bin_attribute *bin_attr,
- char *buf, loff_t pos, size_t count)
+static ssize_t cc_setting_bin_read(struct file *filp, struct kobject *kobj,
+ struct bin_attribute *bin_attr, char *buf,
+ loff_t pos, size_t count)
{
+ struct qib_pportdata *ppd = qib_get_pportdata_kobj(kobj);
int ret;
- struct qib_pportdata *ppd =
- container_of(kobj, struct qib_pportdata, pport_cc_kobj);
if (!qib_cc_table_size || !ppd->congestion_entries_shadow)
return -EINVAL;
@@ -278,67 +274,43 @@ static ssize_t read_cc_setting_bin(struct file *filp, struct kobject *kobj,
return count;
}
+static BIN_ATTR_RO(cc_setting_bin, PAGE_SIZE);
-static const struct bin_attribute cc_setting_bin_attr = {
- .attr = {.name = "cc_settings_bin", .mode = 0444},
- .read = read_cc_setting_bin,
- .size = PAGE_SIZE,
+static struct bin_attribute *port_ccmgta_attributes[] = {
+ &bin_attr_cc_setting_bin,
+ &bin_attr_cc_table_bin,
+ NULL,
};
+static const struct attribute_group port_ccmgta_attribute_group = {
+ .name = "CCMgtA",
+ .bin_attrs = port_ccmgta_attributes,
+};
-static ssize_t qib_portattr_show(struct kobject *kobj,
- struct attribute *attr, char *buf)
-{
- struct qib_port_attr *pattr =
- container_of(attr, struct qib_port_attr, attr);
- struct qib_pportdata *ppd =
- container_of(kobj, struct qib_pportdata, pport_kobj);
-
- if (!pattr->show)
- return -EIO;
+/* Start sl2vl */
- return pattr->show(ppd, buf);
-}
+struct qib_sl2vl_attr {
+ struct ib_port_attribute attr;
+ int sl;
+};
-static ssize_t qib_portattr_store(struct kobject *kobj,
- struct attribute *attr, const char *buf, size_t len)
+static ssize_t sl2vl_attr_show(struct ib_device *ibdev, u32 port_num,
+ struct ib_port_attribute *attr, char *buf)
{
- struct qib_port_attr *pattr =
- container_of(attr, struct qib_port_attr, attr);
- struct qib_pportdata *ppd =
- container_of(kobj, struct qib_pportdata, pport_kobj);
-
- if (!pattr->store)
- return -EIO;
+ struct qib_sl2vl_attr *sattr =
+ container_of(attr, struct qib_sl2vl_attr, attr);
+ struct qib_devdata *dd = dd_from_ibdev(ibdev);
+ struct qib_ibport *qibp = &dd->pport[port_num - 1].ibport_data;
- return pattr->store(ppd, buf, len);
+ return sysfs_emit(buf, "%u\n", qibp->sl_to_vl[sattr->sl]);
}
-
-static const struct sysfs_ops qib_port_ops = {
- .show = qib_portattr_show,
- .store = qib_portattr_store,
-};
-
-static struct kobj_type qib_port_ktype = {
- .release = qib_port_release,
- .sysfs_ops = &qib_port_ops,
- .default_attrs = port_default_attributes
-};
-
-/* Start sl2vl */
-
-#define QIB_SL2VL_ATTR(N) \
- static struct qib_sl2vl_attr qib_sl2vl_attr_##N = { \
- .attr = { .name = __stringify(N), .mode = 0444 }, \
- .sl = N \
+#define QIB_SL2VL_ATTR(N) \
+ static struct qib_sl2vl_attr qib_sl2vl_attr_##N = { \
+ .attr = __ATTR(N, 0444, sl2vl_attr_show, NULL), \
+ .sl = N, \
}
-struct qib_sl2vl_attr {
- struct attribute attr;
- int sl;
-};
-
QIB_SL2VL_ATTR(0);
QIB_SL2VL_ATTR(1);
QIB_SL2VL_ATTR(2);
@@ -356,72 +328,74 @@ QIB_SL2VL_ATTR(13);
QIB_SL2VL_ATTR(14);
QIB_SL2VL_ATTR(15);
-static struct attribute *sl2vl_default_attributes[] = {
- &qib_sl2vl_attr_0.attr,
- &qib_sl2vl_attr_1.attr,
- &qib_sl2vl_attr_2.attr,
- &qib_sl2vl_attr_3.attr,
- &qib_sl2vl_attr_4.attr,
- &qib_sl2vl_attr_5.attr,
- &qib_sl2vl_attr_6.attr,
- &qib_sl2vl_attr_7.attr,
- &qib_sl2vl_attr_8.attr,
- &qib_sl2vl_attr_9.attr,
- &qib_sl2vl_attr_10.attr,
- &qib_sl2vl_attr_11.attr,
- &qib_sl2vl_attr_12.attr,
- &qib_sl2vl_attr_13.attr,
- &qib_sl2vl_attr_14.attr,
- &qib_sl2vl_attr_15.attr,
+static struct attribute *port_sl2vl_attributes[] = {
+ &qib_sl2vl_attr_0.attr.attr,
+ &qib_sl2vl_attr_1.attr.attr,
+ &qib_sl2vl_attr_2.attr.attr,
+ &qib_sl2vl_attr_3.attr.attr,
+ &qib_sl2vl_attr_4.attr.attr,
+ &qib_sl2vl_attr_5.attr.attr,
+ &qib_sl2vl_attr_6.attr.attr,
+ &qib_sl2vl_attr_7.attr.attr,
+ &qib_sl2vl_attr_8.attr.attr,
+ &qib_sl2vl_attr_9.attr.attr,
+ &qib_sl2vl_attr_10.attr.attr,
+ &qib_sl2vl_attr_11.attr.attr,
+ &qib_sl2vl_attr_12.attr.attr,
+ &qib_sl2vl_attr_13.attr.attr,
+ &qib_sl2vl_attr_14.attr.attr,
+ &qib_sl2vl_attr_15.attr.attr,
NULL
};
-static ssize_t sl2vl_attr_show(struct kobject *kobj, struct attribute *attr,
- char *buf)
-{
- struct qib_sl2vl_attr *sattr =
- container_of(attr, struct qib_sl2vl_attr, attr);
- struct qib_pportdata *ppd =
- container_of(kobj, struct qib_pportdata, sl2vl_kobj);
- struct qib_ibport *qibp = &ppd->ibport_data;
-
- return sysfs_emit(buf, "%u\n", qibp->sl_to_vl[sattr->sl]);
-}
-
-static const struct sysfs_ops qib_sl2vl_ops = {
- .show = sl2vl_attr_show,
-};
-
-static struct kobj_type qib_sl2vl_ktype = {
- .release = qib_port_release,
- .sysfs_ops = &qib_sl2vl_ops,
- .default_attrs = sl2vl_default_attributes
+static const struct attribute_group port_sl2vl_group = {
+ .name = "sl2vl",
+ .attrs = port_sl2vl_attributes,
};
/* End sl2vl */
/* Start diag_counters */
-#define QIB_DIAGC_ATTR(N) \
- static struct qib_diagc_attr qib_diagc_attr_##N = { \
- .attr = { .name = __stringify(N), .mode = 0664 }, \
- .counter = offsetof(struct qib_ibport, rvp.n_##N) \
- }
-
-#define QIB_DIAGC_ATTR_PER_CPU(N) \
- static struct qib_diagc_attr qib_diagc_attr_##N = { \
- .attr = { .name = __stringify(N), .mode = 0664 }, \
- .counter = offsetof(struct qib_ibport, rvp.z_##N) \
- }
-
struct qib_diagc_attr {
- struct attribute attr;
+ struct ib_port_attribute attr;
size_t counter;
};
-QIB_DIAGC_ATTR_PER_CPU(rc_acks);
-QIB_DIAGC_ATTR_PER_CPU(rc_qacks);
-QIB_DIAGC_ATTR_PER_CPU(rc_delayed_comp);
+static ssize_t diagc_attr_show(struct ib_device *ibdev, u32 port_num,
+ struct ib_port_attribute *attr, char *buf)
+{
+ struct qib_diagc_attr *dattr =
+ container_of(attr, struct qib_diagc_attr, attr);
+ struct qib_devdata *dd = dd_from_ibdev(ibdev);
+ struct qib_ibport *qibp = &dd->pport[port_num - 1].ibport_data;
+
+ return sysfs_emit(buf, "%llu\n", *((u64 *)qibp + dattr->counter));
+}
+
+static ssize_t diagc_attr_store(struct ib_device *ibdev, u32 port_num,
+ struct ib_port_attribute *attr, const char *buf,
+ size_t count)
+{
+ struct qib_diagc_attr *dattr =
+ container_of(attr, struct qib_diagc_attr, attr);
+ struct qib_devdata *dd = dd_from_ibdev(ibdev);
+ struct qib_ibport *qibp = &dd->pport[port_num - 1].ibport_data;
+ u64 val;
+ int ret;
+
+ ret = kstrtou64(buf, 0, &val);
+ if (ret)
+ return ret;
+ *((u64 *)qibp + dattr->counter) = val;
+ return count;
+}
+
+#define QIB_DIAGC_ATTR(N) \
+ static struct qib_diagc_attr qib_diagc_attr_##N = { \
+ .attr = __ATTR(N, 0664, diagc_attr_show, diagc_attr_store), \
+ .counter = &((struct qib_ibport *)0)->rvp.n_##N - (u64 *)0, \
+ }
QIB_DIAGC_ATTR(rc_resends);
QIB_DIAGC_ATTR(seq_naks);
@@ -437,26 +411,6 @@ QIB_DIAGC_ATTR(rc_dupreq);
QIB_DIAGC_ATTR(rc_seqnak);
QIB_DIAGC_ATTR(rc_crwaits);
-static struct attribute *diagc_default_attributes[] = {
- &qib_diagc_attr_rc_resends.attr,
- &qib_diagc_attr_rc_acks.attr,
- &qib_diagc_attr_rc_qacks.attr,
- &qib_diagc_attr_rc_delayed_comp.attr,
- &qib_diagc_attr_seq_naks.attr,
- &qib_diagc_attr_rdma_seq.attr,
- &qib_diagc_attr_rnr_naks.attr,
- &qib_diagc_attr_other_naks.attr,
- &qib_diagc_attr_rc_timeouts.attr,
- &qib_diagc_attr_loop_pkts.attr,
- &qib_diagc_attr_pkt_drops.attr,
- &qib_diagc_attr_dmawait.attr,
- &qib_diagc_attr_unaligned.attr,
- &qib_diagc_attr_rc_dupreq.attr,
- &qib_diagc_attr_rc_seqnak.attr,
- &qib_diagc_attr_rc_crwaits.attr,
- NULL
-};
-
static u64 get_all_cpu_total(u64 __percpu *cntr)
{
int cpu;
@@ -467,82 +421,115 @@ static u64 get_all_cpu_total(u64 __percpu *cntr)
return counter;
}
-#define def_write_per_cpu(cntr) \
-static void write_per_cpu_##cntr(struct qib_pportdata *ppd, u32 data) \
-{ \
- struct qib_devdata *dd = ppd->dd; \
- struct qib_ibport *qibp = &ppd->ibport_data; \
- /* A write can only zero the counter */ \
- if (data == 0) \
- qibp->rvp.z_##cntr = get_all_cpu_total(qibp->rvp.cntr); \
- else \
- qib_dev_err(dd, "Per CPU cntrs can only be zeroed"); \
+static ssize_t qib_store_per_cpu(struct qib_devdata *dd, const char *buf,
+ size_t count, u64 *zero, u64 cur)
+{
+ u32 val;
+ int ret;
+
+ ret = kstrtou32(buf, 0, &val);
+ if (ret)
+ return ret;
+ if (val != 0) {
+ qib_dev_err(dd, "Per CPU cntrs can only be zeroed");
+ return count;
+ }
+ *zero = cur;
+ return count;
}
-def_write_per_cpu(rc_acks)
-def_write_per_cpu(rc_qacks)
-def_write_per_cpu(rc_delayed_comp)
+static ssize_t rc_acks_show(struct ib_device *ibdev, u32 port_num,
+ struct ib_port_attribute *attr, char *buf)
+{
+ struct qib_devdata *dd = dd_from_ibdev(ibdev);
+ struct qib_ibport *qibp = &dd->pport[port_num - 1].ibport_data;
-#define READ_PER_CPU_CNTR(cntr) (get_all_cpu_total(qibp->rvp.cntr) - \
- qibp->rvp.z_##cntr)
+ return sysfs_emit(buf, "%llu\n",
+ get_all_cpu_total(qibp->rvp.rc_acks) -
+ qibp->rvp.z_rc_acks);
+}
-static ssize_t diagc_attr_show(struct kobject *kobj, struct attribute *attr,
- char *buf)
+static ssize_t rc_acks_store(struct ib_device *ibdev, u32 port_num,
+ struct ib_port_attribute *attr, const char *buf,
+ size_t count)
{
- struct qib_diagc_attr *dattr =
- container_of(attr, struct qib_diagc_attr, attr);
- struct qib_pportdata *ppd =
- container_of(kobj, struct qib_pportdata, diagc_kobj);
- struct qib_ibport *qibp = &ppd->ibport_data;
- u64 val;
+ struct qib_devdata *dd = dd_from_ibdev(ibdev);
+ struct qib_ibport *qibp = &dd->pport[port_num - 1].ibport_data;
- if (!strncmp(dattr->attr.name, "rc_acks", 7))
- val = READ_PER_CPU_CNTR(rc_acks);
- else if (!strncmp(dattr->attr.name, "rc_qacks", 8))
- val = READ_PER_CPU_CNTR(rc_qacks);
- else if (!strncmp(dattr->attr.name, "rc_delayed_comp", 15))
- val = READ_PER_CPU_CNTR(rc_delayed_comp);
- else
- val = *(u32 *)((char *)qibp + dattr->counter);
+ return qib_store_per_cpu(dd, buf, count, &qibp->rvp.z_rc_acks,
+ get_all_cpu_total(qibp->rvp.rc_acks));
+}
+static IB_PORT_ATTR_RW(rc_acks);
- return sysfs_emit(buf, "%llu\n", val);
+static ssize_t rc_qacks_show(struct ib_device *ibdev, u32 port_num,
+ struct ib_port_attribute *attr, char *buf)
+{
+ struct qib_devdata *dd = dd_from_ibdev(ibdev);
+ struct qib_ibport *qibp = &dd->pport[port_num - 1].ibport_data;
+
+ return sysfs_emit(buf, "%llu\n",
+ get_all_cpu_total(qibp->rvp.rc_qacks) -
+ qibp->rvp.z_rc_qacks);
}
-static ssize_t diagc_attr_store(struct kobject *kobj, struct attribute *attr,
- const char *buf, size_t size)
+static ssize_t rc_qacks_store(struct ib_device *ibdev, u32 port_num,
+ struct ib_port_attribute *attr, const char *buf,
+ size_t count)
{
- struct qib_diagc_attr *dattr =
- container_of(attr, struct qib_diagc_attr, attr);
- struct qib_pportdata *ppd =
- container_of(kobj, struct qib_pportdata, diagc_kobj);
- struct qib_ibport *qibp = &ppd->ibport_data;
- u32 val;
- int ret;
+ struct qib_devdata *dd = dd_from_ibdev(ibdev);
+ struct qib_ibport *qibp = &dd->pport[port_num - 1].ibport_data;
- ret = kstrtou32(buf, 0, &val);
- if (ret)
- return ret;
+ return qib_store_per_cpu(dd, buf, count, &qibp->rvp.z_rc_qacks,
+ get_all_cpu_total(qibp->rvp.rc_qacks));
+}
+static IB_PORT_ATTR_RW(rc_qacks);
+
+static ssize_t rc_delayed_comp_show(struct ib_device *ibdev, u32 port_num,
+ struct ib_port_attribute *attr, char *buf)
+{
+ struct qib_devdata *dd = dd_from_ibdev(ibdev);
+ struct qib_ibport *qibp = &dd->pport[port_num - 1].ibport_data;
- if (!strncmp(dattr->attr.name, "rc_acks", 7))
- write_per_cpu_rc_acks(ppd, val);
- else if (!strncmp(dattr->attr.name, "rc_qacks", 8))
- write_per_cpu_rc_qacks(ppd, val);
- else if (!strncmp(dattr->attr.name, "rc_delayed_comp", 15))
- write_per_cpu_rc_delayed_comp(ppd, val);
- else
- *(u32 *)((char *)qibp + dattr->counter) = val;
- return size;
+ return sysfs_emit(buf, "%llu\n",
+ get_all_cpu_total(qibp->rvp.rc_delayed_comp) -
+ qibp->rvp.z_rc_delayed_comp);
}
-static const struct sysfs_ops qib_diagc_ops = {
- .show = diagc_attr_show,
- .store = diagc_attr_store,
+static ssize_t rc_delayed_comp_store(struct ib_device *ibdev, u32 port_num,
+ struct ib_port_attribute *attr,
+ const char *buf, size_t count)
+{
+ struct qib_devdata *dd = dd_from_ibdev(ibdev);
+ struct qib_ibport *qibp = &dd->pport[port_num - 1].ibport_data;
+
+ return qib_store_per_cpu(dd, buf, count, &qibp->rvp.z_rc_delayed_comp,
+ get_all_cpu_total(qibp->rvp.rc_delayed_comp));
+}
+static IB_PORT_ATTR_RW(rc_delayed_comp);
+
+static struct attribute *port_diagc_attributes[] = {
+ &qib_diagc_attr_rc_resends.attr.attr,
+ &qib_diagc_attr_seq_naks.attr.attr,
+ &qib_diagc_attr_rdma_seq.attr.attr,
+ &qib_diagc_attr_rnr_naks.attr.attr,
+ &qib_diagc_attr_other_naks.attr.attr,
+ &qib_diagc_attr_rc_timeouts.attr.attr,
+ &qib_diagc_attr_loop_pkts.attr.attr,
+ &qib_diagc_attr_pkt_drops.attr.attr,
+ &qib_diagc_attr_dmawait.attr.attr,
+ &qib_diagc_attr_unaligned.attr.attr,
+ &qib_diagc_attr_rc_dupreq.attr.attr,
+ &qib_diagc_attr_rc_seqnak.attr.attr,
+ &qib_diagc_attr_rc_crwaits.attr.attr,
+ &ib_port_attr_rc_acks.attr,
+ &ib_port_attr_rc_qacks.attr,
+ &ib_port_attr_rc_delayed_comp.attr,
+ NULL
};
-static struct kobj_type qib_diagc_ktype = {
- .release = qib_port_release,
- .sysfs_ops = &qib_diagc_ops,
- .default_attrs = diagc_default_attributes
+static const struct attribute_group port_diagc_group = {
+ .name = "linkcontrol",
+ .attrs = port_diagc_attributes,
};
/* End diag_counters */
@@ -731,99 +718,19 @@ const struct attribute_group qib_attr_group = {
int qib_create_port_files(struct ib_device *ibdev, u32 port_num,
struct kobject *kobj)
{
- struct qib_pportdata *ppd;
struct qib_devdata *dd = dd_from_ibdev(ibdev);
- int ret;
+ struct qib_pportdata *ppd = &dd->pport[port_num - 1];
+ const struct attribute_group **cur_group;
- if (!port_num || port_num > dd->num_pports) {
- qib_dev_err(dd,
- "Skipping infiniband class with invalid port %u\n",
- port_num);
- ret = -ENODEV;
- goto bail;
- }
- ppd = &dd->pport[port_num - 1];
+ cur_group = &ppd->groups[0];
+ *cur_group++ = &port_linkcontrol_group;
+ *cur_group++ = &port_sl2vl_group;
+ *cur_group++ = &port_diagc_group;
- ret = kobject_init_and_add(&ppd->pport_kobj, &qib_port_ktype, kobj,
- "linkcontrol");
- if (ret) {
- qib_dev_err(dd,
- "Skipping linkcontrol sysfs info, (err %d) port %u\n",
- ret, port_num);
- goto bail_link;
- }
- kobject_uevent(&ppd->pport_kobj, KOBJ_ADD);
+ if (qib_cc_table_size && ppd->congestion_entries_shadow)
+ *cur_group++ = &port_ccmgta_attribute_group;
- ret = kobject_init_and_add(&ppd->sl2vl_kobj, &qib_sl2vl_ktype, kobj,
- "sl2vl");
- if (ret) {
- qib_dev_err(dd,
- "Skipping sl2vl sysfs info, (err %d) port %u\n",
- ret, port_num);
- goto bail_sl;
- }
- kobject_uevent(&ppd->sl2vl_kobj, KOBJ_ADD);
-
- ret = kobject_init_and_add(&ppd->diagc_kobj, &qib_diagc_ktype, kobj,
- "diag_counters");
- if (ret) {
- qib_dev_err(dd,
- "Skipping diag_counters sysfs info, (err %d) port %u\n",
- ret, port_num);
- goto bail_diagc;
- }
- kobject_uevent(&ppd->diagc_kobj, KOBJ_ADD);
-
- if (!qib_cc_table_size || !ppd->congestion_entries_shadow)
- return 0;
-
- ret = kobject_init_and_add(&ppd->pport_cc_kobj, &qib_port_cc_ktype,
- kobj, "CCMgtA");
- if (ret) {
- qib_dev_err(dd,
- "Skipping Congestion Control sysfs info, (err %d) port %u\n",
- ret, port_num);
- goto bail_cc;
- }
-
- kobject_uevent(&ppd->pport_cc_kobj, KOBJ_ADD);
-
- ret = sysfs_create_bin_file(&ppd->pport_cc_kobj,
- &cc_setting_bin_attr);
- if (ret) {
- qib_dev_err(dd,
- "Skipping Congestion Control setting sysfs info, (err %d) port %u\n",
- ret, port_num);
- goto bail_cc;
- }
-
- ret = sysfs_create_bin_file(&ppd->pport_cc_kobj,
- &cc_table_bin_attr);
- if (ret) {
- qib_dev_err(dd,
- "Skipping Congestion Control table sysfs info, (err %d) port %u\n",
- ret, port_num);
- goto bail_cc_entry_bin;
- }
-
- qib_devinfo(dd->pcidev,
- "IB%u: Congestion Control Agent enabled for port %d\n",
- dd->unit, port_num);
-
- return 0;
-
-bail_cc_entry_bin:
- sysfs_remove_bin_file(&ppd->pport_cc_kobj, &cc_setting_bin_attr);
-bail_cc:
- kobject_put(&ppd->pport_cc_kobj);
-bail_diagc:
- kobject_put(&ppd->diagc_kobj);
-bail_sl:
- kobject_put(&ppd->sl2vl_kobj);
-bail_link:
- kobject_put(&ppd->pport_kobj);
-bail:
- return ret;
+ return ib_port_sysfs_create_groups(ibdev, port_num, ppd->groups);
}
/*
@@ -831,21 +738,12 @@ int qib_create_port_files(struct ib_device *ibdev, u32 port_num,
*/
void qib_verbs_unregister_sysfs(struct qib_devdata *dd)
{
- struct qib_pportdata *ppd;
int i;
for (i = 0; i < dd->num_pports; i++) {
- ppd = &dd->pport[i];
- if (qib_cc_table_size &&
- ppd->congestion_entries_shadow) {
- sysfs_remove_bin_file(&ppd->pport_cc_kobj,
- &cc_setting_bin_attr);
- sysfs_remove_bin_file(&ppd->pport_cc_kobj,
- &cc_table_bin_attr);
- kobject_put(&ppd->pport_cc_kobj);
- }
- kobject_put(&ppd->diagc_kobj);
- kobject_put(&ppd->sl2vl_kobj);
- kobject_put(&ppd->pport_kobj);
+ struct qib_pportdata *ppd = &dd->pport[i];
+
+ ib_port_sysfs_remove_groups(&dd->verbs_dev.rdi.ibdev, i,
+ ppd->groups);
}
}
--
2.31.1
^ permalink raw reply related [flat|nested] 35+ messages in thread* Re: [PATCH 11/13] RDMA/qib: Use attributes for the port sysfs
2021-05-17 16:47 ` [PATCH 11/13] RDMA/qib: Use attributes for the port sysfs Jason Gunthorpe
@ 2021-05-17 17:11 ` Greg KH
2021-05-17 17:13 ` Jason Gunthorpe
0 siblings, 1 reply; 35+ messages in thread
From: Greg KH @ 2021-05-17 17:11 UTC (permalink / raw)
To: Jason Gunthorpe
Cc: Dennis Dalessandro, Doug Ledford, linux-rdma, Mike Marciniszyn,
Kees Cook, Nathan Chancellor
On Mon, May 17, 2021 at 01:47:39PM -0300, Jason Gunthorpe wrote:
> qib should not be creating a mess of kobjects to attach to the port
> kobject - this is all attributes. The proper API is to create an
> attribute_group list and create it against the port's kobject.
>
> Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>
> ---
> drivers/infiniband/hw/qib/qib.h | 5 +-
> drivers/infiniband/hw/qib/qib_sysfs.c | 596 +++++++++++---------------
> 2 files changed, 248 insertions(+), 353 deletions(-)
>
> diff --git a/drivers/infiniband/hw/qib/qib.h b/drivers/infiniband/hw/qib/qib.h
> index 88497739029e02..3decd6d0843172 100644
> --- a/drivers/infiniband/hw/qib/qib.h
> +++ b/drivers/infiniband/hw/qib/qib.h
> @@ -521,10 +521,7 @@ struct qib_pportdata {
>
> struct qib_devdata *dd;
> struct qib_chippport_specific *cpspec; /* chip-specific per-port */
> - struct kobject pport_kobj;
> - struct kobject pport_cc_kobj;
> - struct kobject sl2vl_kobj;
> - struct kobject diagc_kobj;
> + const struct attribute_group *groups[5];
As you initialize these all at once, why not just make this:
struct attribute_group **groups;
and then set the groups up at build time instead of runtime as part of a
larger structure like the ATTRIBUTE_GROUPS() macro does for "simple"
drivers? That way you aren't fixed at the array size here and someone
has to go and check to verify you really have properly 0 terminated the
list and set up the pointers properly.
thanks,
greg k-h
^ permalink raw reply [flat|nested] 35+ messages in thread* Re: [PATCH 11/13] RDMA/qib: Use attributes for the port sysfs
2021-05-17 17:11 ` Greg KH
@ 2021-05-17 17:13 ` Jason Gunthorpe
2021-05-17 17:31 ` Greg KH
0 siblings, 1 reply; 35+ messages in thread
From: Jason Gunthorpe @ 2021-05-17 17:13 UTC (permalink / raw)
To: Greg KH
Cc: Dennis Dalessandro, Doug Ledford, linux-rdma, Mike Marciniszyn,
Kees Cook, Nathan Chancellor
On Mon, May 17, 2021 at 07:11:52PM +0200, Greg KH wrote:
> On Mon, May 17, 2021 at 01:47:39PM -0300, Jason Gunthorpe wrote:
> > qib should not be creating a mess of kobjects to attach to the port
> > kobject - this is all attributes. The proper API is to create an
> > attribute_group list and create it against the port's kobject.
> >
> > Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>
> > drivers/infiniband/hw/qib/qib.h | 5 +-
> > drivers/infiniband/hw/qib/qib_sysfs.c | 596 +++++++++++---------------
> > 2 files changed, 248 insertions(+), 353 deletions(-)
> >
> > diff --git a/drivers/infiniband/hw/qib/qib.h b/drivers/infiniband/hw/qib/qib.h
> > index 88497739029e02..3decd6d0843172 100644
> > +++ b/drivers/infiniband/hw/qib/qib.h
> > @@ -521,10 +521,7 @@ struct qib_pportdata {
> >
> > struct qib_devdata *dd;
> > struct qib_chippport_specific *cpspec; /* chip-specific per-port */
> > - struct kobject pport_kobj;
> > - struct kobject pport_cc_kobj;
> > - struct kobject sl2vl_kobj;
> > - struct kobject diagc_kobj;
> > + const struct attribute_group *groups[5];
>
> As you initialize these all at once, why not just make this:
> struct attribute_group **groups;
>
> and then set the groups up at build time instead of runtime as part of a
> larger structure like the ATTRIBUTE_GROUPS() macro does for "simple"
> drivers? That way you aren't fixed at the array size here and someone
> has to go and check to verify you really have properly 0 terminated the
> list and set up the pointers properly.
qib has a variable list of group memberships that can only be
determined at runtime:
if (qib_cc_table_size && ppd->congestion_entries_shadow)
*cur_group++ = &port_ccmgta_attribute_group;
So it can't be setup statically at compile time.
hfi1 was transformed as you describe.
Thanks,
Jason
^ permalink raw reply [flat|nested] 35+ messages in thread* Re: [PATCH 11/13] RDMA/qib: Use attributes for the port sysfs
2021-05-17 17:13 ` Jason Gunthorpe
@ 2021-05-17 17:31 ` Greg KH
2021-05-17 18:44 ` Jason Gunthorpe
0 siblings, 1 reply; 35+ messages in thread
From: Greg KH @ 2021-05-17 17:31 UTC (permalink / raw)
To: Jason Gunthorpe
Cc: Dennis Dalessandro, Doug Ledford, linux-rdma, Mike Marciniszyn,
Kees Cook, Nathan Chancellor
On Mon, May 17, 2021 at 02:13:47PM -0300, Jason Gunthorpe wrote:
> On Mon, May 17, 2021 at 07:11:52PM +0200, Greg KH wrote:
> > On Mon, May 17, 2021 at 01:47:39PM -0300, Jason Gunthorpe wrote:
> > > qib should not be creating a mess of kobjects to attach to the port
> > > kobject - this is all attributes. The proper API is to create an
> > > attribute_group list and create it against the port's kobject.
> > >
> > > Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>
> > > drivers/infiniband/hw/qib/qib.h | 5 +-
> > > drivers/infiniband/hw/qib/qib_sysfs.c | 596 +++++++++++---------------
> > > 2 files changed, 248 insertions(+), 353 deletions(-)
> > >
> > > diff --git a/drivers/infiniband/hw/qib/qib.h b/drivers/infiniband/hw/qib/qib.h
> > > index 88497739029e02..3decd6d0843172 100644
> > > +++ b/drivers/infiniband/hw/qib/qib.h
> > > @@ -521,10 +521,7 @@ struct qib_pportdata {
> > >
> > > struct qib_devdata *dd;
> > > struct qib_chippport_specific *cpspec; /* chip-specific per-port */
> > > - struct kobject pport_kobj;
> > > - struct kobject pport_cc_kobj;
> > > - struct kobject sl2vl_kobj;
> > > - struct kobject diagc_kobj;
> > > + const struct attribute_group *groups[5];
> >
> > As you initialize these all at once, why not just make this:
> > struct attribute_group **groups;
> >
> > and then set the groups up at build time instead of runtime as part of a
> > larger structure like the ATTRIBUTE_GROUPS() macro does for "simple"
> > drivers? That way you aren't fixed at the array size here and someone
> > has to go and check to verify you really have properly 0 terminated the
> > list and set up the pointers properly.
>
> qib has a variable list of group memberships that can only be
> determined at runtime:
>
> if (qib_cc_table_size && ppd->congestion_entries_shadow)
> *cur_group++ = &port_ccmgta_attribute_group;
>
> So it can't be setup statically at compile time.
That attribute group can have a is_visable() callback to allow those
files to show up or not, instead of having to determine this when you
are setting up the list of groups.
But that's your call, not a big deal, overall this series looks like a
lot of good cleanups to me, thanks for doing it.
greg k-h
^ permalink raw reply [flat|nested] 35+ messages in thread* Re: [PATCH 11/13] RDMA/qib: Use attributes for the port sysfs
2021-05-17 17:31 ` Greg KH
@ 2021-05-17 18:44 ` Jason Gunthorpe
0 siblings, 0 replies; 35+ messages in thread
From: Jason Gunthorpe @ 2021-05-17 18:44 UTC (permalink / raw)
To: Greg KH
Cc: Dennis Dalessandro, Doug Ledford, linux-rdma, Mike Marciniszyn,
Kees Cook, Nathan Chancellor
On Mon, May 17, 2021 at 07:31:30PM +0200, Greg KH wrote:
> On Mon, May 17, 2021 at 02:13:47PM -0300, Jason Gunthorpe wrote:
> > On Mon, May 17, 2021 at 07:11:52PM +0200, Greg KH wrote:
> > > On Mon, May 17, 2021 at 01:47:39PM -0300, Jason Gunthorpe wrote:
> > > > qib should not be creating a mess of kobjects to attach to the port
> > > > kobject - this is all attributes. The proper API is to create an
> > > > attribute_group list and create it against the port's kobject.
> > > >
> > > > Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>
> > > > drivers/infiniband/hw/qib/qib.h | 5 +-
> > > > drivers/infiniband/hw/qib/qib_sysfs.c | 596 +++++++++++---------------
> > > > 2 files changed, 248 insertions(+), 353 deletions(-)
> > > >
> > > > diff --git a/drivers/infiniband/hw/qib/qib.h b/drivers/infiniband/hw/qib/qib.h
> > > > index 88497739029e02..3decd6d0843172 100644
> > > > +++ b/drivers/infiniband/hw/qib/qib.h
> > > > @@ -521,10 +521,7 @@ struct qib_pportdata {
> > > >
> > > > struct qib_devdata *dd;
> > > > struct qib_chippport_specific *cpspec; /* chip-specific per-port */
> > > > - struct kobject pport_kobj;
> > > > - struct kobject pport_cc_kobj;
> > > > - struct kobject sl2vl_kobj;
> > > > - struct kobject diagc_kobj;
> > > > + const struct attribute_group *groups[5];
> > >
> > > As you initialize these all at once, why not just make this:
> > > struct attribute_group **groups;
> > >
> > > and then set the groups up at build time instead of runtime as part of a
> > > larger structure like the ATTRIBUTE_GROUPS() macro does for "simple"
> > > drivers? That way you aren't fixed at the array size here and someone
> > > has to go and check to verify you really have properly 0 terminated the
> > > list and set up the pointers properly.
> >
> > qib has a variable list of group memberships that can only be
> > determined at runtime:
> >
> > if (qib_cc_table_size && ppd->congestion_entries_shadow)
> > *cur_group++ = &port_ccmgta_attribute_group;
> >
> > So it can't be setup statically at compile time.
>
> That attribute group can have a is_visable() callback to allow those
> files to show up or not, instead of having to determine this when you
> are setting up the list of groups.
Okay, that looks like it will work out and will be less complicated
code in a driver, and I can get rid of a function callback. See below
> But that's your call, not a big deal, overall this series looks like a
> lot of good cleanups to me, thanks for doing it.
No problem, it deletes a lot of horrible code. Every time I looked at
this in the past I wanted to poke at it. When Kee's pointed out what
it was doing it was just the last straw. I didn't expect it to expand
into cm, qib and hfi1 though..
Thanks,
Jason
diff --git a/drivers/infiniband/hw/qib/qib.h b/drivers/infiniband/hw/qib/qib.h
index 3decd6d0843172..b8a2deb5b4d295 100644
--- a/drivers/infiniband/hw/qib/qib.h
+++ b/drivers/infiniband/hw/qib/qib.h
@@ -521,7 +521,6 @@ struct qib_pportdata {
struct qib_devdata *dd;
struct qib_chippport_specific *cpspec; /* chip-specific per-port */
- const struct attribute_group *groups[5];
/* GUID for this interface, in network order */
__be64 guid;
diff --git a/drivers/infiniband/hw/qib/qib_sysfs.c b/drivers/infiniband/hw/qib/qib_sysfs.c
index 2c81285d245fa7..a1e22c49871266 100644
--- a/drivers/infiniband/hw/qib/qib_sysfs.c
+++ b/drivers/infiniband/hw/qib/qib_sysfs.c
@@ -282,8 +282,19 @@ static struct bin_attribute *port_ccmgta_attributes[] = {
NULL,
};
+static umode_t qib_ccmgta_is_bin_visible(struct kobject *kobj,
+ struct bin_attribute *attr, int n)
+{
+ struct qib_pportdata *ppd = qib_get_pportdata_kobj(kobj);
+
+ if (!qib_cc_table_size || !ppd->congestion_entries_shadow)
+ return 0;
+ return attr->attr.mode;
+}
+
static const struct attribute_group port_ccmgta_attribute_group = {
.name = "CCMgtA",
+ .is_bin_visible = qib_ccmgta_is_bin_visible,
.bin_attrs = port_ccmgta_attributes,
};
@@ -534,6 +545,14 @@ static const struct attribute_group port_diagc_group = {
/* End diag_counters */
+static const struct attribute_group *qib_port_groups[] = {
+ &port_linkcontrol_group,
+ &port_ccmgta_attribute_group,
+ &port_sl2vl_group,
+ &port_diagc_group,
+ NULL,
+};
+
/* end of per-port file structures and support code */
/*
@@ -718,19 +737,7 @@ const struct attribute_group qib_attr_group = {
int qib_create_port_files(struct ib_device *ibdev, u32 port_num,
struct kobject *kobj)
{
- struct qib_devdata *dd = dd_from_ibdev(ibdev);
- struct qib_pportdata *ppd = &dd->pport[port_num - 1];
- const struct attribute_group **cur_group;
-
- cur_group = &ppd->groups[0];
- *cur_group++ = &port_linkcontrol_group;
- *cur_group++ = &port_sl2vl_group;
- *cur_group++ = &port_diagc_group;
-
- if (qib_cc_table_size && ppd->congestion_entries_shadow)
- *cur_group++ = &port_ccmgta_attribute_group;
-
- return ib_port_sysfs_create_groups(ibdev, port_num, ppd->groups);
+ return ib_port_sysfs_create_groups(ibdev, port_num, qib_port_groups);
}
/*
@@ -740,10 +747,7 @@ void qib_verbs_unregister_sysfs(struct qib_devdata *dd)
{
int i;
- for (i = 0; i < dd->num_pports; i++) {
- struct qib_pportdata *ppd = &dd->pport[i];
-
+ for (i = 0; i < dd->num_pports; i++)
ib_port_sysfs_remove_groups(&dd->verbs_dev.rdi.ibdev, i,
- ppd->groups);
- }
+ qib_port_groups);
}
^ permalink raw reply related [flat|nested] 35+ messages in thread
* [PATCH 12/13] RDMA/hfi1: Use attributes for the port sysfs
2021-05-17 16:47 [PATCH 00/13] Reorganize sysfs file creation for struct ib_devices Jason Gunthorpe
` (10 preceding siblings ...)
2021-05-17 16:47 ` [PATCH 11/13] RDMA/qib: Use attributes for the port sysfs Jason Gunthorpe
@ 2021-05-17 16:47 ` Jason Gunthorpe
2021-05-17 16:47 ` [PATCH 13/13] RDMA: Change ops->init_port to ops->get_port_groups Jason Gunthorpe
2021-05-18 23:07 ` [PATCH 00/13] Reorganize sysfs file creation for struct ib_devices Nathan Chancellor
13 siblings, 0 replies; 35+ messages in thread
From: Jason Gunthorpe @ 2021-05-17 16:47 UTC (permalink / raw)
To: Dennis Dalessandro, Doug Ledford, linux-rdma, Mike Marciniszyn
Cc: Greg KH, Kees Cook, Nathan Chancellor
hfi1 should not be creating a mess of kobjects to attach to the port
kobject - this is all attributes. The proper API is to create an
attribute_group list and create it against the port's kobject.
Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>
---
drivers/infiniband/hw/hfi1/hfi.h | 4 -
drivers/infiniband/hw/hfi1/sysfs.c | 529 +++++++++++------------------
2 files changed, 193 insertions(+), 340 deletions(-)
diff --git a/drivers/infiniband/hw/hfi1/hfi.h b/drivers/infiniband/hw/hfi1/hfi.h
index 867ae0b1aa9593..87e101fb1f658a 100644
--- a/drivers/infiniband/hw/hfi1/hfi.h
+++ b/drivers/infiniband/hw/hfi1/hfi.h
@@ -772,10 +772,6 @@ struct hfi1_pportdata {
struct hfi1_ibport ibport_data;
struct hfi1_devdata *dd;
- struct kobject pport_cc_kobj;
- struct kobject sc2vl_kobj;
- struct kobject sl2sc_kobj;
- struct kobject vl2mtu_kobj;
/* PHY support */
struct qsfp_data qsfp_info;
diff --git a/drivers/infiniband/hw/hfi1/sysfs.c b/drivers/infiniband/hw/hfi1/sysfs.c
index eaf441ece25eb8..326c6c2842f2f3 100644
--- a/drivers/infiniband/hw/hfi1/sysfs.c
+++ b/drivers/infiniband/hw/hfi1/sysfs.c
@@ -45,11 +45,21 @@
*
*/
#include <linux/ctype.h>
+#include <rdma/ib_sysfs.h>
#include "hfi.h"
#include "mad.h"
#include "trace.h"
+static struct hfi1_pportdata *hfi1_get_pportdata_kobj(struct kobject *kobj)
+{
+ u32 port_num;
+ struct ib_device *ibdev = ib_port_sysfs_get_ibdev_kobj(kobj, &port_num);
+ struct hfi1_devdata *dd = dd_from_ibdev(ibdev);
+
+ return &dd->pport[port_num - 1];
+}
+
/*
* Start of per-port congestion control structures and support code
*/
@@ -57,13 +67,12 @@
/*
* Congestion control table size followed by table entries
*/
-static ssize_t read_cc_table_bin(struct file *filp, struct kobject *kobj,
- struct bin_attribute *bin_attr,
- char *buf, loff_t pos, size_t count)
+static ssize_t cc_table_bin_read(struct file *filp, struct kobject *kobj,
+ struct bin_attribute *bin_attr, char *buf,
+ loff_t pos, size_t count)
{
int ret;
- struct hfi1_pportdata *ppd =
- container_of(kobj, struct hfi1_pportdata, pport_cc_kobj);
+ struct hfi1_pportdata *ppd = hfi1_get_pportdata_kobj(kobj);
struct cc_state *cc_state;
ret = ppd->total_cct_entry * sizeof(struct ib_cc_table_entry_shadow)
@@ -89,30 +98,19 @@ static ssize_t read_cc_table_bin(struct file *filp, struct kobject *kobj,
return count;
}
-
-static void port_release(struct kobject *kobj)
-{
- /* nothing to do since memory is freed by hfi1_free_devdata() */
-}
-
-static const struct bin_attribute cc_table_bin_attr = {
- .attr = {.name = "cc_table_bin", .mode = 0444},
- .read = read_cc_table_bin,
- .size = PAGE_SIZE,
-};
+static BIN_ATTR_RO(cc_table_bin, PAGE_SIZE);
/*
* Congestion settings: port control, control map and an array of 16
* entries for the congestion entries - increase, timer, event log
* trigger threshold and the minimum injection rate delay.
*/
-static ssize_t read_cc_setting_bin(struct file *filp, struct kobject *kobj,
+static ssize_t cc_setting_bin_read(struct file *filp, struct kobject *kobj,
struct bin_attribute *bin_attr,
char *buf, loff_t pos, size_t count)
{
+ struct hfi1_pportdata *ppd = hfi1_get_pportdata_kobj(kobj);
int ret;
- struct hfi1_pportdata *ppd =
- container_of(kobj, struct hfi1_pportdata, pport_cc_kobj);
struct cc_state *cc_state;
ret = sizeof(struct opa_congestion_setting_attr_shadow);
@@ -136,27 +134,30 @@ static ssize_t read_cc_setting_bin(struct file *filp, struct kobject *kobj,
return count;
}
+static BIN_ATTR_RO(cc_setting_bin, PAGE_SIZE);
-static const struct bin_attribute cc_setting_bin_attr = {
- .attr = {.name = "cc_settings_bin", .mode = 0444},
- .read = read_cc_setting_bin,
- .size = PAGE_SIZE,
-};
-
-struct hfi1_port_attr {
- struct attribute attr;
- ssize_t (*show)(struct hfi1_pportdata *, char *);
- ssize_t (*store)(struct hfi1_pportdata *, const char *, size_t);
+static struct bin_attribute *port_cc_bin_attributes[] = {
+ &bin_attr_cc_setting_bin,
+ &bin_attr_cc_table_bin,
+ NULL
};
-static ssize_t cc_prescan_show(struct hfi1_pportdata *ppd, char *buf)
+static ssize_t cc_prescan_show(struct ib_device *ibdev, u32 port_num,
+ struct ib_port_attribute *attr, char *buf)
{
+ struct hfi1_devdata *dd = dd_from_ibdev(ibdev);
+ struct hfi1_pportdata *ppd = &dd->pport[port_num - 1];
+
return sysfs_emit(buf, "%s\n", ppd->cc_prescan ? "on" : "off");
}
-static ssize_t cc_prescan_store(struct hfi1_pportdata *ppd, const char *buf,
+static ssize_t cc_prescan_store(struct ib_device *ibdev, u32 port_num,
+ struct ib_port_attribute *attr, const char *buf,
size_t count)
{
+ struct hfi1_devdata *dd = dd_from_ibdev(ibdev);
+ struct hfi1_pportdata *ppd = &dd->pport[port_num - 1];
+
if (!memcmp(buf, "on", 2))
ppd->cc_prescan = true;
else if (!memcmp(buf, "off", 3))
@@ -164,60 +165,41 @@ static ssize_t cc_prescan_store(struct hfi1_pportdata *ppd, const char *buf,
return count;
}
+static IB_PORT_ATTR_ADMIN_RW(cc_prescan);
-static struct hfi1_port_attr cc_prescan_attr =
- __ATTR(cc_prescan, 0600, cc_prescan_show, cc_prescan_store);
-
-static ssize_t cc_attr_show(struct kobject *kobj, struct attribute *attr,
- char *buf)
-{
- struct hfi1_port_attr *port_attr =
- container_of(attr, struct hfi1_port_attr, attr);
- struct hfi1_pportdata *ppd =
- container_of(kobj, struct hfi1_pportdata, pport_cc_kobj);
-
- return port_attr->show(ppd, buf);
-}
-
-static ssize_t cc_attr_store(struct kobject *kobj, struct attribute *attr,
- const char *buf, size_t count)
-{
- struct hfi1_port_attr *port_attr =
- container_of(attr, struct hfi1_port_attr, attr);
- struct hfi1_pportdata *ppd =
- container_of(kobj, struct hfi1_pportdata, pport_cc_kobj);
-
- return port_attr->store(ppd, buf, count);
-}
-
-static const struct sysfs_ops port_cc_sysfs_ops = {
- .show = cc_attr_show,
- .store = cc_attr_store
-};
-
-static struct attribute *port_cc_default_attributes[] = {
- &cc_prescan_attr.attr,
+static struct attribute *port_cc_attributes[] = {
+ &ib_port_attr_cc_prescan.attr,
NULL
};
-static struct kobj_type port_cc_ktype = {
- .release = port_release,
- .sysfs_ops = &port_cc_sysfs_ops,
- .default_attrs = port_cc_default_attributes
+static const struct attribute_group port_cc_group = {
+ .name = "CCMgtA",
+ .attrs = port_cc_attributes,
+ .bin_attrs = port_cc_bin_attributes,
};
/* Start sc2vl */
-#define HFI1_SC2VL_ATTR(N) \
- static struct hfi1_sc2vl_attr hfi1_sc2vl_attr_##N = { \
- .attr = { .name = __stringify(N), .mode = 0444 }, \
- .sc = N \
- }
-
struct hfi1_sc2vl_attr {
- struct attribute attr;
+ struct ib_port_attribute attr;
int sc;
};
+static ssize_t sc2vl_attr_show(struct ib_device *ibdev, u32 port_num,
+ struct ib_port_attribute *attr, char *buf)
+{
+ struct hfi1_sc2vl_attr *sattr =
+ container_of(attr, struct hfi1_sc2vl_attr, attr);
+ struct hfi1_devdata *dd = dd_from_ibdev(ibdev);
+
+ return sysfs_emit(buf, "%u\n", *((u8 *)dd->sc2vl + sattr->sc));
+}
+
+#define HFI1_SC2VL_ATTR(N) \
+ static struct hfi1_sc2vl_attr hfi1_sc2vl_attr_##N = { \
+ .attr = __ATTR(N, 0444, sc2vl_attr_show, NULL), \
+ .sc = N, \
+ }
+
HFI1_SC2VL_ATTR(0);
HFI1_SC2VL_ATTR(1);
HFI1_SC2VL_ATTR(2);
@@ -251,78 +233,70 @@ HFI1_SC2VL_ATTR(29);
HFI1_SC2VL_ATTR(30);
HFI1_SC2VL_ATTR(31);
-static struct attribute *sc2vl_default_attributes[] = {
- &hfi1_sc2vl_attr_0.attr,
- &hfi1_sc2vl_attr_1.attr,
- &hfi1_sc2vl_attr_2.attr,
- &hfi1_sc2vl_attr_3.attr,
- &hfi1_sc2vl_attr_4.attr,
- &hfi1_sc2vl_attr_5.attr,
- &hfi1_sc2vl_attr_6.attr,
- &hfi1_sc2vl_attr_7.attr,
- &hfi1_sc2vl_attr_8.attr,
- &hfi1_sc2vl_attr_9.attr,
- &hfi1_sc2vl_attr_10.attr,
- &hfi1_sc2vl_attr_11.attr,
- &hfi1_sc2vl_attr_12.attr,
- &hfi1_sc2vl_attr_13.attr,
- &hfi1_sc2vl_attr_14.attr,
- &hfi1_sc2vl_attr_15.attr,
- &hfi1_sc2vl_attr_16.attr,
- &hfi1_sc2vl_attr_17.attr,
- &hfi1_sc2vl_attr_18.attr,
- &hfi1_sc2vl_attr_19.attr,
- &hfi1_sc2vl_attr_20.attr,
- &hfi1_sc2vl_attr_21.attr,
- &hfi1_sc2vl_attr_22.attr,
- &hfi1_sc2vl_attr_23.attr,
- &hfi1_sc2vl_attr_24.attr,
- &hfi1_sc2vl_attr_25.attr,
- &hfi1_sc2vl_attr_26.attr,
- &hfi1_sc2vl_attr_27.attr,
- &hfi1_sc2vl_attr_28.attr,
- &hfi1_sc2vl_attr_29.attr,
- &hfi1_sc2vl_attr_30.attr,
- &hfi1_sc2vl_attr_31.attr,
+static struct attribute *port_sc2vl_attributes[] = {
+ &hfi1_sc2vl_attr_0.attr.attr,
+ &hfi1_sc2vl_attr_1.attr.attr,
+ &hfi1_sc2vl_attr_2.attr.attr,
+ &hfi1_sc2vl_attr_3.attr.attr,
+ &hfi1_sc2vl_attr_4.attr.attr,
+ &hfi1_sc2vl_attr_5.attr.attr,
+ &hfi1_sc2vl_attr_6.attr.attr,
+ &hfi1_sc2vl_attr_7.attr.attr,
+ &hfi1_sc2vl_attr_8.attr.attr,
+ &hfi1_sc2vl_attr_9.attr.attr,
+ &hfi1_sc2vl_attr_10.attr.attr,
+ &hfi1_sc2vl_attr_11.attr.attr,
+ &hfi1_sc2vl_attr_12.attr.attr,
+ &hfi1_sc2vl_attr_13.attr.attr,
+ &hfi1_sc2vl_attr_14.attr.attr,
+ &hfi1_sc2vl_attr_15.attr.attr,
+ &hfi1_sc2vl_attr_16.attr.attr,
+ &hfi1_sc2vl_attr_17.attr.attr,
+ &hfi1_sc2vl_attr_18.attr.attr,
+ &hfi1_sc2vl_attr_19.attr.attr,
+ &hfi1_sc2vl_attr_20.attr.attr,
+ &hfi1_sc2vl_attr_21.attr.attr,
+ &hfi1_sc2vl_attr_22.attr.attr,
+ &hfi1_sc2vl_attr_23.attr.attr,
+ &hfi1_sc2vl_attr_24.attr.attr,
+ &hfi1_sc2vl_attr_25.attr.attr,
+ &hfi1_sc2vl_attr_26.attr.attr,
+ &hfi1_sc2vl_attr_27.attr.attr,
+ &hfi1_sc2vl_attr_28.attr.attr,
+ &hfi1_sc2vl_attr_29.attr.attr,
+ &hfi1_sc2vl_attr_30.attr.attr,
+ &hfi1_sc2vl_attr_31.attr.attr,
NULL
};
-static ssize_t sc2vl_attr_show(struct kobject *kobj, struct attribute *attr,
- char *buf)
-{
- struct hfi1_sc2vl_attr *sattr =
- container_of(attr, struct hfi1_sc2vl_attr, attr);
- struct hfi1_pportdata *ppd =
- container_of(kobj, struct hfi1_pportdata, sc2vl_kobj);
- struct hfi1_devdata *dd = ppd->dd;
-
- return sysfs_emit(buf, "%u\n", *((u8 *)dd->sc2vl + sattr->sc));
-}
-
-static const struct sysfs_ops hfi1_sc2vl_ops = {
- .show = sc2vl_attr_show,
+static const struct attribute_group port_sc2vl_group = {
+ .name = "sc2vl",
+ .attrs = port_sc2vl_attributes,
};
-
-static struct kobj_type hfi1_sc2vl_ktype = {
- .release = port_release,
- .sysfs_ops = &hfi1_sc2vl_ops,
- .default_attrs = sc2vl_default_attributes
-};
-
/* End sc2vl */
/* Start sl2sc */
-#define HFI1_SL2SC_ATTR(N) \
- static struct hfi1_sl2sc_attr hfi1_sl2sc_attr_##N = { \
- .attr = { .name = __stringify(N), .mode = 0444 }, \
- .sl = N \
- }
-
struct hfi1_sl2sc_attr {
- struct attribute attr;
+ struct ib_port_attribute attr;
int sl;
};
+static ssize_t sl2sc_attr_show(struct ib_device *ibdev, u32 port_num,
+ struct ib_port_attribute *attr, char *buf)
+{
+ struct hfi1_sl2sc_attr *sattr =
+ container_of(attr, struct hfi1_sl2sc_attr, attr);
+ struct hfi1_devdata *dd = dd_from_ibdev(ibdev);
+ struct hfi1_ibport *ibp = &dd->pport[port_num - 1].ibport_data;
+
+ return sysfs_emit(buf, "%u\n", ibp->sl_to_sc[sattr->sl]);
+}
+
+#define HFI1_SL2SC_ATTR(N) \
+ static struct hfi1_sl2sc_attr hfi1_sl2sc_attr_##N = { \
+ .attr = __ATTR(N, 0444, sl2sc_attr_show, NULL), .sl = N \
+ }
+
HFI1_SL2SC_ATTR(0);
HFI1_SL2SC_ATTR(1);
HFI1_SL2SC_ATTR(2);
@@ -356,79 +330,72 @@ HFI1_SL2SC_ATTR(29);
HFI1_SL2SC_ATTR(30);
HFI1_SL2SC_ATTR(31);
-static struct attribute *sl2sc_default_attributes[] = {
- &hfi1_sl2sc_attr_0.attr,
- &hfi1_sl2sc_attr_1.attr,
- &hfi1_sl2sc_attr_2.attr,
- &hfi1_sl2sc_attr_3.attr,
- &hfi1_sl2sc_attr_4.attr,
- &hfi1_sl2sc_attr_5.attr,
- &hfi1_sl2sc_attr_6.attr,
- &hfi1_sl2sc_attr_7.attr,
- &hfi1_sl2sc_attr_8.attr,
- &hfi1_sl2sc_attr_9.attr,
- &hfi1_sl2sc_attr_10.attr,
- &hfi1_sl2sc_attr_11.attr,
- &hfi1_sl2sc_attr_12.attr,
- &hfi1_sl2sc_attr_13.attr,
- &hfi1_sl2sc_attr_14.attr,
- &hfi1_sl2sc_attr_15.attr,
- &hfi1_sl2sc_attr_16.attr,
- &hfi1_sl2sc_attr_17.attr,
- &hfi1_sl2sc_attr_18.attr,
- &hfi1_sl2sc_attr_19.attr,
- &hfi1_sl2sc_attr_20.attr,
- &hfi1_sl2sc_attr_21.attr,
- &hfi1_sl2sc_attr_22.attr,
- &hfi1_sl2sc_attr_23.attr,
- &hfi1_sl2sc_attr_24.attr,
- &hfi1_sl2sc_attr_25.attr,
- &hfi1_sl2sc_attr_26.attr,
- &hfi1_sl2sc_attr_27.attr,
- &hfi1_sl2sc_attr_28.attr,
- &hfi1_sl2sc_attr_29.attr,
- &hfi1_sl2sc_attr_30.attr,
- &hfi1_sl2sc_attr_31.attr,
+static struct attribute *port_sl2sc_attributes[] = {
+ &hfi1_sl2sc_attr_0.attr.attr,
+ &hfi1_sl2sc_attr_1.attr.attr,
+ &hfi1_sl2sc_attr_2.attr.attr,
+ &hfi1_sl2sc_attr_3.attr.attr,
+ &hfi1_sl2sc_attr_4.attr.attr,
+ &hfi1_sl2sc_attr_5.attr.attr,
+ &hfi1_sl2sc_attr_6.attr.attr,
+ &hfi1_sl2sc_attr_7.attr.attr,
+ &hfi1_sl2sc_attr_8.attr.attr,
+ &hfi1_sl2sc_attr_9.attr.attr,
+ &hfi1_sl2sc_attr_10.attr.attr,
+ &hfi1_sl2sc_attr_11.attr.attr,
+ &hfi1_sl2sc_attr_12.attr.attr,
+ &hfi1_sl2sc_attr_13.attr.attr,
+ &hfi1_sl2sc_attr_14.attr.attr,
+ &hfi1_sl2sc_attr_15.attr.attr,
+ &hfi1_sl2sc_attr_16.attr.attr,
+ &hfi1_sl2sc_attr_17.attr.attr,
+ &hfi1_sl2sc_attr_18.attr.attr,
+ &hfi1_sl2sc_attr_19.attr.attr,
+ &hfi1_sl2sc_attr_20.attr.attr,
+ &hfi1_sl2sc_attr_21.attr.attr,
+ &hfi1_sl2sc_attr_22.attr.attr,
+ &hfi1_sl2sc_attr_23.attr.attr,
+ &hfi1_sl2sc_attr_24.attr.attr,
+ &hfi1_sl2sc_attr_25.attr.attr,
+ &hfi1_sl2sc_attr_26.attr.attr,
+ &hfi1_sl2sc_attr_27.attr.attr,
+ &hfi1_sl2sc_attr_28.attr.attr,
+ &hfi1_sl2sc_attr_29.attr.attr,
+ &hfi1_sl2sc_attr_30.attr.attr,
+ &hfi1_sl2sc_attr_31.attr.attr,
NULL
};
-static ssize_t sl2sc_attr_show(struct kobject *kobj, struct attribute *attr,
- char *buf)
-{
- struct hfi1_sl2sc_attr *sattr =
- container_of(attr, struct hfi1_sl2sc_attr, attr);
- struct hfi1_pportdata *ppd =
- container_of(kobj, struct hfi1_pportdata, sl2sc_kobj);
- struct hfi1_ibport *ibp = &ppd->ibport_data;
-
- return sysfs_emit(buf, "%u\n", ibp->sl_to_sc[sattr->sl]);
-}
-
-static const struct sysfs_ops hfi1_sl2sc_ops = {
- .show = sl2sc_attr_show,
-};
-
-static struct kobj_type hfi1_sl2sc_ktype = {
- .release = port_release,
- .sysfs_ops = &hfi1_sl2sc_ops,
- .default_attrs = sl2sc_default_attributes
+static const struct attribute_group port_sl2sc_group = {
+ .name = "sl2sc",
+ .attrs = port_sl2sc_attributes,
};
/* End sl2sc */
/* Start vl2mtu */
-#define HFI1_VL2MTU_ATTR(N) \
- static struct hfi1_vl2mtu_attr hfi1_vl2mtu_attr_##N = { \
- .attr = { .name = __stringify(N), .mode = 0444 }, \
- .vl = N \
- }
-
struct hfi1_vl2mtu_attr {
- struct attribute attr;
+ struct ib_port_attribute attr;
int vl;
};
+static ssize_t vl2mtu_attr_show(struct ib_device *ibdev, u32 port_num,
+ struct ib_port_attribute *attr, char *buf)
+{
+ struct hfi1_vl2mtu_attr *vlattr =
+ container_of(attr, struct hfi1_vl2mtu_attr, attr);
+ struct hfi1_devdata *dd = dd_from_ibdev(ibdev);
+
+ return sysfs_emit(buf, "%u\n", dd->vld[vlattr->vl].mtu);
+}
+
+#define HFI1_VL2MTU_ATTR(N) \
+ static struct hfi1_vl2mtu_attr hfi1_vl2mtu_attr_##N = { \
+ .attr = __ATTR(N, 0444, vl2mtu_attr_show, NULL), \
+ .vl = N, \
+ }
+
HFI1_VL2MTU_ATTR(0);
HFI1_VL2MTU_ATTR(1);
HFI1_VL2MTU_ATTR(2);
@@ -446,46 +413,29 @@ HFI1_VL2MTU_ATTR(13);
HFI1_VL2MTU_ATTR(14);
HFI1_VL2MTU_ATTR(15);
-static struct attribute *vl2mtu_default_attributes[] = {
- &hfi1_vl2mtu_attr_0.attr,
- &hfi1_vl2mtu_attr_1.attr,
- &hfi1_vl2mtu_attr_2.attr,
- &hfi1_vl2mtu_attr_3.attr,
- &hfi1_vl2mtu_attr_4.attr,
- &hfi1_vl2mtu_attr_5.attr,
- &hfi1_vl2mtu_attr_6.attr,
- &hfi1_vl2mtu_attr_7.attr,
- &hfi1_vl2mtu_attr_8.attr,
- &hfi1_vl2mtu_attr_9.attr,
- &hfi1_vl2mtu_attr_10.attr,
- &hfi1_vl2mtu_attr_11.attr,
- &hfi1_vl2mtu_attr_12.attr,
- &hfi1_vl2mtu_attr_13.attr,
- &hfi1_vl2mtu_attr_14.attr,
- &hfi1_vl2mtu_attr_15.attr,
+static struct attribute *port_vl2mtu_attributes[] = {
+ &hfi1_vl2mtu_attr_0.attr.attr,
+ &hfi1_vl2mtu_attr_1.attr.attr,
+ &hfi1_vl2mtu_attr_2.attr.attr,
+ &hfi1_vl2mtu_attr_3.attr.attr,
+ &hfi1_vl2mtu_attr_4.attr.attr,
+ &hfi1_vl2mtu_attr_5.attr.attr,
+ &hfi1_vl2mtu_attr_6.attr.attr,
+ &hfi1_vl2mtu_attr_7.attr.attr,
+ &hfi1_vl2mtu_attr_8.attr.attr,
+ &hfi1_vl2mtu_attr_9.attr.attr,
+ &hfi1_vl2mtu_attr_10.attr.attr,
+ &hfi1_vl2mtu_attr_11.attr.attr,
+ &hfi1_vl2mtu_attr_12.attr.attr,
+ &hfi1_vl2mtu_attr_13.attr.attr,
+ &hfi1_vl2mtu_attr_14.attr.attr,
+ &hfi1_vl2mtu_attr_15.attr.attr,
NULL
};
-static ssize_t vl2mtu_attr_show(struct kobject *kobj, struct attribute *attr,
- char *buf)
-{
- struct hfi1_vl2mtu_attr *vlattr =
- container_of(attr, struct hfi1_vl2mtu_attr, attr);
- struct hfi1_pportdata *ppd =
- container_of(kobj, struct hfi1_pportdata, vl2mtu_kobj);
- struct hfi1_devdata *dd = ppd->dd;
-
- return sysfs_emit(buf, "%u\n", dd->vld[vlattr->vl].mtu);
-}
-
-static const struct sysfs_ops hfi1_vl2mtu_ops = {
- .show = vl2mtu_attr_show,
-};
-
-static struct kobj_type hfi1_vl2mtu_ktype = {
- .release = port_release,
- .sysfs_ops = &hfi1_vl2mtu_ops,
- .default_attrs = vl2mtu_default_attributes
+static const struct attribute_group port_vl2mtu_group = {
+ .name = "vl2mtu",
+ .attrs = port_vl2mtu_attributes,
};
/* end of per-port file structures and support code */
@@ -649,100 +599,17 @@ const struct attribute_group ib_hfi1_attr_group = {
.attrs = hfi1_attributes,
};
+static const struct attribute_group *hfi1_port_groups[] = {
+ &port_sc2vl_group,
+ &port_sl2sc_group,
+ &port_vl2mtu_group,
+ NULL,
+};
+
int hfi1_create_port_files(struct ib_device *ibdev, u32 port_num,
struct kobject *kobj)
{
- struct hfi1_pportdata *ppd;
- struct hfi1_devdata *dd = dd_from_ibdev(ibdev);
- int ret;
-
- if (!port_num || port_num > dd->num_pports) {
- dd_dev_err(dd,
- "Skipping infiniband class with invalid port %u\n",
- port_num);
- return -ENODEV;
- }
- ppd = &dd->pport[port_num - 1];
-
- ret = kobject_init_and_add(&ppd->sc2vl_kobj, &hfi1_sc2vl_ktype, kobj,
- "sc2vl");
- if (ret) {
- dd_dev_err(dd,
- "Skipping sc2vl sysfs info, (err %d) port %u\n",
- ret, port_num);
- /*
- * Based on the documentation for kobject_init_and_add(), the
- * caller should call kobject_put even if this call fails.
- */
- goto bail_sc2vl;
- }
- kobject_uevent(&ppd->sc2vl_kobj, KOBJ_ADD);
-
- ret = kobject_init_and_add(&ppd->sl2sc_kobj, &hfi1_sl2sc_ktype, kobj,
- "sl2sc");
- if (ret) {
- dd_dev_err(dd,
- "Skipping sl2sc sysfs info, (err %d) port %u\n",
- ret, port_num);
- goto bail_sl2sc;
- }
- kobject_uevent(&ppd->sl2sc_kobj, KOBJ_ADD);
-
- ret = kobject_init_and_add(&ppd->vl2mtu_kobj, &hfi1_vl2mtu_ktype, kobj,
- "vl2mtu");
- if (ret) {
- dd_dev_err(dd,
- "Skipping vl2mtu sysfs info, (err %d) port %u\n",
- ret, port_num);
- goto bail_vl2mtu;
- }
- kobject_uevent(&ppd->vl2mtu_kobj, KOBJ_ADD);
-
- ret = kobject_init_and_add(&ppd->pport_cc_kobj, &port_cc_ktype,
- kobj, "CCMgtA");
- if (ret) {
- dd_dev_err(dd,
- "Skipping Congestion Control sysfs info, (err %d) port %u\n",
- ret, port_num);
- goto bail_cc;
- }
-
- kobject_uevent(&ppd->pport_cc_kobj, KOBJ_ADD);
-
- ret = sysfs_create_bin_file(&ppd->pport_cc_kobj, &cc_setting_bin_attr);
- if (ret) {
- dd_dev_err(dd,
- "Skipping Congestion Control setting sysfs info, (err %d) port %u\n",
- ret, port_num);
- goto bail_cc;
- }
-
- ret = sysfs_create_bin_file(&ppd->pport_cc_kobj, &cc_table_bin_attr);
- if (ret) {
- dd_dev_err(dd,
- "Skipping Congestion Control table sysfs info, (err %d) port %u\n",
- ret, port_num);
- goto bail_cc_entry_bin;
- }
-
- dd_dev_info(dd,
- "Congestion Control Agent enabled for port %d\n",
- port_num);
-
- return 0;
-
-bail_cc_entry_bin:
- sysfs_remove_bin_file(&ppd->pport_cc_kobj,
- &cc_setting_bin_attr);
-bail_cc:
- kobject_put(&ppd->pport_cc_kobj);
-bail_vl2mtu:
- kobject_put(&ppd->vl2mtu_kobj);
-bail_sl2sc:
- kobject_put(&ppd->sl2sc_kobj);
-bail_sc2vl:
- kobject_put(&ppd->sc2vl_kobj);
- return ret;
+ return ib_port_sysfs_create_groups(ibdev, port_num, hfi1_port_groups);
}
struct sde_attribute {
@@ -868,23 +735,13 @@ int hfi1_verbs_register_sysfs(struct hfi1_devdata *dd)
*/
void hfi1_verbs_unregister_sysfs(struct hfi1_devdata *dd)
{
- struct hfi1_pportdata *ppd;
int i;
/* Unwind operations in hfi1_verbs_register_sysfs() */
for (i = 0; i < dd->num_sdma; i++)
kobject_put(&dd->per_sdma[i].kobj);
- for (i = 0; i < dd->num_pports; i++) {
- ppd = &dd->pport[i];
-
- sysfs_remove_bin_file(&ppd->pport_cc_kobj,
- &cc_setting_bin_attr);
- sysfs_remove_bin_file(&ppd->pport_cc_kobj,
- &cc_table_bin_attr);
- kobject_put(&ppd->pport_cc_kobj);
- kobject_put(&ppd->vl2mtu_kobj);
- kobject_put(&ppd->sl2sc_kobj);
- kobject_put(&ppd->sc2vl_kobj);
- }
+ for (i = 0; i < dd->num_pports; i++)
+ ib_port_sysfs_remove_groups(&dd->verbs_dev.rdi.ibdev, i + 1,
+ hfi1_port_groups);
}
--
2.31.1
^ permalink raw reply related [flat|nested] 35+ messages in thread* [PATCH 13/13] RDMA: Change ops->init_port to ops->get_port_groups
2021-05-17 16:47 [PATCH 00/13] Reorganize sysfs file creation for struct ib_devices Jason Gunthorpe
` (11 preceding siblings ...)
2021-05-17 16:47 ` [PATCH 12/13] RDMA/hfi1: " Jason Gunthorpe
@ 2021-05-17 16:47 ` Jason Gunthorpe
2021-05-18 23:07 ` [PATCH 00/13] Reorganize sysfs file creation for struct ib_devices Nathan Chancellor
13 siblings, 0 replies; 35+ messages in thread
From: Jason Gunthorpe @ 2021-05-17 16:47 UTC (permalink / raw)
To: Dennis Dalessandro, Doug Ledford, linux-rdma, Mike Marciniszyn
Cc: Greg KH, Kees Cook, Nathan Chancellor
init_port was only being used to register sysfs attributes against the
port kobject. Now that all users are creating attribute_group's we can
simply return the attribute_group list from the driver and the core code
can handle it.
This makes all the sysfs management quite straightforward and prevents any
driver from abusing the naked port kobject in future because no driver
code can access it.
Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>
---
drivers/infiniband/core/device.c | 4 +--
drivers/infiniband/core/sysfs.c | 36 +++++++++------------------
drivers/infiniband/hw/hfi1/hfi.h | 4 +--
drivers/infiniband/hw/hfi1/sysfs.c | 10 +++-----
drivers/infiniband/hw/hfi1/verbs.c | 2 +-
drivers/infiniband/hw/qib/qib.h | 5 ++--
drivers/infiniband/hw/qib/qib_sysfs.c | 22 +++-------------
drivers/infiniband/hw/qib/qib_verbs.c | 4 +--
drivers/infiniband/sw/rdmavt/vt.c | 2 +-
include/rdma/ib_sysfs.h | 4 ---
include/rdma/ib_verbs.h | 5 ++--
11 files changed, 30 insertions(+), 68 deletions(-)
diff --git a/drivers/infiniband/core/device.c b/drivers/infiniband/core/device.c
index 030a4041b2e03b..28898558dda4c3 100644
--- a/drivers/infiniband/core/device.c
+++ b/drivers/infiniband/core/device.c
@@ -1703,7 +1703,7 @@ int ib_device_set_netns_put(struct sk_buff *skb,
* port_cleanup infrastructure is implemented, this limitation will be
* removed.
*/
- if (!dev->ops.disassociate_ucontext || dev->ops.init_port ||
+ if (!dev->ops.disassociate_ucontext || dev->ops.get_port_groups ||
ib_devices_shared_netns) {
ret = -EOPNOTSUPP;
goto ns_err;
@@ -2668,7 +2668,7 @@ void ib_set_device_ops(struct ib_device *dev, const struct ib_device_ops *ops)
SET_DEVICE_OP(dev_ops, get_vf_config);
SET_DEVICE_OP(dev_ops, get_vf_guid);
SET_DEVICE_OP(dev_ops, get_vf_stats);
- SET_DEVICE_OP(dev_ops, init_port);
+ SET_DEVICE_OP(dev_ops, get_port_groups);
SET_DEVICE_OP(dev_ops, iw_accept);
SET_DEVICE_OP(dev_ops, iw_add_ref);
SET_DEVICE_OP(dev_ops, iw_connect);
diff --git a/drivers/infiniband/core/sysfs.c b/drivers/infiniband/core/sysfs.c
index 5d9c8bfc280d8f..595a3d22263bf0 100644
--- a/drivers/infiniband/core/sysfs.c
+++ b/drivers/infiniband/core/sysfs.c
@@ -69,6 +69,7 @@ struct ib_port {
struct attribute_group groups[3];
const struct attribute_group *groups_list[5];
+ const struct attribute_group **driver_groups;
u32 port_num;
struct port_table_attribute attrs_list[];
};
@@ -128,22 +129,6 @@ static ssize_t port_attr_store(struct kobject *kobj,
return port_attr->store(p->ibdev, p->port_num, port_attr, buf, count);
}
-int ib_port_sysfs_create_groups(struct ib_device *ibdev, u32 port_num,
- const struct attribute_group **groups)
-{
- return sysfs_create_groups(&ibdev->port_data[port_num].sysfs->kobj,
- groups);
-}
-EXPORT_SYMBOL(ib_port_sysfs_create_groups);
-
-void ib_port_sysfs_remove_groups(struct ib_device *ibdev, u32 port_num,
- const struct attribute_group **groups)
-{
- return sysfs_remove_groups(&ibdev->port_data[port_num].sysfs->kobj,
- groups);
-}
-EXPORT_SYMBOL(ib_port_sysfs_remove_groups);
-
struct ib_device *ib_port_sysfs_get_ibdev_kobj(struct kobject *kobj,
u32 *port_num)
{
@@ -1253,6 +1238,13 @@ static struct ib_port *setup_port(struct ib_core_device *coredev, int port_num,
ret = sysfs_create_groups(&p->kobj, p->groups_list);
if (ret)
goto err_del;
+ if (device->ops.get_port_groups && is_full_dev) {
+ p->driver_groups =
+ device->ops.get_port_groups(device, port_num);
+ ret = sysfs_create_groups(&p->kobj, p->driver_groups);
+ if (ret)
+ goto err_groups;
+ }
list_add_tail(&p->kobj.entry, &coredev->port_list);
if (device->port_data && is_full_dev)
@@ -1260,6 +1252,8 @@ static struct ib_port *setup_port(struct ib_core_device *coredev, int port_num,
return p;
+err_groups:
+ sysfs_remove_groups(&p->kobj, p->groups_list);
err_del:
kobject_del(&p->kobj);
err_put:
@@ -1273,6 +1267,8 @@ static void destroy_port(struct ib_port *port)
port->ibdev->port_data[port->port_num].sysfs == port)
port->ibdev->port_data[port->port_num].sysfs = NULL;
list_del(&port->kobj.entry);
+ if (port->driver_groups)
+ sysfs_remove_groups(&port->kobj, port->driver_groups);
sysfs_remove_groups(&port->kobj, port->groups_list);
kobject_del(&port->kobj);
kobject_put(&port->kobj);
@@ -1407,7 +1403,6 @@ void ib_free_port_attrs(struct ib_core_device *coredev)
int ib_setup_port_attrs(struct ib_core_device *coredev)
{
struct ib_device *device = rdma_device_to_ibdev(&coredev->dev);
- bool is_full_dev = &device->coredev == coredev;
u32 port_num;
int ret;
@@ -1433,13 +1428,6 @@ int ib_setup_port_attrs(struct ib_core_device *coredev)
ret = setup_gid_attrs(port, &attr);
if (ret)
goto err_put;
-
- if (device->ops.init_port && is_full_dev) {
- ret = device->ops.init_port(device, port_num,
- &port->kobj);
- if (ret)
- goto err_put;
- }
}
return 0;
diff --git a/drivers/infiniband/hw/hfi1/hfi.h b/drivers/infiniband/hw/hfi1/hfi.h
index 87e101fb1f658a..d0d86af19fbf3d 100644
--- a/drivers/infiniband/hw/hfi1/hfi.h
+++ b/drivers/infiniband/hw/hfi1/hfi.h
@@ -2188,8 +2188,8 @@ extern const struct attribute_group ib_hfi1_attr_group;
int hfi1_device_create(struct hfi1_devdata *dd);
void hfi1_device_remove(struct hfi1_devdata *dd);
-int hfi1_create_port_files(struct ib_device *ibdev, u32 port_num,
- struct kobject *kobj);
+const struct attribute_group **hfi1_get_port_groups(struct ib_device *ibdev,
+ u32 port_num);
int hfi1_verbs_register_sysfs(struct hfi1_devdata *dd);
void hfi1_verbs_unregister_sysfs(struct hfi1_devdata *dd);
/* Hook for sysfs read of QSFP */
diff --git a/drivers/infiniband/hw/hfi1/sysfs.c b/drivers/infiniband/hw/hfi1/sysfs.c
index 326c6c2842f2f3..b299ae37023d32 100644
--- a/drivers/infiniband/hw/hfi1/sysfs.c
+++ b/drivers/infiniband/hw/hfi1/sysfs.c
@@ -606,10 +606,10 @@ static const struct attribute_group *hfi1_port_groups[] = {
NULL,
};
-int hfi1_create_port_files(struct ib_device *ibdev, u32 port_num,
- struct kobject *kobj)
+const struct attribute_group **hfi1_get_port_groups(struct ib_device *ibdev,
+ u32 port_num)
{
- return ib_port_sysfs_create_groups(ibdev, port_num, hfi1_port_groups);
+ return hfi1_port_groups;
}
struct sde_attribute {
@@ -740,8 +740,4 @@ void hfi1_verbs_unregister_sysfs(struct hfi1_devdata *dd)
/* Unwind operations in hfi1_verbs_register_sysfs() */
for (i = 0; i < dd->num_sdma; i++)
kobject_put(&dd->per_sdma[i].kobj);
-
- for (i = 0; i < dd->num_pports; i++)
- ib_port_sysfs_remove_groups(&dd->verbs_dev.rdi.ibdev, i + 1,
- hfi1_port_groups);
}
diff --git a/drivers/infiniband/hw/hfi1/verbs.c b/drivers/infiniband/hw/hfi1/verbs.c
index 85deba07a6754b..33c757e6b88d30 100644
--- a/drivers/infiniband/hw/hfi1/verbs.c
+++ b/drivers/infiniband/hw/hfi1/verbs.c
@@ -1791,7 +1791,7 @@ static const struct ib_device_ops hfi1_dev_ops = {
.alloc_rdma_netdev = hfi1_vnic_alloc_rn,
.get_dev_fw_str = hfi1_get_dev_fw_str,
.get_hw_stats = get_hw_stats,
- .init_port = hfi1_create_port_files,
+ .get_port_groups = hfi1_get_port_groups,
.modify_device = modify_device,
/* keep process mad in the driver */
.process_mad = hfi1_process_mad,
diff --git a/drivers/infiniband/hw/qib/qib.h b/drivers/infiniband/hw/qib/qib.h
index 3decd6d0843172..d07b766d50fabc 100644
--- a/drivers/infiniband/hw/qib/qib.h
+++ b/drivers/infiniband/hw/qib/qib.h
@@ -1366,9 +1366,8 @@ extern const struct attribute_group qib_attr_group;
int qib_device_create(struct qib_devdata *);
void qib_device_remove(struct qib_devdata *);
-int qib_create_port_files(struct ib_device *ibdev, u32 port_num,
- struct kobject *kobj);
-void qib_verbs_unregister_sysfs(struct qib_devdata *);
+const struct attribute_group **qib_get_port_groups(struct ib_device *ibdev,
+ u32 port_num);
/* Hook for sysfs read of QSFP */
extern int qib_qsfp_dump(struct qib_pportdata *ppd, char *buf, int len);
diff --git a/drivers/infiniband/hw/qib/qib_sysfs.c b/drivers/infiniband/hw/qib/qib_sysfs.c
index 2c81285d245fa7..835684d43e09f0 100644
--- a/drivers/infiniband/hw/qib/qib_sysfs.c
+++ b/drivers/infiniband/hw/qib/qib_sysfs.c
@@ -715,8 +715,8 @@ const struct attribute_group qib_attr_group = {
.attrs = qib_attributes,
};
-int qib_create_port_files(struct ib_device *ibdev, u32 port_num,
- struct kobject *kobj)
+const struct attribute_group **qib_get_port_groups(struct ib_device *ibdev,
+ u32 port_num)
{
struct qib_devdata *dd = dd_from_ibdev(ibdev);
struct qib_pportdata *ppd = &dd->pport[port_num - 1];
@@ -729,21 +729,5 @@ int qib_create_port_files(struct ib_device *ibdev, u32 port_num,
if (qib_cc_table_size && ppd->congestion_entries_shadow)
*cur_group++ = &port_ccmgta_attribute_group;
-
- return ib_port_sysfs_create_groups(ibdev, port_num, ppd->groups);
-}
-
-/*
- * Unregister and remove our files in /sys/class/infiniband.
- */
-void qib_verbs_unregister_sysfs(struct qib_devdata *dd)
-{
- int i;
-
- for (i = 0; i < dd->num_pports; i++) {
- struct qib_pportdata *ppd = &dd->pport[i];
-
- ib_port_sysfs_remove_groups(&dd->verbs_dev.rdi.ibdev, i,
- ppd->groups);
- }
+ return ppd->groups;
}
diff --git a/drivers/infiniband/hw/qib/qib_verbs.c b/drivers/infiniband/hw/qib/qib_verbs.c
index d17d034ecdfd9f..5552440f843bb1 100644
--- a/drivers/infiniband/hw/qib/qib_verbs.c
+++ b/drivers/infiniband/hw/qib/qib_verbs.c
@@ -1483,7 +1483,7 @@ static const struct ib_device_ops qib_dev_ops = {
.owner = THIS_MODULE,
.driver_id = RDMA_DRIVER_QIB,
- .init_port = qib_create_port_files,
+ .get_port_groups = qib_get_port_groups,
.modify_device = qib_modify_device,
.process_mad = qib_process_mad,
};
@@ -1644,8 +1644,6 @@ void qib_unregister_ib_device(struct qib_devdata *dd)
{
struct qib_ibdev *dev = &dd->verbs_dev;
- qib_verbs_unregister_sysfs(dd);
-
rvt_unregister_device(&dd->verbs_dev.rdi);
if (!list_empty(&dev->piowait))
diff --git a/drivers/infiniband/sw/rdmavt/vt.c b/drivers/infiniband/sw/rdmavt/vt.c
index 12ebe041a5da9b..e701ceff449b4a 100644
--- a/drivers/infiniband/sw/rdmavt/vt.c
+++ b/drivers/infiniband/sw/rdmavt/vt.c
@@ -418,7 +418,7 @@ static noinline int check_support(struct rvt_dev_info *rdi, int verb)
* These functions are not part of verbs specifically but are
* required for rdmavt to function.
*/
- if ((!rdi->ibdev.ops.init_port) ||
+ if ((!rdi->ibdev.ops.get_port_groups) ||
(!rdi->driver_f.get_pci_dev))
return -EINVAL;
break;
diff --git a/include/rdma/ib_sysfs.h b/include/rdma/ib_sysfs.h
index f869d0e4fd3030..3b77cfd74d9a30 100644
--- a/include/rdma/ib_sysfs.h
+++ b/include/rdma/ib_sysfs.h
@@ -31,10 +31,6 @@ struct ib_port_attribute {
#define IB_PORT_ATTR_WO(_name) \
struct ib_port_attribute ib_port_attr_##_name = __ATTR_WO(_name)
-int ib_port_sysfs_create_groups(struct ib_device *ibdev, u32 port_num,
- const struct attribute_group **groups);
-void ib_port_sysfs_remove_groups(struct ib_device *ibdev, u32 port_num,
- const struct attribute_group **groups);
struct ib_device *ib_port_sysfs_get_ibdev_kobj(struct kobject *kobj,
u32 *port_num);
diff --git a/include/rdma/ib_verbs.h b/include/rdma/ib_verbs.h
index 5f70aab1f8663b..5120a279a685e7 100644
--- a/include/rdma/ib_verbs.h
+++ b/include/rdma/ib_verbs.h
@@ -2551,8 +2551,9 @@ struct ib_device_ops {
* This function is called once for each port when a ib device is
* registered.
*/
- int (*init_port)(struct ib_device *device, u32 port_num,
- struct kobject *port_sysfs);
+ const struct attribute_group **(*get_port_groups)(
+ struct ib_device *device, u32 port_num);
+
/**
* Allows rdma drivers to add their own restrack attributes.
*/
--
2.31.1
^ permalink raw reply related [flat|nested] 35+ messages in thread* Re: [PATCH 00/13] Reorganize sysfs file creation for struct ib_devices
2021-05-17 16:47 [PATCH 00/13] Reorganize sysfs file creation for struct ib_devices Jason Gunthorpe
` (12 preceding siblings ...)
2021-05-17 16:47 ` [PATCH 13/13] RDMA: Change ops->init_port to ops->get_port_groups Jason Gunthorpe
@ 2021-05-18 23:07 ` Nathan Chancellor
2021-05-19 13:46 ` Jason Gunthorpe
13 siblings, 1 reply; 35+ messages in thread
From: Nathan Chancellor @ 2021-05-18 23:07 UTC (permalink / raw)
To: Jason Gunthorpe, Potnuri Bharat Teja, clang-built-linux,
Dennis Dalessandro, Devesh Sharma, Doug Ledford, Faisal Latif,
Gal Pressman, Leon Romanovsky, linux-rdma, Mike Marciniszyn,
Naresh Kumar PBS, Nick Desaulniers, Selvin Xavier, Shiraz Saleem,
Yossi Leybovich, Somnath Kotur, Sriharsha Basavapatna,
Yishai Hadas, Zhu Yanjun
Cc: Greg KH, Kees Cook
Hi Jason,
On 5/17/2021 9:47 AM, Jason Gunthorpe wrote:
> IB has a complex sysfs with a deep nesting of attributes. Nathan and Kees
> recently noticed this was not even slightly sane with how it was handling
> attributes and a deeper inspection shows the whole thing is a pretty
> "ick" coding style.
>
> Further review shows the ick extends outward from the ib_port sysfs and
> basically everything is pretty crazy.
>
> Simplify all of it:
>
> - Organize the ib_port and gid_attr's kobj's to have clear setup/destroy
> function pairings that work only on their own kobjs.
>
> - All memory allocated in service of a kobject's attributes is freed as
> part of the kobj release function. Thus all the error handling defers
> the memory frees to a put.
>
> - Build up lists of groups for every kobject and add the entire group
> list as a one-shot operation as the last thing in setup function.
>
> - Remove essentially all the error cleanup. The final kobject_put() will
> always free any memory allocated or do an internal kobject_del() if
> required. The new ordering eliminates all the other cleanup cases.
>
> - Make all attributes use proper typing for the kobj they are attached
> to. Split device and port hw_stats handling.
>
> - Create a ib_port_attribute type and change hfi1, qib and the CM code to
> work with attribute lists of ib_port_attribute type instead of building
> their own kobject madness
>
> This is sort of RFCy in that I qib and hfi1 stuff is complex enough it needs
> Dennis to look at it, and the core stuff has only passed basic testing at this
> moment. Nathan confirmed an earlier version solves the CFI warning.
This series still passes my basic testing of LTP's read_all test case on
/sys with CFI in enforcing mode. If there is any more in-depth testing,
I can put it through, let me know. I'll continue testing the series and
when it is in a mergeable state, I can provide you with a Tested-by tag.
Cheers,
Nathan
^ permalink raw reply [flat|nested] 35+ messages in thread* Re: [PATCH 00/13] Reorganize sysfs file creation for struct ib_devices
2021-05-18 23:07 ` [PATCH 00/13] Reorganize sysfs file creation for struct ib_devices Nathan Chancellor
@ 2021-05-19 13:46 ` Jason Gunthorpe
0 siblings, 0 replies; 35+ messages in thread
From: Jason Gunthorpe @ 2021-05-19 13:46 UTC (permalink / raw)
To: Nathan Chancellor
Cc: Potnuri Bharat Teja, clang-built-linux, Dennis Dalessandro,
Devesh Sharma, Doug Ledford, Faisal Latif, Gal Pressman,
Leon Romanovsky, linux-rdma, Mike Marciniszyn, Naresh Kumar PBS,
Nick Desaulniers, Selvin Xavier, Shiraz Saleem, Yossi Leybovich,
Somnath Kotur, Sriharsha Basavapatna, Yishai Hadas, Zhu Yanjun,
Greg KH, Kees Cook
On Tue, May 18, 2021 at 04:07:49PM -0700, Nathan Chancellor wrote:
> Hi Jason,
>
> On 5/17/2021 9:47 AM, Jason Gunthorpe wrote:
> > IB has a complex sysfs with a deep nesting of attributes. Nathan and Kees
> > recently noticed this was not even slightly sane with how it was handling
> > attributes and a deeper inspection shows the whole thing is a pretty
> > "ick" coding style.
> >
> > Further review shows the ick extends outward from the ib_port sysfs and
> > basically everything is pretty crazy.
> >
> > Simplify all of it:
> >
> > - Organize the ib_port and gid_attr's kobj's to have clear setup/destroy
> > function pairings that work only on their own kobjs.
> >
> > - All memory allocated in service of a kobject's attributes is freed as
> > part of the kobj release function. Thus all the error handling defers
> > the memory frees to a put.
> >
> > - Build up lists of groups for every kobject and add the entire group
> > list as a one-shot operation as the last thing in setup function.
> >
> > - Remove essentially all the error cleanup. The final kobject_put() will
> > always free any memory allocated or do an internal kobject_del() if
> > required. The new ordering eliminates all the other cleanup cases.
> >
> > - Make all attributes use proper typing for the kobj they are attached
> > to. Split device and port hw_stats handling.
> >
> > - Create a ib_port_attribute type and change hfi1, qib and the CM code to
> > work with attribute lists of ib_port_attribute type instead of building
> > their own kobject madness
> >
> > This is sort of RFCy in that I qib and hfi1 stuff is complex enough it needs
> > Dennis to look at it, and the core stuff has only passed basic testing at this
> > moment. Nathan confirmed an earlier version solves the CFI warning.
>
> This series still passes my basic testing of LTP's read_all test case on
> /sys with CFI in enforcing mode. If there is any more in-depth testing, I
> can put it through, let me know. I'll continue testing the series and when
> it is in a mergeable state, I can provide you with a Tested-by tag.
Thanks, I think you can probably ignore the following versions,
confirmation that the approach and root cause is correct is much
appreciated.
Jason
^ permalink raw reply [flat|nested] 35+ messages in thread