* [PATCH v7 for-next 1/2] IB/core: Updating cache for subnet_prefix in config_non_roce_gid_cache()
2021-06-28 9:29 [PATCH v7 for-next 0/2] IB/core: Obtaining subnet_prefix from cache in Anand Khoje
@ 2021-06-28 9:29 ` Anand Khoje
2021-06-28 9:29 ` [PATCH v7 for-next 2/2] IB/core: Read subnet_prefix in ib_query_port via cache Anand Khoje
1 sibling, 0 replies; 3+ messages in thread
From: Anand Khoje @ 2021-06-28 9:29 UTC (permalink / raw)
To: linux-rdma, linux-kernel; +Cc: dledford, jgg, haakon.bugge, leon
Currently cache for subnet_prefix was getting updated by reading port
attributes via ib_query_port. ib_query_port() calls ops.query_gid()
to get subnet_prefix and returns it via port_attr.
In ib_cache_update(), config_non_roce_gid_cache() obtains GIDs
by calling ops.query_gid(). We utilize this to store subnet_prefix
in cache.
Suggested-by: Jason Gunthorpe <jgg@nvidia.com>
Suggested-by: Aru Kolappan <aru.kolappan@oracle.com>
Signed-off-by: Anand Khoje <anand.a.khoje@oracle.com>
Signed-off-by: Haakon Bugge <haakon.bugge@oracle.com>
---
drivers/infiniband/core/cache.c | 8 +++++---
1 file changed, 5 insertions(+), 3 deletions(-)
diff --git a/drivers/infiniband/core/cache.c b/drivers/infiniband/core/cache.c
index c9e9fc8..929399e 100644
--- a/drivers/infiniband/core/cache.c
+++ b/drivers/infiniband/core/cache.c
@@ -1429,7 +1429,7 @@ int rdma_read_gid_l2_fields(const struct ib_gid_attr *attr,
EXPORT_SYMBOL(rdma_read_gid_l2_fields);
static int config_non_roce_gid_cache(struct ib_device *device,
- u32 port, int gid_tbl_len)
+ u32 port, struct ib_port_attr *tprops)
{
struct ib_gid_attr gid_attr = {};
struct ib_gid_table *table;
@@ -1441,7 +1441,7 @@ static int config_non_roce_gid_cache(struct ib_device *device,
table = rdma_gid_table(device, port);
mutex_lock(&table->lock);
- for (i = 0; i < gid_tbl_len; ++i) {
+ for (i = 0; i < tprops->gid_tbl_len; ++i) {
if (!device->ops.query_gid)
continue;
ret = device->ops.query_gid(device, port, i, &gid_attr.gid);
@@ -1452,6 +1452,8 @@ static int config_non_roce_gid_cache(struct ib_device *device,
goto err;
}
gid_attr.index = i;
+ tprops->subnet_prefix =
+ be64_to_cpu(gid_attr.gid.global.subnet_prefix);
add_modify_gid(table, &gid_attr);
}
err:
@@ -1484,7 +1486,7 @@ static int config_non_roce_gid_cache(struct ib_device *device,
if (!rdma_protocol_roce(device, port) && update_gids) {
ret = config_non_roce_gid_cache(device, port,
- tprops->gid_tbl_len);
+ tprops);
if (ret)
goto err;
}
--
1.8.3.1
^ permalink raw reply related [flat|nested] 3+ messages in thread
* [PATCH v7 for-next 2/2] IB/core: Read subnet_prefix in ib_query_port via cache.
2021-06-28 9:29 [PATCH v7 for-next 0/2] IB/core: Obtaining subnet_prefix from cache in Anand Khoje
2021-06-28 9:29 ` [PATCH v7 for-next 1/2] IB/core: Updating cache for subnet_prefix in config_non_roce_gid_cache() Anand Khoje
@ 2021-06-28 9:29 ` Anand Khoje
1 sibling, 0 replies; 3+ messages in thread
From: Anand Khoje @ 2021-06-28 9:29 UTC (permalink / raw)
To: linux-rdma, linux-kernel; +Cc: dledford, jgg, haakon.bugge, leon
ib_query_port() calls device->ops.query_port() to get the port
attributes. The method of querying is device driver specific.
The same function calls device->ops.query_gid() to get the GID and
extract the subnet_prefix (gid_prefix).
The GID and subnet_prefix are stored in a cache. But they do not get
read from the cache if the device is an Infiniband device. The
following change takes advantage of the cached subnet_prefix.
Testing with RDBMS has shown a significant improvement in performance
with this change.
The variable cache_is_initialized is added because ib_query_port()
gets called early in the stage when cache is not built while reading
port immutable properties.
In that case, the default GID still gets read from HCA for IB link-
layer devices.
Fixes: fad61ad ("IB/core: Add subnet prefix to port info")
Signed-off-by: Anand Khoje <anand.a.khoje@oracle.com>
Signed-off-by: Haakon Bugge <haakon.bugge@oracle.com>
---
drivers/infiniband/core/cache.c | 1 +
drivers/infiniband/core/device.c | 9 +++++++++
include/rdma/ib_verbs.h | 1 +
3 files changed, 11 insertions(+)
diff --git a/drivers/infiniband/core/cache.c b/drivers/infiniband/core/cache.c
index 929399e..4150043 100644
--- a/drivers/infiniband/core/cache.c
+++ b/drivers/infiniband/core/cache.c
@@ -1631,6 +1631,7 @@ int ib_cache_setup_one(struct ib_device *device)
err = ib_cache_update(device, p, true, true, true);
if (err)
return err;
+ device->port_data[p].cache_is_initialized = 1;
}
return 0;
diff --git a/drivers/infiniband/core/device.c b/drivers/infiniband/core/device.c
index fa20b18..e078a48 100644
--- a/drivers/infiniband/core/device.c
+++ b/drivers/infiniband/core/device.c
@@ -2063,6 +2063,15 @@ static int __ib_query_port(struct ib_device *device,
IB_LINK_LAYER_INFINIBAND)
return 0;
+ if (!device->port_data[port_num].cache_is_initialized)
+ goto query_gid_from_device;
+
+ ib_get_cached_subnet_prefix(device, port_num,
+ &port_attr->subnet_prefix);
+
+ return 0;
+
+query_gid_from_device:
err = device->ops.query_gid(device, port_num, 0, &gid);
if (err)
return err;
diff --git a/include/rdma/ib_verbs.h b/include/rdma/ib_verbs.h
index 371df1c..e692f9b 100644
--- a/include/rdma/ib_verbs.h
+++ b/include/rdma/ib_verbs.h
@@ -2179,6 +2179,7 @@ struct ib_port_data {
spinlock_t netdev_lock;
+ u8 cache_is_initialized:1;
struct list_head pkey_list;
struct ib_port_cache cache;
--
1.8.3.1
^ permalink raw reply related [flat|nested] 3+ messages in thread