All of lore.kernel.org
 help / color / mirror / Atom feed
* [PATCH v8 for-next 0/3] IB/core: Obtaining subnet_prefix from cache in
@ 2021-07-12 12:26 Anand Khoje
  2021-07-12 12:26 ` [PATCH v8 for-next 1/3] IB/core: Updating cache for subnet_prefix in config_non_roce_gid_cache() Anand Khoje
                   ` (3 more replies)
  0 siblings, 4 replies; 5+ messages in thread
From: Anand Khoje @ 2021-07-12 12:26 UTC (permalink / raw)
  To: linux-rdma, linux-kernel; +Cc: dledford, jgg, haakon.bugge, leon

This v8 of patch series is used to read the port_attribute subnet_prefix
from a valid cache entry instead of having to call
device->ops.query_gid() for Infiniband link-layer devices in
__ib_query_port().

In the event of a cache update, the value for subnet_prefix gets read
using device->ops.query_gid() in config_non_roce_gid_cache().

It also re-orders the initialization of lock cache_lock of struct ib_device
such that the lock is initialized before its first use in __ib_query_port()
during device initialization.

Anand Khoje (3):
  IB/core: Updating cache for subnet_prefix in
    config_non_roce_gid_cache()
  IB/core: Shifting initialization of device->cache_lock
  IB/core: Read subnet_prefix in ib_query_port via cache.

 drivers/infiniband/core/cache.c  | 10 +++++-----
 drivers/infiniband/core/device.c | 10 ++++------
 2 files changed, 9 insertions(+), 11 deletions(-)

---
v1 -> v2:
    -   Split the v1 patch in 3 patches as per Leon's suggestion.

v2 -> v3:
    -   Added changes as per Mark Zhang's suggestion of clearing
        flags in git_table_cleanup_one().
v3 -> v4:
    -   Removed the enum ib_port_data_flags and 8 byte flags from
        struct ib_port_data, and the set_bit()/clear_bit() API
        used to update this flag as that was not necessary.
        Done to keep the code simple.
    -   Added code to read subnet_prefix from updated GID cache in the
        event of cache update. Prior to this change, ib_cache_update
        was reading the value for subnet_prefix via ib_query_port(),
        due to this patch, we ended up reading a stale cached value of
        subnet_prefix.
v4 -> v5:
    -   Removed the code to reset cache_is_initialised bit from cleanup
        as per Leon's suggestion.
    -   Removed ib_cache_is_initialised() function.

v5 -> v6:
    -   Added changes as per Jason's suggestion of updating subnet_prefix
        in config_non_roce_gid_cache() and removing the flag
        cache_is_initialized in __ib_query_port().

v6 -> v7:
    -   Reordering the initialization of cache_lock, as the previous
        version caused an access to uninitialized cache_lock.

v7 -> v8:
    -   Resending the v7 of the patch-set after rebasing on the new rc1.
        There has been no change in the patches between v7 and v8 as v7
        patches applied cleanly after rebasing to 5.14-rc1.
---

-- 
1.8.3.1


^ permalink raw reply	[flat|nested] 5+ messages in thread

* [PATCH v8 for-next 1/3] IB/core: Updating cache for subnet_prefix in config_non_roce_gid_cache()
  2021-07-12 12:26 [PATCH v8 for-next 0/3] IB/core: Obtaining subnet_prefix from cache in Anand Khoje
@ 2021-07-12 12:26 ` Anand Khoje
  2021-07-12 12:26 ` [PATCH v8 for-next 2/3] IB/core: Shifting initialization of device->cache_lock Anand Khoje
                   ` (2 subsequent siblings)
  3 siblings, 0 replies; 5+ messages in thread
From: Anand Khoje @ 2021-07-12 12:26 UTC (permalink / raw)
  To: linux-rdma, linux-kernel; +Cc: dledford, jgg, haakon.bugge, leon

Currently, cache for subnet_prefix was getting updated by reading port
attributes via ib_query_port. ib_query_port() calls ops.query_gid()
to get subnet_prefix and returns it via port_attr.

In ib_cache_update(), config_non_roce_gid_cache() obtains GIDs
by calling ops.query_gid(). We utilize this to store subnet_prefix
in cache.

Suggested-by: Jason Gunthorpe <jgg@nvidia.com>
Suggested-by: Aru Kolappan <aru.kolappan@oracle.com>
Signed-off-by: Anand Khoje <anand.a.khoje@oracle.com>
Signed-off-by: Haakon Bugge <haakon.bugge@oracle.com>
---
 drivers/infiniband/core/cache.c | 8 +++++---
 1 file changed, 5 insertions(+), 3 deletions(-)

diff --git a/drivers/infiniband/core/cache.c b/drivers/infiniband/core/cache.c
index c9e9fc8..929399e 100644
--- a/drivers/infiniband/core/cache.c
+++ b/drivers/infiniband/core/cache.c
@@ -1429,7 +1429,7 @@ int rdma_read_gid_l2_fields(const struct ib_gid_attr *attr,
 EXPORT_SYMBOL(rdma_read_gid_l2_fields);
 
 static int config_non_roce_gid_cache(struct ib_device *device,
-				     u32 port, int gid_tbl_len)
+				     u32 port, struct ib_port_attr *tprops)
 {
 	struct ib_gid_attr gid_attr = {};
 	struct ib_gid_table *table;
@@ -1441,7 +1441,7 @@ static int config_non_roce_gid_cache(struct ib_device *device,
 	table = rdma_gid_table(device, port);
 
 	mutex_lock(&table->lock);
-	for (i = 0; i < gid_tbl_len; ++i) {
+	for (i = 0; i < tprops->gid_tbl_len; ++i) {
 		if (!device->ops.query_gid)
 			continue;
 		ret = device->ops.query_gid(device, port, i, &gid_attr.gid);
@@ -1452,6 +1452,8 @@ static int config_non_roce_gid_cache(struct ib_device *device,
 			goto err;
 		}
 		gid_attr.index = i;
+		tprops->subnet_prefix =
+			be64_to_cpu(gid_attr.gid.global.subnet_prefix);
 		add_modify_gid(table, &gid_attr);
 	}
 err:
@@ -1484,7 +1486,7 @@ static int config_non_roce_gid_cache(struct ib_device *device,
 
 	if (!rdma_protocol_roce(device, port) && update_gids) {
 		ret = config_non_roce_gid_cache(device, port,
-						tprops->gid_tbl_len);
+						tprops);
 		if (ret)
 			goto err;
 	}
-- 
1.8.3.1


^ permalink raw reply related	[flat|nested] 5+ messages in thread

* [PATCH v8 for-next 2/3] IB/core: Shifting initialization of device->cache_lock
  2021-07-12 12:26 [PATCH v8 for-next 0/3] IB/core: Obtaining subnet_prefix from cache in Anand Khoje
  2021-07-12 12:26 ` [PATCH v8 for-next 1/3] IB/core: Updating cache for subnet_prefix in config_non_roce_gid_cache() Anand Khoje
@ 2021-07-12 12:26 ` Anand Khoje
  2021-07-12 12:26 ` [PATCH v8 for-next 3/3] IB/core: Read subnet_prefix in ib_query_port via cache Anand Khoje
  2021-07-16 14:08 ` [PATCH v8 for-next 0/3] IB/core: Obtaining subnet_prefix from cache in Jason Gunthorpe
  3 siblings, 0 replies; 5+ messages in thread
From: Anand Khoje @ 2021-07-12 12:26 UTC (permalink / raw)
  To: linux-rdma, linux-kernel; +Cc: dledford, jgg, haakon.bugge, leon

The lock cache_lock of struct ib_device is initialized in function
ib_cache_setup_one(). This is much later than the device initialization
in _ib_alloc_device().

This change shifts initialization of cache_lock in _ib_alloc_device().

Suggested-by: Haakon Bugge <haakon.bugge@oracle.com>
Signed-off-by: Anand Khoje <anand.a.khoje@oracle.com>
---
 drivers/infiniband/core/cache.c  | 2 --
 drivers/infiniband/core/device.c | 2 ++
 2 files changed, 2 insertions(+), 2 deletions(-)

diff --git a/drivers/infiniband/core/cache.c b/drivers/infiniband/core/cache.c
index 929399e..0c98dd3 100644
--- a/drivers/infiniband/core/cache.c
+++ b/drivers/infiniband/core/cache.c
@@ -1621,8 +1621,6 @@ int ib_cache_setup_one(struct ib_device *device)
 	u32 p;
 	int err;
 
-	rwlock_init(&device->cache_lock);
-
 	err = gid_table_setup_one(device);
 	if (err)
 		return err;
diff --git a/drivers/infiniband/core/device.c b/drivers/infiniband/core/device.c
index fa20b18..ba0ad72 100644
--- a/drivers/infiniband/core/device.c
+++ b/drivers/infiniband/core/device.c
@@ -607,6 +607,8 @@ struct ib_device *_ib_alloc_device(size_t size)
 	for (i = 0; i < ARRAY_SIZE(device->cq_pools); i++)
 		INIT_LIST_HEAD(&device->cq_pools[i]);
 
+	rwlock_init(&device->cache_lock);
+
 	device->uverbs_cmd_mask =
 		BIT_ULL(IB_USER_VERBS_CMD_ALLOC_MW) |
 		BIT_ULL(IB_USER_VERBS_CMD_ALLOC_PD) |
-- 
1.8.3.1


^ permalink raw reply related	[flat|nested] 5+ messages in thread

* [PATCH v8 for-next 3/3] IB/core: Read subnet_prefix in ib_query_port via cache.
  2021-07-12 12:26 [PATCH v8 for-next 0/3] IB/core: Obtaining subnet_prefix from cache in Anand Khoje
  2021-07-12 12:26 ` [PATCH v8 for-next 1/3] IB/core: Updating cache for subnet_prefix in config_non_roce_gid_cache() Anand Khoje
  2021-07-12 12:26 ` [PATCH v8 for-next 2/3] IB/core: Shifting initialization of device->cache_lock Anand Khoje
@ 2021-07-12 12:26 ` Anand Khoje
  2021-07-16 14:08 ` [PATCH v8 for-next 0/3] IB/core: Obtaining subnet_prefix from cache in Jason Gunthorpe
  3 siblings, 0 replies; 5+ messages in thread
From: Anand Khoje @ 2021-07-12 12:26 UTC (permalink / raw)
  To: linux-rdma, linux-kernel; +Cc: dledford, jgg, haakon.bugge, leon

ib_query_port() calls device->ops.query_port() to get the port
attributes. The method of querying is device driver specific.
The same function calls device->ops.query_gid() to get the GID and
extract the subnet_prefix (gid_prefix).

The GID and subnet_prefix are stored in a cache. But they do not get
read from the cache if the device is an Infiniband device. The
following change takes advantage of the cached subnet_prefix.
Testing with RDBMS has shown a significant improvement in performance
with this change.

Fixes: fad61ad ("IB/core: Add subnet prefix to port info")
Signed-off-by: Anand Khoje <anand.a.khoje@oracle.com>
Signed-off-by: Haakon Bugge <haakon.bugge@oracle.com>
---
 drivers/infiniband/core/device.c | 8 ++------
 1 file changed, 2 insertions(+), 6 deletions(-)

diff --git a/drivers/infiniband/core/device.c b/drivers/infiniband/core/device.c
index ba0ad72..9056f48 100644
--- a/drivers/infiniband/core/device.c
+++ b/drivers/infiniband/core/device.c
@@ -2052,7 +2052,6 @@ static int __ib_query_port(struct ib_device *device,
 			   u32 port_num,
 			   struct ib_port_attr *port_attr)
 {
-	union ib_gid gid = {};
 	int err;
 
 	memset(port_attr, 0, sizeof(*port_attr));
@@ -2065,11 +2064,8 @@ static int __ib_query_port(struct ib_device *device,
 	    IB_LINK_LAYER_INFINIBAND)
 		return 0;
 
-	err = device->ops.query_gid(device, port_num, 0, &gid);
-	if (err)
-		return err;
-
-	port_attr->subnet_prefix = be64_to_cpu(gid.global.subnet_prefix);
+	ib_get_cached_subnet_prefix(device, port_num,
+				    &port_attr->subnet_prefix);
 	return 0;
 }
 
-- 
1.8.3.1


^ permalink raw reply related	[flat|nested] 5+ messages in thread

* Re: [PATCH v8 for-next 0/3] IB/core: Obtaining subnet_prefix from cache in
  2021-07-12 12:26 [PATCH v8 for-next 0/3] IB/core: Obtaining subnet_prefix from cache in Anand Khoje
                   ` (2 preceding siblings ...)
  2021-07-12 12:26 ` [PATCH v8 for-next 3/3] IB/core: Read subnet_prefix in ib_query_port via cache Anand Khoje
@ 2021-07-16 14:08 ` Jason Gunthorpe
  3 siblings, 0 replies; 5+ messages in thread
From: Jason Gunthorpe @ 2021-07-16 14:08 UTC (permalink / raw)
  To: Anand Khoje; +Cc: linux-rdma, linux-kernel, dledford, haakon.bugge, leon

On Mon, Jul 12, 2021 at 05:56:22PM +0530, Anand Khoje wrote:
> This v8 of patch series is used to read the port_attribute subnet_prefix
> from a valid cache entry instead of having to call
> device->ops.query_gid() for Infiniband link-layer devices in
> __ib_query_port().
> 
> In the event of a cache update, the value for subnet_prefix gets read
> using device->ops.query_gid() in config_non_roce_gid_cache().
> 
> It also re-orders the initialization of lock cache_lock of struct ib_device
> such that the lock is initialized before its first use in __ib_query_port()
> during device initialization.
> 
> Anand Khoje (3):
>   IB/core: Updating cache for subnet_prefix in
>     config_non_roce_gid_cache()
>   IB/core: Shifting initialization of device->cache_lock
>   IB/core: Read subnet_prefix in ib_query_port via cache.
> 
>  drivers/infiniband/core/cache.c  | 10 +++++-----
>  drivers/infiniband/core/device.c | 10 ++++------
>  2 files changed, 9 insertions(+), 11 deletions(-)

Applied to for-next, thanks

Jason

^ permalink raw reply	[flat|nested] 5+ messages in thread

end of thread, other threads:[~2021-07-16 14:08 UTC | newest]

Thread overview: 5+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-07-12 12:26 [PATCH v8 for-next 0/3] IB/core: Obtaining subnet_prefix from cache in Anand Khoje
2021-07-12 12:26 ` [PATCH v8 for-next 1/3] IB/core: Updating cache for subnet_prefix in config_non_roce_gid_cache() Anand Khoje
2021-07-12 12:26 ` [PATCH v8 for-next 2/3] IB/core: Shifting initialization of device->cache_lock Anand Khoje
2021-07-12 12:26 ` [PATCH v8 for-next 3/3] IB/core: Read subnet_prefix in ib_query_port via cache Anand Khoje
2021-07-16 14:08 ` [PATCH v8 for-next 0/3] IB/core: Obtaining subnet_prefix from cache in Jason Gunthorpe

This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.