* [PATCH v8 for-next 0/3] IB/core: Obtaining subnet_prefix from cache in
@ 2021-07-12 12:26 Anand Khoje
2021-07-12 12:26 ` [PATCH v8 for-next 1/3] IB/core: Updating cache for subnet_prefix in config_non_roce_gid_cache() Anand Khoje
` (3 more replies)
0 siblings, 4 replies; 5+ messages in thread
From: Anand Khoje @ 2021-07-12 12:26 UTC (permalink / raw)
To: linux-rdma, linux-kernel; +Cc: dledford, jgg, haakon.bugge, leon
This v8 of patch series is used to read the port_attribute subnet_prefix
from a valid cache entry instead of having to call
device->ops.query_gid() for Infiniband link-layer devices in
__ib_query_port().
In the event of a cache update, the value for subnet_prefix gets read
using device->ops.query_gid() in config_non_roce_gid_cache().
It also re-orders the initialization of lock cache_lock of struct ib_device
such that the lock is initialized before its first use in __ib_query_port()
during device initialization.
Anand Khoje (3):
IB/core: Updating cache for subnet_prefix in
config_non_roce_gid_cache()
IB/core: Shifting initialization of device->cache_lock
IB/core: Read subnet_prefix in ib_query_port via cache.
drivers/infiniband/core/cache.c | 10 +++++-----
drivers/infiniband/core/device.c | 10 ++++------
2 files changed, 9 insertions(+), 11 deletions(-)
---
v1 -> v2:
- Split the v1 patch in 3 patches as per Leon's suggestion.
v2 -> v3:
- Added changes as per Mark Zhang's suggestion of clearing
flags in git_table_cleanup_one().
v3 -> v4:
- Removed the enum ib_port_data_flags and 8 byte flags from
struct ib_port_data, and the set_bit()/clear_bit() API
used to update this flag as that was not necessary.
Done to keep the code simple.
- Added code to read subnet_prefix from updated GID cache in the
event of cache update. Prior to this change, ib_cache_update
was reading the value for subnet_prefix via ib_query_port(),
due to this patch, we ended up reading a stale cached value of
subnet_prefix.
v4 -> v5:
- Removed the code to reset cache_is_initialised bit from cleanup
as per Leon's suggestion.
- Removed ib_cache_is_initialised() function.
v5 -> v6:
- Added changes as per Jason's suggestion of updating subnet_prefix
in config_non_roce_gid_cache() and removing the flag
cache_is_initialized in __ib_query_port().
v6 -> v7:
- Reordering the initialization of cache_lock, as the previous
version caused an access to uninitialized cache_lock.
v7 -> v8:
- Resending the v7 of the patch-set after rebasing on the new rc1.
There has been no change in the patches between v7 and v8 as v7
patches applied cleanly after rebasing to 5.14-rc1.
---
--
1.8.3.1
^ permalink raw reply [flat|nested] 5+ messages in thread
* [PATCH v8 for-next 1/3] IB/core: Updating cache for subnet_prefix in config_non_roce_gid_cache()
2021-07-12 12:26 [PATCH v8 for-next 0/3] IB/core: Obtaining subnet_prefix from cache in Anand Khoje
@ 2021-07-12 12:26 ` Anand Khoje
2021-07-12 12:26 ` [PATCH v8 for-next 2/3] IB/core: Shifting initialization of device->cache_lock Anand Khoje
` (2 subsequent siblings)
3 siblings, 0 replies; 5+ messages in thread
From: Anand Khoje @ 2021-07-12 12:26 UTC (permalink / raw)
To: linux-rdma, linux-kernel; +Cc: dledford, jgg, haakon.bugge, leon
Currently, cache for subnet_prefix was getting updated by reading port
attributes via ib_query_port. ib_query_port() calls ops.query_gid()
to get subnet_prefix and returns it via port_attr.
In ib_cache_update(), config_non_roce_gid_cache() obtains GIDs
by calling ops.query_gid(). We utilize this to store subnet_prefix
in cache.
Suggested-by: Jason Gunthorpe <jgg@nvidia.com>
Suggested-by: Aru Kolappan <aru.kolappan@oracle.com>
Signed-off-by: Anand Khoje <anand.a.khoje@oracle.com>
Signed-off-by: Haakon Bugge <haakon.bugge@oracle.com>
---
drivers/infiniband/core/cache.c | 8 +++++---
1 file changed, 5 insertions(+), 3 deletions(-)
diff --git a/drivers/infiniband/core/cache.c b/drivers/infiniband/core/cache.c
index c9e9fc8..929399e 100644
--- a/drivers/infiniband/core/cache.c
+++ b/drivers/infiniband/core/cache.c
@@ -1429,7 +1429,7 @@ int rdma_read_gid_l2_fields(const struct ib_gid_attr *attr,
EXPORT_SYMBOL(rdma_read_gid_l2_fields);
static int config_non_roce_gid_cache(struct ib_device *device,
- u32 port, int gid_tbl_len)
+ u32 port, struct ib_port_attr *tprops)
{
struct ib_gid_attr gid_attr = {};
struct ib_gid_table *table;
@@ -1441,7 +1441,7 @@ static int config_non_roce_gid_cache(struct ib_device *device,
table = rdma_gid_table(device, port);
mutex_lock(&table->lock);
- for (i = 0; i < gid_tbl_len; ++i) {
+ for (i = 0; i < tprops->gid_tbl_len; ++i) {
if (!device->ops.query_gid)
continue;
ret = device->ops.query_gid(device, port, i, &gid_attr.gid);
@@ -1452,6 +1452,8 @@ static int config_non_roce_gid_cache(struct ib_device *device,
goto err;
}
gid_attr.index = i;
+ tprops->subnet_prefix =
+ be64_to_cpu(gid_attr.gid.global.subnet_prefix);
add_modify_gid(table, &gid_attr);
}
err:
@@ -1484,7 +1486,7 @@ static int config_non_roce_gid_cache(struct ib_device *device,
if (!rdma_protocol_roce(device, port) && update_gids) {
ret = config_non_roce_gid_cache(device, port,
- tprops->gid_tbl_len);
+ tprops);
if (ret)
goto err;
}
--
1.8.3.1
^ permalink raw reply related [flat|nested] 5+ messages in thread
* [PATCH v8 for-next 2/3] IB/core: Shifting initialization of device->cache_lock
2021-07-12 12:26 [PATCH v8 for-next 0/3] IB/core: Obtaining subnet_prefix from cache in Anand Khoje
2021-07-12 12:26 ` [PATCH v8 for-next 1/3] IB/core: Updating cache for subnet_prefix in config_non_roce_gid_cache() Anand Khoje
@ 2021-07-12 12:26 ` Anand Khoje
2021-07-12 12:26 ` [PATCH v8 for-next 3/3] IB/core: Read subnet_prefix in ib_query_port via cache Anand Khoje
2021-07-16 14:08 ` [PATCH v8 for-next 0/3] IB/core: Obtaining subnet_prefix from cache in Jason Gunthorpe
3 siblings, 0 replies; 5+ messages in thread
From: Anand Khoje @ 2021-07-12 12:26 UTC (permalink / raw)
To: linux-rdma, linux-kernel; +Cc: dledford, jgg, haakon.bugge, leon
The lock cache_lock of struct ib_device is initialized in function
ib_cache_setup_one(). This is much later than the device initialization
in _ib_alloc_device().
This change shifts initialization of cache_lock in _ib_alloc_device().
Suggested-by: Haakon Bugge <haakon.bugge@oracle.com>
Signed-off-by: Anand Khoje <anand.a.khoje@oracle.com>
---
drivers/infiniband/core/cache.c | 2 --
drivers/infiniband/core/device.c | 2 ++
2 files changed, 2 insertions(+), 2 deletions(-)
diff --git a/drivers/infiniband/core/cache.c b/drivers/infiniband/core/cache.c
index 929399e..0c98dd3 100644
--- a/drivers/infiniband/core/cache.c
+++ b/drivers/infiniband/core/cache.c
@@ -1621,8 +1621,6 @@ int ib_cache_setup_one(struct ib_device *device)
u32 p;
int err;
- rwlock_init(&device->cache_lock);
-
err = gid_table_setup_one(device);
if (err)
return err;
diff --git a/drivers/infiniband/core/device.c b/drivers/infiniband/core/device.c
index fa20b18..ba0ad72 100644
--- a/drivers/infiniband/core/device.c
+++ b/drivers/infiniband/core/device.c
@@ -607,6 +607,8 @@ struct ib_device *_ib_alloc_device(size_t size)
for (i = 0; i < ARRAY_SIZE(device->cq_pools); i++)
INIT_LIST_HEAD(&device->cq_pools[i]);
+ rwlock_init(&device->cache_lock);
+
device->uverbs_cmd_mask =
BIT_ULL(IB_USER_VERBS_CMD_ALLOC_MW) |
BIT_ULL(IB_USER_VERBS_CMD_ALLOC_PD) |
--
1.8.3.1
^ permalink raw reply related [flat|nested] 5+ messages in thread
* [PATCH v8 for-next 3/3] IB/core: Read subnet_prefix in ib_query_port via cache.
2021-07-12 12:26 [PATCH v8 for-next 0/3] IB/core: Obtaining subnet_prefix from cache in Anand Khoje
2021-07-12 12:26 ` [PATCH v8 for-next 1/3] IB/core: Updating cache for subnet_prefix in config_non_roce_gid_cache() Anand Khoje
2021-07-12 12:26 ` [PATCH v8 for-next 2/3] IB/core: Shifting initialization of device->cache_lock Anand Khoje
@ 2021-07-12 12:26 ` Anand Khoje
2021-07-16 14:08 ` [PATCH v8 for-next 0/3] IB/core: Obtaining subnet_prefix from cache in Jason Gunthorpe
3 siblings, 0 replies; 5+ messages in thread
From: Anand Khoje @ 2021-07-12 12:26 UTC (permalink / raw)
To: linux-rdma, linux-kernel; +Cc: dledford, jgg, haakon.bugge, leon
ib_query_port() calls device->ops.query_port() to get the port
attributes. The method of querying is device driver specific.
The same function calls device->ops.query_gid() to get the GID and
extract the subnet_prefix (gid_prefix).
The GID and subnet_prefix are stored in a cache. But they do not get
read from the cache if the device is an Infiniband device. The
following change takes advantage of the cached subnet_prefix.
Testing with RDBMS has shown a significant improvement in performance
with this change.
Fixes: fad61ad ("IB/core: Add subnet prefix to port info")
Signed-off-by: Anand Khoje <anand.a.khoje@oracle.com>
Signed-off-by: Haakon Bugge <haakon.bugge@oracle.com>
---
drivers/infiniband/core/device.c | 8 ++------
1 file changed, 2 insertions(+), 6 deletions(-)
diff --git a/drivers/infiniband/core/device.c b/drivers/infiniband/core/device.c
index ba0ad72..9056f48 100644
--- a/drivers/infiniband/core/device.c
+++ b/drivers/infiniband/core/device.c
@@ -2052,7 +2052,6 @@ static int __ib_query_port(struct ib_device *device,
u32 port_num,
struct ib_port_attr *port_attr)
{
- union ib_gid gid = {};
int err;
memset(port_attr, 0, sizeof(*port_attr));
@@ -2065,11 +2064,8 @@ static int __ib_query_port(struct ib_device *device,
IB_LINK_LAYER_INFINIBAND)
return 0;
- err = device->ops.query_gid(device, port_num, 0, &gid);
- if (err)
- return err;
-
- port_attr->subnet_prefix = be64_to_cpu(gid.global.subnet_prefix);
+ ib_get_cached_subnet_prefix(device, port_num,
+ &port_attr->subnet_prefix);
return 0;
}
--
1.8.3.1
^ permalink raw reply related [flat|nested] 5+ messages in thread
* Re: [PATCH v8 for-next 0/3] IB/core: Obtaining subnet_prefix from cache in
2021-07-12 12:26 [PATCH v8 for-next 0/3] IB/core: Obtaining subnet_prefix from cache in Anand Khoje
` (2 preceding siblings ...)
2021-07-12 12:26 ` [PATCH v8 for-next 3/3] IB/core: Read subnet_prefix in ib_query_port via cache Anand Khoje
@ 2021-07-16 14:08 ` Jason Gunthorpe
3 siblings, 0 replies; 5+ messages in thread
From: Jason Gunthorpe @ 2021-07-16 14:08 UTC (permalink / raw)
To: Anand Khoje; +Cc: linux-rdma, linux-kernel, dledford, haakon.bugge, leon
On Mon, Jul 12, 2021 at 05:56:22PM +0530, Anand Khoje wrote:
> This v8 of patch series is used to read the port_attribute subnet_prefix
> from a valid cache entry instead of having to call
> device->ops.query_gid() for Infiniband link-layer devices in
> __ib_query_port().
>
> In the event of a cache update, the value for subnet_prefix gets read
> using device->ops.query_gid() in config_non_roce_gid_cache().
>
> It also re-orders the initialization of lock cache_lock of struct ib_device
> such that the lock is initialized before its first use in __ib_query_port()
> during device initialization.
>
> Anand Khoje (3):
> IB/core: Updating cache for subnet_prefix in
> config_non_roce_gid_cache()
> IB/core: Shifting initialization of device->cache_lock
> IB/core: Read subnet_prefix in ib_query_port via cache.
>
> drivers/infiniband/core/cache.c | 10 +++++-----
> drivers/infiniband/core/device.c | 10 ++++------
> 2 files changed, 9 insertions(+), 11 deletions(-)
Applied to for-next, thanks
Jason
^ permalink raw reply [flat|nested] 5+ messages in thread
end of thread, other threads:[~2021-07-16 14:08 UTC | newest]
Thread overview: 5+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-07-12 12:26 [PATCH v8 for-next 0/3] IB/core: Obtaining subnet_prefix from cache in Anand Khoje
2021-07-12 12:26 ` [PATCH v8 for-next 1/3] IB/core: Updating cache for subnet_prefix in config_non_roce_gid_cache() Anand Khoje
2021-07-12 12:26 ` [PATCH v8 for-next 2/3] IB/core: Shifting initialization of device->cache_lock Anand Khoje
2021-07-12 12:26 ` [PATCH v8 for-next 3/3] IB/core: Read subnet_prefix in ib_query_port via cache Anand Khoje
2021-07-16 14:08 ` [PATCH v8 for-next 0/3] IB/core: Obtaining subnet_prefix from cache in Jason Gunthorpe
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).