[PATCH v7 for-next 0/3] IB/core: Obtaining subnet_prefix from cache in

[PATCH v7 for-next 0/3] IB/core: Obtaining subnet_prefix from cache in

[Anand Khoje]

This v7 of patch series is used to read the port_attribute subnet_prefix
from a valid cache entry instead of having to call
device->ops.query_gid() for Infiniband link-layer devices in
__ib_query_port().

In the event of a cache update, the value for subnet_prefix gets read
using device->ops.query_gid() in config_non_roce_gid_cache().

Anand Khoje (3):
  IB/core: Updating cache for subnet_prefix in
    config_non_roce_gid_cache()
  IB/core: Shifting initialization of device->cache_lock.
  IB/core: Read subnet_prefix in ib_query_port via cache.
---
v1 -> v2:
    -   Split the v1 patch in 3 patches as per Leon's suggestion.

v2 -> v3:
    -   Added changes as per Mark Zhang's suggestion of clearing
        flags in git_table_cleanup_one().
v3 -> v4:
    -   Removed the enum ib_port_data_flags and 8 byte flags from
        struct ib_port_data, and the set_bit()/clear_bit() API
        used to update this flag as that was not necessary.
        Done to keep the code simple.
    -   Added code to read subnet_prefix from updated GID cache in the
        event of cache update. Prior to this change, ib_cache_update
        was reading the value for subnet_prefix via ib_query_port(),
        due to this patch, we ended up reading a stale cached value of
        subnet_prefix.
v4 -> v5:
    -   Removed the code to reset cache_is_initialised bit from cleanup
        as per Leon's suggestion.
    -   Removed ib_cache_is_initialised() function.

v5 -> v6:
    -   Added changes as per Jason's suggestion of updating subnet_prefix
        in config_non_roce_gid_cache() and removing the flag
        cache_is_initialized in __ib_query_port().

v6 -> v7:
    -	Reordering the initialization of cache_lock, as the previous
	version caused an access to uninitialized cache_lock.
---

 drivers/infiniband/core/cache.c  | 10 +++++-----
 drivers/infiniband/core/device.c | 10 ++++------
 2 files changed, 9 insertions(+), 11 deletions(-)

-- 
1.8.3.1

[PATCH v7 for-next 1/3] IB/core: Updating cache for subnet_prefix in config_non_roce_gid_cache()

[Anand Khoje]

Currently cache for subnet_prefix was getting updated by reading port
attributes via ib_query_port. ib_query_port() calls ops.query_gid()
to get subnet_prefix and returns it via port_attr.

In ib_cache_update(), config_non_roce_gid_cache() obtains GIDs
by calling ops.query_gid(). We utilize this to store subnet_prefix
in cache.

Suggested-by: Jason Gunthorpe <jgg@nvidia.com>
Suggested-by: Aru Kolappan <aru.kolappan@oracle.com>
Signed-off-by: Anand Khoje <anand.a.khoje@oracle.com>
Signed-off-by: Haakon Bugge <haakon.bugge@oracle.com>
---
 drivers/infiniband/core/cache.c | 8 +++++---
 1 file changed, 5 insertions(+), 3 deletions(-)

diff --git a/drivers/infiniband/core/cache.c b/drivers/infiniband/core/cache.c
index c9e9fc8..929399e 100644
--- a/drivers/infiniband/core/cache.c
+++ b/drivers/infiniband/core/cache.c
@@ -1429,7 +1429,7 @@ int rdma_read_gid_l2_fields(const struct ib_gid_attr *attr,
 EXPORT_SYMBOL(rdma_read_gid_l2_fields);
 
 static int config_non_roce_gid_cache(struct ib_device *device,
-				     u32 port, int gid_tbl_len)
+				     u32 port, struct ib_port_attr *tprops)
 {
 	struct ib_gid_attr gid_attr = {};
 	struct ib_gid_table *table;
@@ -1441,7 +1441,7 @@ static int config_non_roce_gid_cache(struct ib_device *device,
 	table = rdma_gid_table(device, port);
 
 	mutex_lock(&table->lock);
-	for (i = 0; i < gid_tbl_len; ++i) {
+	for (i = 0; i < tprops->gid_tbl_len; ++i) {
 		if (!device->ops.query_gid)
 			continue;
 		ret = device->ops.query_gid(device, port, i, &gid_attr.gid);
@@ -1452,6 +1452,8 @@ static int config_non_roce_gid_cache(struct ib_device *device,
 			goto err;
 		}
 		gid_attr.index = i;
+		tprops->subnet_prefix =
+			be64_to_cpu(gid_attr.gid.global.subnet_prefix);
 		add_modify_gid(table, &gid_attr);
 	}
 err:
@@ -1484,7 +1486,7 @@ static int config_non_roce_gid_cache(struct ib_device *device,
 
 	if (!rdma_protocol_roce(device, port) && update_gids) {
 		ret = config_non_roce_gid_cache(device, port,
-						tprops->gid_tbl_len);
+						tprops);
 		if (ret)
 			goto err;
 	}
-- 
1.8.3.1

[PATCH v7 for-next 2/3] IB/core: Shifting initialization of device->cache_lock.

[Anand Khoje]

The lock cache_lock of struct ib_device is initialized in function
ib_cache_setup_one(). This is much later than the device initialization
in _ib_alloc_device().

This change shifts initialization of cache_lock in _ib_alloc_device().

Suggested-by: Haakon Bugge <haakon.bugge@oracle.com>
Signed-off-by: Anand Khoje <anand.a.khoje@oracle.com>
---
 drivers/infiniband/core/cache.c  | 2 --
 drivers/infiniband/core/device.c | 2 ++
 2 files changed, 2 insertions(+), 2 deletions(-)

diff --git a/drivers/infiniband/core/cache.c b/drivers/infiniband/core/cache.c
index 929399e..0c98dd3 100644
--- a/drivers/infiniband/core/cache.c
+++ b/drivers/infiniband/core/cache.c
@@ -1621,8 +1621,6 @@ int ib_cache_setup_one(struct ib_device *device)
 	u32 p;
 	int err;
 
-	rwlock_init(&device->cache_lock);
-
 	err = gid_table_setup_one(device);
 	if (err)
 		return err;
diff --git a/drivers/infiniband/core/device.c b/drivers/infiniband/core/device.c
index fa20b18..ba0ad72 100644
--- a/drivers/infiniband/core/device.c
+++ b/drivers/infiniband/core/device.c
@@ -607,6 +607,8 @@ struct ib_device *_ib_alloc_device(size_t size)
 	for (i = 0; i < ARRAY_SIZE(device->cq_pools); i++)
 		INIT_LIST_HEAD(&device->cq_pools[i]);
 
+	rwlock_init(&device->cache_lock);
+
 	device->uverbs_cmd_mask =
 		BIT_ULL(IB_USER_VERBS_CMD_ALLOC_MW) |
 		BIT_ULL(IB_USER_VERBS_CMD_ALLOC_PD) |
-- 
1.8.3.1

[PATCH v7 for-next 3/3] IB/core: Read subnet_prefix in ib_query_port via cache.

[Anand Khoje]

ib_query_port() calls device->ops.query_port() to get the port
attributes. The method of querying is device driver specific.
The same function calls device->ops.query_gid() to get the GID and
extract the subnet_prefix (gid_prefix).

The GID and subnet_prefix are stored in a cache. But they do not get
read from the cache if the device is an Infiniband device. The
following change takes advantage of the cached subnet_prefix.
Testing with RDBMS has shown a significant improvement in performance
with this change.

Fixes: fad61ad ("IB/core: Add subnet prefix to port info")
Signed-off-by: Anand Khoje <anand.a.khoje@oracle.com>
Signed-off-by: Haakon Bugge <haakon.bugge@oracle.com>
---
 drivers/infiniband/core/device.c | 8 ++------
 1 file changed, 2 insertions(+), 6 deletions(-)

diff --git a/drivers/infiniband/core/device.c b/drivers/infiniband/core/device.c
index ba0ad72..9056f48 100644
--- a/drivers/infiniband/core/device.c
+++ b/drivers/infiniband/core/device.c
@@ -2052,7 +2052,6 @@ static int __ib_query_port(struct ib_device *device,
 			   u32 port_num,
 			   struct ib_port_attr *port_attr)
 {
-	union ib_gid gid = {};
 	int err;
 
 	memset(port_attr, 0, sizeof(*port_attr));
@@ -2065,11 +2064,8 @@ static int __ib_query_port(struct ib_device *device,
 	    IB_LINK_LAYER_INFINIBAND)
 		return 0;
 
-	err = device->ops.query_gid(device, port_num, 0, &gid);
-	if (err)
-		return err;
-
-	port_attr->subnet_prefix = be64_to_cpu(gid.global.subnet_prefix);
+	ib_get_cached_subnet_prefix(device, port_num,
+				    &port_attr->subnet_prefix);
 	return 0;
 }
 
-- 
1.8.3.1

Re: [PATCH v7 for-next 0/3] IB/core: Obtaining subnet_prefix from cache in

[Anand Khoje]

On 6/30/2021 3:16 PM, Anand Khoje wrote:
> This v7 of patch series is used to read the port_attribute subnet_prefix
> from a valid cache entry instead of having to call
> device->ops.query_gid() for Infiniband link-layer devices in
> __ib_query_port().
> 
> In the event of a cache update, the value for subnet_prefix gets read
> using device->ops.query_gid() in config_non_roce_gid_cache().
> 
> Anand Khoje (3):
>    IB/core: Updating cache for subnet_prefix in
>      config_non_roce_gid_cache()
>    IB/core: Shifting initialization of device->cache_lock.
>    IB/core: Read subnet_prefix in ib_query_port via cache.
> ---
> v1 -> v2:
>      -   Split the v1 patch in 3 patches as per Leon's suggestion.
> 
> v2 -> v3:
>      -   Added changes as per Mark Zhang's suggestion of clearing
>          flags in git_table_cleanup_one().
> v3 -> v4:
>      -   Removed the enum ib_port_data_flags and 8 byte flags from
>          struct ib_port_data, and the set_bit()/clear_bit() API
>          used to update this flag as that was not necessary.
>          Done to keep the code simple.
>      -   Added code to read subnet_prefix from updated GID cache in the
>          event of cache update. Prior to this change, ib_cache_update
>          was reading the value for subnet_prefix via ib_query_port(),
>          due to this patch, we ended up reading a stale cached value of
>          subnet_prefix.
> v4 -> v5:
>      -   Removed the code to reset cache_is_initialised bit from cleanup
>          as per Leon's suggestion.
>      -   Removed ib_cache_is_initialised() function.
> 
> v5 -> v6:
>      -   Added changes as per Jason's suggestion of updating subnet_prefix
>          in config_non_roce_gid_cache() and removing the flag
>          cache_is_initialized in __ib_query_port().
> 
> v6 -> v7:
>      -	Reordering the initialization of cache_lock, as the previous
> 	version caused an access to uninitialized cache_lock.
> ---
> 
>   drivers/infiniband/core/cache.c  | 10 +++++-----
>   drivers/infiniband/core/device.c | 10 ++++------
>   2 files changed, 9 insertions(+), 11 deletions(-)
> 

Hi,

This is just a reminder note requesting review for this patch-set.

Thanks,
Anand

Re: [PATCH v7 for-next 0/3] IB/core: Obtaining subnet_prefix from cache in

[Jason Gunthorpe]

On Tue, Jul 06, 2021 at 12:55:48PM +0530, Anand Khoje wrote:
> On 6/30/2021 3:16 PM, Anand Khoje wrote:
> > This v7 of patch series is used to read the port_attribute subnet_prefix
> > from a valid cache entry instead of having to call
> > device->ops.query_gid() for Infiniband link-layer devices in
> > __ib_query_port().
> > 
> > In the event of a cache update, the value for subnet_prefix gets read
> > using device->ops.query_gid() in config_non_roce_gid_cache().
> > 
> > Anand Khoje (3):
> >    IB/core: Updating cache for subnet_prefix in
> >      config_non_roce_gid_cache()
> >    IB/core: Shifting initialization of device->cache_lock.
> >    IB/core: Read subnet_prefix in ib_query_port via cache.
> > v1 -> v2:
> >      -   Split the v1 patch in 3 patches as per Leon's suggestion.
> > 
> > v2 -> v3:
> >      -   Added changes as per Mark Zhang's suggestion of clearing
> >          flags in git_table_cleanup_one().
> > v3 -> v4:
> >      -   Removed the enum ib_port_data_flags and 8 byte flags from
> >          struct ib_port_data, and the set_bit()/clear_bit() API
> >          used to update this flag as that was not necessary.
> >          Done to keep the code simple.
> >      -   Added code to read subnet_prefix from updated GID cache in the
> >          event of cache update. Prior to this change, ib_cache_update
> >          was reading the value for subnet_prefix via ib_query_port(),
> >          due to this patch, we ended up reading a stale cached value of
> >          subnet_prefix.
> > v4 -> v5:
> >      -   Removed the code to reset cache_is_initialised bit from cleanup
> >          as per Leon's suggestion.
> >      -   Removed ib_cache_is_initialised() function.
> > 
> > v5 -> v6:
> >      -   Added changes as per Jason's suggestion of updating subnet_prefix
> >          in config_non_roce_gid_cache() and removing the flag
> >          cache_is_initialized in __ib_query_port().
> > 
> > v6 -> v7:
> >      -	Reordering the initialization of cache_lock, as the previous
> > 	version caused an access to uninitialized cache_lock.
> > 
> >   drivers/infiniband/core/cache.c  | 10 +++++-----
> >   drivers/infiniband/core/device.c | 10 ++++------
> >   2 files changed, 9 insertions(+), 11 deletions(-)
> > 
> 
> Hi,
> 
> This is just a reminder note requesting review for this patch-set.

You'll probably have to resend it after the merge window closed on
Monday, rebased on the new rc1

Jason

Re: [PATCH v7 for-next 0/3] IB/core: Obtaining subnet_prefix from cache in

[Anand Khoje]

On 7/9/2021 6:51 PM, Jason Gunthorpe wrote:
> On Tue, Jul 06, 2021 at 12:55:48PM +0530, Anand Khoje wrote:
>> On 6/30/2021 3:16 PM, Anand Khoje wrote:
>>> This v7 of patch series is used to read the port_attribute subnet_prefix
>>> from a valid cache entry instead of having to call
>>> device->ops.query_gid() for Infiniband link-layer devices in
>>> __ib_query_port().
>>>
>>> In the event of a cache update, the value for subnet_prefix gets read
>>> using device->ops.query_gid() in config_non_roce_gid_cache().
>>>
>>> Anand Khoje (3):
>>>     IB/core: Updating cache for subnet_prefix in
>>>       config_non_roce_gid_cache()
>>>     IB/core: Shifting initialization of device->cache_lock.
>>>     IB/core: Read subnet_prefix in ib_query_port via cache.
>>> v1 -> v2:
>>>       -   Split the v1 patch in 3 patches as per Leon's suggestion.
>>>
>>> v2 -> v3:
>>>       -   Added changes as per Mark Zhang's suggestion of clearing
>>>           flags in git_table_cleanup_one().
>>> v3 -> v4:
>>>       -   Removed the enum ib_port_data_flags and 8 byte flags from
>>>           struct ib_port_data, and the set_bit()/clear_bit() API
>>>           used to update this flag as that was not necessary.
>>>           Done to keep the code simple.
>>>       -   Added code to read subnet_prefix from updated GID cache in the
>>>           event of cache update. Prior to this change, ib_cache_update
>>>           was reading the value for subnet_prefix via ib_query_port(),
>>>           due to this patch, we ended up reading a stale cached value of
>>>           subnet_prefix.
>>> v4 -> v5:
>>>       -   Removed the code to reset cache_is_initialised bit from cleanup
>>>           as per Leon's suggestion.
>>>       -   Removed ib_cache_is_initialised() function.
>>>
>>> v5 -> v6:
>>>       -   Added changes as per Jason's suggestion of updating subnet_prefix
>>>           in config_non_roce_gid_cache() and removing the flag
>>>           cache_is_initialized in __ib_query_port().
>>>
>>> v6 -> v7:
>>>       -	Reordering the initialization of cache_lock, as the previous
>>> 	version caused an access to uninitialized cache_lock.
>>>
>>>    drivers/infiniband/core/cache.c  | 10 +++++-----
>>>    drivers/infiniband/core/device.c | 10 ++++------
>>>    2 files changed, 9 insertions(+), 11 deletions(-)
>>>
>>
>> Hi,
>>
>> This is just a reminder note requesting review for this patch-set.
> 
> You'll probably have to resend it after the merge window closed on
> Monday, rebased on the new rc1
> 
> Jason
> 
Hi Jason,

Thanks for the response. Ok, I will rebase and resend this patchset.

Thx, Anand