[PATCH 1/1] net: rds: add per rds connection cache statistics

[PATCH 1/1] net: rds: add per rds connection cache statistics

[Zhu Yanjun]

The variable cache_allocs is to indicate how many frags (KiB) are in one
rds connection frag cache.
The command "rds-info -Iv" will output the rds connection cache
statistics as below:
"
RDS IB Connections:
      LocalAddr RemoteAddr Tos SL  LocalDev            RemoteDev
      1.1.1.14 1.1.1.14   58 255  fe80::2:c903:a:7a31 fe80::2:c903:a:7a31
      send_wr=256, recv_wr=1024, send_sge=8, rdma_mr_max=4096,
      rdma_mr_size=257, cache_allocs=12
"
This means that there are about 12KiB frag in this rds connection frag
 cache.

Tested-by: RDS CI <rdsci_oslo@no.oracle.com>
Signed-off-by: Zhu Yanjun <yanjun.zhu@oracle.com>
---
 include/uapi/linux/rds.h | 2 ++
 net/rds/ib.c             | 2 ++
 2 files changed, 4 insertions(+)

diff --git a/include/uapi/linux/rds.h b/include/uapi/linux/rds.h
index 5d0f76c..fd6b5f6 100644
--- a/include/uapi/linux/rds.h
+++ b/include/uapi/linux/rds.h
@@ -250,6 +250,7 @@ struct rds_info_rdma_connection {
 	__u32		rdma_mr_max;
 	__u32		rdma_mr_size;
 	__u8		tos;
+	__u32		cache_allocs;
 };
 
 struct rds6_info_rdma_connection {
@@ -264,6 +265,7 @@ struct rds6_info_rdma_connection {
 	__u32		rdma_mr_max;
 	__u32		rdma_mr_size;
 	__u8		tos;
+	__u32		cache_allocs;
 };
 
 /* RDS message Receive Path Latency points */
diff --git a/net/rds/ib.c b/net/rds/ib.c
index 2da9b75..f9baf2d 100644
--- a/net/rds/ib.c
+++ b/net/rds/ib.c
@@ -318,6 +318,7 @@ static int rds_ib_conn_info_visitor(struct rds_connection *conn,
 		iinfo->max_recv_wr = ic->i_recv_ring.w_nr;
 		iinfo->max_send_sge = rds_ibdev->max_sge;
 		rds_ib_get_mr_info(rds_ibdev, iinfo);
+		iinfo->cache_allocs = atomic_read(&ic->i_cache_allocs);
 	}
 	return 1;
 }
@@ -351,6 +352,7 @@ static int rds6_ib_conn_info_visitor(struct rds_connection *conn,
 		iinfo6->max_recv_wr = ic->i_recv_ring.w_nr;
 		iinfo6->max_send_sge = rds_ibdev->max_sge;
 		rds6_ib_get_mr_info(rds_ibdev, iinfo6);
+		iinfo6->cache_allocs = atomic_read(&ic->i_cache_allocs);
 	}
 	return 1;
 }
-- 
2.7.4

Re: [PATCH 1/1] net: rds: add per rds connection cache statistics

[santosh.shilimkar]

On 6/1/19 12:54 AM, Zhu Yanjun wrote:
> The variable cache_allocs is to indicate how many frags (KiB) are in one
> rds connection frag cache.
> The command "rds-info -Iv" will output the rds connection cache
> statistics as below:
> "
> RDS IB Connections:
>        LocalAddr RemoteAddr Tos SL  LocalDev            RemoteDev
>        1.1.1.14 1.1.1.14   58 255  fe80::2:c903:a:7a31 fe80::2:c903:a:7a31
>        send_wr=256, recv_wr=1024, send_sge=8, rdma_mr_max=4096,
>        rdma_mr_size=257, cache_allocs=12
> "
> This means that there are about 12KiB frag in this rds connection frag
>   cache.
> 
> Tested-by: RDS CI <rdsci_oslo@no.oracle.com>
Please add some valid email id or drop above. Its expected
that with SOB, patches are tested before testing.

> Signed-off-by: Zhu Yanjun <yanjun.zhu@oracle.com>
> ---
>   include/uapi/linux/rds.h | 2 ++
>   net/rds/ib.c             | 2 ++
>   2 files changed, 4 insertions(+)
> 
> diff --git a/include/uapi/linux/rds.h b/include/uapi/linux/rds.h
> index 5d0f76c..fd6b5f6 100644
> --- a/include/uapi/linux/rds.h
> +++ b/include/uapi/linux/rds.h
> @@ -250,6 +250,7 @@ struct rds_info_rdma_connection {
>   	__u32		rdma_mr_max;
>   	__u32		rdma_mr_size;
>   	__u8		tos;
> +	__u32		cache_allocs;
Some of this header file changes, how is taking care of backward
compatibility with tooling ? This was one of the reason, the
all the fields are not updated.

Regards,
Santosh

Re: [PATCH 1/1] net: rds: add per rds connection cache statistics

[Yanjun Zhu]

On 2019/6/3 11:03, santosh.shilimkar@oracle.com wrote:
> On 6/1/19 12:54 AM, Zhu Yanjun wrote:
>> The variable cache_allocs is to indicate how many frags (KiB) are in one
>> rds connection frag cache.
>> The command "rds-info -Iv" will output the rds connection cache
>> statistics as below:
>> "
>> RDS IB Connections:
>>        LocalAddr RemoteAddr Tos SL  LocalDev RemoteDev
>>        1.1.1.14 1.1.1.14   58 255  fe80::2:c903:a:7a31 
>> fe80::2:c903:a:7a31
>>        send_wr=256, recv_wr=1024, send_sge=8, rdma_mr_max=4096,
>>        rdma_mr_size=257, cache_allocs=12
>> "
>> This means that there are about 12KiB frag in this rds connection frag
>>   cache.
>>
>> Tested-by: RDS CI <rdsci_oslo@no.oracle.com>
> Please add some valid email id or drop above. Its expected
> that with SOB, patches are tested before testing.

Thanks for review.

OK. I will remove this in V2.

>
>> Signed-off-by: Zhu Yanjun <yanjun.zhu@oracle.com>
>> ---
>>   include/uapi/linux/rds.h | 2 ++
>>   net/rds/ib.c             | 2 ++
>>   2 files changed, 4 insertions(+)
>>
>> diff --git a/include/uapi/linux/rds.h b/include/uapi/linux/rds.h
>> index 5d0f76c..fd6b5f6 100644
>> --- a/include/uapi/linux/rds.h
>> +++ b/include/uapi/linux/rds.h
>> @@ -250,6 +250,7 @@ struct rds_info_rdma_connection {
>>       __u32        rdma_mr_max;
>>       __u32        rdma_mr_size;
>>       __u8        tos;
>> +    __u32        cache_allocs;
> Some of this header file changes, how is taking care of backward
> compatibility with tooling ? 

Just now I made tests with rds-tools.

In this commit

"

commit 6c03b61e9097098d35b4c2be16d0f0f9f8357112
Author: Santosh Shilimkar <santosh.shilimkar@oracle.com>
Date:   Wed Mar 9 04:30:48 2016 -0800

     rds-tools: sync up sources with 2.0.7-1.16
"

cache_allocs is added into rds-tools. The diff is as below.

"

@@ -176,6 +191,9 @@ struct rds_info_rdma_connection {
         uint32_t        max_send_sge;
         uint32_t        rdma_mr_max;
         uint32_t        rdma_mr_size;
+       uint8_t         tos;
+       uint8_t         sl;
+       uint32_t        cache_allocs;
  };
"
Then this cache_allocs does not exist in rds-tools 2.0.6 and rds-tools 
2.0.5.

I made tests with 2.0.5 and 2.0.6

"

rds-info -V
rds-info: Invalid option '-V'
rds-info version 2.0.5

[root@ca-dev14 rds-tools]# rds-info -Iv

RDS IB Connections:
       LocalAddr      RemoteAddr LocalDev                        RemoteDev
        1.1.1.14        1.1.1.14 fe80::2:c903:a:7a31              
fe80::2:c903:a:7a31  send_wr=256, recv_wr=1024, send_sge=8, 
rdma_mr_max=4096, rdma_mr_size=257
"

"

[root@ca-dev14 rds-tools]# rds-info -V
rds-info: Invalid option '-V'
rds-info version 2.0.6

[root@ca-dev14 rds-tools]# rds-info -Iv

RDS IB Connections:
       LocalAddr      RemoteAddr LocalDev                        RemoteDev
        1.1.1.14        1.1.1.14 fe80::2:c903:a:7a31              
fe80::2:c903:a:7a31  send_wr=256, recv_wr=1024, send_sge=8, 
rdma_mr_max=4096, rdma_mr_size=257
"

 From output of rds-tools 2.0.5 and 2.0.6, cache_allocs does not appear 
since cache_allocs does not exist in struct rds_info_rdma_connection.

But in rds-tools 2.0.7, cache_allocs exists in struct 
rds_info_rdma_connection.

"

[root@ca-dev14 rds-tools]# rds-info -V
rds-info: invalid option -- 'V'

rds-info version 2.0.7

[root@ca-dev14 rds-tools]# rds-info -Iv

RDS IB Connections:
       LocalAddr      RemoteAddr  Tos  SL 
LocalDev                        RemoteDev
        1.1.1.14        1.1.1.14    5 255 
fe80::2:c903:a:7a31              fe80::2:c903:a:7a31  send_wr=256, 
recv_wr=1024, send_sge=8, rdma_mr_max=4096, rdma_mr_size=257, 
cache_allocs=12
"

So do not worry about backward compatibility.  This commit will work 
well with older rds-tools2.0.5 and 2.0.6.

I will send V2 soon.

Thanks

Zhu Yanjun

> This was one of the reason, the
> all the fields are not updated.
>
> Regards,
> Santosh