[RFC v1 for verbs counters]

[RFC v1 for verbs counters]

[Guy Shattah]

RFC: Verbs Counters


There is a constant demand to know about connections used in verbs (and and other aspects).
Some vendors have been offering hardware counters for a long time by using sysfs. 
Those counters, however - are not available per connection but for the whole system.

One way to do it is for each vendor to offer their vendor specific counters, which will probably
 not be generic since each vendor could have their own implementation of counters hence
 the verbs interface will not be generic for the rest of them. 


We present a generic interface for using counters in verbs.
Let's have some definition before going into details:

Object:          an existing structure in verbs which describes physical entity
                                 e.g.: QP/FLOW/DEVICE/
Counter:         a single attribute which is use to count events/statistics
                                 on object
Counter-set:     a set of counters that belongs to one specific object.


A generic interface with  following functionality is presented :


1.      A way to list of all the available counter-sets in the device.
        Per each counter-set:
        - What do the counters within the set measure? is it QP? Flow? other?
        - A Unique identifier per counter-set.
        - A list of names for all the counters within each counter-set.
           since each vendor has their own counters/stats. Each vendor 
           could use their own names for a counter. This suggestion 
           aims to replace vendor-specific-api with predefined 
           enums/names for each counter/stats.
        - Additional Meta-data about a counter-set (for example - is it cached?)
        
2.      Operations available per counter-set:
2.1     Bind and unbind:
        a counter-set has to be attached to an object in order for any
        counter within a counter-set to count. the attaching action
        is referred to as 'bind' and the opposite action is referred to
        as 'unbind'.
        Rather than having specific generic operation for bind and unbind 
        I choose to use existing verbs methods. The existing methods could 
        be modified with small changes (like adding a new flag) to bind 
        (or unbind) a counter-set to an object.


2.2     Counter-set may be created: counter-set instance is allocated and 
          created on a ibv context and belongs to that context.

2.3     Counter-set may be destroyed: counter-set instance is destroyed and 
        de-allocated. If counter-set is bonded to object then it is the 
        responsibility of the driver either to unbind prior to  hardware 
        de-allocation or to notify the user that driver is unable to destroy 
        a counter-set and it is the user responsibility to unbind prior to 
        destruction.

2.4     Counter-set may be queried:  the user supply counter-set instance and 
        output address. The hardware queries the counter-set and writes the 
        output to the  address as an array of uint64_t. Each entry in the 
        uint64_t array represents a single counter. 


The user is expected to query the device on startup, 
   find which counter-sets are supported and to which objects
   each counter-set may be bonded. During this scan the user
    also finds out which counters are supported for which object.
        
Example for a way to list of all the available counter-sets in the device.


We modify the method  int query_device_ex() by adding a new flag to the 
enum ibv_device_attr_mask:


+          IBV_DEVICE_ATTR_COUNTER_SET        = 1 <<  1


When using this flag, the device will response with struct ibv_device_attr_ex
with a new attribute:


+          uint64_t      max_supported_counter_sets;


And then a user can use a new API to get the description for each counter-set.
number of counter set is specified by a counter-set-id.
a counter-set id is a number from 0 to max_supported_counter_sets.
that is - the number returned from the query_device_ex() call.


int ibv_query_counter_set_description(struct ibv_context *context, \
                                      uint64_t counter_set_id, \
                                      struct ibv_counter_set_description * out)
- return 0 on success
- return -1 when counter_set_id is invalid.


The API writes to out the following structure:


struct ibv_counter_set_description {
            // Which type does this set refers to?                                              
            // value is taken from enum ibv_counter_set_counted_type  
                        uint8_t            counted_type;                                     
            // Number of instances of this counter-set available in the hardware
                        uint64_t           number_of_counter_sets;                   
            // Attributes of the set (bit mask)
            // value is taken from enum ibv_counter_set_attributes
                        uint32_t           attributes;                                                                    
            // number of entries
                        uint8_t              entries_count;
            // List of entries, 
                        struct ibv_counter_entry  entry[256];              
            }


Where:
struct ibv_counter_entry {
	     // name of the entry. last entry contains NULL
             char       name[32];
}           
===========================


Brief explanation for the fields inside struct ibv_counter_set_description:


counted_type - contain id for which this counter_set is related to.
the id is a value from ibv_counter_set_counted_type  (see below)
Each counter-set relates to a verbs object, which is the verbs object this 
counter-set aim to count (i.e. measure), such as QP or Flow.


enum  ibv_counter_set_counted_type  {
                IBV_COUNTER_IBV_QP = 0,
                IBV_COUNTER_IBV_FLOW,
                ...
                }

number_of_counter_sets - how many counters does this device supports?
Note that this value can be interpreted in more than one way. Either how many 
counter_sets are currently available or what is the total (max) number of 
counter_sets the device supports. this is seen as the max limit of count-set
which the process is allowed to create.


attributes   - special attributes which this counter-set might have 
either in software or hardware. 
For example we can have cached counter-set. Which means that every query 
for that set is read from the cache. Unless a request to read the values from 
the hardware was specially specified.


enum ibv_counter_set_attributes {
         // the counter-set value is cached by default 
          IBV_COUNTER_ATTR_CACHED                         = 1 <<  1   
};


entries_count - number of entries in the counter_set

struct ibv_counter_entry  entry[256] - an ordered list of counter names
           where the last name in the array is empty (NULL)

====


Example:




ibv_counter_set_description is a struct to describe other structs.
                for example, we have the following struct:


                private struct guy_counter {
                                uint64_t apples_kg;
                                uint64_t apples_count;
                }


                
                - note that all counters are 64bit entries
                
                The ibv_counter_set_desc looks like this:
                   // according to the ibv_counter_set_type enum
                ibv_counter_set_desc.type = IBV_COUNTER_IBV_GUYGUY                                       
                ibv_counter_set_desc.attributes = 0;
                ibv_counter_set_desc.number_of_counter_sets = 1000;
                ibv_counter_set_desc.entry[0].name =  "apples [Kg]"
                ibv_counter_set_desc.entry[1].name =  "apples [Count]"
                ibv_counter_set_desc.entry[2].name =  \0
                
====


How to fill the internal tables?


On initialization - driver should query device capabilities to see how many 
                    counter-set are supported. for each  supported 
                    counter-set the driver will act as following:
1. Allocate counter_set_id 
2. Register counter_set_id with pointer to data structures with list of counters.
5. Finally - The driver returns number of counter_sets supported in ibv_qeury_device_ex()




Operations available for each counter-set:
Each ibv_counter is represented by the following structure:


struct ibv_counter_set {
        struct ibv_context       *context;
        uint64_t                 handle;  
}




THE NEW API


struct ibv_counter_set* ibv_create_counter_set(struct ibv_context *context,  \
                        uint16_t counter_set_id)


Method returns struct ibv_counter_set which contains context+handle.
Actions: Method Allocates memory for struct ibv_counter_set and then calls 
the driver to allocate the actual hardware counter-set.
If successful method returns pointer to struct ibv_counter_set on the heap
which contains context+handle.
If unsuccessful - method returns NULL and set errno accordingly.


int ibv_destroy_counter_set(struct ibv_counter_set* counter_set)


Methods destroys input counter_set and free the allocated memory.
Actions: Method attempts to remove hardware counter-set and then input struct 
is released (deleted). In the kernel the code checks if caller is 
allowed to destroy counter_set (by comparing pid) and then released 
hardware-resource.


If unsuccessful method returns -1 and set errno accordingly. 
If successful method returns 0.


int ibv_query_counter_set(struct ibv_query_counter_set_attr, uint64_t * out)

Method receives query structure and output address, then query the 
hardware and writes output to the uint64_t * out address.
Actions: Method recives  struct ibv_query_counter_set_attr, parse the query
and then send it to execution in kernel.
In the kernel the code checks if caller is allowed to query
the hardware, executes the query and then writes to *out.
If unsuccessful method returns -1 and set errno accordingly. 
If successful method returns 0


Where:


struct ibv_query_counter_set_attr {
                        uint32_t          comp_mask
                        ibv_counter_set   *counter_set;
                        enum ibv_query_counter_set_attr_params  *query_params;
}


enum ibv_query_counter_set_attr_params {
        // force hardware query instead of cached value
        IBV_COUNTER_FORCE_UPDATE                = 1 <<  1  
};




int ibv_query_counter_set_description(struct ibv_context *context, \
                                      uint64_t counter_set_id, \
                                      struct ibv_counter_set_description * out)

Method writes out a struct ibv_counter_set_description which contains a description 
of a counter-set. 
User should allocate sizeof(struct ibv_counter_set_description) for *out;


- return 0 on success
- return -1 when counter_set_id is invalid.




Example on using counter_sets:




void foo(struct ibv_context *context, int counter_set_id) 
{
        // an array of attributes. this is a container of counter-set values
        uint64_t my_counter[256];

	struct ibv_counter_set_description my_description;
        struct ibv_counter_set* my_counter_set ;

	ibv_query_counter_set_description(context, counter_set_id, &my_description);
                
        my_counter_set = ibv_create_counter_set(context,counter_set_id);


        // let's define the query object
        struct ibv_query_counter_set_attr my_query;
        my_query.comp_mask = 0;
        my_query.counter_set = &my_counter_set;
        my_query.query_params = 0;
                
        // finally - do the query.
        
         if(-1 == ibv_query_counter_set(my_query, my_counter)) {
                printf("query failed")
        }
        else  {
                  for(int i = 0 ; i < my_description->entries_count ; ++i)   {
                                 printf("name %d = %lu, \
                        my_description->entries[i].name, my_counter[i]); 
                  }
        }
}
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: [RFC v1 for verbs counters]

[Amrani, Ram]

> RFC: Verbs Counters
> 
> 
> There is a constant demand to know about connections used in verbs (and and other aspects).
> Some vendors have been offering hardware counters for a long time by using sysfs.
> Those counters, however - are not available per connection but for the whole system.
> 
> One way to do it is for each vendor to offer their vendor specific counters, which will probably
>  not be generic since each vendor could have their own implementation of counters hence
>  the verbs interface will not be generic for the rest of them.
> 
> 
> We present a generic interface for using counters in verbs.
> Let's have some definition before going into details:
> 
> Object:          an existing structure in verbs which describes physical entity
>                                  e.g.: QP/FLOW/DEVICE/
> Counter:         a single attribute which is use to count events/statistics
>                                  on object
> Counter-set:     a set of counters that belongs to one specific object.
> 
> 
> A generic interface with  following functionality is presented :
> 
> 
> 1.      A way to list of all the available counter-sets in the device.
>         Per each counter-set:
>         - What do the counters within the set measure? is it QP? Flow? other?
>         - A Unique identifier per counter-set.
>         - A list of names for all the counters within each counter-set.
>            since each vendor has their own counters/stats. Each vendor
>            could use their own names for a counter. This suggestion
>            aims to replace vendor-specific-api with predefined
>            enums/names for each counter/stats.
>         - Additional Meta-data about a counter-set (for example - is it cached?)
> 
> 2.      Operations available per counter-set:
> 2.1     Bind and unbind:
>         a counter-set has to be attached to an object in order for any
>         counter within a counter-set to count. the attaching action
>         is referred to as 'bind' and the opposite action is referred to
>         as 'unbind'.
>         Rather than having specific generic operation for bind and unbind
>         I choose to use existing verbs methods. The existing methods could
>         be modified with small changes (like adding a new flag) to bind
>         (or unbind) a counter-set to an object.
> 
> 
> 2.2     Counter-set may be created: counter-set instance is allocated and
>           created on a ibv context and belongs to that context.
> 
> 2.3     Counter-set may be destroyed: counter-set instance is destroyed and
>         de-allocated. If counter-set is bonded to object then it is the
>         responsibility of the driver either to unbind prior to  hardware
>         de-allocation or to notify the user that driver is unable to destroy
>         a counter-set and it is the user responsibility to unbind prior to
>         destruction.
> 
> 2.4     Counter-set may be queried:  the user supply counter-set instance and
>         output address. The hardware queries the counter-set and writes the
>         output to the  address as an array of uint64_t. Each entry in the
>         uint64_t array represents a single counter.
> 
> 
> The user is expected to query the device on startup,
>    find which counter-sets are supported and to which objects
>    each counter-set may be bonded. During this scan the user
>     also finds out which counters are supported for which object.
> 
> Example for a way to list of all the available counter-sets in the device.
> 
> 
> We modify the method  int query_device_ex() by adding a new flag to the
> enum ibv_device_attr_mask:
> 
> 
> +          IBV_DEVICE_ATTR_COUNTER_SET        = 1 <<  1
> 
> 
> When using this flag, the device will response with struct ibv_device_attr_ex
> with a new attribute:
> 
> 
> +          uint64_t      max_supported_counter_sets;
> 
> 
> And then a user can use a new API to get the description for each counter-set.
> number of counter set is specified by a counter-set-id.
> a counter-set id is a number from 0 to max_supported_counter_sets.
> that is - the number returned from the query_device_ex() call.
> 
> 
> int ibv_query_counter_set_description(struct ibv_context *context, \
>                                       uint64_t counter_set_id, \
>                                       struct ibv_counter_set_description * out)
> - return 0 on success
> - return -1 when counter_set_id is invalid.
> 
> 
> The API writes to out the following structure:
> 
> 
> struct ibv_counter_set_description {
>             // Which type does this set refers to?
>             // value is taken from enum ibv_counter_set_counted_type
>                         uint8_t            counted_type;
>             // Number of instances of this counter-set available in the hardware
>                         uint64_t           number_of_counter_sets;
>             // Attributes of the set (bit mask)
>             // value is taken from enum ibv_counter_set_attributes
>                         uint32_t           attributes;
>             // number of entries
>                         uint8_t              entries_count;
>             // List of entries,
>                         struct ibv_counter_entry  entry[256];
>             }
> 
> 
> Where:
> struct ibv_counter_entry {
> 	     // name of the entry. last entry contains NULL
>              char       name[32];
> }
> ===========================
> 
> 
> Brief explanation for the fields inside struct ibv_counter_set_description:
> 
> 
> counted_type - contain id for which this counter_set is related to.
> the id is a value from ibv_counter_set_counted_type  (see below)
> Each counter-set relates to a verbs object, which is the verbs object this
> counter-set aim to count (i.e. measure), such as QP or Flow.
> 
> 
> enum  ibv_counter_set_counted_type  {
>                 IBV_COUNTER_IBV_QP = 0,
>                 IBV_COUNTER_IBV_FLOW,
>                 ...
>                 }
> 
> number_of_counter_sets - how many counters does this device supports?
> Note that this value can be interpreted in more than one way. Either how many
> counter_sets are currently available or what is the total (max) number of
> counter_sets the device supports. this is seen as the max limit of count-set
> which the process is allowed to create.
> 
> 
> attributes   - special attributes which this counter-set might have
> either in software or hardware.
> For example we can have cached counter-set. Which means that every query
> for that set is read from the cache. Unless a request to read the values from
> the hardware was specially specified.
> 
> 
> enum ibv_counter_set_attributes {
>          // the counter-set value is cached by default
>           IBV_COUNTER_ATTR_CACHED                         = 1 <<  1
> };
> 
> 
> entries_count - number of entries in the counter_set
> 
> struct ibv_counter_entry  entry[256] - an ordered list of counter names
>            where the last name in the array is empty (NULL)
> 
> ====
> 
> 
> Example:
> 
> 
> 
> 
> ibv_counter_set_description is a struct to describe other structs.
>                 for example, we have the following struct:
> 
> 
>                 private struct guy_counter {
>                                 uint64_t apples_kg;
>                                 uint64_t apples_count;
>                 }
> 
> 
> 
>                 - note that all counters are 64bit entries
> 
>                 The ibv_counter_set_desc looks like this:
>                    // according to the ibv_counter_set_type enum
>                 ibv_counter_set_desc.type = IBV_COUNTER_IBV_GUYGUY
>                 ibv_counter_set_desc.attributes = 0;
>                 ibv_counter_set_desc.number_of_counter_sets = 1000;
>                 ibv_counter_set_desc.entry[0].name =  "apples [Kg]"
>                 ibv_counter_set_desc.entry[1].name =  "apples [Count]"
>                 ibv_counter_set_desc.entry[2].name =  \0
> 
> ====
> 
> 
> How to fill the internal tables?
> 
> 
> On initialization - driver should query device capabilities to see how many
>                     counter-set are supported. for each  supported
>                     counter-set the driver will act as following:
> 1. Allocate counter_set_id
> 2. Register counter_set_id with pointer to data structures with list of counters.
> 5. Finally - The driver returns number of counter_sets supported in ibv_qeury_device_ex()
> 
> 
> 
> 
> Operations available for each counter-set:
> Each ibv_counter is represented by the following structure:
> 
> 
> struct ibv_counter_set {
>         struct ibv_context       *context;
>         uint64_t                 handle;
> }
> 
> 
> 
> 
> THE NEW API
> 
> 
> struct ibv_counter_set* ibv_create_counter_set(struct ibv_context *context,  \
>                         uint16_t counter_set_id)
> 
> 
> Method returns struct ibv_counter_set which contains context+handle.
> Actions: Method Allocates memory for struct ibv_counter_set and then calls
> the driver to allocate the actual hardware counter-set.
> If successful method returns pointer to struct ibv_counter_set on the heap
> which contains context+handle.
> If unsuccessful - method returns NULL and set errno accordingly.
> 
> 
> int ibv_destroy_counter_set(struct ibv_counter_set* counter_set)
> 
> 
> Methods destroys input counter_set and free the allocated memory.
> Actions: Method attempts to remove hardware counter-set and then input struct
> is released (deleted). In the kernel the code checks if caller is
> allowed to destroy counter_set (by comparing pid) and then released
> hardware-resource.
> 
> 
> If unsuccessful method returns -1 and set errno accordingly.
> If successful method returns 0.
> 
> 
> int ibv_query_counter_set(struct ibv_query_counter_set_attr, uint64_t * out)
> 
> Method receives query structure and output address, then query the
> hardware and writes output to the uint64_t * out address.
> Actions: Method recives  struct ibv_query_counter_set_attr, parse the query
> and then send it to execution in kernel.
> In the kernel the code checks if caller is allowed to query
> the hardware, executes the query and then writes to *out.
> If unsuccessful method returns -1 and set errno accordingly.
> If successful method returns 0
> 
> 
> Where:
> 
> 
> struct ibv_query_counter_set_attr {
>                         uint32_t          comp_mask
>                         ibv_counter_set   *counter_set;
>                         enum ibv_query_counter_set_attr_params  *query_params;
> }
> 
> 
> enum ibv_query_counter_set_attr_params {
>         // force hardware query instead of cached value
>         IBV_COUNTER_FORCE_UPDATE                = 1 <<  1
> };
> 
> 
> 
> 
> int ibv_query_counter_set_description(struct ibv_context *context, \
>                                       uint64_t counter_set_id, \
>                                       struct ibv_counter_set_description * out)
> 
> Method writes out a struct ibv_counter_set_description which contains a description
> of a counter-set.
> User should allocate sizeof(struct ibv_counter_set_description) for *out;
> 
> 
> - return 0 on success
> - return -1 when counter_set_id is invalid.
> 
> 
> 
> 
> Example on using counter_sets:
> 
> 
> 
> 
> void foo(struct ibv_context *context, int counter_set_id)
> {
>         // an array of attributes. this is a container of counter-set values
>         uint64_t my_counter[256];
> 
> 	struct ibv_counter_set_description my_description;
>         struct ibv_counter_set* my_counter_set ;
> 
> 	ibv_query_counter_set_description(context, counter_set_id, &my_description);
> 
>         my_counter_set = ibv_create_counter_set(context,counter_set_id);
> 
> 
>         // let's define the query object
>         struct ibv_query_counter_set_attr my_query;
>         my_query.comp_mask = 0;
>         my_query.counter_set = &my_counter_set;
>         my_query.query_params = 0;
> 
>         // finally - do the query.
> 
>          if(-1 == ibv_query_counter_set(my_query, my_counter)) {
>                 printf("query failed")
>         }
>         else  {
>                   for(int i = 0 ; i < my_description->entries_count ; ++i)   {
>                                  printf("name %d = %lu, \
>                         my_description->entries[i].name, my_counter[i]);
>                   }
>         }
> }

I like this solution, even though I was aiming for a different problem :-).

Regarding this solution. It allows polling statistics from the driver/hardware
per demand in a generic way. Generic means in this case - the number,
the scope and the meaning of the statistics. I understand that it assumes
that the resources to count these statistics are limited and hence the usage
should be only after allocation by the "user".
It isn't clear to me if the user is the application or libibverbs. Anyway, some
Cleanup should be considered to avoid driver/HW resources being wasted
due to improper behavior. If the NIC's HW supports this then the implications
on the performance is little. If it doesn't, then it is up to the vendor to decide
if to support and to what extent.
The vendor should be able to decide what scope it allows/supports.
I'm not sure what is the added value of caching statistics.

I'm not sure how this would relate to "rdmatool - tool for RDMA users" [1].
Is this an alternative or parallel solution? Will the code be re-usable?

Regarding what issue was planning to focus on, I will comment in Alex's e-mail.

Thanks,
Ram

[1] https://www.spinics.net/lists/linux-rdma/msg45250.html
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [RFC v1 for verbs counters]

[Guy Shattah]

    
>> RFC: Verbs Counters
>> 
>> 
>> There is a constant demand to know about connections used in verbs (and and other aspects).
>> Some vendors have been offering hardware counters for a long time by using sysfs.
>> Those counters, however - are not available per connection but for the whole system.
>> 
>> One way to do it is for each vendor to offer their vendor specific counters, which will probably
>>  not be generic since each vendor could have their own implementation of counters hence
>>  the verbs interface will not be generic for the rest of them.
>> 
>> 
>> We present a generic interface for using counters in verbs.
>> Let's have some definition before going into details:
>> 
>> Object:          an existing structure in verbs which describes physical entity
>>                                  e.g.: QP/FLOW/DEVICE/
>> Counter:         a single attribute which is use to count events/statistics
>>                                  on object
>> Counter-set:     a set of counters that belongs to one specific object.
>> 
>> 
>> A generic interface with  following functionality is presented :
>> 
>> 
>> 1.      A way to list of all the available counter-sets in the device.
>>         Per each counter-set:
>>         - What do the counters within the set measure? is it QP? Flow? other?
>>         - A Unique identifier per counter-set.
>>         - A list of names for all the counters within each counter-set.
>>            since each vendor has their own counters/stats. Each vendor
>>            could use their own names for a counter. This suggestion
>>            aims to replace vendor-specific-api with predefined
>>            enums/names for each counter/stats.
>>         - Additional Meta-data about a counter-set (for example - is it cached?)
>> 
>> 2.      Operations available per counter-set:
>> 2.1     Bind and unbind:
>>         a counter-set has to be attached to an object in order for any
>>         counter within a counter-set to count. the attaching action
>>         is referred to as 'bind' and the opposite action is referred to
>>         as 'unbind'.
>>         Rather than having specific generic operation for bind and unbind
>>         I choose to use existing verbs methods. The existing methods could
>>         be modified with small changes (like adding a new flag) to bind
>>         (or unbind) a counter-set to an object.
>> 
>> 
>> 2.2     Counter-set may be created: counter-set instance is allocated and
>>           created on a ibv context and belongs to that context.
>> 
>> 2.3     Counter-set may be destroyed: counter-set instance is destroyed and
>>         de-allocated. If counter-set is bonded to object then it is the
>>         responsibility of the driver either to unbind prior to  hardware
>>         de-allocation or to notify the user that driver is unable to destroy
>>         a counter-set and it is the user responsibility to unbind prior to
>>         destruction.
>> 
>> 2.4     Counter-set may be queried:  the user supply counter-set instance and
>>         output address. The hardware queries the counter-set and writes the
>>         output to the  address as an array of uint64_t. Each entry in the
>>         uint64_t array represents a single counter.
>> 
>> 
>> The user is expected to query the device on startup,
>>    find which counter-sets are supported and to which objects
>>    each counter-set may be bonded. During this scan the user
>>     also finds out which counters are supported for which object.
>> 
>> Example for a way to list of all the available counter-sets in the device.
>> 
>> 
>> We modify the method  int query_device_ex() by adding a new flag to the
>> enum ibv_device_attr_mask:
>> 
>> 
>> +          IBV_DEVICE_ATTR_COUNTER_SET        = 1 <<  1
>> 
>> 
>> When using this flag, the device will response with struct ibv_device_attr_ex
>> with a new attribute:
>> 
>> 
>> +          uint64_t      max_supported_counter_sets;
>> 
>> 
>> And then a user can use a new API to get the description for each counter-set.
>> number of counter set is specified by a counter-set-id.
>> a counter-set id is a number from 0 to max_supported_counter_sets.
>> that is - the number returned from the query_device_ex() call.
>> 
>> 
>> int ibv_query_counter_set_description(struct ibv_context *context, \
>>                                       uint64_t counter_set_id, \
>>                                       struct ibv_counter_set_description * out)
>> - return 0 on success
>> - return -1 when counter_set_id is invalid.
>> 
>> 
>> The API writes to out the following structure:
>> 
>> 
>> struct ibv_counter_set_description {
>>             // Which type does this set refers to?
>>             // value is taken from enum ibv_counter_set_counted_type
>>                         uint8_t            counted_type;
>>             // Number of instances of this counter-set available in the hardware
>>                         uint64_t           number_of_counter_sets;
>>             // Attributes of the set (bit mask)
>>             // value is taken from enum ibv_counter_set_attributes
>>                         uint32_t           attributes;
>>             // number of entries
>>                         uint8_t              entries_count;
>>             // List of entries,
>>                         struct ibv_counter_entry  entry[256];
>>             }
>> 
>> 
>> Where:
>> struct ibv_counter_entry {
>>             // name of the entry. last entry contains NULL
>>              char       name[32];
>> }
>> ===========================
>> 
>> 
>> Brief explanation for the fields inside struct ibv_counter_set_description:
>> 
>> 
>> counted_type - contain id for which this counter_set is related to.
>> the id is a value from ibv_counter_set_counted_type  (see below)
>> Each counter-set relates to a verbs object, which is the verbs object this
>> counter-set aim to count (i.e. measure), such as QP or Flow.
>> 
>> 
>> enum  ibv_counter_set_counted_type  {
>>                 IBV_COUNTER_IBV_QP = 0,
>>                 IBV_COUNTER_IBV_FLOW,
>>                 ...
>>                 }
>> 
>> number_of_counter_sets - how many counters does this device supports?
>> Note that this value can be interpreted in more than one way. Either how many
>> counter_sets are currently available or what is the total (max) number of
>> counter_sets the device supports. this is seen as the max limit of count-set
>> which the process is allowed to create.
>> 
>> 
>> attributes   - special attributes which this counter-set might have
>> either in software or hardware.
>> For example we can have cached counter-set. Which means that every query
>> for that set is read from the cache. Unless a request to read the values from
>> the hardware was specially specified.
>> 
>> 
>> enum ibv_counter_set_attributes {
>>          // the counter-set value is cached by default
>>           IBV_COUNTER_ATTR_CACHED                         = 1 <<  1
>> };
>> 
>> 
>> entries_count - number of entries in the counter_set
>> 
>> struct ibv_counter_entry  entry[256] - an ordered list of counter names
>>            where the last name in the array is empty (NULL)
>> 
>> ====
>> 
>> 
>> Example:
>> 
>> 
>> 
>> 
>> ibv_counter_set_description is a struct to describe other structs.
>>                 for example, we have the following struct:
>> 
>> 
>>                 private struct guy_counter {
>>                                 uint64_t apples_kg;
>>                                 uint64_t apples_count;
>>                 }
>> 
>> 
>> 
>>                 - note that all counters are 64bit entries
>> 
>>                 The ibv_counter_set_desc looks like this:
>>                    // according to the ibv_counter_set_type enum
>>                 ibv_counter_set_desc.type = IBV_COUNTER_IBV_GUYGUY
>>                 ibv_counter_set_desc.attributes = 0;
>>                 ibv_counter_set_desc.number_of_counter_sets = 1000;
>>                 ibv_counter_set_desc.entry[0].name =  "apples [Kg]"
>>                 ibv_counter_set_desc.entry[1].name =  "apples [Count]"
>>                 ibv_counter_set_desc.entry[2].name =  \0
>> 
>> ====
>> 
>> 
>> How to fill the internal tables?
>> 
>> 
>> On initialization - driver should query device capabilities to see how many
>>                     counter-set are supported. for each  supported
>>                     counter-set the driver will act as following:
>> 1. Allocate counter_set_id
>> 2. Register counter_set_id with pointer to data structures with list of counters.
>> 5. Finally - The driver returns number of counter_sets supported in ibv_qeury_device_ex()
>> 
>> 
>> 
>> 
>> Operations available for each counter-set:
>> Each ibv_counter is represented by the following structure:
>> 
>> 
>> struct ibv_counter_set {
>>         struct ibv_context       *context;
>>         uint64_t                 handle;
>> }
>> 
>> 
>> 
>> 
>> THE NEW API
>> 
>> 
>> struct ibv_counter_set* ibv_create_counter_set(struct ibv_context *context,  \
>>                         uint16_t counter_set_id)
>> 
>> 
>> Method returns struct ibv_counter_set which contains context+handle.
>> Actions: Method Allocates memory for struct ibv_counter_set and then calls
>> the driver to allocate the actual hardware counter-set.
>> If successful method returns pointer to struct ibv_counter_set on the heap
>> which contains context+handle.
>> If unsuccessful - method returns NULL and set errno accordingly.
>> 
>> 
>> int ibv_destroy_counter_set(struct ibv_counter_set* counter_set)
>> 
>> 
>> Methods destroys input counter_set and free the allocated memory.
>> Actions: Method attempts to remove hardware counter-set and then input struct
>> is released (deleted). In the kernel the code checks if caller is
>> allowed to destroy counter_set (by comparing pid) and then released
>> hardware-resource.
>> 
>> 
>> If unsuccessful method returns -1 and set errno accordingly.
>> If successful method returns 0.
>> 
>> 
>> int ibv_query_counter_set(struct ibv_query_counter_set_attr, uint64_t * out)
>> 
>> Method receives query structure and output address, then query the
>> hardware and writes output to the uint64_t * out address.
>> Actions: Method recives  struct ibv_query_counter_set_attr, parse the query
>> and then send it to execution in kernel.
>> In the kernel the code checks if caller is allowed to query
>> the hardware, executes the query and then writes to *out.
>> If unsuccessful method returns -1 and set errno accordingly.
>> If successful method returns 0
>> 
>> 
>> Where:
>> 
>> 
>> struct ibv_query_counter_set_attr {
>>                         uint32_t          comp_mask
>>                         ibv_counter_set   *counter_set;
>>                         enum ibv_query_counter_set_attr_params  *query_params;
>> }
>> 
>> 
>> enum ibv_query_counter_set_attr_params {
>>         // force hardware query instead of cached value
>>         IBV_COUNTER_FORCE_UPDATE                = 1 <<  1
>> };
>> 
>> 
>> 
>> 
>> int ibv_query_counter_set_description(struct ibv_context *context, \
>>                                       uint64_t counter_set_id, \
>>                                       struct ibv_counter_set_description * out)
>> 
>> Method writes out a struct ibv_counter_set_description which contains a description
>> of a counter-set.
>> User should allocate sizeof(struct ibv_counter_set_description) for *out;
>> 
>> 
>> - return 0 on success
>> - return -1 when counter_set_id is invalid.
>> 
>> 
>> 
>> 
>> Example on using counter_sets:
>> 
>> 
>> 
>> 
>> void foo(struct ibv_context *context, int counter_set_id)
>> {
>>         // an array of attributes. this is a container of counter-set values
>>         uint64_t my_counter[256];
>> 
>>        struct ibv_counter_set_description my_description;
>>         struct ibv_counter_set* my_counter_set ;
>> 
>>        ibv_query_counter_set_description(context, counter_set_id, &my_description);
>> 
>>         my_counter_set = ibv_create_counter_set(context,counter_set_id);
>> 
>> 
>>         // let's define the query object
>>         struct ibv_query_counter_set_attr my_query;
>>         my_query.comp_mask = 0;
>>         my_query.counter_set = &my_counter_set;
>>         my_query.query_params = 0;
>> 
>>         // finally - do the query.
>> 
>>          if(-1 == ibv_query_counter_set(my_query, my_counter)) {
>>                 printf("query failed")
>>         }
>>         else  {
>>                   for(int i = 0 ; i < my_description->entries_count ; ++i)   {
>>                                  printf("name %d = %lu, \
>>                         my_description->entries[i].name, my_counter[i]);
>>                   }
>>         }
>> }
>>
>>I like this solution, even though I was aiming for a different problem :-).
>>
>>Regarding this solution. It allows polling statistics from the driver/hardware
>>per demand in a generic way. Generic means in this case - the number,
>>the scope and the meaning of the statistics. I understand that it assumes
>>that the resources to count these statistics are limited and hence the usage
>>should be only after allocation by the "user".
>>It isn't clear to me if the user is the application or libibverbs. Anyway, some
>>Cleanup should be considered to avoid driver/HW resources being wasted
>>due to improper behavior. If the NIC's HW supports this then the implications
>>on the performance is little. If it doesn't, then it is up to the vendor to decide
>>if to support and to what extent.
>>The vendor should be able to decide what scope it allows/supports.
>>I'm not sure what is the added value of caching statistics.
>>
>>I'm not sure how this would relate to "rdmatool - tool for RDMA users" [1].
>>Is this an alternative or parallel solution? Will the code be re-usable?
>>
>>Regarding what issue was planning to focus on, I will comment in Alex's e-mail.
>>
>>Thanks,
>>Ram
>>
>>[1] https://www.spinics.net/lists/linux-rdma/msg45250.html

Hi Ram,
I'm glad to know that you like this solution.
However, I believe that we were aiming for the same thing. trying to know 
more about the QP/Connection. What I offered in this RFC is a generic infrastructure
to be utilized by the vendors to implement a solution that suits your needs.

as for your concerns,
the user is the programmer who is utilizing the newly suggested verbs interface
to know more about the system.
Cleanup is supported as intergal part of the API (destroy_counter_set)
and it is part of the user responsability to free resources.
it is up to the vendor to make sure resources are not getting wasted.
for example - if a user attached (binds) a counter_set to QP and the program terminates.
I expect the vendor to make sure their driver releases hardware resources for both the QP
and the attached counter_set. if the vendor fails to release any resources it is 
resources leak. 

the took you mentioned in [1] could possibly utilize the interface I suggested
in order to supply more information and/or statistics about the system.

for examle, the tool you mentioned describes the following actions:
[1] Query RDMA device capabilities
- capabilities are constant values/attributes, hence counter_set do not fit this option.
[2] Query RDMA device status and current open resources
- status and least of open resources:
  I could think of a counter_set that is binded to the ibv_context and bring
  a with values, such as number of open QPs and other resources.
[3] Fetching RDMA statistics
- the suggested API fits  RDMA statistics very well and is capable of fulfilling these needs.
[4] Configure RDMA device
- sorry, the API suggested here is 'read-only'. maybe someone else will offer a generic API
  to configure. however - it is out of the scope of the suggested API.


re--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [RFC v1 for verbs counters]

[Dennis Dalessandro]

On 03/13/2017 04:36 PM, Guy Shattah wrote:
>
> RFC: Verbs Counters
>
>
> There is a constant demand to know about connections used in verbs (and and other aspects).
> Some vendors have been offering hardware counters for a long time by using sysfs.
> Those counters, however - are not available per connection but for the whole system.

Are you proposing to remove the system wide counters from sysfs?

> ibv_counter_set_description is a struct to describe other structs.
>                 for example, we have the following struct:
>
>
>                 private struct guy_counter {
>                                 uint64_t apples_kg;
>                                 uint64_t apples_count;
>                 }
>
>
>
>                 - note that all counters are 64bit entries

Might be a bit wasteful for some things. Could you not have an attribute 
which says if its 64bit or not?

-Denny


--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: [RFC v1 for verbs counters]

[Guy Shattah]

>On 03/13/2017 04:36 PM, Guy Shattah wrote:
>>
>> RFC: Verbs Counters
>>
>>
>> There is a constant demand to know about connections used in verbs (and and other aspects).
>> Some vendors have been offering hardware counters for a long time by using sysfs.
>> Those counters, however - are not available per connection but for the whole system.
>
>Are you proposing to remove the system wide counters from sysfs?
>
>> ibv_counter_set_description is a struct to describe other structs.
>>                 for example, we have the following struct:
>>
>>
>>                 private struct guy_counter {
>>                                 uint64_t apples_kg;
>>                                 uint64_t apples_count;
>>                 }
>>
>>
>>
>>                 - note that all counters are 64bit entries
>
>Might be a bit wasteful for some things. Could you not have an attribute which says if its 64bit or not?
>
>-Denny

HI,
I'm not offering to remove the system-wide counters. I'm offering another way to view counters 
in the system. My suggestion is aimed for the developer looking to know more about
Their connections, flows or other aspects which are not system-wide statistics.
such counters have not been exposed so far.


Regarding counters width  - After giving it a lot of thought. I believe that 64bit counters are the best
decision.
a) Allowing more than one size of counters will make the memory allocation and data parsing harder.

b) 64bit is exactly the right size. It is not too small for the terabytes of data passing in the system
      and it is not bit enough (like 128bit) so vendor may still choose to implement 128bit width
      counters as a pair of 64bit (high + low)

Guy

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [RFC v1 for verbs counters]

[Dennis Dalessandro]

On 03/19/2017 06:07 AM, Guy Shattah wrote:
>> On 03/13/2017 04:36 PM, Guy Shattah wrote:
>>>
>>> RFC: Verbs Counters
>>>
>>>
>>> There is a constant demand to know about connections used in verbs (and and other aspects).
>>> Some vendors have been offering hardware counters for a long time by using sysfs.
>>> Those counters, however - are not available per connection but for the whole system.
>>
>> Are you proposing to remove the system wide counters from sysfs?
>>
>>> ibv_counter_set_description is a struct to describe other structs.
>>>                 for example, we have the following struct:
>>>
>>>
>>>                 private struct guy_counter {
>>>                                 uint64_t apples_kg;
>>>                                 uint64_t apples_count;
>>>                 }
>>>
>>>
>>>
>>>                 - note that all counters are 64bit entries
>>
>> Might be a bit wasteful for some things. Could you not have an attribute which says if its 64bit or not?
>>
>> -Denny
>
> HI,
> I'm not offering to remove the system-wide counters. I'm offering another way to view counters
> in the system. My suggestion is aimed for the developer looking to know more about
> Their connections, flows or other aspects which are not system-wide statistics.
> such counters have not been exposed so far.
>
>
> Regarding counters width  - After giving it a lot of thought. I believe that 64bit counters are the best
> decision.
> a) Allowing more than one size of counters will make the memory allocation and data parsing harder.

I would agree, but I'm not sure it's that hard.

> b) 64bit is exactly the right size. It is not too small for the terabytes of data passing in the system
>       and it is not bit enough (like 128bit) so vendor may still choose to implement 128bit width
>       counters as a pair of 64bit (high + low)
>
> Guy

What about something that counts errors? I would hope 64bits isn't 
needed for that. It all depends on what you are counting really. If this 
is meant to be a generic interface I would think it should be as 
flexible as possible. However maybe the small counters are rare and it's 
just not worth the extra implementation complexity. I'm on the fence 
here and won't argue too hard one way or the other.

-Denny

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: [RFC v1 for verbs counters]

[Hefty, Sean]

> Regarding counters width  - After giving it a lot of thought. I believe
> that 64bit counters are the best
> decision.

I agree - unless we're talking about thousands of counters, smaller ones don't seem worth the additional complexity to save a few bytes.

> a) Allowing more than one size of counters will make the memory
> allocation and data parsing harder.
> 
> b) 64bit is exactly the right size.

Well, I wouldn't go this far.

Is there a need to indicate if hardware doesn't actually support 64-bits?
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: [RFC v1 for verbs counters]

[Amrani, Ram]

> > Regarding counters width  - After giving it a lot of thought. I believe
> > that 64bit counters are the best
> > decision.
> 
> I agree - unless we're talking about thousands of counters, smaller ones don't seem worth the additional complexity to
> save a few bytes.
> 
> > a) Allowing more than one size of counters will make the memory
> > allocation and data parsing harder.
> >
> > b) 64bit is exactly the right size.

I agree on this size.

> 
> Well, I wouldn't go this far.
> 
> Is there a need to indicate if hardware doesn't actually support 64-bits?

I'm guessing that most, if not all, HW support 64-bits counters.
If not then the driver can overcome this by storing the previously read value and
reporting a 64-bit value (aggregate). This is cumbersome for the driver but simplifies
the common implementation.

Ram






--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html