Re: [PATCH mlx5-next 3/5] RDMA/mlx5: Change the cache to hold mkeys instead of MRs

[Jason Gunthorpe <jgg@nvidia.com>]

On Tue, Jun 22, 2021 at 03:08:21PM +0300, Leon Romanovsky wrote:
> From: Aharon Landau <aharonl@nvidia.com>
> 
> Today the cache is an MR-cache, however, all members of MR, except for
> mkey, are not being used in the cache.
> Therefore, changing it to an mkey-cache so that the cache has its own
> memory and holds only the values needed for the cache.

This patch is quite big and seems to be doing a lot more than just
this

Frankly, I'm not sure what it is trying to do

> diff --git a/drivers/infiniband/hw/mlx5/mlx5_ib.h b/drivers/infiniband/hw/mlx5/mlx5_ib.h
> index af11a0d8ebc0..ffb6f1d41f3d 100644
> +++ b/drivers/infiniband/hw/mlx5/mlx5_ib.h
> @@ -634,6 +634,15 @@ struct mlx5_user_mmap_entry {
>  #define mlx5_update_odp_stats(mr, counter_name, value)		\
>  	atomic64_add(value, &((mr)->odp_stats.counter_name))
>  
> +struct mlx5r_cache_mkey {
> +	u32 key;
> +	struct mlx5_cache_ent *cache_ent;
> +	u32 out[MLX5_ST_SZ_DW(create_mkey_out)];
> +	struct mlx5_async_work cb_work;
> +	/* Cache list element */
> +	struct list_head list;
> +};

This is the point, right? Lift these members out of the mlx5_ib_mr?

But out abd cb_work shouldn't be stored in perpetuity in the cache, it
is only needed short-term as part of the callback for async mkey
creation.

This should also be organized to not have so many alignment holes

Actually the only thing it does is store a u32 attached to each rbtree
so this looks like rather a lot of memory overhead, plus the
kfree/allocs.

I'd probably do this with an xarray on the mlx5_cache_ent
instead. Store the 'tail index' and adding is
'xa_insert(tail_index++)' and removing is 'xa_erase(tail_index--)'

Use xa_mk_value() and I think we have less than 31 bits of mkey,
right?

>  static inline bool is_odp_mr(struct mlx5_ib_mr *mr)
>  {
>  	return IS_ENABLED(CONFIG_INFINIBAND_ON_DEMAND_PAGING) && mr->umem &&
> @@ -763,16 +758,16 @@ struct mlx5_cache_ent {
>  	u8 fill_to_high_water:1;
>  
>  	/*
> -	 * - available_mrs is the length of list head, ie the number of MRs
> +	 * - available_mkeys is the length of list head, ie the number of Mkeys
>  	 *   available for immediate allocation.
> -	 * - total_mrs is available_mrs plus all in use MRs that could be
> +	 * - total_mkeys is available_mkeys plus all in use Mkeys that could be
>  	 *   returned to the cache.
> -	 * - limit is the low water mark for available_mrs, 2* limit is the
> +	 * - limit is the low water mark for available_mkeys, 2* limit is the
>  	 *   upper water mark.
> -	 * - pending is the number of MRs currently being created
> +	 * - pending is the number of Mkeys currently being created
>  	 */
> -	u32 total_mrs;
> -	u32 available_mrs;
> +	u32 total_mkeys;
> +	u32 available_mkeys;
>  	u32 limit;
>  	u32 pending;

Put all the renaming in another patch, maybe as the last patch in the
series and do everything. Much too hard to read when renaming is
muddled with logic changes

Jason