Re: [PATCH rdma-next 4/4] RDMA/mlx5: Use query_special_contexts for mkeys

From: Jason Gunthorpe <jgg@nvidia.com>
To: Leon Romanovsky <leon@kernel.org>
Cc: Or Har-Toov <ohartoov@nvidia.com>,
	"David S. Miller" <davem@davemloft.net>,
	Eric Dumazet <edumazet@google.com>,
	Jakub Kicinski <kuba@kernel.org>,
	linux-rdma@vger.kernel.org,
	Michael Guralnik <michaelgur@nvidia.com>,
	netdev@vger.kernel.org, Paolo Abeni <pabeni@redhat.com>,
	Saeed Mahameed <saeedm@nvidia.com>
Subject: Re: [PATCH rdma-next 4/4] RDMA/mlx5: Use query_special_contexts for mkeys
Date: Wed, 4 Jan 2023 09:56:21 -0400	[thread overview]
Message-ID: <Y7WFhXk6UhGulLKi@nvidia.com> (raw)
In-Reply-To: <Y7WFTULBGZ1WczxV@unreal>

On Wed, Jan 04, 2023 at 03:55:25PM +0200, Leon Romanovsky wrote:
> On Wed, Jan 04, 2023 at 09:13:10AM -0400, Jason Gunthorpe wrote:
> > On Wed, Jan 04, 2023 at 03:09:54PM +0200, Leon Romanovsky wrote:
> > > On Wed, Jan 04, 2023 at 09:03:06AM -0400, Jason Gunthorpe wrote:
> > > > On Wed, Jan 04, 2023 at 10:11:25AM +0200, Leon Romanovsky wrote:
> > > > > -int mlx5_cmd_null_mkey(struct mlx5_core_dev *dev, u32 *null_mkey)
> > > > > -{
> > > > > -	u32 out[MLX5_ST_SZ_DW(query_special_contexts_out)] = {};
> > > > > -	u32 in[MLX5_ST_SZ_DW(query_special_contexts_in)] = {};
> > > > > -	int err;
> > > > > +	err = mlx5_cmd_exec_inout(dev->mdev, query_special_contexts, in, out);
> > > > > +	if (err)
> > > > > +		return err;
> > > > >  
> > > > > -	MLX5_SET(query_special_contexts_in, in, opcode,
> > > > > -		 MLX5_CMD_OP_QUERY_SPECIAL_CONTEXTS);
> > > > > -	err = mlx5_cmd_exec_inout(dev, query_special_contexts, in, out);
> > > > > -	if (!err)
> > > > > -		*null_mkey = MLX5_GET(query_special_contexts_out, out,
> > > > > -				      null_mkey);
> > > > > -	return err;
> > > > > +	if (MLX5_CAP_GEN(dev->mdev, dump_fill_mkey))
> > > > > +		dev->mkeys.dump_fill_mkey = MLX5_GET(query_special_contexts_out,
> > > > > +						     out, dump_fill_mkey);
> > > > > +
> > > > > +	if (MLX5_CAP_GEN(dev->mdev, null_mkey))
> > > > > +		dev->mkeys.null_mkey = cpu_to_be32(
> > > > > +			MLX5_GET(query_special_contexts_out, out, null_mkey));
> > > > > +
> > > > > +	if (MLX5_CAP_GEN(dev->mdev, terminate_scatter_list_mkey)) {
> > > > > +		dev->mkeys.terminate_scatter_list_mkey =
> > > > > +			cpu_to_be32(MLX5_GET(query_special_contexts_out, out,
> > > > > +					     terminate_scatter_list_mkey));
> > > > > +		return 0;
> > > > > +	}
> > > > > +	dev->mkeys.terminate_scatter_list_mkey =
> > > > > +		MLX5_TERMINATE_SCATTER_LIST_LKEY;
> > > > 
> > > > This is already stored in the core dev, why are you recalculating it
> > > > here?
> > > 
> > > It is not recalculating but setting default value. In core dev, we will
> > > have value only if MLX5_CAP_GEN(dev->mdev, terminate_scatter_list_mkey)
> > > is true.
> > 
> > No, it has the identical code:
> > 
> > +static int mlx5_get_terminate_scatter_list_mkey(struct mlx5_core_dev *dev)
> > +{
> > +       if (MLX5_CAP_GEN(dev, terminate_scatter_list_mkey)) {
> > +               dev->terminate_scatter_list_mkey =
> > +                       cpu_to_be32(MLX5_GET(query_special_contexts_out, out,
> > +                                            terminate_scatter_list_mkey));
> > +               return 0;
> > +       }
> > +       dev->terminate_scatter_list_mkey = MLX5_TERMINATE_SCATTER_LIST_LKEY;
> 
> Ahh, you are talking about that.
> terminate_scatter_list_mkey is part of an output from MLX5_CMD_OP_QUERY_SPECIAL_CONTEXTS,
> which is needed to get other mkeys. So instead of doing special logic
> for the terminate_scatter_list_mkey, we decided to use same pattern as
> for other mkeys, which don't belong to core.

Regardless, don't duplicate the code and maybe don't even duplicate
the storage of the terminate_scatter_list_mkey

Jason