Re: [PATCH rdma-next v1 2/2] RDMA/mlx5: Allow modifying Relaxed Ordering via fast registration

From: Jason Gunthorpe <jgg@nvidia.com>
To: Leon Romanovsky <leon@kernel.org>
Cc: Doug Ledford <dledford@redhat.com>,
	Avihai Horon <avihaih@nvidia.com>,
	linux-kernel@vger.kernel.org, linux-rdma@vger.kernel.org
Subject: Re: [PATCH rdma-next v1 2/2] RDMA/mlx5: Allow modifying Relaxed Ordering via fast registration
Date: Wed, 26 May 2021 16:49:06 -0300	[thread overview]
Message-ID: <20210526194906.GA3646419@nvidia.com> (raw)
In-Reply-To: <9442b0de75f4ee029e7c306fce34b1f6f94a9e34.1621505111.git.leonro@nvidia.com>

On Thu, May 20, 2021 at 01:13:36PM +0300, Leon Romanovsky wrote:
> From: Avihai Horon <avihaih@nvidia.com>
> 
> Relaxed Ordering is enabled by default for kernel ULPs, and is set
> during MKey creation, yet it cannot be modified by them afterwards.
> 
> Allow modifying Relaxed Ordering via fast registration work request.
> This is done by setting the relevant flags in the MKey context mask and
> the Relaxed Ordering flags in the MKey context itself.
> 
> Only ConnectX-7 supports modifying Relaxed Ordering via fast
> registration, and HCA capabilities indicate it. These capabilities are
> checked, and if a fast registration work request tries to modify Relaxed
> Ordering and the capabilities are not present, the work request will fail.

> @@ -762,23 +786,33 @@ static void set_sig_mkey_segment(struct mlx5_mkey_seg *seg,
>  	seg->len = cpu_to_be64(length);
>  	seg->xlt_oct_size = cpu_to_be32(get_xlt_octo(size));
>  	seg->bsfs_octo_size = cpu_to_be32(MLX5_MKEY_BSF_OCTO_SIZE);
> +
> +	if (!(access_flags & IB_ACCESS_DISABLE_RELAXED_ORDERING)) {
> +		MLX5_SET(mkc, seg, relaxed_ordering_write,
> +			 MLX5_CAP_GEN(dev->mdev, relaxed_ordering_write_umr));
> +		MLX5_SET(mkc, seg, relaxed_ordering_read,
> +			 MLX5_CAP_GEN(dev->mdev, relaxed_ordering_read_umr));
> +	}
>  }

I don't quite get this patch

FRWR can only be used with kernel MRs

All kernel MRs are created with relaxed ordering set

Nothing does a FRWR with IB_ACCESS_DISABLE_RELAXED_ORDERING set

So why not leave the relaxed ordering bits masked in the UMR for FWRW
so that the UMR doesn't change them at all and fail/panic if the
caller requests IB_ACCESS_DISABLE_RELAXED_ORDERING ?

Jason