Re: [for-next PATCH v5 00/11] RDMA/rxe: Add RDMA FLUSH operation

From: "lizhijian@fujitsu.com" <lizhijian@fujitsu.com>
To: Yanjun Zhu <yanjun.zhu@linux.dev>, Jason Gunthorpe <jgg@nvidia.com>
Cc: Bob Pearson <rpearsonhpe@gmail.com>,
	Leon Romanovsky <leon@kernel.org>,
	"linux-rdma@vger.kernel.org" <linux-rdma@vger.kernel.org>,
	Zhu Yanjun <zyjzyj2000@gmail.com>,
	"yangx.jy@fujitsu.com" <yangx.jy@fujitsu.com>,
	"Yasunori Gotou (Fujitsu)" <y-goto@fujitsu.com>,
	"mbloch@nvidia.com" <mbloch@nvidia.com>,
	"liangwenpeng@huawei.com" <liangwenpeng@huawei.com>,
	"tom@talpey.com" <tom@talpey.com>,
	"tomasz.gromadzki@intel.com" <tomasz.gromadzki@intel.com>,
	"dan.j.williams@intel.com" <dan.j.williams@intel.com>,
	"linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>
Subject: Re: [for-next PATCH v5 00/11] RDMA/rxe: Add RDMA FLUSH operation
Date: Fri, 11 Nov 2022 06:10:22 +0000	[thread overview]
Message-ID: <9fa30c8b-cb8d-efec-5896-b96177929d08@fujitsu.com> (raw)
In-Reply-To: <b9f3550a-6fbc-b279-22a3-50285da82e5d@linux.dev>

On 11/11/2022 13:52, Yanjun Zhu wrote:
> 在 2022/11/11 13:10, lizhijian@fujitsu.com 写道:
>>
>>
>> On 11/11/2022 10:49, Yanjun Zhu wrote:
>>> 在 2022/10/29 1:57, Jason Gunthorpe 写道:
>>>> On Tue, Sep 27, 2022 at 01:53:26PM +0800, Li Zhijian wrote:
>>>>> Hey folks,
>>>>>
>>>>> Firstly i want to say thank you to all you guys, especially Bob, who
>>>>> in the
>>>>> past 1+ month, gave me a lots of idea and inspiration.
>>>>
>>>> I would like it if someone familiar with rxe could reviewed-by the
>>>> protocol parts.
>>>
>>> Hi, Jason
>>>
>>> I reviewed these patches. I am fine with these patches.
>>>
>>> Hi, Zhijian
>>>
>>> I noticed the followings:
>>> "
>>> $ ./rdma_flush_server -s [server_address] -p [port_number]
>>> client:
>>> $ ./rdma_flush_client -s [server_address] -p [port_number]
>>> "
>>> Can you merge the server and the client to rdma-core?
>>
>> Yanjun,
>>
>> Yes, there was already a draft PR here
>> https://github.com/linux-rdma/rdma-core/pull/1181, but it cannot go
>> ahead until the kernel's patches are merged.
>>
>> and i will post a new version these days, would you mind if i add your
>> "Reviewed-by" in next version ?
> 
> Reviewed-by: Zhu Yanjun <yanjun.zhu@linux.dev>
> Thanks.
> 
> Another problem, normally rxe should connect to physical ib devices, 
> such as mlx ib device. That is, one host is rxe, the other host is mlx 
> ib device. The rdma connection should be created between the 2 hosts.

it's fully compatible with old operation.

> 
> Do you connect to mlx ib device with this RDMA FLUSH operation?
> And what is the test result?

Yes, i tested it.

After these patches, only RXE device can register *FLUSHABLE* MRs 
successfully. If mlx try that, EOPNOSUPP will be returned.

Similarly, Since other hardwares(MLX for example) have not supported 
FLUSH operation, EOPNOSUPP will be returned if users try to to that.

In short, for RXE requester, MLX responder will return error for the 
request. MLX requester is not able to request a FLUSH operation.

Thanks
Zhijian

> 
> Thanks a lot.
> Zhu Yanjun
> 
>>
>>
>>
>>>
>>> Thanks,
>>> Zhu Yanjun
>>>
>>>>
>>>> Jason
> 