Re: FW: [PATCH for-next] RDMA/rxe: Remove pkey table

[Zhu Yanjun <zyjzyj2000@gmail.com>]

On 7/29/2020 1:42 AM, Kamal Heib wrote:
> On Tue, Jul 28, 2020 at 11:46:36PM +0800, Zhu Yanjun wrote:
>> On 7/28/2020 9:44 PM, Kamal Heib wrote:
>>> On Tue, Jul 28, 2020 at 09:21:06PM +0800, Zhu Yanjun wrote:
>>>> On 7/28/2020 4:35 PM, Kamal Heib wrote:
>>>>> On Thu, Jul 23, 2020 at 11:15:00PM +0800, Zhu Yanjun wrote:
>>>>>> On 7/23/2020 9:15 PM, Jason Gunthorpe wrote:
>>>>>>> On Thu, Jul 23, 2020 at 09:08:39PM +0800, Zhu Yanjun wrote:
>>>>>>>> On 7/23/2020 3:25 PM, Kamal Heib wrote:
>>>>>>>>> On Thu, Jul 23, 2020 at 02:58:41PM +0800, Zhu Yanjun wrote:
>>>>>>>>>> On 7/23/2020 1:57 PM, Kamal Heib wrote:
>>>>>>>>>>> On Wed, Jul 22, 2020 at 10:09:04AM +0800, Zhu Yanjun wrote:
>>>>>>>>>>>> On Tue, Jul 21, 2020 at 7:28 PM Yanjun Zhu <yanjunz@mellanox.com> wrote:
>>>>>>>>>>>>> From: Kamal Heib <kamalheib1@gmail.com>
>>>>>>>>>>>>> Sent: Tuesday, July 21, 2020 6:16 PM
>>>>>>>>>>>>> To: linux-rdma@vger.kernel.org
>>>>>>>>>>>>> Cc: Yanjun Zhu <yanjunz@mellanox.com>; Doug Ledford <dledford@redhat.com>; Jason Gunthorpe <jgg@ziepe.ca>; Kamal Heib <kamalheib1@gmail.com>
>>>>>>>>>>>>> Subject: [PATCH for-next] RDMA/rxe: Remove pkey table
>>>>>>>>>>>>>
>>>>>>>>>>>>> The RoCE spec require from RoCE devices to support only the defualt pkey, While the rxe driver maintain a 64 enties pkey table and use only the first entry. With that said remove the maintaing of the pkey table and used the default pkey when needed.
>>>>>>>>>>>>>
>>>>>>>>>>>> Hi Kamal
>>>>>>>>>>>>
>>>>>>>>>>>> After this patch is applied, do you make tests with SoftRoCE and mlx hardware?
>>>>>>>>>>>>
>>>>>>>>>>>> The SoftRoCE should work well with the mlx hardware.
>>>>>>>>>>>>
>>>>>>>>>>>> Zhu Yanjun
>>>>>>>>>>>>
>>>>>>>>>>> Hi Zhu,
>>>>>>>>>>>
>>>>>>>>>>> Yes, please see below:
>>>>>>>>>>>
>>>>>>>>>>> $ ibv_rc_pingpong -d mlx5_0 -g 11
>>>>>>>>>>>         local address:  LID 0x0000, QPN 0x0000e3, PSN 0x728a4f, GID ::ffff:172.31.40.121
>>>>>>>>>> Can you make tests with GSI QP?
>>>>>>>>>>
>>>>>>>>>> Zhu Yanjun
>>>>>>>>>>
>>>>>>>> Is this the GSI ?
>>>>>>>>
>>>>>>>> Please check GSI in "InfiniBandTM Architecture Specification Volume 1
>>>>>>>> Release 1.3"
>>>>>>>>
>>>>>>>> Then make tests with GSI again.
>>>>>> The followings are also removed by this commit. Not sure if it is good.
>>>>>>
>>>>>> "
>>>>>>
>>>>>> C9-42: If the destination QP is QP1, the BTH:P_Key shall be compared to the
>>>>>> set of P_Keys associated with the port on which the packet arrived. If the
>>>>>> P_Key matches any of the keys associated with the port, it shall be
>>>>>> considered valid.
>>>>>>
>>>>>> "
>>>>>>
>>>>> The above is correct for ports that configured to work in InfiniBand
>>>>> mode, while in RoCEv2 mode only the default P_Key should be associated
>>>>> with the port (Please see below from "ANNEX A17:   ROCEV2 (IP ROUTABLE
>>>>> ROCE)):
>>>>>
>>>>> """
>>>>> 17.7.1 LOADING THE P_KEY TABLE
>>>>>
>>>>> Compliance statement C17-7: on page 1193 describes requirements for
>>>>> setting the P_Key table based on an assumption that the P_Key table is
>>>>> set directly by a Subnet Manager. However, RoCEv2 ports do not support
>>>>> InfiniBand Subnet Management. Therefore, compliance statement C17-7:
>>>>> on page 1193 does not apply to RoCEv2 ports.
>>>> "
>>>>
>>>> C17-7: An HCA shall require no OS involvement to set the P_Key table;
>>>>
>>>> the P_Key table shall be set directly by Subnet Manager MADs.
>>>>
>>>> "
>>>>
>>>> In SoftRoCE, what set the P_Key table?
>>>>
>>> No one is setting the P_Key table in SoftRoCE, and no subnet manager in
>>> the RoCE fabric.
>>>
>>> Could you please tell me what is wrong with this patch?
>> Please read the mail thread again.
>>
>> GSI QP number is 1. In your commits, the handle of qpn == 1 is removed.
>>
>> It seems that it conflicts with IB specification.
>>
>> Not sure if it is good.
>>
> Could you please read my patch again and point to what do you think is
> wrong?

What I said is very clear. Good luck

Zhu Yanjun

>
> What I did in this patch is to verify that the pkey value in the
> received packet is the default P_Key regardless of the qpn, because RoCE
> devices should maintain only the default P_Key.
>
> Thanks,
> Kamal
>
>>> Thanks,
>>> Kamal
>>>
>>>>> Methods for setting the P_Key table associated with a RoCEv2 port are
>>>>> not defined in this specification, except for the requirements for a
>>>>> default P_Key described elsewhere in this annex.
>>>>> """
>>>>>
>>>>> Thanks,
>>>>> Kamal
>>>>>
>>>>>
>>>>>>> rping uses RDMA CM which goes over the GSI
>>>>>>>
>>>>>>> Jason
>>