Linux-RDMA Archive on lore.kernel.org
 help / color / Atom feed
From: Zhu Yanjun <zyjzyj2000@gmail.com>
To: Kamal Heib <kamalheib1@gmail.com>
Cc: Jason Gunthorpe <jgg@ziepe.ca>, Yanjun Zhu <yanjunz@mellanox.com>,
	linux-rdma@vger.kernel.org, Doug Ledford <dledford@redhat.com>
Subject: Re: FW: [PATCH for-next] RDMA/rxe: Remove pkey table
Date: Tue, 28 Jul 2020 23:46:36 +0800
Message-ID: <93160a8d-fca7-defc-b39e-e6e5a97ddb87@gmail.com> (raw)
In-Reply-To: <20200728134442.GA29573@kheib-workstation>

On 7/28/2020 9:44 PM, Kamal Heib wrote:
> On Tue, Jul 28, 2020 at 09:21:06PM +0800, Zhu Yanjun wrote:
>> On 7/28/2020 4:35 PM, Kamal Heib wrote:
>>> On Thu, Jul 23, 2020 at 11:15:00PM +0800, Zhu Yanjun wrote:
>>>> On 7/23/2020 9:15 PM, Jason Gunthorpe wrote:
>>>>> On Thu, Jul 23, 2020 at 09:08:39PM +0800, Zhu Yanjun wrote:
>>>>>> On 7/23/2020 3:25 PM, Kamal Heib wrote:
>>>>>>> On Thu, Jul 23, 2020 at 02:58:41PM +0800, Zhu Yanjun wrote:
>>>>>>>> On 7/23/2020 1:57 PM, Kamal Heib wrote:
>>>>>>>>> On Wed, Jul 22, 2020 at 10:09:04AM +0800, Zhu Yanjun wrote:
>>>>>>>>>> On Tue, Jul 21, 2020 at 7:28 PM Yanjun Zhu <yanjunz@mellanox.com> wrote:
>>>>>>>>>>> From: Kamal Heib <kamalheib1@gmail.com>
>>>>>>>>>>> Sent: Tuesday, July 21, 2020 6:16 PM
>>>>>>>>>>> To: linux-rdma@vger.kernel.org
>>>>>>>>>>> Cc: Yanjun Zhu <yanjunz@mellanox.com>; Doug Ledford <dledford@redhat.com>; Jason Gunthorpe <jgg@ziepe.ca>; Kamal Heib <kamalheib1@gmail.com>
>>>>>>>>>>> Subject: [PATCH for-next] RDMA/rxe: Remove pkey table
>>>>>>>>>>>
>>>>>>>>>>> The RoCE spec require from RoCE devices to support only the defualt pkey, While the rxe driver maintain a 64 enties pkey table and use only the first entry. With that said remove the maintaing of the pkey table and used the default pkey when needed.
>>>>>>>>>>>
>>>>>>>>>> Hi Kamal
>>>>>>>>>>
>>>>>>>>>> After this patch is applied, do you make tests with SoftRoCE and mlx hardware?
>>>>>>>>>>
>>>>>>>>>> The SoftRoCE should work well with the mlx hardware.
>>>>>>>>>>
>>>>>>>>>> Zhu Yanjun
>>>>>>>>>>
>>>>>>>>> Hi Zhu,
>>>>>>>>>
>>>>>>>>> Yes, please see below:
>>>>>>>>>
>>>>>>>>> $ ibv_rc_pingpong -d mlx5_0 -g 11
>>>>>>>>>        local address:  LID 0x0000, QPN 0x0000e3, PSN 0x728a4f, GID ::ffff:172.31.40.121
>>>>>>>> Can you make tests with GSI QP?
>>>>>>>>
>>>>>>>> Zhu Yanjun
>>>>>>>>
>>>>>> Is this the GSI ?
>>>>>>
>>>>>> Please check GSI in "InfiniBandTM Architecture Specification Volume 1
>>>>>> Release 1.3"
>>>>>>
>>>>>> Then make tests with GSI again.
>>>> The followings are also removed by this commit. Not sure if it is good.
>>>>
>>>> "
>>>>
>>>> C9-42: If the destination QP is QP1, the BTH:P_Key shall be compared to the
>>>> set of P_Keys associated with the port on which the packet arrived. If the
>>>> P_Key matches any of the keys associated with the port, it shall be
>>>> considered valid.
>>>>
>>>> "
>>>>
>>> The above is correct for ports that configured to work in InfiniBand
>>> mode, while in RoCEv2 mode only the default P_Key should be associated
>>> with the port (Please see below from "ANNEX A17:   ROCEV2 (IP ROUTABLE
>>> ROCE)):
>>>
>>> """
>>> 17.7.1 LOADING THE P_KEY TABLE
>>>
>>> Compliance statement C17-7: on page 1193 describes requirements for
>>> setting the P_Key table based on an assumption that the P_Key table is
>>> set directly by a Subnet Manager. However, RoCEv2 ports do not support
>>> InfiniBand Subnet Management. Therefore, compliance statement C17-7:
>>> on page 1193 does not apply to RoCEv2 ports.
>> "
>>
>> C17-7: An HCA shall require no OS involvement to set the P_Key table;
>>
>> the P_Key table shall be set directly by Subnet Manager MADs.
>>
>> "
>>
>> In SoftRoCE, what set the P_Key table?
>>
> No one is setting the P_Key table in SoftRoCE, and no subnet manager in
> the RoCE fabric.
>
> Could you please tell me what is wrong with this patch?

Please read the mail thread again.

GSI QP number is 1. In your commits, the handle of qpn == 1 is removed.

It seems that it conflicts with IB specification.

Not sure if it is good.

>
> Thanks,
> Kamal
>
>>> Methods for setting the P_Key table associated with a RoCEv2 port are
>>> not defined in this specification, except for the requirements for a
>>> default P_Key described elsewhere in this annex.
>>> """
>>>
>>> Thanks,
>>> Kamal
>>>
>>>
>>>>> rping uses RDMA CM which goes over the GSI
>>>>>
>>>>> Jason



  reply index

Thread overview: 16+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2020-07-21 10:16 Kamal Heib
     [not found] ` <AM6PR05MB6263CFB337190B1740CDF4B7D8780@AM6PR05MB6263.eurprd05.prod.outlook.com>
2020-07-22  2:09   ` FW: " Zhu Yanjun
2020-07-23  5:57     ` Kamal Heib
2020-07-23  6:58       ` Zhu Yanjun
2020-07-23  7:25         ` Kamal Heib
2020-07-23 13:08           ` Zhu Yanjun
2020-07-23 13:15             ` Jason Gunthorpe
2020-07-23 15:15               ` Zhu Yanjun
2020-07-28  8:35                 ` Kamal Heib
2020-07-28 13:21                   ` Zhu Yanjun
2020-07-28 13:44                     ` Kamal Heib
2020-07-28 15:46                       ` Zhu Yanjun [this message]
2020-07-28 17:42                         ` Kamal Heib
2020-07-28 23:45                           ` Zhu Yanjun
2020-07-29  1:36                             ` Mark Bloch
2020-07-31 19:22 ` Jason Gunthorpe

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=93160a8d-fca7-defc-b39e-e6e5a97ddb87@gmail.com \
    --to=zyjzyj2000@gmail.com \
    --cc=dledford@redhat.com \
    --cc=jgg@ziepe.ca \
    --cc=kamalheib1@gmail.com \
    --cc=linux-rdma@vger.kernel.org \
    --cc=yanjunz@mellanox.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

Linux-RDMA Archive on lore.kernel.org

Archives are clonable:
	git clone --mirror https://lore.kernel.org/linux-rdma/0 linux-rdma/git/0.git

	# If you have public-inbox 1.1+ installed, you may
	# initialize and index your mirror using the following commands:
	public-inbox-init -V2 linux-rdma linux-rdma/ https://lore.kernel.org/linux-rdma \
		linux-rdma@vger.kernel.org
	public-inbox-index linux-rdma

Example config snippet for mirrors

Newsgroup available over NNTP:
	nntp://nntp.lore.kernel.org/org.kernel.vger.linux-rdma


AGPL code for this site: git clone https://public-inbox.org/public-inbox.git