Linux-RDMA Archive on lore.kernel.org
 help / color / Atom feed
From: Mark Bloch <markb@mellanox.com>
To: Zhu Yanjun <zyjzyj2000@gmail.com>, Kamal Heib <kamalheib1@gmail.com>
Cc: Jason Gunthorpe <jgg@ziepe.ca>, Yanjun Zhu <yanjunz@mellanox.com>,
	linux-rdma@vger.kernel.org, Doug Ledford <dledford@redhat.com>
Subject: Re: FW: [PATCH for-next] RDMA/rxe: Remove pkey table
Date: Tue, 28 Jul 2020 18:36:18 -0700
Message-ID: <9e2beff9-ff0a-a6e6-c4b5-621650480044@mellanox.com> (raw)
In-Reply-To: <b12cf75a-1459-bee9-8d38-19a73d048a62@gmail.com>



On 7/28/2020 16:45, Zhu Yanjun wrote:
> On 7/29/2020 1:42 AM, Kamal Heib wrote:
>> On Tue, Jul 28, 2020 at 11:46:36PM +0800, Zhu Yanjun wrote:
>>> On 7/28/2020 9:44 PM, Kamal Heib wrote:
>>>> On Tue, Jul 28, 2020 at 09:21:06PM +0800, Zhu Yanjun wrote:
>>>>> On 7/28/2020 4:35 PM, Kamal Heib wrote:
>>>>>> On Thu, Jul 23, 2020 at 11:15:00PM +0800, Zhu Yanjun wrote:
>>>>>>> On 7/23/2020 9:15 PM, Jason Gunthorpe wrote:
>>>>>>>> On Thu, Jul 23, 2020 at 09:08:39PM +0800, Zhu Yanjun wrote:
>>>>>>>>> On 7/23/2020 3:25 PM, Kamal Heib wrote:
>>>>>>>>>> On Thu, Jul 23, 2020 at 02:58:41PM +0800, Zhu Yanjun wrote:
>>>>>>>>>>> On 7/23/2020 1:57 PM, Kamal Heib wrote:
>>>>>>>>>>>> On Wed, Jul 22, 2020 at 10:09:04AM +0800, Zhu Yanjun wrote:
>>>>>>>>>>>>> On Tue, Jul 21, 2020 at 7:28 PM Yanjun Zhu <yanjunz@mellanox.com> wrote:
>>>>>>>>>>>>>> From: Kamal Heib <kamalheib1@gmail.com>
>>>>>>>>>>>>>> Sent: Tuesday, July 21, 2020 6:16 PM
>>>>>>>>>>>>>> To: linux-rdma@vger.kernel.org
>>>>>>>>>>>>>> Cc: Yanjun Zhu <yanjunz@mellanox.com>; Doug Ledford <dledford@redhat.com>; Jason Gunthorpe <jgg@ziepe.ca>; Kamal Heib <kamalheib1@gmail.com>
>>>>>>>>>>>>>> Subject: [PATCH for-next] RDMA/rxe: Remove pkey table
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> The RoCE spec require from RoCE devices to support only the defualt pkey, While the rxe driver maintain a 64 enties pkey table and use only the first entry. With that said remove the maintaing of the pkey table and used the default pkey when needed.
>>>>>>>>>>>>>>
>>>>>>>>>>>>> Hi Kamal
>>>>>>>>>>>>>
>>>>>>>>>>>>> After this patch is applied, do you make tests with SoftRoCE and mlx hardware?
>>>>>>>>>>>>>
>>>>>>>>>>>>> The SoftRoCE should work well with the mlx hardware.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Zhu Yanjun
>>>>>>>>>>>>>
>>>>>>>>>>>> Hi Zhu,
>>>>>>>>>>>>
>>>>>>>>>>>> Yes, please see below:
>>>>>>>>>>>>
>>>>>>>>>>>> $ ibv_rc_pingpong -d mlx5_0 -g 11
>>>>>>>>>>>>         local address:  LID 0x0000, QPN 0x0000e3, PSN 0x728a4f, GID ::ffff:172.31.40.121
>>>>>>>>>>> Can you make tests with GSI QP?
>>>>>>>>>>>
>>>>>>>>>>> Zhu Yanjun
>>>>>>>>>>>
>>>>>>>>> Is this the GSI ?
>>>>>>>>>
>>>>>>>>> Please check GSI in "InfiniBandTM Architecture Specification Volume 1
>>>>>>>>> Release 1.3"
>>>>>>>>>
>>>>>>>>> Then make tests with GSI again.
>>>>>>> The followings are also removed by this commit. Not sure if it is good.
>>>>>>>
>>>>>>> "
>>>>>>>
>>>>>>> C9-42: If the destination QP is QP1, the BTH:P_Key shall be compared to the
>>>>>>> set of P_Keys associated with the port on which the packet arrived. If the
>>>>>>> P_Key matches any of the keys associated with the port, it shall be
>>>>>>> considered valid.
>>>>>>>
>>>>>>> "
>>>>>>>
>>>>>> The above is correct for ports that configured to work in InfiniBand
>>>>>> mode, while in RoCEv2 mode only the default P_Key should be associated
>>>>>> with the port (Please see below from "ANNEX A17:   ROCEV2 (IP ROUTABLE
>>>>>> ROCE)):
>>>>>>
>>>>>> """
>>>>>> 17.7.1 LOADING THE P_KEY TABLE
>>>>>>
>>>>>> Compliance statement C17-7: on page 1193 describes requirements for
>>>>>> setting the P_Key table based on an assumption that the P_Key table is
>>>>>> set directly by a Subnet Manager. However, RoCEv2 ports do not support
>>>>>> InfiniBand Subnet Management. Therefore, compliance statement C17-7:
>>>>>> on page 1193 does not apply to RoCEv2 ports.
>>>>> "
>>>>>
>>>>> C17-7: An HCA shall require no OS involvement to set the P_Key table;
>>>>>
>>>>> the P_Key table shall be set directly by Subnet Manager MADs.
>>>>>
>>>>> "
>>>>>
>>>>> In SoftRoCE, what set the P_Key table?
>>>>>
>>>> No one is setting the P_Key table in SoftRoCE, and no subnet manager in
>>>> the RoCE fabric.
>>>>
>>>> Could you please tell me what is wrong with this patch?
>>> Please read the mail thread again.
>>>
>>> GSI QP number is 1. In your commits, the handle of qpn == 1 is removed.
>>>
>>> It seems that it conflicts with IB specification.
>>>
>>> Not sure if it is good.
>>>
>> Could you please read my patch again and point to what do you think is
>> wrong?
> 
> What I said is very clear. Good luck

qpn == 1, qpn == x it, qpn == i, it doesn't matter. Please read the RoCEv2 annex:

A17.4.7 INFINIBAND PARTITIONING
Methods to populate the P_Key table associated with a RoCEv2 port are
outside the scope of this annex. Note that this annex relies on the partition
table being initialized at power on time with at least the default P_Key as
described in Chapter 10 (Software Transport Interface) of the base specification. The P_Key contained in the BTH is validated for inbound packets
as required by the packet header validation protocols defined in Chapter
9 of the base specification.

A17.7.1 LOADING THE P_KEY TABLE
Compliance statement C17-7 describes requirements for setting the
P_Key table based on an assumption that the P_Key table is set directly
by a Subnet Manager. However, RoCEv2 ports do not support InfiniBand
Subnet Management. Therefore, compliance statement C17-7 does not
apply to RoCEv2 ports.
Methods for setting the P_Key table associated with a RoCEv2 port are
not defined in this specification, except for the requirements for a default
P_Key described elsewhere in this annex.

rxe =  Software RDMA over Ethernet, so we are dealing only with RoCE traffic (no IB).
We don't have an SM.
The spec requires that at least the default pkey is defined (and rxe defines only the default p_pkey).
Kamel is doing just that.

Mark

> 
> Zhu Yanjun
> 
>>
>> What I did in this patch is to verify that the pkey value in the
>> received packet is the default P_Key regardless of the qpn, because RoCE
>> devices should maintain only the default P_Key.
>>
>> Thanks,
>> Kamal
>>
>>>> Thanks,
>>>> Kamal
>>>>
>>>>>> Methods for setting the P_Key table associated with a RoCEv2 port are
>>>>>> not defined in this specification, except for the requirements for a
>>>>>> default P_Key described elsewhere in this annex.
>>>>>> """
>>>>>>
>>>>>> Thanks,
>>>>>> Kamal
>>>>>>
>>>>>>
>>>>>>>> rping uses RDMA CM which goes over the GSI
>>>>>>>>
>>>>>>>> Jason
>>>
> 

  reply index

Thread overview: 16+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2020-07-21 10:16 Kamal Heib
     [not found] ` <AM6PR05MB6263CFB337190B1740CDF4B7D8780@AM6PR05MB6263.eurprd05.prod.outlook.com>
2020-07-22  2:09   ` FW: " Zhu Yanjun
2020-07-23  5:57     ` Kamal Heib
2020-07-23  6:58       ` Zhu Yanjun
2020-07-23  7:25         ` Kamal Heib
2020-07-23 13:08           ` Zhu Yanjun
2020-07-23 13:15             ` Jason Gunthorpe
2020-07-23 15:15               ` Zhu Yanjun
2020-07-28  8:35                 ` Kamal Heib
2020-07-28 13:21                   ` Zhu Yanjun
2020-07-28 13:44                     ` Kamal Heib
2020-07-28 15:46                       ` Zhu Yanjun
2020-07-28 17:42                         ` Kamal Heib
2020-07-28 23:45                           ` Zhu Yanjun
2020-07-29  1:36                             ` Mark Bloch [this message]
2020-07-31 19:22 ` Jason Gunthorpe

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=9e2beff9-ff0a-a6e6-c4b5-621650480044@mellanox.com \
    --to=markb@mellanox.com \
    --cc=dledford@redhat.com \
    --cc=jgg@ziepe.ca \
    --cc=kamalheib1@gmail.com \
    --cc=linux-rdma@vger.kernel.org \
    --cc=yanjunz@mellanox.com \
    --cc=zyjzyj2000@gmail.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

Linux-RDMA Archive on lore.kernel.org

Archives are clonable:
	git clone --mirror https://lore.kernel.org/linux-rdma/0 linux-rdma/git/0.git

	# If you have public-inbox 1.1+ installed, you may
	# initialize and index your mirror using the following commands:
	public-inbox-init -V2 linux-rdma linux-rdma/ https://lore.kernel.org/linux-rdma \
		linux-rdma@vger.kernel.org
	public-inbox-index linux-rdma

Example config snippet for mirrors

Newsgroup available over NNTP:
	nntp://nntp.lore.kernel.org/org.kernel.vger.linux-rdma


AGPL code for this site: git clone https://public-inbox.org/public-inbox.git