From: Kamal Heib <kamalheib1@gmail.com>
To: Zhu Yanjun <zyjzyj2000@gmail.com>
Cc: Jason Gunthorpe <jgg@ziepe.ca>, Yanjun Zhu <yanjunz@mellanox.com>,
linux-rdma@vger.kernel.org, Doug Ledford <dledford@redhat.com>
Subject: Re: FW: [PATCH for-next] RDMA/rxe: Remove pkey table
Date: Tue, 28 Jul 2020 20:42:25 +0300 [thread overview]
Message-ID: <20200728174225.GA52282@kheib-workstation> (raw)
In-Reply-To: <93160a8d-fca7-defc-b39e-e6e5a97ddb87@gmail.com>
On Tue, Jul 28, 2020 at 11:46:36PM +0800, Zhu Yanjun wrote:
> On 7/28/2020 9:44 PM, Kamal Heib wrote:
> > On Tue, Jul 28, 2020 at 09:21:06PM +0800, Zhu Yanjun wrote:
> > > On 7/28/2020 4:35 PM, Kamal Heib wrote:
> > > > On Thu, Jul 23, 2020 at 11:15:00PM +0800, Zhu Yanjun wrote:
> > > > > On 7/23/2020 9:15 PM, Jason Gunthorpe wrote:
> > > > > > On Thu, Jul 23, 2020 at 09:08:39PM +0800, Zhu Yanjun wrote:
> > > > > > > On 7/23/2020 3:25 PM, Kamal Heib wrote:
> > > > > > > > On Thu, Jul 23, 2020 at 02:58:41PM +0800, Zhu Yanjun wrote:
> > > > > > > > > On 7/23/2020 1:57 PM, Kamal Heib wrote:
> > > > > > > > > > On Wed, Jul 22, 2020 at 10:09:04AM +0800, Zhu Yanjun wrote:
> > > > > > > > > > > On Tue, Jul 21, 2020 at 7:28 PM Yanjun Zhu <yanjunz@mellanox.com> wrote:
> > > > > > > > > > > > From: Kamal Heib <kamalheib1@gmail.com>
> > > > > > > > > > > > Sent: Tuesday, July 21, 2020 6:16 PM
> > > > > > > > > > > > To: linux-rdma@vger.kernel.org
> > > > > > > > > > > > Cc: Yanjun Zhu <yanjunz@mellanox.com>; Doug Ledford <dledford@redhat.com>; Jason Gunthorpe <jgg@ziepe.ca>; Kamal Heib <kamalheib1@gmail.com>
> > > > > > > > > > > > Subject: [PATCH for-next] RDMA/rxe: Remove pkey table
> > > > > > > > > > > >
> > > > > > > > > > > > The RoCE spec require from RoCE devices to support only the defualt pkey, While the rxe driver maintain a 64 enties pkey table and use only the first entry. With that said remove the maintaing of the pkey table and used the default pkey when needed.
> > > > > > > > > > > >
> > > > > > > > > > > Hi Kamal
> > > > > > > > > > >
> > > > > > > > > > > After this patch is applied, do you make tests with SoftRoCE and mlx hardware?
> > > > > > > > > > >
> > > > > > > > > > > The SoftRoCE should work well with the mlx hardware.
> > > > > > > > > > >
> > > > > > > > > > > Zhu Yanjun
> > > > > > > > > > >
> > > > > > > > > > Hi Zhu,
> > > > > > > > > >
> > > > > > > > > > Yes, please see below:
> > > > > > > > > >
> > > > > > > > > > $ ibv_rc_pingpong -d mlx5_0 -g 11
> > > > > > > > > > local address: LID 0x0000, QPN 0x0000e3, PSN 0x728a4f, GID ::ffff:172.31.40.121
> > > > > > > > > Can you make tests with GSI QP?
> > > > > > > > >
> > > > > > > > > Zhu Yanjun
> > > > > > > > >
> > > > > > > Is this the GSI ?
> > > > > > >
> > > > > > > Please check GSI in "InfiniBandTM Architecture Specification Volume 1
> > > > > > > Release 1.3"
> > > > > > >
> > > > > > > Then make tests with GSI again.
> > > > > The followings are also removed by this commit. Not sure if it is good.
> > > > >
> > > > > "
> > > > >
> > > > > C9-42: If the destination QP is QP1, the BTH:P_Key shall be compared to the
> > > > > set of P_Keys associated with the port on which the packet arrived. If the
> > > > > P_Key matches any of the keys associated with the port, it shall be
> > > > > considered valid.
> > > > >
> > > > > "
> > > > >
> > > > The above is correct for ports that configured to work in InfiniBand
> > > > mode, while in RoCEv2 mode only the default P_Key should be associated
> > > > with the port (Please see below from "ANNEX A17: ROCEV2 (IP ROUTABLE
> > > > ROCE)):
> > > >
> > > > """
> > > > 17.7.1 LOADING THE P_KEY TABLE
> > > >
> > > > Compliance statement C17-7: on page 1193 describes requirements for
> > > > setting the P_Key table based on an assumption that the P_Key table is
> > > > set directly by a Subnet Manager. However, RoCEv2 ports do not support
> > > > InfiniBand Subnet Management. Therefore, compliance statement C17-7:
> > > > on page 1193 does not apply to RoCEv2 ports.
> > > "
> > >
> > > C17-7: An HCA shall require no OS involvement to set the P_Key table;
> > >
> > > the P_Key table shall be set directly by Subnet Manager MADs.
> > >
> > > "
> > >
> > > In SoftRoCE, what set the P_Key table?
> > >
> > No one is setting the P_Key table in SoftRoCE, and no subnet manager in
> > the RoCE fabric.
> >
> > Could you please tell me what is wrong with this patch?
>
> Please read the mail thread again.
>
> GSI QP number is 1. In your commits, the handle of qpn == 1 is removed.
>
> It seems that it conflicts with IB specification.
>
> Not sure if it is good.
>
Could you please read my patch again and point to what do you think is
wrong?
What I did in this patch is to verify that the pkey value in the
received packet is the default P_Key regardless of the qpn, because RoCE
devices should maintain only the default P_Key.
Thanks,
Kamal
> >
> > Thanks,
> > Kamal
> >
> > > > Methods for setting the P_Key table associated with a RoCEv2 port are
> > > > not defined in this specification, except for the requirements for a
> > > > default P_Key described elsewhere in this annex.
> > > > """
> > > >
> > > > Thanks,
> > > > Kamal
> > > >
> > > >
> > > > > > rping uses RDMA CM which goes over the GSI
> > > > > >
> > > > > > Jason
>
>
next prev parent reply other threads:[~2020-07-28 17:42 UTC|newest]
Thread overview: 16+ messages / expand[flat|nested] mbox.gz Atom feed top
2020-07-21 10:16 [PATCH for-next] RDMA/rxe: Remove pkey table Kamal Heib
[not found] ` <AM6PR05MB6263CFB337190B1740CDF4B7D8780@AM6PR05MB6263.eurprd05.prod.outlook.com>
2020-07-22 2:09 ` FW: " Zhu Yanjun
2020-07-23 5:57 ` Kamal Heib
2020-07-23 6:58 ` Zhu Yanjun
2020-07-23 7:25 ` Kamal Heib
2020-07-23 13:08 ` Zhu Yanjun
2020-07-23 13:15 ` Jason Gunthorpe
2020-07-23 15:15 ` Zhu Yanjun
2020-07-28 8:35 ` Kamal Heib
2020-07-28 13:21 ` Zhu Yanjun
2020-07-28 13:44 ` Kamal Heib
2020-07-28 15:46 ` Zhu Yanjun
2020-07-28 17:42 ` Kamal Heib [this message]
2020-07-28 23:45 ` Zhu Yanjun
2020-07-29 1:36 ` Mark Bloch
2020-07-31 19:22 ` Jason Gunthorpe
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20200728174225.GA52282@kheib-workstation \
--to=kamalheib1@gmail.com \
--cc=dledford@redhat.com \
--cc=jgg@ziepe.ca \
--cc=linux-rdma@vger.kernel.org \
--cc=yanjunz@mellanox.com \
--cc=zyjzyj2000@gmail.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).