[PATCH for-next 00/13] Implement the xrc transport

[PATCH for-next 00/13] Implement the xrc transport

[Bob Pearson]

This patch series implements the xrc transport for the rdma_rxe driver.
It is based on the current for-next branch of rdma-linux.
The first two patches in the series do some cleanup which is helpful
for this effort. The remaining patches implement the xrc functionality.
There is a matching patch set for the user space rxe provider driver.
The communications between these is accomplished without making an
ABI change by taking advantage of the space freed up by a recent
patch called "Remove redundant num_sge fields" which is a reprequisite
for this patch series.

The two patch sets have been tested with the pyverbs regression test
suite with and without each set installed. This series enables 5 of
the 6 xrc test cases in pyverbs. The ODP case does is currently skipped
but should work once the ODP patch series is accepted.

Signed-off-by: Bob Pearson <rpearsonhpe@gmail.com>

Bob Pearson (13):
  RDMA/rxe: Replace START->FIRST, END->LAST
  RDMA/rxe: Move next_opcode() to rxe_opcode.c
  RDMA: Add xrc opcodes to ib_pack.h
  RDMA/rxe: Extend opcodes and headers to support xrc
  RDMA/rxe: Add xrc opcodes to next_opcode()
  RDMA/rxe: Implement open_xrcd and close_xrcd
  RDMA/rxe: Extend srq verbs to support xrcd
  RDMA/rxe: Extend rxe_qp.c to support xrc qps
  RDMA/rxe: Extend rxe_recv.c to support xrc
  RDMA/rxe: Extend rxe_comp.c to support xrc qps
  RDMA/rxe: Extend rxe_req.c to support xrc qps
  RDMA/rxe: Extend rxe_net.c to support xrc qps
  RDMA/rxe: Extend rxe_resp.c to support xrc qps

 drivers/infiniband/sw/rxe/rxe.c        |   2 +
 drivers/infiniband/sw/rxe/rxe_av.c     |   3 +-
 drivers/infiniband/sw/rxe/rxe_comp.c   |  51 +-
 drivers/infiniband/sw/rxe/rxe_hdr.h    |  41 +-
 drivers/infiniband/sw/rxe/rxe_loc.h    |  13 +-
 drivers/infiniband/sw/rxe/rxe_mw.c     |  14 +-
 drivers/infiniband/sw/rxe/rxe_net.c    |  23 +-
 drivers/infiniband/sw/rxe/rxe_opcode.c | 766 +++++++++++++++++++++----
 drivers/infiniband/sw/rxe/rxe_opcode.h |   9 +-
 drivers/infiniband/sw/rxe/rxe_param.h  |   3 +
 drivers/infiniband/sw/rxe/rxe_pool.c   |   8 +
 drivers/infiniband/sw/rxe/rxe_pool.h   |   1 +
 drivers/infiniband/sw/rxe/rxe_qp.c     | 307 ++++++----
 drivers/infiniband/sw/rxe/rxe_recv.c   |  79 ++-
 drivers/infiniband/sw/rxe/rxe_req.c    | 204 +------
 drivers/infiniband/sw/rxe/rxe_resp.c   | 165 ++++--
 drivers/infiniband/sw/rxe/rxe_srq.c    | 131 +++--
 drivers/infiniband/sw/rxe/rxe_verbs.c  |  58 +-
 drivers/infiniband/sw/rxe/rxe_verbs.h  |  20 +-
 include/rdma/ib_pack.h                 |  32 +-
 include/uapi/rdma/rdma_user_rxe.h      |   4 +-
 21 files changed, 1338 insertions(+), 596 deletions(-)


base-commit: db77d84cfe3608eac938302f8f7178e44415bcba
-- 
2.34.1

RE: [PATCH for-next 00/13] Implement the xrc transport

[matsuda-daisuke]

On Fri, Oct 14, 2022 2:18 AM Bob Pearson wrote:
> On 10/12/22 02:41, matsuda-daisuke@fujitsu.com wrote:
> > Hi Bob,
> >
> > I am ready and willing to review your workqueue implementation.
> > Could you inform me of the current status of the patch series?
> > Are you having trouble rebasing it?
> >
> > Daisuke
> >
> >>>> The ODP patch series is the one I posted for rxe this month:
> >>>>   [RFC PATCH 0/7] RDMA/rxe: On-Demand Paging on SoftRoCE
> >>>>   https://lore.kernel.org/lkml/cover.1662461897.git.matsuda-daisuke@fujitsu.com/
> >>>>   https://patchwork.kernel.org/project/linux-rdma/list/?series=674699&state=%2A&archive=both
> >>>>
> >>>> We had an argument about the way to use workqueues instead of tasklets.
> >>>> Some prefer to get rid of tasklets, but others prefer finding a way to switch
> >>>> between the bottom halves for performance reasons. I am currently taking
> >>>> some data to continue the discussion while waiting for Bob to post their(HPE's)
> >>>> implementation that enables the switching. I think I can resend the ODP patches
> >>>> without an RFC tag once we reach an agreement on this point.
> >>>
> >>> I tried to get Ian Ziemba, who wrote the patch series, to send it in but he is very busy
> >>> and finally after a long while he asked me to do that. I have to rebase it from
> >>> an older kernel to head of tree. I hope to have that done in a day or two.
> >>>
> >>> Bob
> >>
> >> Thank you for the update.
> >> I am glad to hear your work is in progress just now.
> >> I am looking forward to seeing your work!
> >>
> >> Daisuke
> >
> 
> I know. Thanks. It's ready to send in. I am testing the performance and was surprised that the tasklet performance has
> gotten better in 6.0. Not sure why. I'll send it today.
> 
> Bob

Thank you for the reply.
The performance change sounds interesting.

BTW, would you please send it in as soon as possible?
I would like to see if I can easily rebase my ODP patches on it,
and also I will start testing and reviewing your change so that
it will be merged safely and quickly.

Daisuke

Re: [PATCH for-next 00/13] Implement the xrc transport

[Bob Pearson]

On 10/12/22 02:41, matsuda-daisuke@fujitsu.com wrote:
> Hi Bob,
> 
> I am ready and willing to review your workqueue implementation.
> Could you inform me of the current status of the patch series?
> Are you having trouble rebasing it? 
> 
> Daisuke
> 
>>>> The ODP patch series is the one I posted for rxe this month:
>>>>   [RFC PATCH 0/7] RDMA/rxe: On-Demand Paging on SoftRoCE
>>>>   https://lore.kernel.org/lkml/cover.1662461897.git.matsuda-daisuke@fujitsu.com/
>>>>   https://patchwork.kernel.org/project/linux-rdma/list/?series=674699&state=%2A&archive=both
>>>>
>>>> We had an argument about the way to use workqueues instead of tasklets.
>>>> Some prefer to get rid of tasklets, but others prefer finding a way to switch
>>>> between the bottom halves for performance reasons. I am currently taking
>>>> some data to continue the discussion while waiting for Bob to post their(HPE's)
>>>> implementation that enables the switching. I think I can resend the ODP patches
>>>> without an RFC tag once we reach an agreement on this point.
>>>
>>> I tried to get Ian Ziemba, who wrote the patch series, to send it in but he is very busy
>>> and finally after a long while he asked me to do that. I have to rebase it from
>>> an older kernel to head of tree. I hope to have that done in a day or two.
>>>
>>> Bob
>>
>> Thank you for the update.
>> I am glad to hear your work is in progress just now.
>> I am looking forward to seeing your work!
>>
>> Daisuke
> 

I know. Thanks. It's ready to send in. I am testing the performance and was surprised that the tasklet performance has gotten better in 6.0. Not sure why. I'll send it today.

Bob

Re: [PATCH for-next 00/13] Implement the xrc transport

[matsuda-daisuke]

Hi Bob,

I am ready and willing to review your workqueue implementation.
Could you inform me of the current status of the patch series?
Are you having trouble rebasing it? 

Daisuke

> > > The ODP patch series is the one I posted for rxe this month:
> > >   [RFC PATCH 0/7] RDMA/rxe: On-Demand Paging on SoftRoCE
> > >   https://lore.kernel.org/lkml/cover.1662461897.git.matsuda-daisuke@fujitsu.com/
> > >   https://patchwork.kernel.org/project/linux-rdma/list/?series=674699&state=%2A&archive=both
> > >
> > > We had an argument about the way to use workqueues instead of tasklets.
> > > Some prefer to get rid of tasklets, but others prefer finding a way to switch
> > > between the bottom halves for performance reasons. I am currently taking
> > > some data to continue the discussion while waiting for Bob to post their(HPE's)
> > > implementation that enables the switching. I think I can resend the ODP patches
> > > without an RFC tag once we reach an agreement on this point.
> >
> > I tried to get Ian Ziemba, who wrote the patch series, to send it in but he is very busy
> > and finally after a long while he asked me to do that. I have to rebase it from
> > an older kernel to head of tree. I hope to have that done in a day or two.
> >
> > Bob
> 
> Thank you for the update.
> I am glad to hear your work is in progress just now.
> I am looking forward to seeing your work!
> 
> Daisuke

RE: [PATCH for-next 00/13] Implement the xrc transport

[matsuda-daisuke]

On Fri, Sep 30, 2022 12:59 AM Bob Pearson wrote:
> On 9/26/22 20:38, matsuda-daisuke@fujitsu.com wrote:
> > On Tue, Sep 27, 2022 8:14 AM, Jason Gunthorpe wrote:
> >> On Fri, Sep 16, 2022 at 10:10:51PM -0500, Bob Pearson wrote:
> >>> This patch series implements the xrc transport for the rdma_rxe driver.
> >>> It is based on the current for-next branch of rdma-linux.
> >>> The first two patches in the series do some cleanup which is helpful
> >>> for this effort. The remaining patches implement the xrc functionality.
> >>> There is a matching patch set for the user space rxe provider driver.
> >>> The communications between these is accomplished without making an
> >>> ABI change by taking advantage of the space freed up by a recent
> >>> patch called "Remove redundant num_sge fields" which is a reprequisite
> >>> for this patch series.
> >>>
> >>> The two patch sets have been tested with the pyverbs regression test
> >>> suite with and without each set installed. This series enables 5 of
> >>> the 6 xrc test cases in pyverbs. The ODP case does is currently skipped
> >>> but should work once the ODP patch series is accepted.
> >>
> >> The ODP patch isn't even on patchworks any more, so it needs
> >> resending. I can't remember why it needed respin now.
> >
> > The ODP patch series is the one I posted for rxe this month:
> >   [RFC PATCH 0/7] RDMA/rxe: On-Demand Paging on SoftRoCE
> >   https://lore.kernel.org/lkml/cover.1662461897.git.matsuda-daisuke@fujitsu.com/
> >   https://patchwork.kernel.org/project/linux-rdma/list/?series=674699&state=%2A&archive=both
> >
> > We had an argument about the way to use workqueues instead of tasklets.
> > Some prefer to get rid of tasklets, but others prefer finding a way to switch
> > between the bottom halves for performance reasons. I am currently taking
> > some data to continue the discussion while waiting for Bob to post their(HPE's)
> > implementation that enables the switching. I think I can resend the ODP patches
> > without an RFC tag once we reach an agreement on this point.
> 
> I tried to get Ian Ziemba, who wrote the patch series, to send it in but he is very busy
> and finally after a long while he asked me to do that. I have to rebase it from
> an older kernel to head of tree. I hope to have that done in a day or two.
> 
> Bob

Thank you for the update.
I am glad to hear your work is in progress just now.
I am looking forward to seeing your work!

Daisuke

> >
> > Thanks,
> > Daisuke Matsuda
> >
> >>
> >> I'm inclined to apply this without really looking closely at the rxe
> >> code. If someone has other ideas please chime in. It looks like it
> >> needs rebasing, yes?
> >>
> >> Jason
> 

Re: [PATCH for-next 00/13] Implement the xrc transport

[Bob Pearson]

On 9/26/22 18:13, Jason Gunthorpe wrote:
> On Fri, Sep 16, 2022 at 10:10:51PM -0500, Bob Pearson wrote:
>> This patch series implements the xrc transport for the rdma_rxe driver.
>> It is based on the current for-next branch of rdma-linux.
>> The first two patches in the series do some cleanup which is helpful
>> for this effort. The remaining patches implement the xrc functionality.
>> There is a matching patch set for the user space rxe provider driver.
>> The communications between these is accomplished without making an
>> ABI change by taking advantage of the space freed up by a recent
>> patch called "Remove redundant num_sge fields" which is a reprequisite
>> for this patch series.
>>
>> The two patch sets have been tested with the pyverbs regression test
>> suite with and without each set installed. This series enables 5 of
>> the 6 xrc test cases in pyverbs. The ODP case does is currently skipped
>> but should work once the ODP patch series is accepted.
> 
> The ODP patch isn't even on patchworks any more, so it needs
> resending. I can't remember why it needed respin now.
> 
> I'm inclined to apply this without really looking closely at the rxe
> code. If someone has other ideas please chime in. It looks like it
> needs rebasing, yes?

I will do that today and resend if it needs to change.

Bob
> 
> Jason

Re: [PATCH for-next 00/13] Implement the xrc transport

[Bob Pearson]

On 9/26/22 20:38, matsuda-daisuke@fujitsu.com wrote:
> On Tue, Sep 27, 2022 8:14 AM, Jason Gunthorpe wrote:
>> On Fri, Sep 16, 2022 at 10:10:51PM -0500, Bob Pearson wrote:
>>> This patch series implements the xrc transport for the rdma_rxe driver.
>>> It is based on the current for-next branch of rdma-linux.
>>> The first two patches in the series do some cleanup which is helpful
>>> for this effort. The remaining patches implement the xrc functionality.
>>> There is a matching patch set for the user space rxe provider driver.
>>> The communications between these is accomplished without making an
>>> ABI change by taking advantage of the space freed up by a recent
>>> patch called "Remove redundant num_sge fields" which is a reprequisite
>>> for this patch series.
>>>
>>> The two patch sets have been tested with the pyverbs regression test
>>> suite with and without each set installed. This series enables 5 of
>>> the 6 xrc test cases in pyverbs. The ODP case does is currently skipped
>>> but should work once the ODP patch series is accepted.
>>
>> The ODP patch isn't even on patchworks any more, so it needs
>> resending. I can't remember why it needed respin now.
> 
> The ODP patch series is the one I posted for rxe this month:
>   [RFC PATCH 0/7] RDMA/rxe: On-Demand Paging on SoftRoCE
>   https://lore.kernel.org/lkml/cover.1662461897.git.matsuda-daisuke@fujitsu.com/
>   https://patchwork.kernel.org/project/linux-rdma/list/?series=674699&state=%2A&archive=both
> 
> We had an argument about the way to use workqueues instead of tasklets.
> Some prefer to get rid of tasklets, but others prefer finding a way to switch
> between the bottom halves for performance reasons. I am currently taking
> some data to continue the discussion while waiting for Bob to post their(HPE's)
> implementation that enables the switching. I think I can resend the ODP patches
> without an RFC tag once we reach an agreement on this point.

I tried to get Ian Ziemba, who wrote the patch series, to send it in but he is very busy
and finally after a long while he asked me to do that. I have to rebase it from
an older kernel to head of tree. I hope to have that done in a day or two.

Bob
> 
> Thanks,
> Daisuke Matsuda
> 
>>
>> I'm inclined to apply this without really looking closely at the rxe
>> code. If someone has other ideas please chime in. It looks like it
>> needs rebasing, yes?
>>
>> Jason

RE: [PATCH for-next 00/13] Implement the xrc transport

[matsuda-daisuke]

On Tue, Sep 27, 2022 8:14 AM, Jason Gunthorpe wrote:
> On Fri, Sep 16, 2022 at 10:10:51PM -0500, Bob Pearson wrote:
> > This patch series implements the xrc transport for the rdma_rxe driver.
> > It is based on the current for-next branch of rdma-linux.
> > The first two patches in the series do some cleanup which is helpful
> > for this effort. The remaining patches implement the xrc functionality.
> > There is a matching patch set for the user space rxe provider driver.
> > The communications between these is accomplished without making an
> > ABI change by taking advantage of the space freed up by a recent
> > patch called "Remove redundant num_sge fields" which is a reprequisite
> > for this patch series.
> >
> > The two patch sets have been tested with the pyverbs regression test
> > suite with and without each set installed. This series enables 5 of
> > the 6 xrc test cases in pyverbs. The ODP case does is currently skipped
> > but should work once the ODP patch series is accepted.
> 
> The ODP patch isn't even on patchworks any more, so it needs
> resending. I can't remember why it needed respin now.

The ODP patch series is the one I posted for rxe this month:
  [RFC PATCH 0/7] RDMA/rxe: On-Demand Paging on SoftRoCE
  https://lore.kernel.org/lkml/cover.1662461897.git.matsuda-daisuke@fujitsu.com/
  https://patchwork.kernel.org/project/linux-rdma/list/?series=674699&state=%2A&archive=both

We had an argument about the way to use workqueues instead of tasklets.
Some prefer to get rid of tasklets, but others prefer finding a way to switch
between the bottom halves for performance reasons. I am currently taking
some data to continue the discussion while waiting for Bob to post their(HPE's)
implementation that enables the switching. I think I can resend the ODP patches
without an RFC tag once we reach an agreement on this point.

Thanks,
Daisuke Matsuda

> 
> I'm inclined to apply this without really looking closely at the rxe
> code. If someone has other ideas please chime in. It looks like it
> needs rebasing, yes?
> 
> Jason

Re: [PATCH for-next 00/13] Implement the xrc transport

[Jason Gunthorpe]

On Fri, Sep 16, 2022 at 10:10:51PM -0500, Bob Pearson wrote:
> This patch series implements the xrc transport for the rdma_rxe driver.
> It is based on the current for-next branch of rdma-linux.
> The first two patches in the series do some cleanup which is helpful
> for this effort. The remaining patches implement the xrc functionality.
> There is a matching patch set for the user space rxe provider driver.
> The communications between these is accomplished without making an
> ABI change by taking advantage of the space freed up by a recent
> patch called "Remove redundant num_sge fields" which is a reprequisite
> for this patch series.
> 
> The two patch sets have been tested with the pyverbs regression test
> suite with and without each set installed. This series enables 5 of
> the 6 xrc test cases in pyverbs. The ODP case does is currently skipped
> but should work once the ODP patch series is accepted.

The ODP patch isn't even on patchworks any more, so it needs
resending. I can't remember why it needed respin now.

I'm inclined to apply this without really looking closely at the rxe
code. If someone has other ideas please chime in. It looks like it
needs rebasing, yes?

Jason

[PATCH for-next 00/13] Implement the xrc transport

[Bob Pearson]

This patch series implements the xrc transport for the rdma_rxe driver.
It is based on the current for-next branch of rdma-linux.
The first two patches in the series do some cleanup which is helpful
for this effort. The remaining patches implement the xrc functionality.
There is a matching patch set for the user space rxe provider driver.
The communications between these is accomplished without making an
ABI change by taking advantage of the space freed up by a recent
patch called "Remove redundant num_sge fields" which is a reprequisite
for this patch series.

The two patch sets have been tested with the pyverbs regression test
suite with and without each set installed. This series enables 5 of
the 6 xrc test cases in pyverbs. The ODP case does is currently skipped
but should work once the ODP patch series is accepted.

Signed-off-by: Bob Pearson <rpearsonhpe@gmail.com>

Bob Pearson (13):
  RDMA/rxe: Replace START->FIRST, END->LAST
  RDMA/rxe: Move next_opcode() to rxe_opcode.c
  RDMA: Add xrc opcodes to ib_pack.h
  RDMA/rxe: Extend opcodes and headers to support xrc
  RDMA/rxe: Add xrc opcodes to next_opcode()
  RDMA/rxe: Implement open_xrcd and close_xrcd
  RDMA/rxe: Extend srq verbs to support xrcd
  RDMA/rxe: Extend rxe_qp.c to support xrc qps
  RDMA/rxe: Extend rxe_recv.c to support xrc
  RDMA/rxe: Extend rxe_comp.c to support xrc qps
  RDMA/rxe: Extend rxe_req.c to support xrc qps
  RDMA/rxe: Extend rxe_net.c to support xrc qps
  RDMA/rxe: Extend rxe_resp.c to support xrc qps

 drivers/infiniband/sw/rxe/rxe.c        |   2 +
 drivers/infiniband/sw/rxe/rxe_av.c     |   3 +-
 drivers/infiniband/sw/rxe/rxe_comp.c   |  51 +-
 drivers/infiniband/sw/rxe/rxe_hdr.h    |  41 +-
 drivers/infiniband/sw/rxe/rxe_loc.h    |  13 +-
 drivers/infiniband/sw/rxe/rxe_mw.c     |  14 +-
 drivers/infiniband/sw/rxe/rxe_net.c    |  23 +-
 drivers/infiniband/sw/rxe/rxe_opcode.c | 766 +++++++++++++++++++++----
 drivers/infiniband/sw/rxe/rxe_opcode.h |   9 +-
 drivers/infiniband/sw/rxe/rxe_param.h  |   3 +
 drivers/infiniband/sw/rxe/rxe_pool.c   |   8 +
 drivers/infiniband/sw/rxe/rxe_pool.h   |   1 +
 drivers/infiniband/sw/rxe/rxe_qp.c     | 307 ++++++----
 drivers/infiniband/sw/rxe/rxe_recv.c   |  79 ++-
 drivers/infiniband/sw/rxe/rxe_req.c    | 204 +------
 drivers/infiniband/sw/rxe/rxe_resp.c   | 165 ++++--
 drivers/infiniband/sw/rxe/rxe_srq.c    | 131 +++--
 drivers/infiniband/sw/rxe/rxe_verbs.c  |  58 +-
 drivers/infiniband/sw/rxe/rxe_verbs.h  |  20 +-
 include/rdma/ib_pack.h                 |  32 +-
 include/uapi/rdma/rdma_user_rxe.h      |   4 +-
 21 files changed, 1338 insertions(+), 596 deletions(-)


base-commit: db77d84cfe3608eac938302f8f7178e44415bcba
-- 
2.34.1