[PATCH 0/3] RDMA/core: iWARP Port Mapper Overview

[PATCH 0/3] RDMA/core: iWARP Port Mapper Overview

[Tatyana Nikolova]

Hello All,

This patch series adds iWARP Port Mapper (IWPM) Version 2 support
in RDMA/core, RDMA/nes driver and RDMA/cxgb4 driver.
The iWARP Port Mapper implementation is based on the port mapper specification
section in the Sockets Direct Protocol paper -
http://www.rdmaconsortium.org/home/draft-pinkerton-iwarp-sdp-v1.0.pdf 

Existing iWARP RDMA providers use the same IP address as the native TCP/IP
stack when creating RDMA connections. They need to ensure a mechanism to claim 
the TCP ports used for RDMA connections, thereby preventing TCP port collisions
when other host applications are consuming TCP ports. The iWARP Port Mapper
provides a standard mechanism to accomplish this. Without this service it is
possible for RDMA application to bind/listen on the same port which is already
being used by native TCP host application. If that happens the incoming TCP
connection data can be passed to the RDMA stack with error.

The iWARP Port Mapper solution doesn't contain any changes to the existing
network stack in the kernel space. All the changes are contained with the
infiniband tree and also in user space.

The iWARP Port Mapper service is implemented as a user space daemon process.
Interested readers could look at the source code of the IWPM service located at
http://git.openfabrics.org/git?p=~tnikolova/libiwpm-1.0.0/.git;a=summary

The iWARP driver (port mapper client) sends to the IWPM service the local IP
address and TCP port it has received from the RDMA application, when starting a
connection. The IWPM service performs a socket bind from user space to get
an available TCP port, called a mapped port, and communicates it back to the
client. In that sense, the IWPM service is used to map the TCP port, which the
RDMA application uses to any port available from the host TCP port space. The
mapped ports are used in iWARP RDMA connections to avoid collisions with native
TCP stack which is aware that these ports are taken. When an RDMA connection
using a mapped port is terminated, the client notifies the IWPM service, which
then releases the TCP port.

The message exchange between the IWPM service and the iWARP drivers
(between user space and kernel space) is implemented using netlink sockets.

This patch series adds iWarp driver support for sending/receiving/parsing
netlink messages, to make the communication with the IWPM service possible.

This is the second submission for the iWarp Port Mapper and
we have considered and incorporated feedback after the first submission
to improve the IWPM Version 2 patch series.

The following items are improvements and fixes to IWPM V2:

1) The IWPM functionality, common for both iWarp drivers (nes and cxgb4)
   is refactored from the drivers source files and is moved to new shared
   files in infiniband/core which are compiled as part of the iw_cm module.

2) Hash table data structure is implemented to store the drivers mapping
   information and enable efficient searching. (Hash table implementation
   is based on original work by Vipul Pandya)

3) Fix for a timer bug is provided as the unnecessary timer scheduling is
   removed.

The patches are built against Roland's infiniband tree for-next branch.

Thank you,

Tatyana Nikolova (2):
  RDMA/core: Add support for iWarp Port Mapper V2 user space service
  RDMA/nes: Add support for iWarp Port Mapper V2 user space service
Steve Wise (1):
  RDMA/cxgb4: Add support for iWarp Port Mapper V2 user space service

 drivers/infiniband/core/Makefile       |    2 +-
 drivers/infiniband/core/cma.c          |    3 +-
 drivers/infiniband/core/iwpm_msg.c     |  691 ++++++++++++++++++++++++++++++++
 drivers/infiniband/core/iwpm_util.c    |  609 ++++++++++++++++++++++++++++
 drivers/infiniband/core/iwpm_util.h    |  238 +++++++++++
 drivers/infiniband/core/netlink.c      |   18 +-
 drivers/infiniband/hw/cxgb4/cm.c       |  180 +++++++--
 drivers/infiniband/hw/cxgb4/device.c   |   81 ++++-
 drivers/infiniband/hw/cxgb4/iw_cxgb4.h |   44 ++
 drivers/infiniband/hw/nes/nes.c        |   25 ++-
 drivers/infiniband/hw/nes/nes.h        |    3 +
 drivers/infiniband/hw/nes/nes_cm.c     |  320 ++++++++++++---
 drivers/infiniband/hw/nes/nes_cm.h     |   12 +-
 include/rdma/iw_portmap.h              |  199 +++++++++
 include/rdma/rdma_netlink.h            |   23 +-
 include/uapi/rdma/rdma_netlink.h       |   96 +++++-
 16 files changed, 2431 insertions(+), 113 deletions(-)
 create mode 100644 drivers/infiniband/core/iwpm_msg.c
 create mode 100644 drivers/infiniband/core/iwpm_util.c
 create mode 100644 drivers/infiniband/core/iwpm_util.h
 create mode 100644 include/rdma/iw_portmap.h

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: [PATCH 0/3] RDMA/core: iWARP Port Mapper Overview

[Steve Wise]

Any review comments on this?


> -----Original Message-----
> From: Tatyana Nikolova [mailto:Tatyana.E.Nikolova-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org]
> Sent: Wednesday, March 26, 2014 5:07 PM
> To: Roland Dreier
> Cc: john.s.lacombe-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org; swise-7bPotxP6k4+P2YhJcF5u+vpXobYPEAuW@public.gmane.org;
> peter.p.waskiewicz.jr-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org; sean.hefty-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org; linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> Subject: [PATCH 0/3] RDMA/core: iWARP Port Mapper Overview
> 
> 
> Hello All,
> 
> This patch series adds iWARP Port Mapper (IWPM) Version 2 support
> in RDMA/core, RDMA/nes driver and RDMA/cxgb4 driver.
> The iWARP Port Mapper implementation is based on the port mapper specification
> section in the Sockets Direct Protocol paper -
> http://www.rdmaconsortium.org/home/draft-pinkerton-iwarp-sdp-v1.0.pdf
> 
> Existing iWARP RDMA providers use the same IP address as the native TCP/IP
> stack when creating RDMA connections. They need to ensure a mechanism to claim
> the TCP ports used for RDMA connections, thereby preventing TCP port collisions
> when other host applications are consuming TCP ports. The iWARP Port Mapper
> provides a standard mechanism to accomplish this. Without this service it is
> possible for RDMA application to bind/listen on the same port which is already
> being used by native TCP host application. If that happens the incoming TCP
> connection data can be passed to the RDMA stack with error.
> 
> The iWARP Port Mapper solution doesn't contain any changes to the existing
> network stack in the kernel space. All the changes are contained with the
> infiniband tree and also in user space.
> 
> The iWARP Port Mapper service is implemented as a user space daemon process.
> Interested readers could look at the source code of the IWPM service located at
> http://git.openfabrics.org/git?p=~tnikolova/libiwpm-1.0.0/.git;a=summary
> 
> The iWARP driver (port mapper client) sends to the IWPM service the local IP
> address and TCP port it has received from the RDMA application, when starting a
> connection. The IWPM service performs a socket bind from user space to get
> an available TCP port, called a mapped port, and communicates it back to the
> client. In that sense, the IWPM service is used to map the TCP port, which the
> RDMA application uses to any port available from the host TCP port space. The
> mapped ports are used in iWARP RDMA connections to avoid collisions with native
> TCP stack which is aware that these ports are taken. When an RDMA connection
> using a mapped port is terminated, the client notifies the IWPM service, which
> then releases the TCP port.
> 
> The message exchange between the IWPM service and the iWARP drivers
> (between user space and kernel space) is implemented using netlink sockets.
> 
> This patch series adds iWarp driver support for sending/receiving/parsing
> netlink messages, to make the communication with the IWPM service possible.
> 
> This is the second submission for the iWarp Port Mapper and
> we have considered and incorporated feedback after the first submission
> to improve the IWPM Version 2 patch series.
> 
> The following items are improvements and fixes to IWPM V2:
> 
> 1) The IWPM functionality, common for both iWarp drivers (nes and cxgb4)
>    is refactored from the drivers source files and is moved to new shared
>    files in infiniband/core which are compiled as part of the iw_cm module.
> 
> 2) Hash table data structure is implemented to store the drivers mapping
>    information and enable efficient searching. (Hash table implementation
>    is based on original work by Vipul Pandya)
> 
> 3) Fix for a timer bug is provided as the unnecessary timer scheduling is
>    removed.
> 
> The patches are built against Roland's infiniband tree for-next branch.
> 
> Thank you,
> 
> Tatyana Nikolova (2):
>   RDMA/core: Add support for iWarp Port Mapper V2 user space service
>   RDMA/nes: Add support for iWarp Port Mapper V2 user space service
> Steve Wise (1):
>   RDMA/cxgb4: Add support for iWarp Port Mapper V2 user space service
> 
>  drivers/infiniband/core/Makefile       |    2 +-
>  drivers/infiniband/core/cma.c          |    3 +-
>  drivers/infiniband/core/iwpm_msg.c     |  691 ++++++++++++++++++++++++++++++++
>  drivers/infiniband/core/iwpm_util.c    |  609 ++++++++++++++++++++++++++++
>  drivers/infiniband/core/iwpm_util.h    |  238 +++++++++++
>  drivers/infiniband/core/netlink.c      |   18 +-
>  drivers/infiniband/hw/cxgb4/cm.c       |  180 +++++++--
>  drivers/infiniband/hw/cxgb4/device.c   |   81 ++++-
>  drivers/infiniband/hw/cxgb4/iw_cxgb4.h |   44 ++
>  drivers/infiniband/hw/nes/nes.c        |   25 ++-
>  drivers/infiniband/hw/nes/nes.h        |    3 +
>  drivers/infiniband/hw/nes/nes_cm.c     |  320 ++++++++++++---
>  drivers/infiniband/hw/nes/nes_cm.h     |   12 +-
>  include/rdma/iw_portmap.h              |  199 +++++++++
>  include/rdma/rdma_netlink.h            |   23 +-
>  include/uapi/rdma/rdma_netlink.h       |   96 +++++-
>  16 files changed, 2431 insertions(+), 113 deletions(-)
>  create mode 100644 drivers/infiniband/core/iwpm_msg.c
>  create mode 100644 drivers/infiniband/core/iwpm_util.c
>  create mode 100644 drivers/infiniband/core/iwpm_util.h
>  create mode 100644 include/rdma/iw_portmap.h

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH 0/3] RDMA/core: iWARP Port Mapper Overview

[Roland Dreier]

On Wed, Mar 26, 2014 at 3:07 PM, Tatyana Nikolova
<Tatyana.E.Nikolova-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org> wrote:
> 1) The IWPM functionality, common for both iWarp drivers (nes and cxgb4)
>    is refactored from the drivers source files and is moved to new shared
>    files in infiniband/core which are compiled as part of the iw_cm module.


So I'm looking to include this for 3.16.  One question: I assume
unpatched iwarp drivers (eg cxgb3) continue to work as before?  In
other words they don't get the new port mapper support but at least
they don't regress?

 - R.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH 0/3] RDMA/core: iWARP Port Mapper Overview

[Or Gerlitz]

On 27/03/2014 00:08, Tatyana Nikolova wrote:
> This patch series adds iWARP Port Mapper (IWPM) Version 2 support
> in RDMA/core, RDMA/nes driver and RDMA/cxgb4 driver.

Guys, the link here 
http://vger.kernel.org/netconf2011_slides/pj_netconf2011.ppt
seems to point to a damaged PPT, can you send me updated copy

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: [PATCH 0/3] RDMA/core: iWARP Port Mapper Overview

[Steve Wise]

> -----Original Message-----
> From: linux-rdma-owner-u79uwXL29TY76Z2rM5mHXA@public.gmane.org [mailto:linux-rdma-owner-u79uwXL29TY76Z2rM5mHXA@public.gmane.org] On
> Behalf Of Roland Dreier
> Sent: Monday, May 19, 2014 8:10 PM
> To: Tatyana Nikolova
> Cc: Lacombe, John S; Steve Wise; Sean Hefty; linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> Subject: Re: [PATCH 0/3] RDMA/core: iWARP Port Mapper Overview
> 
> On Wed, Mar 26, 2014 at 3:07 PM, Tatyana Nikolova
> <Tatyana.E.Nikolova-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org> wrote:
> > 1) The IWPM functionality, common for both iWarp drivers (nes and cxgb4)
> >    is refactored from the drivers source files and is moved to new shared
> >    files in infiniband/core which are compiled as part of the iw_cm module.
> 
> 
> So I'm looking to include this for 3.16.  One question: I assume
> unpatched iwarp drivers (eg cxgb3) continue to work as before?  In
> other words they don't get the new port mapper support but at least
> they don't regress?
> 

Hey Roland,

cxgb3<->cxgb3 would behave as before; no regression.  cxgb3<->cxgb4/nes would interoperate only if the port mapping service is disabled on the cxgb4/nes nodes.  

Steve.

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH 0/3] RDMA/core: iWARP Port Mapper Overview

[Or Gerlitz]

On 20/05/2014 16:46, Steve Wise wrote:
>   But here is it attached
thanks
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: [PATCH 0/3] RDMA/core: iWARP Port Mapper Overview

[Nikolova, Tatyana E]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain; charset="utf-8", Size: 1750 bytes --]

Hello Rolland,

Thanks for reviewing the port mapper patches.

Steve's reply is correct. 
Unpatched nes will not regress and will continue working as before. 

Tatyana

-----Original Message-----
From: Steve Wise [mailto:swise@opengridcomputing.com] 
Sent: Tuesday, May 20, 2014 8:45 AM
To: 'Roland Dreier'; Nikolova, Tatyana E
Cc: Lacombe, John S; Hefty, Sean; linux-rdma@vger.kernel.org; 'Doug Ledford'
Subject: RE: [PATCH 0/3] RDMA/core: iWARP Port Mapper Overview



> -----Original Message-----
> From: linux-rdma-owner@vger.kernel.org 
> [mailto:linux-rdma-owner@vger.kernel.org] On Behalf Of Roland Dreier
> Sent: Monday, May 19, 2014 8:10 PM
> To: Tatyana Nikolova
> Cc: Lacombe, John S; Steve Wise; Sean Hefty; 
> linux-rdma@vger.kernel.org
> Subject: Re: [PATCH 0/3] RDMA/core: iWARP Port Mapper Overview
> 
> On Wed, Mar 26, 2014 at 3:07 PM, Tatyana Nikolova 
> <Tatyana.E.Nikolova@intel.com> wrote:
> > 1) The IWPM functionality, common for both iWarp drivers (nes and cxgb4)
> >    is refactored from the drivers source files and is moved to new shared
> >    files in infiniband/core which are compiled as part of the iw_cm module.
> 
> 
> So I'm looking to include this for 3.16.  One question: I assume 
> unpatched iwarp drivers (eg cxgb3) continue to work as before?  In 
> other words they don't get the new port mapper support but at least 
> they don't regress?
> 

Hey Roland,

cxgb3<->cxgb3 would behave as before; no regression.  cxgb3<->cxgb4/nes would interoperate only if the port mapping service is disabled on the cxgb4/nes nodes.  

Steve.

N‹§²æìr¸›yúèšØb²X¬¶Ç§vØ^–)Þº{.nÇ+‰·¥Š{±­ÙšŠ{ayº\x1dʇڙë,j\a­¢f£¢·hš‹»öì\x17/oSc¾™Ú³9˜uÀ¦æå‰È&jw¨®\x03(­éšŽŠÝ¢j"ú\x1a¶^[m§ÿïêäz¹Þ–Šàþf£¢·hšˆ§~ˆmš

Re: [PATCH 0/3] RDMA/core: iWARP Port Mapper Overview

[Or Gerlitz]

On Tue, May 20, 2014 at 4:10 AM, Roland Dreier <roland-DgEjT+Ai2ygdnm+yROfE0A@public.gmane.org> wrote:

> So I'm looking to include this for 3.16.

Roland, can you reword these patches and remove Peter's email address,
it bounces back (just tried it, SB) and I don't see the point to
introduce an email in a  signature which is broken from day one.

Or.

Delivery to the following recipient failed permanently:

     peter.p.waskiewicz.jr-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org

Technical details of permanent failure:
Google tried to deliver your message, but it was rejected by the
server for the recipient domain intel.com by mga01.intel.com.
[192.55.52.88].

The error that the other server returned was:
550 #5.1.0 Address rejected.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH 0/3] RDMA/core: iWARP Port Mapper Overview

[Or Gerlitz]

Tatyana Nikolova <Tatyana.E.Nikolova-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org> wrote:

> This patch series adds iWARP Port Mapper (IWPM) Version 2 support
> in RDMA/core, RDMA/nes driver and RDMA/cxgb4 driver.

Hi Steve, Tatyana

Reading this cover letter, I think it's good idea to follow Steve's
suggestion and add portions from here into the IB/core change you
introduced (patch #1 in this series) and is now merged into Roland's
for-next -- before Roland's sends his 3.16 merge window pull request
to Linus (there is enough time!!), since the current change-log
http://git.kernel.org/cgit/linux/kernel/git/roland/infiniband.git/commit/?h=for-next&id=1943d56776ad31a119517efeb309786db3b4383e
practically says nothing on what is the PM about

2nd, there's nothing specific to iWARP in the port mapper implementation, right?

What it does is the following:

1. kernel rdma driver tells a user space daemon they want to reserve
(claim) the combination of IP address X and TCP port Y for the sake of
RDMA connections

2. user space daemon opens a socket and binds to X:Y

Specifically, down the road, more use cases, not only the current
iWARP case may pop up.

So taking into account that the implementation doesn't have anything
whivh is IW specific, and that future non-IW use case might show up,
why we need to fill the IB core code with IW_PM notation? why not just
use RDMA_PM notation? this can be achieved quickly with

$ sed -i g/iw_pm/rdma_pm/g *

and friends on Roland's tree and later squashed into the relevant patches.

I understand that my comment came little late... but taking into
account the ease of the change (sed exercise), why not doing it? the
IB core patch exposes API to user-space, so we should make sure to do
it right

Or.


> The iWARP Port Mapper implementation is based on the port mapper specification
> section in the Sockets Direct Protocol paper -
> http://www.rdmaconsortium.org/home/draft-pinkerton-iwarp-sdp-v1.0.pdf
>
> Existing iWARP RDMA providers use the same IP address as the native TCP/IP
> stack when creating RDMA connections. They need to ensure a mechanism to claim
> the TCP ports used for RDMA connections, thereby preventing TCP port collisions
> when other host applications are consuming TCP ports. The iWARP Port Mapper
> provides a standard mechanism to accomplish this. Without this service it is
> possible for RDMA application to bind/listen on the same port which is already
> being used by native TCP host application. If that happens the incoming TCP
> connection data can be passed to the RDMA stack with error.
>
> The iWARP Port Mapper solution doesn't contain any changes to the existing
> network stack in the kernel space. All the changes are contained with the
> infiniband tree and also in user space.
>
> The iWARP Port Mapper service is implemented as a user space daemon process.
> Interested readers could look at the source code of the IWPM service located at
> http://git.openfabrics.org/git?p=~tnikolova/libiwpm-1.0.0/.git;a=summary
>
> The iWARP driver (port mapper client) sends to the IWPM service the local IP
> address and TCP port it has received from the RDMA application, when starting a
> connection. The IWPM service performs a socket bind from user space to get
> an available TCP port, called a mapped port, and communicates it back to the
> client. In that sense, the IWPM service is used to map the TCP port, which the
> RDMA application uses to any port available from the host TCP port space. The
> mapped ports are used in iWARP RDMA connections to avoid collisions with native
> TCP stack which is aware that these ports are taken. When an RDMA connection
> using a mapped port is terminated, the client notifies the IWPM service, which
> then releases the TCP port.
>
> The message exchange between the IWPM service and the iWARP drivers
> (between user space and kernel space) is implemented using netlink sockets.
>
> This patch series adds iWarp driver support for sending/receiving/parsing
> netlink messages, to make the communication with the IWPM service possible.
>
> This is the second submission for the iWarp Port Mapper and
> we have considered and incorporated feedback after the first submission
> to improve the IWPM Version 2 patch series.
>
> The following items are improvements and fixes to IWPM V2:
>
> 1) The IWPM functionality, common for both iWarp drivers (nes and cxgb4)
>    is refactored from the drivers source files and is moved to new shared
>    files in infiniband/core which are compiled as part of the iw_cm module.
>
> 2) Hash table data structure is implemented to store the drivers mapping
>    information and enable efficient searching. (Hash table implementation
>    is based on original work by Vipul Pandya)
>
> 3) Fix for a timer bug is provided as the unnecessary timer scheduling is
>    removed.
>
> The patches are built against Roland's infiniband tree for-next branch.
>
> Thank you,
>
> Tatyana Nikolova (2):
>   RDMA/core: Add support for iWarp Port Mapper V2 user space service
>   RDMA/nes: Add support for iWarp Port Mapper V2 user space service
> Steve Wise (1):
>   RDMA/cxgb4: Add support for iWarp Port Mapper V2 user space service
>
>  drivers/infiniband/core/Makefile       |    2 +-
>  drivers/infiniband/core/cma.c          |    3 +-
>  drivers/infiniband/core/iwpm_msg.c     |  691 ++++++++++++++++++++++++++++++++
>  drivers/infiniband/core/iwpm_util.c    |  609 ++++++++++++++++++++++++++++
>  drivers/infiniband/core/iwpm_util.h    |  238 +++++++++++
>  drivers/infiniband/core/netlink.c      |   18 +-
>  drivers/infiniband/hw/cxgb4/cm.c       |  180 +++++++--
>  drivers/infiniband/hw/cxgb4/device.c   |   81 ++++-
>  drivers/infiniband/hw/cxgb4/iw_cxgb4.h |   44 ++
>  drivers/infiniband/hw/nes/nes.c        |   25 ++-
>  drivers/infiniband/hw/nes/nes.h        |    3 +
>  drivers/infiniband/hw/nes/nes_cm.c     |  320 ++++++++++++---
>  drivers/infiniband/hw/nes/nes_cm.h     |   12 +-
>  include/rdma/iw_portmap.h              |  199 +++++++++
>  include/rdma/rdma_netlink.h            |   23 +-
>  include/uapi/rdma/rdma_netlink.h       |   96 +++++-
>  16 files changed, 2431 insertions(+), 113 deletions(-)
>  create mode 100644 drivers/infiniband/core/iwpm_msg.c
>  create mode 100644 drivers/infiniband/core/iwpm_util.c
>  create mode 100644 drivers/infiniband/core/iwpm_util.h
>  create mode 100644 include/rdma/iw_portmap.h
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: [PATCH 0/3] RDMA/core: iWARP Port Mapper Overview

[Steve Wise]

> 
> Hi Steve, Tatyana
> 
> Reading this cover letter, I think it's good idea to follow Steve's
> suggestion and add portions from here into the IB/core change you
> introduced (patch #1 in this series) and is now merged into Roland's
> for-next -- before Roland's sends his 3.16 merge window pull request
> to Linus (there is enough time!!), since the current change-log
> http://git.kernel.org/cgit/linux/kernel/git/roland/infiniband.git/commit/?h=for-
> next&id=1943d56776ad31a119517efeb309786db3b4383e
> practically says nothing on what is the PM about
> 

Roland, can you do this?

> 2nd, there's nothing specific to iWARP in the port mapper implementation, right?
> 
> What it does is the following:
> 
> 1. kernel rdma driver tells a user space daemon they want to reserve
> (claim) the combination of IP address X and TCP port Y for the sake of
> RDMA connections
> 
> 2. user space daemon opens a socket and binds to X:Y
> 

No, the use space daemon opens a socket and binds to address X port 0 and thus allocates an ephemeral port number Z.  The mapping of X->Z is then made available to clients trying to connect to this service.

> Specifically, down the road, more use cases, not only the current
> iWARP case may pop up.
> 

It is really only needed for iWARP and not IB because IB doesn't share the port space with the host TCP stack.  iWARP does.

> So taking into account that the implementation doesn't have anything
> whivh is IW specific, and that future non-IW use case might show up,
> why we need to fill the IB core code with IW_PM notation? why not just
> use RDMA_PM notation? this can be achieved quickly with
> 
> $ sed -i g/iw_pm/rdma_pm/g *
> 
> and friends on Roland's tree and later squashed into the relevant patches.
> 
> I understand that my comment came little late... but taking into
> account the ease of the change (sed exercise), why not doing it? the
> IB core patch exposes API to user-space, so we should make sure to do
> it right
> 

I'm not convinced this is desirable.

Steve.

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH 0/3] RDMA/core: iWARP Port Mapper Overview

[Jason Gunthorpe]

On Sun, Jun 08, 2014 at 10:34:30PM +0300, Or Gerlitz wrote:
> What it does is the following:
> 
> 1. kernel rdma driver tells a user space daemon they want to reserve
> (claim) the combination of IP address X and TCP port Y for the sake of
> RDMA connections
> 
> 2. user space daemon opens a socket and binds to X:Y
> 
> Specifically, down the road, more use cases, not only the current
> iWARP case may pop up.

This really seems horrible, using user space to circumvent the kernel
stack because kernel maintainers don't want this kind of integration
is not going to make people very happy.

IIRC the patch set that tried to do this directly in the kernel was
NAK'd, adding a userspace round trip doesn't really change anything.

The message from netdev has, IMHO, always been pretty clear - offload
can live in it's own little side world but cannot appear to the user
to be integrated to the main stack (because it isn't).

Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH 0/3] RDMA/core: iWARP Port Mapper Overview

[Christoph Lameter]

On Mon, 9 Jun 2014, Jason Gunthorpe wrote:

> The message from netdev has, IMHO, always been pretty clear - offload
> can live in it's own little side world but cannot appear to the user
> to be integrated to the main stack (because it isn't).

Hmmm... The flow steering API already does what the port mapper does and
its already merged. Why do we need another implementation?



--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: [PATCH 0/3] RDMA/core: iWARP Port Mapper Overview

[Steve Wise]

> -----Original Message-----
> From: Jason Gunthorpe [mailto:jgunthorpe-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org]
> Sent: Monday, June 09, 2014 11:51 AM
> To: Or Gerlitz
> Cc: Tatyana Nikolova; Steve Wise; Roland Dreier; Lacombe, John S; Sean Hefty; linux-rdma
> Subject: Re: [PATCH 0/3] RDMA/core: iWARP Port Mapper Overview
> 
> On Sun, Jun 08, 2014 at 10:34:30PM +0300, Or Gerlitz wrote:
> > What it does is the following:
> >
> > 1. kernel rdma driver tells a user space daemon they want to reserve
> > (claim) the combination of IP address X and TCP port Y for the sake of
> > RDMA connections
> >
> > 2. user space daemon opens a socket and binds to X:Y
> >
> > Specifically, down the road, more use cases, not only the current
> > iWARP case may pop up.
> 
> This really seems horrible, using user space to circumvent the kernel
> stack because kernel maintainers don't want this kind of integration
> is not going to make people very happy.
> 
> IIRC the patch set that tried to do this directly in the kernel was
> NAK'd, adding a userspace round trip doesn't really change anything.
> 

The patch set you refer to tried to _unify_ the port space and was rejected.  The only
other alternative is to pick ephemeral ports and maintain a mapping for RDMA services.

> The message from netdev has, IMHO, always been pretty clear - offload
> can live in it's own little side world but cannot appear to the user
> to be integrated to the main stack (because it isn't).

That is what this design does...

Steve.

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: [PATCH 0/3] RDMA/core: iWARP Port Mapper Overview

[Steve Wise]

> -----Original Message-----
> From: Christoph Lameter [mailto:cl-gkYfJU5Cukgdnm+yROfE0A@public.gmane.org]
> Sent: Monday, June 09, 2014 1:11 PM
> To: Jason Gunthorpe
> Cc: Or Gerlitz; Tatyana Nikolova; Steve Wise; Roland Dreier; Lacombe, John S; Sean
Hefty;
> linux-rdma
> Subject: Re: [PATCH 0/3] RDMA/core: iWARP Port Mapper Overview
> 
> On Mon, 9 Jun 2014, Jason Gunthorpe wrote:
> 
> > The message from netdev has, IMHO, always been pretty clear - offload
> > can live in it's own little side world but cannot appear to the user
> > to be integrated to the main stack (because it isn't).
> 
> Hmmm... The flow steering API already does what the port mapper does and
> its already merged. Why do we need another implementation?
> 

I don't see how the flow steering API does anything like the port mapping. 

Steve.


--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: [PATCH 0/3] RDMA/core: iWARP Port Mapper Overview

[Christoph Lameter]

On Mon, 9 Jun 2014, Steve Wise wrote:

> I don't see how the flow steering API does anything like the port mapping.

You can redirect flows with a particular port number. Thats the same as
port mappig right?

linux/include/rdma/ib_verbs.h:

struct ib_flow_spec_ipv4 {
        enum ib_flow_spec_type     type;
        u16                        size;
        struct ib_flow_ipv4_filter val;
        struct ib_flow_ipv4_filter mask;
};

struct ib_flow_tcp_udp_filter {
        __be16  dst_port;
        __be16  src_port;
};

struct ib_flow_spec_tcp_udp {
        enum ib_flow_spec_type        type;
        u16                           size;
        struct ib_flow_tcp_udp_filter val;
        struct ib_flow_tcp_udp_filter mask;
};

union ib_flow_spec {
        struct {
                enum ib_flow_spec_type  type;
                u16                     size;
        };
        struct ib_flow_spec_eth         eth;
        struct ib_flow_spec_ib          ib;
        struct ib_flow_spec_ipv4        ipv4;
        struct ib_flow_spec_tcp_udp     tcp_udp;
};

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: [PATCH 0/3] RDMA/core: iWARP Port Mapper Overview

[Steve Wise]

> -----Original Message-----
> From: Christoph Lameter [mailto:cl-gkYfJU5Cukgdnm+yROfE0A@public.gmane.org]
> Sent: Monday, June 09, 2014 1:28 PM
> To: Steve Wise
> Cc: 'Jason Gunthorpe'; 'Or Gerlitz'; 'Tatyana Nikolova'; 'Roland Dreier'; 'Lacombe, John
S';
> 'Sean Hefty'; 'linux-rdma'
> Subject: RE: [PATCH 0/3] RDMA/core: iWARP Port Mapper Overview
> 
> On Mon, 9 Jun 2014, Steve Wise wrote:
> 
> > I don't see how the flow steering API does anything like the port mapping.
> 
> You can redirect flows with a particular port number. Thats the same as
> port mappig right?
> 

We don't want to redirect any flows.  We want to map "tcp" port numbers to ephemeral port
numbers so a offload iWARP connection 4-tuple doesn't collide with a host tcp connection
4-tuple. 

Steve.

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH 0/3] RDMA/core: iWARP Port Mapper Overview

[Jason Gunthorpe]

On Mon, Jun 09, 2014 at 01:12:50PM -0500, Steve Wise wrote:

> > > 1. kernel rdma driver tells a user space daemon they want to reserve
> > > (claim) the combination of IP address X and TCP port Y for the sake of
> > > RDMA connections
> > >
> > > 2. user space daemon opens a socket and binds to X:Y
> > >
> > > Specifically, down the road, more use cases, not only the current
> > > iWARP case may pop up.
> > 
> > This really seems horrible, using user space to circumvent the kernel
> > stack because kernel maintainers don't want this kind of integration
> > is not going to make people very happy.
> > 
> > IIRC the patch set that tried to do this directly in the kernel was
> > NAK'd, adding a userspace round trip doesn't really change anything.

> The patch set you refer to tried to _unify_ the port space and was
> rejected.  The only other alternative is to pick ephemeral ports and
> maintain a mapping for RDMA services.

I thought both approaches were tried in kernel and NAK'd by netdev?
This looks very similar to the 2010 patchset, except the dummy socket
allocation is living in userspace in this version.

Again, from here, this looks like another patch to do an end-run
around netdev's NAK by hiding the offending stuff in userspace. That
is no good either...

> > The message from netdev has, IMHO, always been pretty clear - offload
> > can live in it's own little side world but cannot appear to the user
> > to be integrated to the main stack (because it isn't).
> 
> That is what this design does...

Not really, it is still sharing an IP used by the main stack,
otherwise you wouldn't have this problem in the first place.

Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: FW: [PATCH 0/3] RDMA/core: iWARP Port Mapper Overview

[PJ Waskiewicz]

> -----Original Message-----
> From: linux-rdma-owner-u79uwXL29TY76Z2rM5mHXA@public.gmane.org [mailto:linux-rdma-owner-u79uwXL29TY76Z2rM5mHXA@public.gmane.org] On Behalf Of Jason Gunthorpe
> Sent: Monday, June 09, 2014 1:42 PM
> To: Steve Wise
> Cc: 'Or Gerlitz'; Nikolova, Tatyana E; 'Roland Dreier'; Lacombe, John S; Hefty, Sean; 'linux-rdma'
> Subject: Re: [PATCH 0/3] RDMA/core: iWARP Port Mapper Overview
>
> On Mon, Jun 09, 2014 at 01:12:50PM -0500, Steve Wise wrote:
>
> > > > 1. kernel rdma driver tells a user space daemon they want to
> > > > reserve
> > > > (claim) the combination of IP address X and TCP port Y for the
> > > > sake of RDMA connections
> > > >
> > > > 2. user space daemon opens a socket and binds to X:Y
> > > >
> > > > Specifically, down the road, more use cases, not only the current
> > > > iWARP case may pop up.
> > >
> > > This really seems horrible, using user space to circumvent the
> > > kernel stack because kernel maintainers don't want this kind of
> > > integration is not going to make people very happy.
> > >
> > > IIRC the patch set that tried to do this directly in the kernel was
> > > NAK'd, adding a userspace round trip doesn't really change anything.
>
> > The patch set you refer to tried to _unify_ the port space and was
> > rejected.  The only other alternative is to pick ephemeral ports and
> > maintain a mapping for RDMA services.
>
> I thought both approaches were tried in kernel and NAK'd by netdev?
> This looks very similar to the 2010 patchset, except the dummy socket allocation is living in userspace in this version.

This was something I presented and discussed at NetConf 2011 with
David and other netdev core folks:
http://vger.kernel.org/netconf2011.html.  The main issue David had
with the original approach was the socket was being consumed by
something else inside the kernel.  It circumvented the TCP socket
allocation/reservation code, which was frowned upon.

The compromise we came up with is the approach in this patchset.  As
long as the socket is claimed via userspace, David had no problem with
the approach.

> Again, from here, this looks like another patch to do an end-run around netdev's NAK by hiding the offending stuff in userspace. That is no good either...

See above.

> > > The message from netdev has, IMHO, always been pretty clear -
> > > offload can live in it's own little side world but cannot appear to
> > > the user to be integrated to the main stack (because it isn't).
> >
> > That is what this design does...
>
> Not really, it is still sharing an IP used by the main stack, otherwise you wouldn't have this problem in the first place.

The reason we didn't back down from having separate IP's is because
there are customers using iWARP that want the traffic converged, using
a single IP.  This approach is the best way we came up with as a
group, it just took a bit of time to get headcount onto it and
developed/tested.

Cheers,
-PJ
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: FW: [PATCH 0/3] RDMA/core: iWARP Port Mapper Overview

[Or Gerlitz]

On 10/06/2014 03:08, PJ Waskiewicz wrote:
> This was something I presented and discussed at NetConf 2011 

Peter, the email address from your signature on the patches isn't 
functional anymore, is it OK
if the patches will be re-worded with this (or any other address you 
want to) one such that we don't
introduce (3.16) patches which has wrong email from day one? Roland, can 
you make that?

> with David and other netdev core folks: 
> http://vger.kernel.org/netconf2011.html. The main issue David had with 
> the original approach was the socket was being consumed by something 
> else inside the kernel. It circumvented the TCP socket 
> allocation/reservation code, which was frowned upon. The compromise we 
> came up with is the approach in this patchset. As long as the socket 
> is claimed via userspace, David had no problem with the approach. 

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: FW: [PATCH 0/3] RDMA/core: iWARP Port Mapper Overview

[PJ Waskiewicz]

On Mon, Jun 9, 2014 at 11:10 PM, Or Gerlitz <ogerlitz-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org> wrote:
>
> On 10/06/2014 03:08, PJ Waskiewicz wrote:
>>
>> This was something I presented and discussed at NetConf 2011
>
>
> Peter, the email address from your signature on the patches isn't functional anymore, is it OK
> if the patches will be re-worded with this (or any other address you want to) one such that we don't
> introduce (3.16) patches which has wrong email from day one? Roland, can you make that?

Hi Or,

Sorry about that, I made some changes (obviously) between when the
patches were finished up and sent and now.

You can add my new info:

Reviewed-by: PJ Waskiewicz <pj.waskiewicz-PrFOU5CTvK1l57MIdRCFDg@public.gmane.org>

Cheers,
-PJ
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: FW: [PATCH 0/3] RDMA/core: iWARP Port Mapper Overview

[Jason Gunthorpe]

On Mon, Jun 09, 2014 at 05:08:09PM -0700, PJ Waskiewicz wrote:
> > > The patch set you refer to tried to _unify_ the port space and was
> > > rejected.  The only other alternative is to pick ephemeral ports and
> > > maintain a mapping for RDMA services.
> >
> > I thought both approaches were tried in kernel and NAK'd by
> > netdev?  This looks very similar to the 2010 patchset, except the
> > dummy socket allocation is living in userspace in this version.
> 
> This was something I presented and discussed at NetConf 2011 with
> David and other netdev core folks:
> http://vger.kernel.org/netconf2011.html.  The main issue David had
> with the original approach was the socket was being consumed by
> something else inside the kernel.  It circumvented the TCP socket
> allocation/reservation code, which was frowned upon.

If you have/can get agreement from netdev that this is acceptable then
I think it is OK from the RDMA side. It would be a good touch to CC
the netdev list with the cover letter describing this mechanism
properly, maybe referencing this discussion just so there are no
surprises.

BTW, you can purge my jgg-8fiUuRrzOP0dnm+yROfE0A@public.gmane.org email from your address book,
that hasn't worked in over 10 years..

Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: FW: [PATCH 0/3] RDMA/core: iWARP Port Mapper Overview

[PJ Waskiewicz]

On Tue, Jun 10, 2014 at 9:52 AM, Jason Gunthorpe
<jgunthorpe-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org> wrote:
> On Mon, Jun 09, 2014 at 05:08:09PM -0700, PJ Waskiewicz wrote:
>> > > The patch set you refer to tried to _unify_ the port space and was
>> > > rejected.  The only other alternative is to pick ephemeral ports and
>> > > maintain a mapping for RDMA services.
>> >
>> > I thought both approaches were tried in kernel and NAK'd by
>> > netdev?  This looks very similar to the 2010 patchset, except the
>> > dummy socket allocation is living in userspace in this version.
>>
>> This was something I presented and discussed at NetConf 2011 with
>> David and other netdev core folks:
>> http://vger.kernel.org/netconf2011.html.  The main issue David had
>> with the original approach was the socket was being consumed by
>> something else inside the kernel.  It circumvented the TCP socket
>> allocation/reservation code, which was frowned upon.
>
> If you have/can get agreement from netdev that this is acceptable then
> I think it is OK from the RDMA side. It would be a good touch to CC
> the netdev list with the cover letter describing this mechanism
> properly, maybe referencing this discussion just so there are no
> surprises.

Bob and/or Tatyana, did you want to re-send and make sure netdev is
included?  Even though I did the initial championing of the design,
I'm no longer directly involved, and it would be better coming from
you guys.

> BTW, you can purge my jgg-8fiUuRrzOP0dnm+yROfE0A@public.gmane.org email from your address book,
> that hasn't worked in over 10 years..

Yeah, that's my bad.  I didn't have the original email from the
thread, so I had to make "guesses" of some of the email addresses.

Cheers,
-PJ
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html