[PATCH 1/1] RDMA over Fibre Channel

[PATCH 1/1] RDMA over Fibre Channel

[Christoph Hellwig]

On Wed, Apr 18, 2018@02:42:40AM -0700, muneendra.kumar@broadcom.com wrote:
> Eventhough it is inspired from the Soft RoCE driver, the underlying
> transport layer is FC-NVMe (short for 'NVMe over fibre channel').
> The request, response and completion state machines in the driver
> have been heavily modified to adapt to the Exchange based Data
> transfer mechanism of Fibre channel.

That sounds like a bad joke.  Please stop abusing the NVMe code for this
otherwise reasonable idea.  You should be able to layer this over plain
FCP just fine.

[PATCH 1/1] RDMA over Fibre Channel

[Muneendra Kumar M]

Hi Christoph,

The current implementation of RDMA over Fibre channel uses NVMe for the
below reasons:
1. Existing FC-NVMe HBA and FC network can be used without requiring any
changes.
2. NVMe namespace based discovery is used for RDMA node discovery.
3. FC-NVMe provides us a way to achieve Zero copy TX/RX for non-block
workloads.

Although we concur with the idea of RDMA directly over Fibre channel, the
actual implementation addressing the above reasons requires
standardization and coordination with FC HBA vendors and other SAN
ecosystem players. This effort is ongoing within our organization (Brocade
at Broadcom).   However, there is a business case for the current soft
RDMA implementation for FC, i.e. RDMA over FC-NVMe, as it provides
existing Fibre channel customers a way to utilize existing FC network to
transport RDMA workloads as well.  While doing this we are making sure
NVMe block traffic also can happen on the same FC network.

The below link  gives more technical details. We are glad to discuss any
further details on the same.

https://github.com/brocade/RDMAoverFC/blob/master/RDMA%20over%20FC.pdf

Regards,
Muneendra, Amit & Anand.


-----Original Message-----
From: Christoph Hellwig [mailto:hch@infradead.org]
Sent: Wednesday, April 18, 2018 3:52 PM
To: muneendra.kumar at broadcom.com
Cc: linux-rdma at vger.kernel.org; amit.tyagi at broadcom.com;
anand.sundaram at broadcom.com; linux-nvme at lists.infradead.org
Subject: Re: [PATCH 1/1] RDMA over Fibre Channel

On Wed, Apr 18, 2018 at 02:42:40AM -0700, muneendra.kumar at broadcom.com
wrote:
> Eventhough it is inspired from the Soft RoCE driver, the underlying
> transport layer is FC-NVMe (short for 'NVMe over fibre channel').
> The request, response and completion state machines in the driver have
> been heavily modified to adapt to the Exchange based Data transfer
> mechanism of Fibre channel.

That sounds like a bad joke.  Please stop abusing the NVMe code for this
otherwise reasonable idea.  You should be able to layer this over plain
FCP just fine.

[PATCH 1/1] RDMA over Fibre Channel

[Christoph Hellwig]

On Wed, Apr 18, 2018@05:17:25PM +0530, Muneendra Kumar M wrote:
> Although we concur with the idea of RDMA directly over Fibre channel, the
> actual implementation addressing the above reasons requires
> standardization and coordination with FC HBA vendors and other SAN
> ecosystem players. This effort is ongoing within our organization (Brocade
> at Broadcom).   However, there is a business case for the current soft
> RDMA implementation for FC, i.e. RDMA over FC-NVMe, as it provides
> existing Fibre channel customers a way to utilize existing FC network to
> transport RDMA workloads as well.  While doing this we are making sure
> NVMe block traffic also can happen on the same FC network.

There might be a business case for you, but with my Linux NVMe
(co-)maintainer hat on I'll have to tell you that this abuse of the
Linux nvme code is a complete no-go.

And even if it wasn't we'd still require the protocol be ratified by the
NVMe technical working group first.

[PATCH 1/1] RDMA over Fibre Channel

[Anand Nataraja Sundaram]

Hi Christoph,

Just wanted to understand more on your concerns on the mods done to Linux
NVMe.

The whole work was to tunnel IB protocol over existing NVMe protocol.  To
do this we first made sure NVMe stack (host, target) is able to send block
traffic and non-block (object based ) traffic. To do this, no changes were
required in the NVMe protocol itself. Only the target stack needed some
modifications to vector
  (a) NVMe block traffic to backend NVMe Namespace block driver
  (b) non-block  IB protocol traffic to RFC transport layer

The NVMe changes are restricted to below:
drivers/nvme/target/fc.c                        |   94 +-
drivers/nvme/target/io-cmd.c                    |   44 +-
include/linux/nvme-fc-driver.h                  |    6 +

Of the above only the io-cmd.c  is generic target component. The rest are
contained under FC stack.

Finally I respect your decision.  The above intent is to demonstrate
existing NVMe protocol can support both block and non-block workloads. If
this requires working with NVMe core working group, kindly let us know how
to accomplish this.

Thanks,
-anand


-----Original Message-----
From: Christoph Hellwig [mailto:hch@infradead.org]
Sent: Wednesday, April 18, 2018 6:49 PM
To: Muneendra Kumar M <muneendra.kumar at broadcom.com>
Cc: Christoph Hellwig <hch at infradead.org>; linux-rdma at vger.kernel.org;
Amit Kumar Tyagi <amit.tyagi at broadcom.com>; Anand Nataraja Sundaram
<anand.sundaram at broadcom.com>; linux-nvme at lists.infradead.org
Subject: Re: [PATCH 1/1] RDMA over Fibre Channel

On Wed, Apr 18, 2018@05:17:25PM +0530, Muneendra Kumar M wrote:
> Although we concur with the idea of RDMA directly over Fibre channel,
> the actual implementation addressing the above reasons requires
> standardization and coordination with FC HBA vendors and other SAN
> ecosystem players. This effort is ongoing within our organization
(Brocade
> at Broadcom).   However, there is a business case for the current soft
> RDMA implementation for FC, i.e. RDMA over FC-NVMe, as it provides
> existing Fibre channel customers a way to utilize existing FC network
> to transport RDMA workloads as well.  While doing this we are making
> sure NVMe block traffic also can happen on the same FC network.

There might be a business case for you, but with my Linux NVMe
(co-)maintainer hat on I'll have to tell you that this abuse of the Linux
nvme code is a complete no-go.

And even if it wasn't we'd still require the protocol be ratified by the
NVMe technical working group first.

[PATCH 1/1] RDMA over Fibre Channel

[Christoph Hellwig]

On Wed, Apr 18, 2018@10:23:45PM +0530, Anand Nataraja Sundaram wrote:
> Just wanted to understand more on your concerns on the mods done to Linux
> NVMe.
> 
> The whole work was to tunnel IB protocol over existing NVMe protocol.  To
> do this we first made sure NVMe stack (host, target) is able to send block
> traffic and non-block (object based ) traffic. To do this, no changes were
> required in the NVMe protocol itself. Only the target stack needed some
> modifications to vector
>   (a) NVMe block traffic to backend NVMe Namespace block driver
>   (b) non-block  IB protocol traffic to RFC transport layer
> 
> The NVMe changes are restricted to below:
> drivers/nvme/target/fc.c                        |   94 +-
> drivers/nvme/target/io-cmd.c                    |   44 +-
> include/linux/nvme-fc-driver.h                  |    6 +

You forgot the larger chunks of Linux NVMe code you copied while
stripping the copyrights and incorrectly relicensing it to a BSD-like
license.

The point is that IFF you really want to do RDMA over NVMe you need to
defined a new NVMe I/O command set for it and get it standardized.  If
that is done we could do a proper upper level protocol interface for it,
instead of just hacking it into the protocol and code through the
backdoor.  But as said before there is no upside of using NVMe, I can
see the interest in layering on top of FCP to reuse existing hardware
accelerations, similar to how NVMe layers on top of FCP for that reason,
but there isn't really any value in throwing in another NVMe layer.

[PATCH 1/1] RDMA over Fibre Channel

[Anand Nataraja Sundaram]

Agreed some of the host nvme code was wrongly duplicated under BSD-like
license in
drivers/infiniband/sw/rfc/rfc_tb.c              |  795 +++++++++++++

Bottom line: We need both NVMe host and NVMe target stack changes to
tunnel RDMA over FC-NVMe.

This exercise just proved that RDMA can be tunneled over FC-NVMe. I agree
we need some standardization to transport RDMA workload over FC networks.

The clear advantage of doing RDMA over NVMe is that we could do end to end
zero-copy between RDMA applications. Doing RDMA over SCSI-FCP incurs a
one-copy penalty between RDMA applications.
However doing RDMA over FC directly (as a new FC-4 Upper Level Protocol
type) is also a possibility. Here FC-VI could also be considered. All this
required using new HBAs.

As a FC-SAN Community, we will work out which is the best route for RDMA
over FC standardization.

Thanks for your feedback,
-anand




-----Original Message-----
From: Christoph Hellwig [mailto:hch@infradead.org]
Sent: Thursday, April 19, 2018 3:10 PM
To: Anand Nataraja Sundaram <anand.sundaram at broadcom.com>
Cc: Christoph Hellwig <hch at infradead.org>; Muneendra Kumar M
<muneendra.kumar at broadcom.com>; linux-rdma at vger.kernel.org; Amit Kumar
Tyagi <amit.tyagi at broadcom.com>; linux-nvme at lists.infradead.org
Subject: Re: [PATCH 1/1] RDMA over Fibre Channel

On Wed, Apr 18, 2018@10:23:45PM +0530, Anand Nataraja Sundaram wrote:
> Just wanted to understand more on your concerns on the mods done to
> Linux NVMe.
>
> The whole work was to tunnel IB protocol over existing NVMe protocol.
> To do this we first made sure NVMe stack (host, target) is able to
> send block traffic and non-block (object based ) traffic. To do this,
> no changes were required in the NVMe protocol itself. Only the target
> stack needed some modifications to vector
>   (a) NVMe block traffic to backend NVMe Namespace block driver
>   (b) non-block  IB protocol traffic to RFC transport layer
>
> The NVMe changes are restricted to below:
> drivers/nvme/target/fc.c                        |   94 +-
> drivers/nvme/target/io-cmd.c                    |   44 +-
> include/linux/nvme-fc-driver.h                  |    6 +

You forgot the larger chunks of Linux NVMe code you copied while stripping
the copyrights and incorrectly relicensing it to a BSD-like license.

The point is that IFF you really want to do RDMA over NVMe you need to
defined a new NVMe I/O command set for it and get it standardized.  If
that is done we could do a proper upper level protocol interface for it,
instead of just hacking it into the protocol and code through the
backdoor.  But as said before there is no upside of using NVMe, I can see
the interest in layering on top of FCP to reuse existing hardware
accelerations, similar to how NVMe layers on top of FCP for that reason,
but there isn't really any value in throwing in another NVMe layer.

[PATCH 1/1] RDMA over Fibre Channel

[Bart Van Assche]

On Wed, 2018-04-18@17:17 +0530, Muneendra Kumar M wrote:
> Although we concur with the idea of RDMA directly over Fibre channel, the
> actual implementation addressing the above reasons requires
> standardization and coordination with FC HBA vendors and other SAN
> ecosystem players. This effort is ongoing within our organization (Brocade
> at Broadcom).

What makes you think that a new standardization effort would have to be
launched to make RDMA over FC possible? RDMA over FC was standardized a long
time ago and is called "FC-VI". Why aren't you using that standard? A quote
from the FC-VI document:

Table 1 ? VI and FC-VI functions
................................
VI function                   FC-VI equivalent
VI Message                    Exchange
Request/Response primitives   Sequence
VI Connect/Disconnect request Unsolicited control IU
VI Connect response           Solicited control IU
VI Send Data                  Solicited data IU
VI Send Response              Command status IU
VI RDMA Write Request         Solicited data IU
VI RDMA Write Response        Command status IU
VI RDMA Read Request          Unsolicited command IU
VI RDMA Read Response         Solicited data IU

See also ANSI/INCITS 357:2002, "Fibre Channel Virtual Interface Architecture
Mapping Protocol (FC-VI)", NCITS Project 1332-D, July 2000.

Bart.