Re: [PATCH 00/24] InfiniBand Transport (IBTRS) and Network Block Device (IBNBD)

From: Danil Kipnis <danil.kipnis@profitbricks.com>
To: dledford@redhat.com
Cc: Bart Van Assche <Bart.VanAssche@wdc.com>,
	Roman Penyaev <roman.penyaev@profitbricks.com>,
	linux-block@vger.kernel.org, linux-rdma@vger.kernel.org,
	Christoph Hellwig <hch@infradead.org>,
	ogerlitz@mellanox.com, Jack Wang <jinpu.wang@profitbricks.com>,
	axboe@kernel.dk, Sagi Grimberg <sagi@grimberg.me>
Subject: Re: [PATCH 00/24] InfiniBand Transport (IBTRS) and Network Block Device (IBNBD)
Date: Mon, 4 Jun 2018 14:14:08 +0200	[thread overview]
Message-ID: <CAHg0HuwhyWXHPhr-91vNxLsyFS8Ka9OtxS-yud7jQmK0Bc48-w@mail.gmail.com> (raw)
In-Reply-To: <1517589637.3936.16.camel@redhat.com>

[-- Attachment #1: Type: text/plain, Size: 3181 bytes --]

Hi Doug,

thanks for the feedback. You read the cover letter correctly: our transport
library implements multipath (load balancing and failover) on top of RDMA
API. Its name "IBTRS" is slightly misleading in that regard: it can sit on
top of ROCE as well. The library allows for "bundling" multiple rdma
"paths" (source addr - destination addr pair) into one "session". So our
session consists of one or more paths and each path under the hood consists
of as many QPs (each connecting source with destination) as there are CPUs
on the client system. The user load (In our case IBNBD is a block device
and generates some block requests) is load-balanced on per cpu-basis.
I understand, this is something very different to what smc-r is doing. Am I
right? Do you know what stage MP-RDMA development currently is?

Best,

Danil Kipnis.

On Fri, Feb 2, 2018 at 5:40 PM Doug Ledford <dledford@redhat.com> wrote:

> On Fri, 2018-02-02 at 16:07 +0000, Bart Van Assche wrote:
> > On Fri, 2018-02-02 at 15:08 +0100, Roman Pen wrote:
> > > Since the first version the following was changed:
> > >
> > >    - Load-balancing and IO fail-over using multipath features were
> added.
> > >    - Major parts of the code were rewritten, simplified and overall
> code
> > >      size was reduced by a quarter.
> >
> > That is interesting to know, but what happened to the feedback that Sagi
> and
> > I provided on v1? Has that feedback been addressed? See also
> > https://www.spinics.net/lists/linux-rdma/msg47819.html and
> > https://www.spinics.net/lists/linux-rdma/msg47879.html.
> >
> > Regarding multipath support: there are already two multipath
> implementations
> > upstream (dm-mpath and the multipath implementation in the NVMe
> initiator).
> > I'm not sure we want a third multipath implementation in the Linux
> kernel.
>
> There's more than that.  There was also md-multipath, and smc-r includes
> another version of multipath, plus I assume we support mptcp as well.
>
> But, to be fair, the different multipaths in this list serve different
> purposes and I'm not sure they could all be generalized out and served
> by a single multipath code.  Although, fortunately, md-multipath is
> deprecated, so no need to worry about it, and it is only dm-multipath
> and nvme multipath that deal directly with block devices and assume
> block semantics.  If I read the cover letter right (and I haven't dug
> into the code to confirm this), the ibtrs multipath has much more in
> common with smc-r multipath, where it doesn't really assume a block
> layer device sits on top of it, it's more of a pure network multipath,
> which the implementation of smc-r is and mptcp would be too.  I would
> like to see a core RDMA multipath implementation soon that would
> abstract out some of these multipath tasks, at least across RDMA links,
> and that didn't have the current limitations (smc-r only supports RoCE
> links, and it sounds like ibtrs only supports IB like links, but maybe
> I'm wrong there, I haven't looked at the patches yet).
>
> --
> Doug Ledford <dledford@redhat.com>
>     GPG KeyID: B826A3330E572FDD
>     Key fingerprint = AE6B 1BDA 122B 23B4 265B  1274 B826 A333 0E57 2FDD

[-- Attachment #2: Type: text/html, Size: 4250 bytes --]