Re: [PATCH-for-9.1 v2 2/3] migration: Remove RDMA protocol handling

From: Peter Xu <peterx@redhat.com>
To: "Gonglei (Arei)" <arei.gonglei@huawei.com>
Cc: "Daniel P. Berrangé" <berrange@redhat.com>,
	"Markus Armbruster" <armbru@redhat.com>,
	"Michael Galaxy" <mgalaxy@akamai.com>,
	"Yu Zhang" <yu.zhang@ionos.com>,
	"Zhijian Li (Fujitsu)" <lizhijian@fujitsu.com>,
	"Jinpu Wang" <jinpu.wang@ionos.com>,
	"Elmar Gerdes" <elmar.gerdes@ionos.com>,
	"qemu-devel@nongnu.org" <qemu-devel@nongnu.org>,
	"Yuval Shaia" <yuval.shaia.ml@gmail.com>,
	"Kevin Wolf" <kwolf@redhat.com>,
	"Prasanna Kumar Kalever" <prasanna.kalever@redhat.com>,
	"Cornelia Huck" <cohuck@redhat.com>,
	"Michael Roth" <michael.roth@amd.com>,
	"Prasanna Kumar Kalever" <prasanna4324@gmail.com>,
	"integration@gluster.org" <integration@gluster.org>,
	"Paolo Bonzini" <pbonzini@redhat.com>,
	"qemu-block@nongnu.org" <qemu-block@nongnu.org>,
	"devel@lists.libvirt.org" <devel@lists.libvirt.org>,
	"Hanna Reitz" <hreitz@redhat.com>,
	"Michael S. Tsirkin" <mst@redhat.com>,
	"Thomas Huth" <thuth@redhat.com>,
	"Eric Blake" <eblake@redhat.com>,
	"Song Gao" <gaosong@loongson.cn>,
	"Marc-André Lureau" <marcandre.lureau@redhat.com>,
	"Alex Bennée" <alex.bennee@linaro.org>,
	"Wainer dos Santos Moschetta" <wainersm@redhat.com>,
	"Beraldo Leal" <bleal@redhat.com>,
	Pannengyuan <pannengyuan@huawei.com>,
	Xiexiangyou <xiexiangyou@huawei.com>,
	zhengchuan <zhengchuan@huawei.com>
Subject: Re: [PATCH-for-9.1 v2 2/3] migration: Remove RDMA protocol handling
Date: Tue, 7 May 2024 12:28:08 -0400	[thread overview]
Message-ID: <ZjpWmG2aUJLkYxJm@x1n> (raw)
In-Reply-To: <addaa8d094904315a466533763689ead@huawei.com>

On Tue, May 07, 2024 at 01:50:43AM +0000, Gonglei (Arei) wrote:
> Hello,
> 
> > -----Original Message-----
> > From: Peter Xu [mailto:peterx@redhat.com]
> > Sent: Monday, May 6, 2024 11:18 PM
> > To: Gonglei (Arei) <arei.gonglei@huawei.com>
> > Cc: Daniel P. Berrangé <berrange@redhat.com>; Markus Armbruster
> > <armbru@redhat.com>; Michael Galaxy <mgalaxy@akamai.com>; Yu Zhang
> > <yu.zhang@ionos.com>; Zhijian Li (Fujitsu) <lizhijian@fujitsu.com>; Jinpu Wang
> > <jinpu.wang@ionos.com>; Elmar Gerdes <elmar.gerdes@ionos.com>;
> > qemu-devel@nongnu.org; Yuval Shaia <yuval.shaia.ml@gmail.com>; Kevin Wolf
> > <kwolf@redhat.com>; Prasanna Kumar Kalever
> > <prasanna.kalever@redhat.com>; Cornelia Huck <cohuck@redhat.com>;
> > Michael Roth <michael.roth@amd.com>; Prasanna Kumar Kalever
> > <prasanna4324@gmail.com>; integration@gluster.org; Paolo Bonzini
> > <pbonzini@redhat.com>; qemu-block@nongnu.org; devel@lists.libvirt.org;
> > Hanna Reitz <hreitz@redhat.com>; Michael S. Tsirkin <mst@redhat.com>;
> > Thomas Huth <thuth@redhat.com>; Eric Blake <eblake@redhat.com>; Song
> > Gao <gaosong@loongson.cn>; Marc-André Lureau
> > <marcandre.lureau@redhat.com>; Alex Bennée <alex.bennee@linaro.org>;
> > Wainer dos Santos Moschetta <wainersm@redhat.com>; Beraldo Leal
> > <bleal@redhat.com>; Pannengyuan <pannengyuan@huawei.com>;
> > Xiexiangyou <xiexiangyou@huawei.com>
> > Subject: Re: [PATCH-for-9.1 v2 2/3] migration: Remove RDMA protocol handling
> > 
> > On Mon, May 06, 2024 at 02:06:28AM +0000, Gonglei (Arei) wrote:
> > > Hi, Peter
> > 
> > Hey, Lei,
> > 
> > Happy to see you around again after years.
> > 
> Haha, me too.
> 
> > > RDMA features high bandwidth, low latency (in non-blocking lossless
> > > network), and direct remote memory access by bypassing the CPU (As you
> > > know, CPU resources are expensive for cloud vendors, which is one of
> > > the reasons why we introduced offload cards.), which TCP does not have.
> > 
> > It's another cost to use offload cards, v.s. preparing more cpu resources?
> > 
> Software and hardware offload converged architecture is the way to go for all cloud vendors 
> (Including comprehensive benefits in terms of performance, cost, security, and innovation speed), 
> it's not just a matter of adding the resource of a DPU card.
> 
> > > In some scenarios where fast live migration is needed (extremely short
> > > interruption duration and migration duration) is very useful. To this
> > > end, we have also developed RDMA support for multifd.
> > 
> > Will any of you upstream that work?  I'm curious how intrusive would it be
> > when adding it to multifd, if it can keep only 5 exported functions like what
> > rdma.h does right now it'll be pretty nice.  We also want to make sure it works
> > with arbitrary sized loads and buffers, e.g. vfio is considering to add IO loads to
> > multifd channels too.
> > 
> 
> In fact, we sent the patchset to the community in 2021. Pls see:
> https://lore.kernel.org/all/20210203185906.GT2950@work-vm/T/

I wasn't aware of that for sure in the past..

Multifd has changed quite a bit in the last 9.0 release, that may not apply
anymore.  One thing to mention is please look at Dan's comment on possible
use of rsocket.h:

https://lore.kernel.org/all/ZjJm6rcqS5EhoKgK@redhat.com/

And Jinpu did help provide an initial test result over the library:

https://lore.kernel.org/qemu-devel/CAMGffEk8wiKNQmoUYxcaTHGtiEm2dwoCF_W7T0vMcD-i30tUkA@mail.gmail.com/

It looks like we have a chance to apply that in QEMU.

> 
> 
> > One thing to note that the question here is not about a pure performance
> > comparison between rdma and nics only.  It's about help us make a decision
> > on whether to drop rdma, iow, even if rdma performs well, the community still
> > has the right to drop it if nobody can actively work and maintain it.
> > It's just that if nics can perform as good it's more a reason to drop, unless
> > companies can help to provide good support and work together.
> > 
> 
> We are happy to provide the necessary review and maintenance work for RDMA
> if the community needs it.
> 
> CC'ing Chuan Zheng.

I'm not sure whether you and Jinpu's team would like to work together and
provide a final solution for rdma over multifd.  It could be much simpler
than the original 2021 proposal if the rsocket API will work out.

Thanks,

-- 
Peter Xu