Re: Outline for VHOST_USER_PROTOCOL_F_VDPA

From: Jason Wang <jasowang@redhat.com>
To: "Stefan Hajnoczi" <stefanha@redhat.com>,
	"Marc-André Lureau" <marcandre.lureau@redhat.com>
Cc: lulu@redhat.com, tiwei.bie@intel.com,
	"Michael S. Tsirkin" <mst@redhat.com>,
	qemu-devel <qemu-devel@nongnu.org>,
	Raphael Norwitz <raphael.norwitz@nutanix.com>,
	"Coquelin, Maxime" <maxime.coquelin@redhat.com>,
	"Hoffmann, Gerd" <kraxel@redhat.com>,
	Felipe Franciosi <felipe@nutanix.com>,
	Nikos Dragazis <ndragazis@arrikto.com>,
	"Liu, Changpeng" <changpeng.liu@intel.com>,
	Daniele Buono <dbuono@us.ibm.com>
Subject: Re: Outline for VHOST_USER_PROTOCOL_F_VDPA
Date: Mon, 12 Oct 2020 10:56:14 +0800	[thread overview]
Message-ID: <d140fcd1-8a3f-ed24-1ef0-82b4c68746e8@redhat.com> (raw)
In-Reply-To: <20200928153257.GA173977@stefanha-x1.localdomain>

On 2020/9/28 下午11:32, Stefan Hajnoczi wrote:
> On Mon, Sep 28, 2020 at 03:21:56PM +0400, Marc-André Lureau wrote:
>> On Mon, Sep 28, 2020 at 1:25 PM Stefan Hajnoczi <stefanha@redhat.com wrote:
>>> Where this converges with multi-process QEMU
>>> --------------------------------------------
>>> At this point QEMU can run ad-hoc vhost-user backends using existing
>>> VIRTIO device models. It is possible to go further by creating a
>>> qemu-dev launcher executable that implements the vhost-user spec's
>>> "Backend program conventions". This way a minimal device emulator
>>> executable hosts the device instead of a full system emulator.
>>>
>>> The requirements for this are similar to the multi-process QEMU effort,
>>> which aims to run QEMU devices as separate processes. One of the main
>>> open questions is how to design build system and Kconfig support for
>>> building minimal device emulator executables.
>>>
>>> In the case of vhost-user-net the qemu-dev-vhost-user-net executable
>>> would contain virtio-net-device, vhost-user-backend, any netdevs the
>>> user wishes to include, a QMP monitor, and a vhost-user backend
>>> command-line interface.
>>>
>>> Where does this leave us? QEMU's existing VIRTIO device models can be
>>> used as vhost-user devices and run in a separate processes from the VMM.
>>> It's a great way of reusing code and having the flexibility to deploy it
>>> in the way that makes most sense for the intended use case.
>>>
>> My understanding is that this would only be able to expose virtio
>> devices from external processes. But vfio-user could expose more kinds
>> of devices, including the virtio devices.
>>
>> Shouldn't we focus on vfio-user now, as the general out-of-process
>> device solution?

Similar question could be asked for vDPA(kernel) vs VFIO(kernel).

> Eventually vfio-user can replace vhost-user. However, vfio-user
> development will take longer so for anyone already comfortable with
> vhost-user I think extending the protocol with vDPA ioctls is
> attractive.

My understanding is for vhost-user may advantages:

- well defined interface, this helps a lot for e.g live migration (cross 
migration among different vendors), backend disconnection, device 
failover and there will be no vendor lock
- high level abstraction, not tie to a specific bus implementation, 
micro VM that want to get rid of PCI can use MMIO transport

So it doesn't conflict with vfio(-user) which is more suitable for any 
vendor specific device (API)s.

Thanks

>
> Maybe we can get more organized around vfio-user and make progress
> quicker?
>
> Stefan