[Qemu-devel] vhost-pci and virtio-vhost-user

[Qemu-devel] vhost-pci and virtio-vhost-user

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 2625 bytes --]

Hi Wei,
I wanted to summarize the differences between the vhost-pci and
virtio-vhost-user approaches because previous discussions may have been
confusing.

vhost-pci defines a new virtio device type for each vhost device type
(net, scsi, blk).  It therefore requires a virtio device driver for each
device type inside the slave VM.

Adding a new device type requires:
1. Defining a new virtio device type in the VIRTIO specification.
3. Implementing a new QEMU device model.
2. Implementing a new virtio driver.

virtio-vhost-user is a single virtio device that acts as a vhost-user
protocol transport for any vhost device type.  It requires one virtio
driver inside the slave VM and device types are implemented using
existing vhost-user slave libraries (librte_vhost in DPDK and
libvhost-user in QEMU).

Adding a new device type to virtio-vhost-user involves:
1. Adding any new vhost-user protocol messages to the QEMU
   virtio-vhost-user device model.
2. Adding any new vhost-user protocol messages to the vhost-user slave
   library.
3. Implementing the new device slave.

The simplest case is when no new vhost-user protocol messages are
required for the new device.  Then all that's needed for
virtio-vhost-user is a device slave implementation (#3).  That slave
implementation will also work with AF_UNIX because the vhost-user slave
library hides the transport (AF_UNIX vs virtio-vhost-user).  Even
better, if another person has already implemented that device slave to
use with AF_UNIX then no new code is needed for virtio-vhost-user
support at all!

If you compare this to vhost-pci, it would be necessary to design a new
virtio device, implement it in QEMU, and implement the virtio driver.
Much of virtio driver is more or less the same thing as the vhost-user
device slave but it cannot be reused because the vhost-user protocol
isn't being used by the virtio device.  The result is a lot of
duplication in DPDK and other codebases that implement vhost-user
slaves.

The way that vhost-pci is designed means that anyone wishing to support
a new device type has to become a virtio device designer.  They need to
map vhost-user protocol concepts to a new virtio device type.  This will
be time-consuming for everyone involved (e.g. the developer, the VIRTIO
community, etc).

The virtio-vhost-user approach stays at the vhost-user protocol level as
much as possible.  This way there are fewer concepts that need to be
mapped by people adding new device types.  As a result, it will allow
virtio-vhost-user to keep up with AF_UNIX vhost-user and grow because
it's easier to work with.

What do you think?

Stefan

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 455 bytes --]

Re: [Qemu-devel] vhost-pci and virtio-vhost-user

[Wei Wang]

On 01/11/2018 12:14 AM, Stefan Hajnoczi wrote:
> Hi Wei,
> I wanted to summarize the differences between the vhost-pci and
> virtio-vhost-user approaches because previous discussions may have been
> confusing.
>
> vhost-pci defines a new virtio device type for each vhost device type
> (net, scsi, blk).  It therefore requires a virtio device driver for each
> device type inside the slave VM.
>
> Adding a new device type requires:
> 1. Defining a new virtio device type in the VIRTIO specification.
> 3. Implementing a new QEMU device model.
> 2. Implementing a new virtio driver.
>
> virtio-vhost-user is a single virtio device that acts as a vhost-user
> protocol transport for any vhost device type.  It requires one virtio
> driver inside the slave VM and device types are implemented using
> existing vhost-user slave libraries (librte_vhost in DPDK and
> libvhost-user in QEMU).
>
> Adding a new device type to virtio-vhost-user involves:
> 1. Adding any new vhost-user protocol messages to the QEMU
>     virtio-vhost-user device model.
> 2. Adding any new vhost-user protocol messages to the vhost-user slave
>     library.
> 3. Implementing the new device slave.
>
> The simplest case is when no new vhost-user protocol messages are
> required for the new device.  Then all that's needed for
> virtio-vhost-user is a device slave implementation (#3).  That slave
> implementation will also work with AF_UNIX because the vhost-user slave
> library hides the transport (AF_UNIX vs virtio-vhost-user).  Even
> better, if another person has already implemented that device slave to
> use with AF_UNIX then no new code is needed for virtio-vhost-user
> support at all!
>
> If you compare this to vhost-pci, it would be necessary to design a new
> virtio device, implement it in QEMU, and implement the virtio driver.
> Much of virtio driver is more or less the same thing as the vhost-user
> device slave but it cannot be reused because the vhost-user protocol
> isn't being used by the virtio device.  The result is a lot of
> duplication in DPDK and other codebases that implement vhost-user
> slaves.
>
> The way that vhost-pci is designed means that anyone wishing to support
> a new device type has to become a virtio device designer.  They need to
> map vhost-user protocol concepts to a new virtio device type.  This will
> be time-consuming for everyone involved (e.g. the developer, the VIRTIO
> community, etc).
>
> The virtio-vhost-user approach stays at the vhost-user protocol level as
> much as possible.  This way there are fewer concepts that need to be
> mapped by people adding new device types.  As a result, it will allow
> virtio-vhost-user to keep up with AF_UNIX vhost-user and grow because
> it's easier to work with.
>
> What do you think?
>

Thanks Stefan for the clarification.

I agree with idea of making one single device for all device types. 
Would you think it is also possible with vhost-pci? (Fundamentally, the 
duty of the device is to use a bar to expose the master guest memory, 
and passes the master vring address info and memory region info, which 
has no dependency on device types)

If you agree with the above, I think the main difference is what to pass 
to the driver. I think vhost-pci is simpler because it only passes the 
above mentioned info, which is sufficient.

Relaying needs to
1) pass all the vhost-user messages to the driver, and
2) requires the driver to join the vhost-user negotiation.
Without above two, the solution already works well, so I'm not sure why 
would we need the above two from functionality point of view.

Finally, either we choose vhost-pci or virtio-vhost-user, future 
developers will need to study vhost-user protocol and virtio spec (one 
device). This wouldn't make much difference, right?

Best,
Wei

Re: [Qemu-devel] vhost-pci and virtio-vhost-user

[Stefan Hajnoczi]

On Thu, Jan 11, 2018 at 6:31 AM, Wei Wang <wei.w.wang@intel.com> wrote:
> On 01/11/2018 12:14 AM, Stefan Hajnoczi wrote:
>>
>> Hi Wei,
>> I wanted to summarize the differences between the vhost-pci and
>> virtio-vhost-user approaches because previous discussions may have been
>> confusing.
>>
>> vhost-pci defines a new virtio device type for each vhost device type
>> (net, scsi, blk).  It therefore requires a virtio device driver for each
>> device type inside the slave VM.
>>
>> Adding a new device type requires:
>> 1. Defining a new virtio device type in the VIRTIO specification.
>> 3. Implementing a new QEMU device model.
>> 2. Implementing a new virtio driver.
>>
>> virtio-vhost-user is a single virtio device that acts as a vhost-user
>> protocol transport for any vhost device type.  It requires one virtio
>> driver inside the slave VM and device types are implemented using
>> existing vhost-user slave libraries (librte_vhost in DPDK and
>> libvhost-user in QEMU).
>>
>> Adding a new device type to virtio-vhost-user involves:
>> 1. Adding any new vhost-user protocol messages to the QEMU
>>     virtio-vhost-user device model.
>> 2. Adding any new vhost-user protocol messages to the vhost-user slave
>>     library.
>> 3. Implementing the new device slave.
>>
>> The simplest case is when no new vhost-user protocol messages are
>> required for the new device.  Then all that's needed for
>> virtio-vhost-user is a device slave implementation (#3).  That slave
>> implementation will also work with AF_UNIX because the vhost-user slave
>> library hides the transport (AF_UNIX vs virtio-vhost-user).  Even
>> better, if another person has already implemented that device slave to
>> use with AF_UNIX then no new code is needed for virtio-vhost-user
>> support at all!
>>
>> If you compare this to vhost-pci, it would be necessary to design a new
>> virtio device, implement it in QEMU, and implement the virtio driver.
>> Much of virtio driver is more or less the same thing as the vhost-user
>> device slave but it cannot be reused because the vhost-user protocol
>> isn't being used by the virtio device.  The result is a lot of
>> duplication in DPDK and other codebases that implement vhost-user
>> slaves.
>>
>> The way that vhost-pci is designed means that anyone wishing to support
>> a new device type has to become a virtio device designer.  They need to
>> map vhost-user protocol concepts to a new virtio device type.  This will
>> be time-consuming for everyone involved (e.g. the developer, the VIRTIO
>> community, etc).
>>
>> The virtio-vhost-user approach stays at the vhost-user protocol level as
>> much as possible.  This way there are fewer concepts that need to be
>> mapped by people adding new device types.  As a result, it will allow
>> virtio-vhost-user to keep up with AF_UNIX vhost-user and grow because
>> it's easier to work with.
>>
>> What do you think?
>>
>
> Thanks Stefan for the clarification.
>
> I agree with idea of making one single device for all device types.

Great!

> Would
> you think it is also possible with vhost-pci? (Fundamentally, the duty of
> the device is to use a bar to expose the master guest memory, and passes the
> master vring address info and memory region info, which has no dependency on
> device types)

Yes, it's possible to have a single virtio device with vhost-pci but...

> If you agree with the above, I think the main difference is what to pass to
> the driver. I think vhost-pci is simpler because it only passes the above
> mentioned info, which is sufficient.

...the current vhost-pci patch series exposes a smaller interface to
the driver only because the code is incomplete.  Once you add fully
implement vhost-user-net, reconnection, and allow other device types,
then the vhost-pci interface will just be an extra layer on top of the
vhost-user protocol - it will not be simpler.  I'll explain why I say
this below.

> Relaying needs to
> 1) pass all the vhost-user messages to the driver, and

vhost-pci will need to pass at least some messages to the user.  For
example, VHOST_USER_SEND_RARP and VHOST_USER_NET_SET_MTU are needed
for vhost-user-net.  Once you add a mechanism to relay certain
messages then vhost-pci already starts to look more like
virtio-vhost-user.

> 2) requires the driver to join the vhost-user negotiation.

The driver must participate in vhost-user negotiation.  The vhost-pci
patches try to avoid this by taking features bits on the QEMU
command-line and hardcoding the number of supported virtqueues.  That
doesn't work in production environments because:
1. What if the driver inside the VM has been updated and now supports
different features?
2. What if the user isn't allowed to modify the VM configuration?
3. What if the management software doesn't expose the feature bits
command-line parameter?
4. What if the number of virtqueues must be different from QEMU's
default value to limit resource consumption?

Users will find it incovenient to manually enter feature bits for the
driver they are using.  The driver needs to be part of negotiation so
it can indicate which features are supported, how many virtqueues,
etc.

> Without above two, the solution already works well, so I'm not sure why would we need the above two from functionality point of view.

The "[PATCH v3 0/7] Vhost-pci for inter-VM communication" series is
incomplete.  It is a subset of vhost-user-net and it works only for
poll-mode drivers.  It's the requirements that haven't been covered by
the vhost-pci patch series yet that make me prefer the
virtio-vhost-user approach.

The virtio device design needs to be capable of supporting the rest of
vhost-user functionality in the future.  Once the code is merged in
QEMU and DPDK it will be very difficult to make changes to the virtio
device.

It's simpler to relay vhost-user protocol messages than to try to hide
them from the driver.  Mistakes will be made when designing a virtio
device interface that hides them.

vhost-pci also cannot share the slave driver code with AF_UNIX
vhost-user.  I don't see something that makes up for this
disadvantage.

> Finally, either we choose vhost-pci or virtio-vhost-user, future developers
> will need to study vhost-user protocol and virtio spec (one device). This
> wouldn't make much difference, right?

It's easier to read the spec than to change a virtio device.  The
virtio-vhost-user approach is focussed on the vhost-user protocol with
a straightforward mapping to virtio.  The vhost-pci approach tries to
consume virtio-protocol messages and provide a new virtio device
interface - this requires more knowledge of virtio device design.  The
difference is smaller now that we are in agreement that there should
only be one virtio device though.

Stefan

Re: [Qemu-devel] vhost-pci and virtio-vhost-user

[Jason Wang]

On 2018年01月11日 00:14, Stefan Hajnoczi wrote:
> Hi Wei,
> I wanted to summarize the differences between the vhost-pci and
> virtio-vhost-user approaches because previous discussions may have been
> confusing.
>
> vhost-pci defines a new virtio device type for each vhost device type
> (net, scsi, blk).  It therefore requires a virtio device driver for each
> device type inside the slave VM.
>
> Adding a new device type requires:
> 1. Defining a new virtio device type in the VIRTIO specification.
> 3. Implementing a new QEMU device model.
> 2. Implementing a new virtio driver.
>
> virtio-vhost-user is a single virtio device that acts as a vhost-user
> protocol transport for any vhost device type.  It requires one virtio
> driver inside the slave VM and device types are implemented using
> existing vhost-user slave libraries (librte_vhost in DPDK and
> libvhost-user in QEMU).
>
> Adding a new device type to virtio-vhost-user involves:
> 1. Adding any new vhost-user protocol messages to the QEMU
>     virtio-vhost-user device model.
> 2. Adding any new vhost-user protocol messages to the vhost-user slave
>     library.
> 3. Implementing the new device slave.
>
> The simplest case is when no new vhost-user protocol messages are
> required for the new device.  Then all that's needed for
> virtio-vhost-user is a device slave implementation (#3).  That slave
> implementation will also work with AF_UNIX because the vhost-user slave
> library hides the transport (AF_UNIX vs virtio-vhost-user).  Even
> better, if another person has already implemented that device slave to
> use with AF_UNIX then no new code is needed for virtio-vhost-user
> support at all!
>
> If you compare this to vhost-pci, it would be necessary to design a new
> virtio device, implement it in QEMU, and implement the virtio driver.
> Much of virtio driver is more or less the same thing as the vhost-user
> device slave but it cannot be reused because the vhost-user protocol
> isn't being used by the virtio device.  The result is a lot of
> duplication in DPDK and other codebases that implement vhost-user
> slaves.
>
> The way that vhost-pci is designed means that anyone wishing to support
> a new device type has to become a virtio device designer.  They need to
> map vhost-user protocol concepts to a new virtio device type.  This will
> be time-consuming for everyone involved (e.g. the developer, the VIRTIO
> community, etc).
>
> The virtio-vhost-user approach stays at the vhost-user protocol level as
> much as possible.  This way there are fewer concepts that need to be
> mapped by people adding new device types.  As a result, it will allow
> virtio-vhost-user to keep up with AF_UNIX vhost-user and grow because
> it's easier to work with.
>
> What do you think?
>
> Stefan

So a question is what's the motivation here?

Form what I'm understanding, vhost-pci tries to build a scalable V2V 
private datapath. But according to what you describe here, 
virito-vhost-user tries to make it possible to implement the device 
inside another VM. I understand the goal of vhost-pci could be done on 
top, but it looks to me it would then rather similar to the design of 
Xen driver domain. So I can not figure out how it can be done in a high 
performance way.

Thanks

Re: [Qemu-devel] vhost-pci and virtio-vhost-user

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 4228 bytes --]

On Thu, Jan 11, 2018 at 06:57:03PM +0800, Jason Wang wrote:
> 
> 
> On 2018年01月11日 00:14, Stefan Hajnoczi wrote:
> > Hi Wei,
> > I wanted to summarize the differences between the vhost-pci and
> > virtio-vhost-user approaches because previous discussions may have been
> > confusing.
> > 
> > vhost-pci defines a new virtio device type for each vhost device type
> > (net, scsi, blk).  It therefore requires a virtio device driver for each
> > device type inside the slave VM.
> > 
> > Adding a new device type requires:
> > 1. Defining a new virtio device type in the VIRTIO specification.
> > 3. Implementing a new QEMU device model.
> > 2. Implementing a new virtio driver.
> > 
> > virtio-vhost-user is a single virtio device that acts as a vhost-user
> > protocol transport for any vhost device type.  It requires one virtio
> > driver inside the slave VM and device types are implemented using
> > existing vhost-user slave libraries (librte_vhost in DPDK and
> > libvhost-user in QEMU).
> > 
> > Adding a new device type to virtio-vhost-user involves:
> > 1. Adding any new vhost-user protocol messages to the QEMU
> >     virtio-vhost-user device model.
> > 2. Adding any new vhost-user protocol messages to the vhost-user slave
> >     library.
> > 3. Implementing the new device slave.
> > 
> > The simplest case is when no new vhost-user protocol messages are
> > required for the new device.  Then all that's needed for
> > virtio-vhost-user is a device slave implementation (#3).  That slave
> > implementation will also work with AF_UNIX because the vhost-user slave
> > library hides the transport (AF_UNIX vs virtio-vhost-user).  Even
> > better, if another person has already implemented that device slave to
> > use with AF_UNIX then no new code is needed for virtio-vhost-user
> > support at all!
> > 
> > If you compare this to vhost-pci, it would be necessary to design a new
> > virtio device, implement it in QEMU, and implement the virtio driver.
> > Much of virtio driver is more or less the same thing as the vhost-user
> > device slave but it cannot be reused because the vhost-user protocol
> > isn't being used by the virtio device.  The result is a lot of
> > duplication in DPDK and other codebases that implement vhost-user
> > slaves.
> > 
> > The way that vhost-pci is designed means that anyone wishing to support
> > a new device type has to become a virtio device designer.  They need to
> > map vhost-user protocol concepts to a new virtio device type.  This will
> > be time-consuming for everyone involved (e.g. the developer, the VIRTIO
> > community, etc).
> > 
> > The virtio-vhost-user approach stays at the vhost-user protocol level as
> > much as possible.  This way there are fewer concepts that need to be
> > mapped by people adding new device types.  As a result, it will allow
> > virtio-vhost-user to keep up with AF_UNIX vhost-user and grow because
> > it's easier to work with.
> > 
> > What do you think?
> > 
> > Stefan
> 
> So a question is what's the motivation here?
> 
> Form what I'm understanding, vhost-pci tries to build a scalable V2V private
> datapath. But according to what you describe here, virito-vhost-user tries
> to make it possible to implement the device inside another VM. I understand
> the goal of vhost-pci could be done on top, but it looks to me it would then
> rather similar to the design of Xen driver domain. So I can not figure out
> how it can be done in a high performance way.

vhost-pci and virtio-vhost-user both have the same goal.  They allow
a VM to implement a vhost device (net, scsi, blk, etc).  This allows
software defined network or storage appliances running inside a VM to
provide I/O services to other VMs.  To the other VMs the devices look
like regular virtio devices.

I'm not sure I understand your reference to the Xen driver domain or
performance.  Both vhost-pci and virtio-vhost-user work using shared
memory access to the guest RAM of the other VM.  Therefore they can poll
virtqueues and avoid vmexit.  They do also support cross-VM interrupts,
thanks to QEMU setting up irqfd/ioeventfd appropriately on the host.

Stefan

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 455 bytes --]

Re: [Qemu-devel] vhost-pci and virtio-vhost-user

[Jason Wang]

On 2018年01月11日 23:23, Stefan Hajnoczi wrote:
> On Thu, Jan 11, 2018 at 06:57:03PM +0800, Jason Wang wrote:
>>
>> On 2018年01月11日 00:14, Stefan Hajnoczi wrote:
>>> Hi Wei,
>>> I wanted to summarize the differences between the vhost-pci and
>>> virtio-vhost-user approaches because previous discussions may have been
>>> confusing.
>>>
>>> vhost-pci defines a new virtio device type for each vhost device type
>>> (net, scsi, blk).  It therefore requires a virtio device driver for each
>>> device type inside the slave VM.
>>>
>>> Adding a new device type requires:
>>> 1. Defining a new virtio device type in the VIRTIO specification.
>>> 3. Implementing a new QEMU device model.
>>> 2. Implementing a new virtio driver.
>>>
>>> virtio-vhost-user is a single virtio device that acts as a vhost-user
>>> protocol transport for any vhost device type.  It requires one virtio
>>> driver inside the slave VM and device types are implemented using
>>> existing vhost-user slave libraries (librte_vhost in DPDK and
>>> libvhost-user in QEMU).
>>>
>>> Adding a new device type to virtio-vhost-user involves:
>>> 1. Adding any new vhost-user protocol messages to the QEMU
>>>      virtio-vhost-user device model.
>>> 2. Adding any new vhost-user protocol messages to the vhost-user slave
>>>      library.
>>> 3. Implementing the new device slave.
>>>
>>> The simplest case is when no new vhost-user protocol messages are
>>> required for the new device.  Then all that's needed for
>>> virtio-vhost-user is a device slave implementation (#3).  That slave
>>> implementation will also work with AF_UNIX because the vhost-user slave
>>> library hides the transport (AF_UNIX vs virtio-vhost-user).  Even
>>> better, if another person has already implemented that device slave to
>>> use with AF_UNIX then no new code is needed for virtio-vhost-user
>>> support at all!
>>>
>>> If you compare this to vhost-pci, it would be necessary to design a new
>>> virtio device, implement it in QEMU, and implement the virtio driver.
>>> Much of virtio driver is more or less the same thing as the vhost-user
>>> device slave but it cannot be reused because the vhost-user protocol
>>> isn't being used by the virtio device.  The result is a lot of
>>> duplication in DPDK and other codebases that implement vhost-user
>>> slaves.
>>>
>>> The way that vhost-pci is designed means that anyone wishing to support
>>> a new device type has to become a virtio device designer.  They need to
>>> map vhost-user protocol concepts to a new virtio device type.  This will
>>> be time-consuming for everyone involved (e.g. the developer, the VIRTIO
>>> community, etc).
>>>
>>> The virtio-vhost-user approach stays at the vhost-user protocol level as
>>> much as possible.  This way there are fewer concepts that need to be
>>> mapped by people adding new device types.  As a result, it will allow
>>> virtio-vhost-user to keep up with AF_UNIX vhost-user and grow because
>>> it's easier to work with.
>>>
>>> What do you think?
>>>
>>> Stefan
>> So a question is what's the motivation here?
>>
>> Form what I'm understanding, vhost-pci tries to build a scalable V2V private
>> datapath. But according to what you describe here, virito-vhost-user tries
>> to make it possible to implement the device inside another VM. I understand
>> the goal of vhost-pci could be done on top, but it looks to me it would then
>> rather similar to the design of Xen driver domain. So I can not figure out
>> how it can be done in a high performance way.
> vhost-pci and virtio-vhost-user both have the same goal.  They allow
> a VM to implement a vhost device (net, scsi, blk, etc).

Looks not, if I read the code correctly, vhost-pci has a device 
implementation in qemu, and in slave VM it only have a vhost-pci-net driver.

> This allows
> software defined network or storage appliances running inside a VM to
> provide I/O services to other VMs.

Well, I think we can do it even with the existed virtio or whatever 
other emulated device which should not be bounded to any specific kind 
of device.  And what's more important, according to the kvm 2016 slides 
of vhost-pci, the motivation of vhost-pci is not building SDN but a 
chain of VNFs. So bypassing the central vswitch through a private VM2VM 
path does make sense. (Though whether or not vhost-pci is the best 
choice is still questionable).

>   To the other VMs the devices look
> like regular virtio devices.
>
> I'm not sure I understand your reference to the Xen driver domain or
> performance.

So what proposed here is basically memory sharing and event notification 
through eventfd, this model have been used by Xen for many years through 
grant table and event channel. Xen use this to move the backend 
implementation from dom0 to a driver domain which has direct access to 
some hardwares. Consider the case of network, it can then implement xen 
netback inside driver domain which can access hardware NIC directly.

This makes sense for Xen and for performance since driver domain 
(backend) can access hardware directly and event was triggered through 
lower overhead hypercall (or it can do busypolling). But for 
virtio-vhost-user, unless you want SRIOV based solutions inside the 
slave VM, I believe we won't want to go back to Xen since the hardware 
virtualization can bring extra overheads.

>   Both vhost-pci and virtio-vhost-user work using shared
> memory access to the guest RAM of the other VM.  Therefore they can poll
> virtqueues and avoid vmexit.  They do also support cross-VM interrupts,
> thanks to QEMU setting up irqfd/ioeventfd appropriately on the host.
>
> Stefan

So in conclusion, consider the complexity, I would suggest to figure out 
whether or not this (either vhost-pci or virito-vhost-user) is really 
required before moving ahead. E.g, for VM2VM direct network path, this 
looks simply an issue of network topology instead of the problem of 
device, so there's a lot of trick, for vhost-user one can easily image 
to write an application (or use testpmd) to build a zerocopied VM2VM 
datapath, isn't this not sufficient for the case?

Thanks

Re: [Qemu-devel] vhost-pci and virtio-vhost-user

[Yang, Zhiyong]

Hi Jason,

> -----Original Message-----
> From: Qemu-devel [mailto:qemu-devel-
> bounces+zhiyong.yang=intel.com@nongnu.org] On Behalf Of Jason Wang
> Sent: Friday, January 12, 2018 11:33 AM
> To: Stefan Hajnoczi <stefanha@redhat.com>
> Cc: Wang, Wei W <wei.w.wang@intel.com>; qemu-devel@nongnu.org
> Subject: Re: [Qemu-devel] vhost-pci and virtio-vhost-user
> 
> 
> 
> On 2018年01月11日 23:23, Stefan Hajnoczi wrote:
> > On Thu, Jan 11, 2018 at 06:57:03PM +0800, Jason Wang wrote:
> >>
> >> On 2018年01月11日 00:14, Stefan Hajnoczi wrote:
> >>> Hi Wei,
> >>> I wanted to summarize the differences between the vhost-pci and
> >>> virtio-vhost-user approaches because previous discussions may have
> >>> been confusing.
> >>>
> >>> vhost-pci defines a new virtio device type for each vhost device
> >>> type (net, scsi, blk).  It therefore requires a virtio device driver
> >>> for each device type inside the slave VM.
> >>>
> >>> Adding a new device type requires:
> >>> 1. Defining a new virtio device type in the VIRTIO specification.
> >>> 3. Implementing a new QEMU device model.
> >>> 2. Implementing a new virtio driver.
> >>>
> >>> virtio-vhost-user is a single virtio device that acts as a
> >>> vhost-user protocol transport for any vhost device type.  It
> >>> requires one virtio driver inside the slave VM and device types are
> >>> implemented using existing vhost-user slave libraries (librte_vhost
> >>> in DPDK and libvhost-user in QEMU).
> >>>
> >>> Adding a new device type to virtio-vhost-user involves:
> >>> 1. Adding any new vhost-user protocol messages to the QEMU
> >>>      virtio-vhost-user device model.
> >>> 2. Adding any new vhost-user protocol messages to the vhost-user slave
> >>>      library.
> >>> 3. Implementing the new device slave.
> >>>
> >>> The simplest case is when no new vhost-user protocol messages are
> >>> required for the new device.  Then all that's needed for
> >>> virtio-vhost-user is a device slave implementation (#3).  That slave
> >>> implementation will also work with AF_UNIX because the vhost-user
> >>> slave library hides the transport (AF_UNIX vs virtio-vhost-user).
> >>> Even better, if another person has already implemented that device
> >>> slave to use with AF_UNIX then no new code is needed for
> >>> virtio-vhost-user support at all!
> >>>
> >>> If you compare this to vhost-pci, it would be necessary to design a
> >>> new virtio device, implement it in QEMU, and implement the virtio driver.
> >>> Much of virtio driver is more or less the same thing as the
> >>> vhost-user device slave but it cannot be reused because the
> >>> vhost-user protocol isn't being used by the virtio device.  The
> >>> result is a lot of duplication in DPDK and other codebases that
> >>> implement vhost-user slaves.
> >>>
> >>> The way that vhost-pci is designed means that anyone wishing to
> >>> support a new device type has to become a virtio device designer.
> >>> They need to map vhost-user protocol concepts to a new virtio device
> >>> type.  This will be time-consuming for everyone involved (e.g. the
> >>> developer, the VIRTIO community, etc).
> >>>
> >>> The virtio-vhost-user approach stays at the vhost-user protocol
> >>> level as much as possible.  This way there are fewer concepts that
> >>> need to be mapped by people adding new device types.  As a result,
> >>> it will allow virtio-vhost-user to keep up with AF_UNIX vhost-user
> >>> and grow because it's easier to work with.
> >>>
> >>> What do you think?
> >>>
> >>> Stefan
> >> So a question is what's the motivation here?
> >>
> >> Form what I'm understanding, vhost-pci tries to build a scalable V2V
> >> private datapath. But according to what you describe here,
> >> virito-vhost-user tries to make it possible to implement the device
> >> inside another VM. I understand the goal of vhost-pci could be done
> >> on top, but it looks to me it would then rather similar to the design
> >> of Xen driver domain. So I can not figure out how it can be done in a high
> performance way.
> > vhost-pci and virtio-vhost-user both have the same goal.  They allow a
> > VM to implement a vhost device (net, scsi, blk, etc).
> 
> Looks not, if I read the code correctly, vhost-pci has a device implementation
> in qemu, and in slave VM it only have a vhost-pci-net driver.
> 
> > This allows
> > software defined network or storage appliances running inside a VM to
> > provide I/O services to other VMs.
> 
> Well, I think we can do it even with the existed virtio or whatever other
> emulated device which should not be bounded to any specific kind of
> device.  And what's more important, according to the kvm 2016 slides of
> vhost-pci, the motivation of vhost-pci is not building SDN but a chain of VNFs.
> So bypassing the central vswitch through a private VM2VM path does make
> sense. (Though whether or not vhost-pci is the best choice is still
> questionable).
> 
> >   To the other VMs the devices look
> > like regular virtio devices.
> >
> > I'm not sure I understand your reference to the Xen driver domain or
> > performance.
> 
> So what proposed here is basically memory sharing and event notification
> through eventfd, this model have been used by Xen for many years through
> grant table and event channel. Xen use this to move the backend
> implementation from dom0 to a driver domain which has direct access to
> some hardwares. Consider the case of network, it can then implement xen
> netback inside driver domain which can access hardware NIC directly.
> 
> This makes sense for Xen and for performance since driver domain
> (backend) can access hardware directly and event was triggered through
> lower overhead hypercall (or it can do busypolling). But for virtio-vhost-user,
> unless you want SRIOV based solutions inside the slave VM, I believe we
> won't want to go back to Xen since the hardware virtualization can bring
> extra overheads.
> 
> >   Both vhost-pci and virtio-vhost-user work using shared memory access
> > to the guest RAM of the other VM.  Therefore they can poll virtqueues
> > and avoid vmexit.  They do also support cross-VM interrupts, thanks to
> > QEMU setting up irqfd/ioeventfd appropriately on the host.
> >
> > Stefan
> 
> So in conclusion, consider the complexity, I would suggest to figure out
> whether or not this (either vhost-pci or virito-vhost-user) is really required
> before moving ahead. E.g, for VM2VM direct network path, this looks simply
> an issue of network topology instead of the problem of device, so there's a
> lot of trick, for vhost-user one can easily image to write an application (or use
> testpmd) to build a zerocopied VM2VM datapath, isn't this not sufficient for
> the case?

As far as I know,  dequeue zero copied feature of vhost user PMD can't help improve throughput for small packest ,such as 64 bytes.
On the contrary, it causes perf drop.  The feature mainly helps large packets throughput.
   
Vhostpci can bring the following advantages compared to traditional solution(vhost/virtio PMD pairs)
1.  higher throughput for two VMs. ( Let us see the following  case,  if we use NIC passthrough way to two 2 VMs,  vhostpci RX or TX is handled  running 1 single core in VM1,  virtio PMD  is similar on VM2, 
Only RX or TX is handled running on one single core.
for traditional solution,  except each virtio PMD is running inside each VM,  at least one extra core is needed for vhost user RX and TX as an mediator. 
In this case, the bottleneck lies in the two vhost user ports running on one single core, which has double workload. 

2. Low latencies (have shorter data path than tradition soluton, doesn't need to pass host OS any more by vhost user)

3. reduce nearly 50% cores  because  OVS is not involved again if we apply vhostpci/virtio to VMs-chain case.

Thanks
Zhiyong

> 
> Thanks

Re: [Qemu-devel] vhost-pci and virtio-vhost-user

[Wei Wang]

On 01/11/2018 05:56 PM, Stefan Hajnoczi wrote:
> On Thu, Jan 11, 2018 at 6:31 AM, Wei Wang <wei.w.wang@intel.com> wrote:
>> On 01/11/2018 12:14 AM, Stefan Hajnoczi wrote:
>>
>> 2) requires the driver to join the vhost-user negotiation.
> The driver must participate in vhost-user negotiation.  The vhost-pci
> patches try to avoid this by taking features bits on the QEMU
> command-line and hardcoding the number of supported virtqueues.  That
> doesn't work in production environments because:
> 1. What if the driver inside the VM has been updated and now supports
> different features?
> 2. What if the user isn't allowed to modify the VM configuration?
> 3. What if the management software doesn't expose the feature bits
> command-line parameter?
> 4. What if the number of virtqueues must be different from QEMU's
> default value to limit resource consumption?
>
> Users will find it incovenient to manually enter feature bits for the
> driver they are using.  The driver needs to be part of negotiation so
> it can indicate which features are supported, how many virtqueues,
> etc.
>

OK, the advantages of letting the guest join the vhost-user negotiation 
sounds convincing to me. Thanks.



>> Without above two, the solution already works well, so I'm not sure why would we need the above two from functionality point of view.
> The "[PATCH v3 0/7] Vhost-pci for inter-VM communication" series is
> incomplete.  It is a subset of vhost-user-net and it works only for
> poll-mode drivers.  It's the requirements that haven't been covered by
> the vhost-pci patch series yet that make me prefer the
> virtio-vhost-user approach.
>
> The virtio device design needs to be capable of supporting the rest of
> vhost-user functionality in the future.  Once the code is merged in
> QEMU and DPDK it will be very difficult to make changes to the virtio
> device.

This is how virtio works. A new feature with a new feature bit. Now, we 
let the guest driver join the vhost-user negotiation (including feature 
negotiation), the default device/driver feature negotiation is free to use.
I'm thinking if it is worthwhile to do some kind of mediated 
passthrough, which passes the selected messages only. Because many 
messages are not necessary to be passed down (e.g. VHOST_USER_SEND_RARP 
is not needed for simple VM-to-VM communication), though might be safe 
to do. I plan to see your full passthrough code first, and see if 
changing to mediated passthrough would be simpler.

Best,
Wei

Re: [Qemu-devel] vhost-pci and virtio-vhost-user

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 7992 bytes --]

On Fri, Jan 12, 2018 at 11:32:56AM +0800, Jason Wang wrote:
> 
> 
> On 2018年01月11日 23:23, Stefan Hajnoczi wrote:
> > On Thu, Jan 11, 2018 at 06:57:03PM +0800, Jason Wang wrote:
> > > 
> > > On 2018年01月11日 00:14, Stefan Hajnoczi wrote:
> > > > Hi Wei,
> > > > I wanted to summarize the differences between the vhost-pci and
> > > > virtio-vhost-user approaches because previous discussions may have been
> > > > confusing.
> > > > 
> > > > vhost-pci defines a new virtio device type for each vhost device type
> > > > (net, scsi, blk).  It therefore requires a virtio device driver for each
> > > > device type inside the slave VM.
> > > > 
> > > > Adding a new device type requires:
> > > > 1. Defining a new virtio device type in the VIRTIO specification.
> > > > 3. Implementing a new QEMU device model.
> > > > 2. Implementing a new virtio driver.
> > > > 
> > > > virtio-vhost-user is a single virtio device that acts as a vhost-user
> > > > protocol transport for any vhost device type.  It requires one virtio
> > > > driver inside the slave VM and device types are implemented using
> > > > existing vhost-user slave libraries (librte_vhost in DPDK and
> > > > libvhost-user in QEMU).
> > > > 
> > > > Adding a new device type to virtio-vhost-user involves:
> > > > 1. Adding any new vhost-user protocol messages to the QEMU
> > > >      virtio-vhost-user device model.
> > > > 2. Adding any new vhost-user protocol messages to the vhost-user slave
> > > >      library.
> > > > 3. Implementing the new device slave.
> > > > 
> > > > The simplest case is when no new vhost-user protocol messages are
> > > > required for the new device.  Then all that's needed for
> > > > virtio-vhost-user is a device slave implementation (#3).  That slave
> > > > implementation will also work with AF_UNIX because the vhost-user slave
> > > > library hides the transport (AF_UNIX vs virtio-vhost-user).  Even
> > > > better, if another person has already implemented that device slave to
> > > > use with AF_UNIX then no new code is needed for virtio-vhost-user
> > > > support at all!
> > > > 
> > > > If you compare this to vhost-pci, it would be necessary to design a new
> > > > virtio device, implement it in QEMU, and implement the virtio driver.
> > > > Much of virtio driver is more or less the same thing as the vhost-user
> > > > device slave but it cannot be reused because the vhost-user protocol
> > > > isn't being used by the virtio device.  The result is a lot of
> > > > duplication in DPDK and other codebases that implement vhost-user
> > > > slaves.
> > > > 
> > > > The way that vhost-pci is designed means that anyone wishing to support
> > > > a new device type has to become a virtio device designer.  They need to
> > > > map vhost-user protocol concepts to a new virtio device type.  This will
> > > > be time-consuming for everyone involved (e.g. the developer, the VIRTIO
> > > > community, etc).
> > > > 
> > > > The virtio-vhost-user approach stays at the vhost-user protocol level as
> > > > much as possible.  This way there are fewer concepts that need to be
> > > > mapped by people adding new device types.  As a result, it will allow
> > > > virtio-vhost-user to keep up with AF_UNIX vhost-user and grow because
> > > > it's easier to work with.
> > > > 
> > > > What do you think?
> > > > 
> > > > Stefan
> > > So a question is what's the motivation here?
> > > 
> > > Form what I'm understanding, vhost-pci tries to build a scalable V2V private
> > > datapath. But according to what you describe here, virito-vhost-user tries
> > > to make it possible to implement the device inside another VM. I understand
> > > the goal of vhost-pci could be done on top, but it looks to me it would then
> > > rather similar to the design of Xen driver domain. So I can not figure out
> > > how it can be done in a high performance way.
> > vhost-pci and virtio-vhost-user both have the same goal.  They allow
> > a VM to implement a vhost device (net, scsi, blk, etc).
> 
> Looks not, if I read the code correctly, vhost-pci has a device
> implementation in qemu, and in slave VM it only have a vhost-pci-net driver.

You are right that the current "[PATCH v3 0/7] Vhost-pci for inter-VM
communication" does not reach this goal yet.  The patch series focusses
on a subset of vhost-user-net for poll mode drivers.

But the goal is to eventually let VMs implement any vhost device type.
Even if Wei, you, or I don't implement scsi, for example, someone else
should be able to do it based on vhost-pci or virtio-vhost-user.

Wei: Do you agree?

> > This allows
> > software defined network or storage appliances running inside a VM to
> > provide I/O services to other VMs.
> 
> Well, I think we can do it even with the existed virtio or whatever other
> emulated device which should not be bounded to any specific kind of device. 

Please explain the approach you have in mind.

> And what's more important, according to the kvm 2016 slides of vhost-pci,
> the motivation of vhost-pci is not building SDN but a chain of VNFs. So
> bypassing the central vswitch through a private VM2VM path does make sense.
> (Though whether or not vhost-pci is the best choice is still questionable).

This is probably my fault.  Maybe my networking terminology is wrong.  I
consider "virtual network functions" to be part of "software-defined
networking" use cases.  I'm not implying there must be a central virtual
switch.

To rephrase: vhost-pci enables exitless VM2VM communication.

> >   To the other VMs the devices look
> > like regular virtio devices.
> > 
> > I'm not sure I understand your reference to the Xen driver domain or
> > performance.
> 
> So what proposed here is basically memory sharing and event notification
> through eventfd, this model have been used by Xen for many years through
> grant table and event channel. Xen use this to move the backend
> implementation from dom0 to a driver domain which has direct access to some
> hardwares. Consider the case of network, it can then implement xen netback
> inside driver domain which can access hardware NIC directly.
>
> This makes sense for Xen and for performance since driver domain (backend)
> can access hardware directly and event was triggered through lower overhead
> hypercall (or it can do busypolling). But for virtio-vhost-user, unless you
> want SRIOV based solutions inside the slave VM, I believe we won't want to
> go back to Xen since the hardware virtualization can bring extra overheads.

Okay, this point is about the NFV use case.  I can't answer that because
I'm not familiar with it.

Even if the NFV use case is not ideal for VMs, there are many other use
cases for VMs implementing vhost devices.  In the cloud the VM is the
first-class object that users can manage.  They do not have the ability
to run vhost-user processes on the host.  Therefore I/O appliances need
to be able to run as VMs and vhost-pci (or virtio-vhost-user) solve that
problem.

> >   Both vhost-pci and virtio-vhost-user work using shared
> > memory access to the guest RAM of the other VM.  Therefore they can poll
> > virtqueues and avoid vmexit.  They do also support cross-VM interrupts,
> > thanks to QEMU setting up irqfd/ioeventfd appropriately on the host.
> > 
> > Stefan
> 
> So in conclusion, consider the complexity, I would suggest to figure out
> whether or not this (either vhost-pci or virito-vhost-user) is really
> required before moving ahead. E.g, for VM2VM direct network path, this looks
> simply an issue of network topology instead of the problem of device, so
> there's a lot of trick, for vhost-user one can easily image to write an
> application (or use testpmd) to build a zerocopied VM2VM datapath, isn't
> this not sufficient for the case?

See above, I described the general cloud I/O appliance use case.

Stefan

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 455 bytes --]

Re: [Qemu-devel] vhost-pci and virtio-vhost-user

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 2493 bytes --]

On Fri, Jan 12, 2018 at 02:44:00PM +0800, Wei Wang wrote:
> On 01/11/2018 05:56 PM, Stefan Hajnoczi wrote:
> > On Thu, Jan 11, 2018 at 6:31 AM, Wei Wang <wei.w.wang@intel.com> wrote:
> > > On 01/11/2018 12:14 AM, Stefan Hajnoczi wrote:
> > > Without above two, the solution already works well, so I'm not sure why would we need the above two from functionality point of view.
> > The "[PATCH v3 0/7] Vhost-pci for inter-VM communication" series is
> > incomplete.  It is a subset of vhost-user-net and it works only for
> > poll-mode drivers.  It's the requirements that haven't been covered by
> > the vhost-pci patch series yet that make me prefer the
> > virtio-vhost-user approach.
> > 
> > The virtio device design needs to be capable of supporting the rest of
> > vhost-user functionality in the future.  Once the code is merged in
> > QEMU and DPDK it will be very difficult to make changes to the virtio
> > device.
> 
> This is how virtio works. A new feature with a new feature bit.

Although it is possible to add a feature bit that radically changes the
virtio device's interface, the existing driver software would need to be
largely rewritten - all the way up to the net, scsi, blk, etc devices.

We cannot easily migrate between the vhost-pci to virtio-vhost-user
approaches later.

> Now, we let
> the guest driver join the vhost-user negotiation (including feature
> negotiation), the default device/driver feature negotiation is free to use.
> I'm thinking if it is worthwhile to do some kind of mediated passthrough,
> which passes the selected messages only. Because many messages are not
> necessary to be passed down (e.g. VHOST_USER_SEND_RARP is not needed for
> simple VM-to-VM communication), though might be safe to do. I plan to see
> your full passthrough code first, and see if changing to mediated
> passthrough would be simpler.

I expect vhost-pci to require fewer code changes.  If you judge
"simpler" just by the patch count or size, then vhost-pci will win.

The reason for that is virtio-vhost-user integrates with librte_vhost.
This requires refactoring librte_vhost to support multiple transports.

I think the virtio-vhost-user end result is worth it though: vhost
devices like examples/vhost/ and examples/vhost/scsi/ will work with
both AF_UNIX and virtio-vhost-user.  This makes it simpler for users and
vhost device developers - you only have one implementation of net, scsi,
blk, etc devices.

Stefan

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 455 bytes --]

Re: [Qemu-devel] vhost-pci and virtio-vhost-user

[Wang, Wei W]

On Friday, January 12, 2018 6:38 PM, Stefan Hajnoczi wrote:
> On Fri, Jan 12, 2018 at 02:44:00PM +0800, Wei Wang wrote:
> > On 01/11/2018 05:56 PM, Stefan Hajnoczi wrote:
> > > On Thu, Jan 11, 2018 at 6:31 AM, Wei Wang <wei.w.wang@intel.com>
> wrote:
> > > > On 01/11/2018 12:14 AM, Stefan Hajnoczi wrote:
> >
> I expect vhost-pci to require fewer code changes.  If you judge "simpler" just
> by the patch count or size, then vhost-pci will win.
> 
> The reason for that is virtio-vhost-user integrates with librte_vhost.
> This requires refactoring librte_vhost to support multiple transports.
> 

I think the driver having more code is fine. The device part is what will go into the spec, and people have the freedom to implement their own driver (either the dpdk or kernel driver) in the future, if they don't want the librte_vhost based one.

So, for the first version, I think it would be better to not make the device full-featured (e.g. pass through as little messages as possible), this would give users a chance to get a device and driver with the least lines of code.  


> I think the virtio-vhost-user end result is worth it though: vhost devices like
> examples/vhost/ and examples/vhost/scsi/ will work with both AF_UNIX and
> virtio-vhost-user.  This makes it simpler for users and vhost device
> developers - you only have one implementation of net, scsi, blk, etc devices.

For the driver part, isn't it that net, scsi, blk will have their own separate implementation on top of the common libret_vhost?

Best,
Wei

Re: [Qemu-devel] vhost-pci and virtio-vhost-user

[Jason Wang]

On 2018年01月12日 13:20, Yang, Zhiyong wrote:
>>>    Both vhost-pci and virtio-vhost-user work using shared memory access
>>> to the guest RAM of the other VM.  Therefore they can poll virtqueues
>>> and avoid vmexit.  They do also support cross-VM interrupts, thanks to
>>> QEMU setting up irqfd/ioeventfd appropriately on the host.
>>>
>>> Stefan
>> So in conclusion, consider the complexity, I would suggest to figure out
>> whether or not this (either vhost-pci or virito-vhost-user) is really required
>> before moving ahead. E.g, for VM2VM direct network path, this looks simply
>> an issue of network topology instead of the problem of device, so there's a
>> lot of trick, for vhost-user one can easily image to write an application (or use
>> testpmd) to build a zerocopied VM2VM datapath, isn't this not sufficient for
>> the case?
> As far as I know,  dequeue zero copied feature of vhost user PMD can't help improve throughput for small packest ,such as 64 bytes.
> On the contrary, it causes perf drop.  The feature mainly helps large packets throughput.

Can you explain why? And what's the number of:

1) 64B/1500B zerocopy
2) 64B/1500B datacopy
3) 64B/1500B vhost-pci

It makes make feel that vhost-pci is dedicated for small bytes? We 
probably don't want a solution for just a specific size of packets.

>     
> Vhostpci can bring the following advantages compared to traditional solution(vhost/virtio PMD pairs)
> 1.  higher throughput for two VMs. ( Let us see the following  case,  if we use NIC passthrough way to two 2 VMs,  vhostpci RX or TX is handled  running 1 single core in VM1,  virtio PMD  is similar on VM2,
> Only RX or TX is handled running on one single core.
> for traditional solution,  except each virtio PMD is running inside each VM,  at least one extra core is needed for vhost user RX and TX as an mediator.
> In this case, the bottleneck lies in the two vhost user ports running on one single core, which has double workload.

Does this still make sense for packet size other than 64 byte (e.g 1500B)?

>   
>
> 2. Low latencies (have shorter data path than tradition soluton, doesn't need to pass host OS any more by vhost user)

Is this still true if you do busy polling on both sides?

>
> 3. reduce nearly 50% cores  because  OVS is not involved again if we apply vhostpci/virtio to VMs-chain case.

Well the differences to me is, copy in guest vs copy in host.

- vhost-pci move the copy from host process to pmd in guest, it probably 
save cores but sacrifice the performance of pmd which needs do copy now
- exist OVS may occupy more cores in host, but if saves the ability of 
guest pmd

 From the view of performance, it looks to me that copy in host is 
faster since it has less overhead e.g vmexits. Vhost-pci probably needs 
more vcpus to compete with current solution.

Thanks

>
> Thanks
> Zhiyong
>

Re: [Qemu-devel] vhost-pci and virtio-vhost-user

[Jason Wang]

On 2018年01月12日 18:18, Stefan Hajnoczi wrote:
>>>> Form what I'm understanding, vhost-pci tries to build a scalable V2V private
>>>> datapath. But according to what you describe here, virito-vhost-user tries
>>>> to make it possible to implement the device inside another VM. I understand
>>>> the goal of vhost-pci could be done on top, but it looks to me it would then
>>>> rather similar to the design of Xen driver domain. So I can not figure out
>>>> how it can be done in a high performance way.
>>> vhost-pci and virtio-vhost-user both have the same goal.  They allow
>>> a VM to implement a vhost device (net, scsi, blk, etc).
>> Looks not, if I read the code correctly, vhost-pci has a device
>> implementation in qemu, and in slave VM it only have a vhost-pci-net driver.
> You are right that the current "[PATCH v3 0/7] Vhost-pci for inter-VM
> communication" does not reach this goal yet.  The patch series focusses
> on a subset of vhost-user-net for poll mode drivers.
>
> But the goal is to eventually let VMs implement any vhost device type.
> Even if Wei, you, or I don't implement scsi, for example, someone else
> should be able to do it based on vhost-pci or virtio-vhost-user.
>
> Wei: Do you agree?
>
>>> This allows
>>> software defined network or storage appliances running inside a VM to
>>> provide I/O services to other VMs.
>> Well, I think we can do it even with the existed virtio or whatever other
>> emulated device which should not be bounded to any specific kind of device.
> Please explain the approach you have in mind.

I just fail understand why we can't do software defined network or 
storage with exist virtio device/drivers (or are there any shortcomings 
that force us to invent new infrastructure).

>
>> And what's more important, according to the kvm 2016 slides of vhost-pci,
>> the motivation of vhost-pci is not building SDN but a chain of VNFs. So
>> bypassing the central vswitch through a private VM2VM path does make sense.
>> (Though whether or not vhost-pci is the best choice is still questionable).
> This is probably my fault.  Maybe my networking terminology is wrong.  I
> consider "virtual network functions" to be part of "software-defined
> networking" use cases.  I'm not implying there must be a central virtual
> switch.
>
> To rephrase: vhost-pci enables exitless VM2VM communication.

The problem is, exitless is not what vhost-pci invents, it could be 
achieved now when both sides are doing busypolling.

>
>>>    To the other VMs the devices look
>>> like regular virtio devices.
>>>
>>> I'm not sure I understand your reference to the Xen driver domain or
>>> performance.
>> So what proposed here is basically memory sharing and event notification
>> through eventfd, this model have been used by Xen for many years through
>> grant table and event channel. Xen use this to move the backend
>> implementation from dom0 to a driver domain which has direct access to some
>> hardwares. Consider the case of network, it can then implement xen netback
>> inside driver domain which can access hardware NIC directly.
>>
>> This makes sense for Xen and for performance since driver domain (backend)
>> can access hardware directly and event was triggered through lower overhead
>> hypercall (or it can do busypolling). But for virtio-vhost-user, unless you
>> want SRIOV based solutions inside the slave VM, I believe we won't want to
>> go back to Xen since the hardware virtualization can bring extra overheads.
> Okay, this point is about the NFV use case.  I can't answer that because
> I'm not familiar with it.
>
> Even if the NFV use case is not ideal for VMs, there are many other use
> cases for VMs implementing vhost devices.  In the cloud the VM is the
> first-class object that users can manage.  They do not have the ability
> to run vhost-user processes on the host.  Therefore I/O appliances need
> to be able to run as VMs and vhost-pci (or virtio-vhost-user) solve that
> problem.

The question is why must use vhost-user? E.g in the case of SDN, you can 
easily deploy an OVS instance with openflow inside a VM and it works 
like a charm.

>
>>>    Both vhost-pci and virtio-vhost-user work using shared
>>> memory access to the guest RAM of the other VM.  Therefore they can poll
>>> virtqueues and avoid vmexit.  They do also support cross-VM interrupts,
>>> thanks to QEMU setting up irqfd/ioeventfd appropriately on the host.
>>>
>>> Stefan
>> So in conclusion, consider the complexity, I would suggest to figure out
>> whether or not this (either vhost-pci or virito-vhost-user) is really
>> required before moving ahead. E.g, for VM2VM direct network path, this looks
>> simply an issue of network topology instead of the problem of device, so
>> there's a lot of trick, for vhost-user one can easily image to write an
>> application (or use testpmd) to build a zerocopied VM2VM datapath, isn't
>> this not sufficient for the case?
> See above, I described the general cloud I/O appliance use case.
>
> Stefan

So I understand vhost-user could be used to build I/O appliance. What I 
don't understand is, the advantages of using vhost-user or why we must 
use it inside a guest.

Thanks

Re: [Qemu-devel] vhost-pci and virtio-vhost-user

[Wei Wang]

On 01/12/2018 06:18 PM, Stefan Hajnoczi wrote:
> On Fri, Jan 12, 2018 at 11:32:56AM +0800, Jason Wang wrote:
>>
>> On 2018年01月11日 23:23, Stefan Hajnoczi wrote:
>>> On Thu, Jan 11, 2018 at 06:57:03PM +0800, Jason Wang wrote:
>>>> On 2018年01月11日 00:14, Stefan Hajnoczi wrote:
>>>
>> Looks not, if I read the code correctly, vhost-pci has a device
>> implementation in qemu, and in slave VM it only have a vhost-pci-net driver.
> You are right that the current "[PATCH v3 0/7] Vhost-pci for inter-VM
> communication" does not reach this goal yet.  The patch series focusses
> on a subset of vhost-user-net for poll mode drivers.
>
> But the goal is to eventually let VMs implement any vhost device type.
> Even if Wei, you, or I don't implement scsi, for example, someone else
> should be able to do it based on vhost-pci or virtio-vhost-user.
>
> Wei: Do you agree?

Yes. I agree it would be good to have only one device, which performs 
the fundamental emulation (e.g. expose master VM memory via a bar) that 
is common to all the types (net, scsi..). The device should be able to 
tell the driver which type the master side is. So, at the QEMU device 
layer, we don't need to distinguish between device types, while 
regarding the driver, each device type has its own specific driver 
implementation, which is based on a common vhost-pci driver layer.


Best,
Wei

Re: [Qemu-devel] vhost-pci and virtio-vhost-user

[Wei Wang]

On 01/15/2018 02:56 PM, Jason Wang wrote:
>
>
> On 2018年01月12日 18:18, Stefan Hajnoczi wrote:
>>
>
> I just fail understand why we can't do software defined network or 
> storage with exist virtio device/drivers (or are there any 
> shortcomings that force us to invent new infrastructure).
>

Existing virtio-net works with a host central vSwitch, and it has the 
following disadvantages:
1) long code/data path;
2) poor scalability; and
3) host CPU sacrifice

Vhost-pci solves the above issues by providing a point-to-point 
communication between VMs. No matter how the control path would look 
like finally, the key point is that the data path is P2P between VMs.

Best,
Wei

Re: [Qemu-devel] vhost-pci and virtio-vhost-user

[Jason Wang]

On 2018年01月15日 15:59, Wei Wang wrote:
> On 01/15/2018 02:56 PM, Jason Wang wrote:
>>
>>
>> On 2018年01月12日 18:18, Stefan Hajnoczi wrote:
>>>
>>
>> I just fail understand why we can't do software defined network or 
>> storage with exist virtio device/drivers (or are there any 
>> shortcomings that force us to invent new infrastructure).
>>
>
> Existing virtio-net works with a host central vSwitch, and it has the 
> following disadvantages:
> 1) long code/data path;
> 2) poor scalability; and
> 3) host CPU sacrifice

Please show me the numbers.

>
> Vhost-pci solves the above issues by providing a point-to-point 
> communication between VMs. No matter how the control path would look 
> like finally, the key point is that the data path is P2P between VMs.
>
> Best,
> Wei
>
>

Well, I think I've pointed out several times in the replies of previous 
versions. Both vhost-pci-net and virtio-net is an ethernet device, which 
is not tied to a central vswitch for sure. There're just too many 
methods or tricks which can be used to build a point to point data path.

The problem is not whether or not you can build direct path between VMs 
through vhost-pci, the problem is why you must do it through vhost-pci.

Thanks

Re: [Qemu-devel] vhost-pci and virtio-vhost-user

[Wei Wang]

On 01/15/2018 04:34 PM, Jason Wang wrote:
>
>
> On 2018年01月15日 15:59, Wei Wang wrote:
>> On 01/15/2018 02:56 PM, Jason Wang wrote:
>>>
>>>
>>> On 2018年01月12日 18:18, Stefan Hajnoczi wrote:
>>>>
>>>
>>> I just fail understand why we can't do software defined network or 
>>> storage with exist virtio device/drivers (or are there any 
>>> shortcomings that force us to invent new infrastructure).
>>>
>>
>> Existing virtio-net works with a host central vSwitch, and it has the 
>> following disadvantages:
>> 1) long code/data path;
>> 2) poor scalability; and
>> 3) host CPU sacrifice
>
> Please show me the numbers.

Sure. For 64B packet transmission between two VMs: vhost-user reports 
~6.8Mpps, and vhost-pci reports ~11Mpps, which is ~1.62x faster.


>
>>
>> Vhost-pci solves the above issues by providing a point-to-point 
>> communication between VMs. No matter how the control path would look 
>> like finally, the key point is that the data path is P2P between VMs.
>>
>> Best,
>> Wei
>>
>>
>
> Well, I think I've pointed out several times in the replies of 
> previous versions. Both vhost-pci-net and virtio-net is an ethernet 
> device, which is not tied to a central vswitch for sure. There're just 
> too many methods or tricks which can be used to build a point to point 
> data path.


Could you please show an existing example that makes virtio-net work 
without a host vswitch/bridge?
Could you also share other p2p data path solutions that you have in 
mind? Thanks.


Best,
Wei

Re: [Qemu-devel] vhost-pci and virtio-vhost-user

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 1372 bytes --]

On Mon, Jan 15, 2018 at 02:56:31PM +0800, Jason Wang wrote:
> On 2018年01月12日 18:18, Stefan Hajnoczi wrote:
> > > And what's more important, according to the kvm 2016 slides of vhost-pci,
> > > the motivation of vhost-pci is not building SDN but a chain of VNFs. So
> > > bypassing the central vswitch through a private VM2VM path does make sense.
> > > (Though whether or not vhost-pci is the best choice is still questionable).
> > This is probably my fault.  Maybe my networking terminology is wrong.  I
> > consider "virtual network functions" to be part of "software-defined
> > networking" use cases.  I'm not implying there must be a central virtual
> > switch.
> > 
> > To rephrase: vhost-pci enables exitless VM2VM communication.
> 
> The problem is, exitless is not what vhost-pci invents, it could be achieved
> now when both sides are doing busypolling.

The only way I'm aware of is ivshmem.  But ivshmem lacks a family of
standard device types that allows different implementations to
interoperate.  We already have the virtio family of device types, so it
makes sense to work on a virtio-based solution.

Perhaps I've missed a different approach for exitless VM2VM
communication.  Please explain how VM1 and VM2 can do exitless network
communication today?

Also, how can VM1 provide SCSI I/O services to VM2 today?

Stefan

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 455 bytes --]

Re: [Qemu-devel] vhost-pci and virtio-vhost-user

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 1046 bytes --]

On Sun, Jan 14, 2018 at 03:36:19AM +0000, Wang, Wei W wrote:
> On Friday, January 12, 2018 6:38 PM, Stefan Hajnoczi wrote:
> > On Fri, Jan 12, 2018 at 02:44:00PM +0800, Wei Wang wrote:
> > > On 01/11/2018 05:56 PM, Stefan Hajnoczi wrote:
> > > > On Thu, Jan 11, 2018 at 6:31 AM, Wei Wang <wei.w.wang@intel.com>
> > wrote:
> > > > > On 01/11/2018 12:14 AM, Stefan Hajnoczi wrote:
> > I think the virtio-vhost-user end result is worth it though: vhost devices like
> > examples/vhost/ and examples/vhost/scsi/ will work with both AF_UNIX and
> > virtio-vhost-user.  This makes it simpler for users and vhost device
> > developers - you only have one implementation of net, scsi, blk, etc devices.
> 
> For the driver part, isn't it that net, scsi, blk will have their own separate implementation on top of the common libret_vhost?

Yes.

The following DPDK components use librte_vhost:

drivers/net/vhost
examples/tep_termination
examples/vhost
examples/vhost_scsi

They will all be able to use virtio-vhost-user.

Stefan

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 455 bytes --]

Re: [Qemu-devel] vhost-pci and virtio-vhost-user

[Jason Wang]

On 2018年01月15日 18:43, Wei Wang wrote:
> On 01/15/2018 04:34 PM, Jason Wang wrote:
>>
>>
>> On 2018年01月15日 15:59, Wei Wang wrote:
>>> On 01/15/2018 02:56 PM, Jason Wang wrote:
>>>>
>>>>
>>>> On 2018年01月12日 18:18, Stefan Hajnoczi wrote:
>>>>>
>>>>
>>>> I just fail understand why we can't do software defined network or 
>>>> storage with exist virtio device/drivers (or are there any 
>>>> shortcomings that force us to invent new infrastructure).
>>>>
>>>
>>> Existing virtio-net works with a host central vSwitch, and it has 
>>> the following disadvantages:
>>> 1) long code/data path;
>>> 2) poor scalability; and
>>> 3) host CPU sacrifice
>>
>> Please show me the numbers.
>
> Sure. For 64B packet transmission between two VMs: vhost-user reports 
> ~6.8Mpps, and vhost-pci reports ~11Mpps, which is ~1.62x faster.
>

This result is kind of incomplete. So still many questions left:

- What's the configuration of the vhost-user?
- What's the result of e.g 1500 byte?
- You said it improves scalability, at least I can't get this conclusion 
just from what you provide here
- You suspect long code/data path, but no latency numbers to prove it

>
>>
>>>
>>> Vhost-pci solves the above issues by providing a point-to-point 
>>> communication between VMs. No matter how the control path would look 
>>> like finally, the key point is that the data path is P2P between VMs.
>>>
>>> Best,
>>> Wei
>>>
>>>
>>
>> Well, I think I've pointed out several times in the replies of 
>> previous versions. Both vhost-pci-net and virtio-net is an ethernet 
>> device, which is not tied to a central vswitch for sure. There're 
>> just too many methods or tricks which can be used to build a point to 
>> point data path.
>
>
> Could you please show an existing example that makes virtio-net work 
> without a host vswitch/bridge?

For vhost-user, it's as simple as a testpmd which does io forwarding 
between two vhost ports? For kernel, you can do even more tricks, tc, 
bpf or whatever others.

> Could you also share other p2p data path solutions that you have in 
> mind? Thanks.
>
>
> Best,
> Wei
>

So my point stands still: both vhost-pci-net and virtio-net are ethernet 
devices, any ethernet device can connect to each other directly without 
switch. Saying virtio-net can not connect to each other directly without 
a switch obviously make no sense, it's a network topology issue for 
sure. Even if it was not a typical setup or configuration, extending the 
exist backends is 1st choice unless you can prove there're any design 
limitations of exist solutions.

Thanks

Re: [Qemu-devel] vhost-pci and virtio-vhost-user

[Jason Wang]

On 2018年01月15日 21:56, Stefan Hajnoczi wrote:
> On Mon, Jan 15, 2018 at 02:56:31PM +0800, Jason Wang wrote:
>> On 2018年01月12日 18:18, Stefan Hajnoczi wrote:
>>>> And what's more important, according to the kvm 2016 slides of vhost-pci,
>>>> the motivation of vhost-pci is not building SDN but a chain of VNFs. So
>>>> bypassing the central vswitch through a private VM2VM path does make sense.
>>>> (Though whether or not vhost-pci is the best choice is still questionable).
>>> This is probably my fault.  Maybe my networking terminology is wrong.  I
>>> consider "virtual network functions" to be part of "software-defined
>>> networking" use cases.  I'm not implying there must be a central virtual
>>> switch.
>>>
>>> To rephrase: vhost-pci enables exitless VM2VM communication.
>> The problem is, exitless is not what vhost-pci invents, it could be achieved
>> now when both sides are doing busypolling.
> The only way I'm aware of is ivshmem.  But ivshmem lacks a family of
> standard device types that allows different implementations to
> interoperate.  We already have the virtio family of device types, so it
> makes sense to work on a virtio-based solution.
>
> Perhaps I've missed a different approach for exitless VM2VM
> communication.  Please explain how VM1 and VM2 can do exitless network
> communication today?

I'm not sure we're talking the same thing. For VM2VM, do you mean only 
for shared memory? I thought we can treat any backends that can transfer 
data directly between two VMs for a VM2VM solution. In this case, if 
virtqueue notifications were disabled by all sides (e.g busy polling), 
there will be no exits at all.

And if you want a virtio version of shared memory, it's another kind of 
motivation at least from my point of view.

>
> Also, how can VM1 provide SCSI I/O services to VM2 today?
>
> Stefan

I know little about storage, but it looks to me iSCSI can do this.

Thanks

Re: [Qemu-devel] vhost-pci and virtio-vhost-user

[Wei Wang]

On 01/16/2018 01:33 PM, Jason Wang wrote:
>
>
> On 2018年01月15日 18:43, Wei Wang wrote:
>> On 01/15/2018 04:34 PM, Jason Wang wrote:
>>>
>>>
>>> On 2018年01月15日 15:59, Wei Wang wrote:
>>>> On 01/15/2018 02:56 PM, Jason Wang wrote:
>>>>>
>>>>>
>>>>> On 2018年01月12日 18:18, Stefan Hajnoczi wrote:
>>>>>>
>>>>>
>>>>> I just fail understand why we can't do software defined network or 
>>>>> storage with exist virtio device/drivers (or are there any 
>>>>> shortcomings that force us to invent new infrastructure).
>>>>>
>>>>
>>>> Existing virtio-net works with a host central vSwitch, and it has 
>>>> the following disadvantages:
>>>> 1) long code/data path;
>>>> 2) poor scalability; and
>>>> 3) host CPU sacrifice
>>>
>>> Please show me the numbers.
>>
>> Sure. For 64B packet transmission between two VMs: vhost-user reports 
>> ~6.8Mpps, and vhost-pci reports ~11Mpps, which is ~1.62x faster.
>>
>
> This result is kind of incomplete. So still many questions left:
>
> - What's the configuration of the vhost-user?
> - What's the result of e.g 1500 byte?
> - You said it improves scalability, at least I can't get this 
> conclusion just from what you provide here
> - You suspect long code/data path, but no latency numbers to prove it
>

Had an offline meeting with Jason. The future discussion will be more 
focused on the design.

Here is a conclusion about more results we collected for 64B packet 
transmission, compared to ovs-dpdk (though we are comparing to ovs-dpdk 
here, but vhost-pci isn't meant to replace ovs-dpdk. It's for inter-VM 
communication, and packets going to the outside world will go from the 
traditional backend like ovs-dpdk):

1) 2VM communication: over 1.6x higher throughput;
2) 22% shorter latency;
3) in the 5-VM chain communication tests, vhost-pci shows ~6.5x higher 
throughput thanks to its better scalability

We'll provide 1500B test results later.

Best,
Wei

Re: [Qemu-devel] vhost-pci and virtio-vhost-user

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 2264 bytes --]

On Tue, Jan 16, 2018 at 01:41:37PM +0800, Jason Wang wrote:
> 
> 
> On 2018年01月15日 21:56, Stefan Hajnoczi wrote:
> > On Mon, Jan 15, 2018 at 02:56:31PM +0800, Jason Wang wrote:
> > > On 2018年01月12日 18:18, Stefan Hajnoczi wrote:
> > > > > And what's more important, according to the kvm 2016 slides of vhost-pci,
> > > > > the motivation of vhost-pci is not building SDN but a chain of VNFs. So
> > > > > bypassing the central vswitch through a private VM2VM path does make sense.
> > > > > (Though whether or not vhost-pci is the best choice is still questionable).
> > > > This is probably my fault.  Maybe my networking terminology is wrong.  I
> > > > consider "virtual network functions" to be part of "software-defined
> > > > networking" use cases.  I'm not implying there must be a central virtual
> > > > switch.
> > > > 
> > > > To rephrase: vhost-pci enables exitless VM2VM communication.
> > > The problem is, exitless is not what vhost-pci invents, it could be achieved
> > > now when both sides are doing busypolling.
> > The only way I'm aware of is ivshmem.  But ivshmem lacks a family of
> > standard device types that allows different implementations to
> > interoperate.  We already have the virtio family of device types, so it
> > makes sense to work on a virtio-based solution.
> > 
> > Perhaps I've missed a different approach for exitless VM2VM
> > communication.  Please explain how VM1 and VM2 can do exitless network
> > communication today?
> 
> I'm not sure we're talking the same thing. For VM2VM, do you mean only for
> shared memory? I thought we can treat any backends that can transfer data
> directly between two VMs for a VM2VM solution. In this case, if virtqueue
> notifications were disabled by all sides (e.g busy polling), there will be
> no exits at all.
> 
> And if you want a virtio version of shared memory, it's another kind of
> motivation at least from my point of view.

I'm confused, we're probably not talking about the same thing.

You said that exitless "could be achieved now when both sides are doing
busypolling".  Can you post a QEMU command-line that does this?

In other words, what exactly are you proposing as an alternative to
vhost-pci?

Stefan

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 455 bytes --]

Re: [Qemu-devel] vhost-pci and virtio-vhost-user

[Jason Wang]

On 2018年01月18日 18:51, Stefan Hajnoczi wrote:
> On Tue, Jan 16, 2018 at 01:41:37PM +0800, Jason Wang wrote:
>>
>> On 2018年01月15日 21:56, Stefan Hajnoczi wrote:
>>> On Mon, Jan 15, 2018 at 02:56:31PM +0800, Jason Wang wrote:
>>>> On 2018年01月12日 18:18, Stefan Hajnoczi wrote:
>>>>>> And what's more important, according to the kvm 2016 slides of vhost-pci,
>>>>>> the motivation of vhost-pci is not building SDN but a chain of VNFs. So
>>>>>> bypassing the central vswitch through a private VM2VM path does make sense.
>>>>>> (Though whether or not vhost-pci is the best choice is still questionable).
>>>>> This is probably my fault.  Maybe my networking terminology is wrong.  I
>>>>> consider "virtual network functions" to be part of "software-defined
>>>>> networking" use cases.  I'm not implying there must be a central virtual
>>>>> switch.
>>>>>
>>>>> To rephrase: vhost-pci enables exitless VM2VM communication.
>>>> The problem is, exitless is not what vhost-pci invents, it could be achieved
>>>> now when both sides are doing busypolling.
>>> The only way I'm aware of is ivshmem.  But ivshmem lacks a family of
>>> standard device types that allows different implementations to
>>> interoperate.  We already have the virtio family of device types, so it
>>> makes sense to work on a virtio-based solution.
>>>
>>> Perhaps I've missed a different approach for exitless VM2VM
>>> communication.  Please explain how VM1 and VM2 can do exitless network
>>> communication today?
>> I'm not sure we're talking the same thing. For VM2VM, do you mean only for
>> shared memory? I thought we can treat any backends that can transfer data
>> directly between two VMs for a VM2VM solution. In this case, if virtqueue
>> notifications were disabled by all sides (e.g busy polling), there will be
>> no exits at all.
>>
>> And if you want a virtio version of shared memory, it's another kind of
>> motivation at least from my point of view.
> I'm confused, we're probably not talking about the same thing.
>
> You said that exitless "could be achieved now when both sides are doing
> busypolling".  Can you post a QEMU command-line that does this?

If exitless means no virtqueue kick and interrupt. It does not require 
any special command line, just start a testpmd in both guest and host.

>
> In other words, what exactly are you proposing as an alternative to
> vhost-pci?

I don't propose any new idea. I just want to know what's the advantage 
of vhost-pci over zerocopy. Both needs one time of copy, the difference 
is the vhost-pci did it inside a guest but zerocopy did in on host.

Thanks

>
> Stefan

Re: [Qemu-devel] vhost-pci and virtio-vhost-user

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 3217 bytes --]

On Thu, Jan 18, 2018 at 07:51:49PM +0800, Jason Wang wrote:
> 
> 
> On 2018年01月18日 18:51, Stefan Hajnoczi wrote:
> > On Tue, Jan 16, 2018 at 01:41:37PM +0800, Jason Wang wrote:
> > > 
> > > On 2018年01月15日 21:56, Stefan Hajnoczi wrote:
> > > > On Mon, Jan 15, 2018 at 02:56:31PM +0800, Jason Wang wrote:
> > > > > On 2018年01月12日 18:18, Stefan Hajnoczi wrote:
> > > > > > > And what's more important, according to the kvm 2016 slides of vhost-pci,
> > > > > > > the motivation of vhost-pci is not building SDN but a chain of VNFs. So
> > > > > > > bypassing the central vswitch through a private VM2VM path does make sense.
> > > > > > > (Though whether or not vhost-pci is the best choice is still questionable).
> > > > > > This is probably my fault.  Maybe my networking terminology is wrong.  I
> > > > > > consider "virtual network functions" to be part of "software-defined
> > > > > > networking" use cases.  I'm not implying there must be a central virtual
> > > > > > switch.
> > > > > > 
> > > > > > To rephrase: vhost-pci enables exitless VM2VM communication.
> > > > > The problem is, exitless is not what vhost-pci invents, it could be achieved
> > > > > now when both sides are doing busypolling.
> > > > The only way I'm aware of is ivshmem.  But ivshmem lacks a family of
> > > > standard device types that allows different implementations to
> > > > interoperate.  We already have the virtio family of device types, so it
> > > > makes sense to work on a virtio-based solution.
> > > > 
> > > > Perhaps I've missed a different approach for exitless VM2VM
> > > > communication.  Please explain how VM1 and VM2 can do exitless network
> > > > communication today?
> > > I'm not sure we're talking the same thing. For VM2VM, do you mean only for
> > > shared memory? I thought we can treat any backends that can transfer data
> > > directly between two VMs for a VM2VM solution. In this case, if virtqueue
> > > notifications were disabled by all sides (e.g busy polling), there will be
> > > no exits at all.
> > > 
> > > And if you want a virtio version of shared memory, it's another kind of
> > > motivation at least from my point of view.
> > I'm confused, we're probably not talking about the same thing.
> > 
> > You said that exitless "could be achieved now when both sides are doing
> > busypolling".  Can you post a QEMU command-line that does this?
> 
> If exitless means no virtqueue kick and interrupt. It does not require any
> special command line, just start a testpmd in both guest and host.
> 
> > 
> > In other words, what exactly are you proposing as an alternative to
> > vhost-pci?
> 
> I don't propose any new idea. I just want to know what's the advantage of
> vhost-pci over zerocopy. Both needs one time of copy, the difference is the
> vhost-pci did it inside a guest but zerocopy did in on host.

Exitless VM2VM communication is desirable if you cannot run software on
the host or if both endpoints are already in VMs.  In that case running
one thing in a VM and another on the host doesn't make sense.

The obvious environment where this applies is in the cloud where
everything is a VM.

Stefan

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 455 bytes --]

Re: [Qemu-devel] vhost-pci and virtio-vhost-user

[Jason Wang]

On 2018年01月20日 01:20, Stefan Hajnoczi wrote:
>> I don't propose any new idea. I just want to know what's the advantage of
>> vhost-pci over zerocopy. Both needs one time of copy, the difference is the
>> vhost-pci did it inside a guest but zerocopy did in on host.
> Exitless VM2VM communication is desirable if you cannot run software on
> the host or if both endpoints are already in VMs.  In that case running
> one thing in a VM and another on the host doesn't make sense.

Well, I must have missed anything, I don't see why we can not run 
virtio-net backend on host. Especially it only does L2 stuffs, higher 
level of service could be provided by another VM for sure. So it looks 
to me virtio-vhost-user is just a split device implementation which is 
irreverent to the service it could provide.

Maybe you can provide a concrete examples of virtio-vhost-user and its 
advantages?

>
> The obvious environment where this applies is in the cloud where
> everything is a VM.

So a typical setup makes the VMs can already talk to each other through 
ethernet(virtio-net). Virtio-vhost-user looks much less flexible than 
exist stuffs. The only possible advantage of virtio-vhost-user is its 
performance or security which still need to be proved.

Thanks

Re: [Qemu-devel] vhost-pci and virtio-vhost-user

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 2148 bytes --]

On Mon, Jan 22, 2018 at 11:54:41AM +0800, Jason Wang wrote:
> On 2018年01月20日 01:20, Stefan Hajnoczi wrote:
> > > I don't propose any new idea. I just want to know what's the advantage of
> > > vhost-pci over zerocopy. Both needs one time of copy, the difference is the
> > > vhost-pci did it inside a guest but zerocopy did in on host.
> > Exitless VM2VM communication is desirable if you cannot run software on
> > the host or if both endpoints are already in VMs.  In that case running
> > one thing in a VM and another on the host doesn't make sense.
> 
> Well, I must have missed anything, I don't see why we can not run virtio-net
> backend on host. Especially it only does L2 stuffs, higher level of service
> could be provided by another VM for sure. So it looks to me
> virtio-vhost-user is just a split device implementation which is irreverent
> to the service it could provide.
> 
> Maybe you can provide a concrete examples of virtio-vhost-user and its
> advantages?
[...]
> > 
> > The obvious environment where this applies is in the cloud where
> > everything is a VM.
> 
> So a typical setup makes the VMs can already talk to each other through
> ethernet(virtio-net). Virtio-vhost-user looks much less flexible than exist
> stuffs. The only possible advantage of virtio-vhost-user is its performance
> or security which still need to be proved.

I look forward to performance results but in the meantime consider
existing vhost-user use cases.  Those applications could just use
network communication too (e.g. iSCSI instead of vhost-user-scsi).  They
choose vhost-user mainly because it's faster (zerocopy, no vmexits, and
no network protocol overhead).

There are other advantages too.  In the iSCSI case, guests need
configuration details to establish the iSCSI connection.  In the
virtio-scsi case the guests only need the virtio-scsi driver and no
configuration to access the LUNs.  It's simpler on the guest side!

So the reasons for virtio-vhost-user are the same as vhost-user PLUS
some users choose to deploy everything in VMs and cannot run host
userspace process.

Stefan

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 455 bytes --]