[Qemu-devel] [RFC 0/2] virtio-vhost-user: add virtio-vhost-user device

[Qemu-devel] [RFC 0/2] virtio-vhost-user: add virtio-vhost-user device

[Stefan Hajnoczi]

These patches implement the virtio-vhost-user device design that I have
described here:
https://stefanha.github.io/virtio/vhost-user-slave.html#x1-2830007

The goal is to let the guest act as the vhost device backend for other guests.
This allows virtual networking and storage appliances to run inside guests.
This device is particularly interesting for poll mode drivers where exitless
VM-to-VM communication is possible, completely bypassing the hypervisor in the
data path.

The DPDK driver is here:
https://github.com/stefanha/dpdk/tree/virtio-vhost-user

For more information, see
https://wiki.qemu.org/Features/VirtioVhostUser.

virtio-vhost-user is inspired by Wei Wang and Zhiyong Yang's vhost-pci work.
It differs from vhost-pci in that it has:
1. Vhost-user protocol message tunneling, allowing existing vhost-user
   slave software to be reused inside the guest.
2. Support for all vhost device types.
3. Disconnected operation and reconnection support.
4. Asynchronous vhost-user socket implementation that avoids blocking.

I have written this code to demonstrate how the virtio-vhost-user approach
works and that it is more maintainable than vhost-pci because vhost-user slave
software can use both AF_UNIX and virtio-vhost-user without significant code
changes to the vhost device backends.

One of the main concerns about virtio-vhost-user was that the QEMU
virtio-vhost-user device implementation could be complex because it needs to
parse all messages.  I hope this patch series shows that it's actually very
simple because most messages are passed through.

After this patch series has been reviewed, we need to decide whether to follow
the original vhost-pci approach or to use this one.  Either way, both patch
series still require improvements before they can be merged.  Here are my todos
for this series:

 * Implement "Additional Device Resources over PCI" for shared memory,
   doorbells, and notifications instead of hardcoding a BAR with magic
   offsets into virtio-vhost-user:
   https://stefanha.github.io/virtio/vhost-user-slave.html#x1-2920007
 * Implement the VRING_KICK eventfd - currently vhost-user slaves must be poll
   mode drivers.
 * Optimize VRING_CALL doorbell with ioeventfd to avoid QEMU exit.
 * vhost-user log feature
 * UUID config field for stable device identification regardless of PCI
   bus addresses.
 * vhost-user IOMMU and SLAVE_REQ_FD feature
 * VhostUserMsg little-endian conversion for cross-endian support
 * Chardev disconnect using qemu_chr_fe_set_watch() since CHR_CLOSED is
   only emitted while a read callback is registered.  We don't keep a
   read callback registered all the time.
 * Drain txq on reconnection to discard stale messages.

Stefan Hajnoczi (1):
  virtio-vhost-user: add virtio-vhost-user device

Wei Wang (1):
  vhost-user: share the vhost-user protocol related structures

 configure                                   |   18 +
 hw/virtio/Makefile.objs                     |    1 +
 hw/virtio/virtio-pci.h                      |   21 +
 include/hw/pci/pci.h                        |    1 +
 include/hw/virtio/vhost-user.h              |  106 +++
 include/hw/virtio/virtio-vhost-user.h       |   88 +++
 include/standard-headers/linux/virtio_ids.h |    1 +
 hw/virtio/vhost-user.c                      |  100 +--
 hw/virtio/virtio-pci.c                      |   61 ++
 hw/virtio/virtio-vhost-user.c               | 1047 +++++++++++++++++++++++++++
 hw/virtio/trace-events                      |   22 +
 11 files changed, 1367 insertions(+), 99 deletions(-)
 create mode 100644 include/hw/virtio/vhost-user.h
 create mode 100644 include/hw/virtio/virtio-vhost-user.h
 create mode 100644 hw/virtio/virtio-vhost-user.c

-- 
2.14.3

[Qemu-devel] [RFC 1/2] vhost-user: share the vhost-user protocol related structures

[Stefan Hajnoczi]

From: Wei Wang <wei.w.wang@intel.com>

Put the vhost-user protocol related data structures to vhost-user.h,
so that they can be used in other implementations (e.g. a slave
implementation).

Signed-off-by: Wei Wang <wei.w.wang@intel.com>
---
 include/hw/virtio/vhost-user.h | 106 +++++++++++++++++++++++++++++++++++++++++
 hw/virtio/vhost-user.c         | 100 +-------------------------------------
 2 files changed, 107 insertions(+), 99 deletions(-)
 create mode 100644 include/hw/virtio/vhost-user.h

diff --git a/include/hw/virtio/vhost-user.h b/include/hw/virtio/vhost-user.h
new file mode 100644
index 0000000000..d76e9ada31
--- /dev/null
+++ b/include/hw/virtio/vhost-user.h
@@ -0,0 +1,106 @@
+#ifndef VHOST_USER_H
+#define VHOST_USER_H
+
+#include <linux/vhost.h>
+
+#define VHOST_MEMORY_MAX_NREGIONS    8
+#define VHOST_USER_F_PROTOCOL_FEATURES 30
+
+enum VhostUserProtocolFeature {
+    VHOST_USER_PROTOCOL_F_MQ = 0,
+    VHOST_USER_PROTOCOL_F_LOG_SHMFD = 1,
+    VHOST_USER_PROTOCOL_F_RARP = 2,
+    VHOST_USER_PROTOCOL_F_REPLY_ACK = 3,
+    VHOST_USER_PROTOCOL_F_NET_MTU = 4,
+    VHOST_USER_PROTOCOL_F_SLAVE_REQ = 5,
+    VHOST_USER_PROTOCOL_F_CROSS_ENDIAN = 6,
+
+    VHOST_USER_PROTOCOL_F_MAX
+};
+
+#define VHOST_USER_PROTOCOL_FEATURE_MASK ((1 << VHOST_USER_PROTOCOL_F_MAX) - 1)
+
+typedef enum VhostUserRequest {
+    VHOST_USER_NONE = 0,
+    VHOST_USER_GET_FEATURES = 1,
+    VHOST_USER_SET_FEATURES = 2,
+    VHOST_USER_SET_OWNER = 3,
+    VHOST_USER_RESET_OWNER = 4,
+    VHOST_USER_SET_MEM_TABLE = 5,
+    VHOST_USER_SET_LOG_BASE = 6,
+    VHOST_USER_SET_LOG_FD = 7,
+    VHOST_USER_SET_VRING_NUM = 8,
+    VHOST_USER_SET_VRING_ADDR = 9,
+    VHOST_USER_SET_VRING_BASE = 10,
+    VHOST_USER_GET_VRING_BASE = 11,
+    VHOST_USER_SET_VRING_KICK = 12,
+    VHOST_USER_SET_VRING_CALL = 13,
+    VHOST_USER_SET_VRING_ERR = 14,
+    VHOST_USER_GET_PROTOCOL_FEATURES = 15,
+    VHOST_USER_SET_PROTOCOL_FEATURES = 16,
+    VHOST_USER_GET_QUEUE_NUM = 17,
+    VHOST_USER_SET_VRING_ENABLE = 18,
+    VHOST_USER_SEND_RARP = 19,
+    VHOST_USER_NET_SET_MTU = 20,
+    VHOST_USER_SET_SLAVE_REQ_FD = 21,
+    VHOST_USER_IOTLB_MSG = 22,
+    VHOST_USER_SET_VRING_ENDIAN = 23,
+    VHOST_USER_MAX
+} VhostUserRequest;
+
+typedef enum VhostUserSlaveRequest {
+    VHOST_USER_SLAVE_NONE = 0,
+    VHOST_USER_SLAVE_IOTLB_MSG = 1,
+    VHOST_USER_SLAVE_MAX
+}  VhostUserSlaveRequest;
+
+typedef struct VhostUserMemoryRegion {
+    uint64_t guest_phys_addr;
+    uint64_t memory_size;
+    uint64_t userspace_addr;
+    uint64_t mmap_offset;
+} VhostUserMemoryRegion;
+
+typedef struct VhostUserMemory {
+    uint32_t nregions;
+    uint32_t padding;
+    VhostUserMemoryRegion regions[VHOST_MEMORY_MAX_NREGIONS];
+} VhostUserMemory;
+
+typedef struct VhostUserLog {
+    uint64_t mmap_size;
+    uint64_t mmap_offset;
+} VhostUserLog;
+
+typedef struct VhostUserMsg {
+    VhostUserRequest request;
+
+#define VHOST_USER_VERSION_MASK     (0x3)
+#define VHOST_USER_REPLY_MASK       (0x1 << 2)
+#define VHOST_USER_NEED_REPLY_MASK  (0x1 << 3)
+    uint32_t flags;
+    /* The following payload size */
+    uint32_t size;
+    union {
+#define VHOST_USER_VRING_IDX_MASK   (0xff)
+#define VHOST_USER_VRING_NOFD_MASK  (0x1 << 8)
+        uint64_t u64;
+        struct vhost_vring_state state;
+        struct vhost_vring_addr addr;
+        VhostUserMemory memory;
+        VhostUserLog log;
+        struct vhost_iotlb_msg iotlb;
+    } payload;
+} QEMU_PACKED VhostUserMsg;
+
+static VhostUserMsg m __attribute__ ((unused));
+#define VHOST_USER_HDR_SIZE (sizeof(m.request) \
+                            + sizeof(m.flags) \
+                            + sizeof(m.size))
+
+#define VHOST_USER_PAYLOAD_SIZE (sizeof(m) - VHOST_USER_HDR_SIZE)
+
+/* The version of the protocol we support */
+#define VHOST_USER_VERSION    (0x1)
+
+#endif
diff --git a/hw/virtio/vhost-user.c b/hw/virtio/vhost-user.c
index 093675ed98..e512f5a463 100644
--- a/hw/virtio/vhost-user.c
+++ b/hw/virtio/vhost-user.c
@@ -12,6 +12,7 @@
 #include "qapi/error.h"
 #include "hw/virtio/vhost.h"
 #include "hw/virtio/vhost-backend.h"
+#include "hw/virtio/vhost-user.h"
 #include "hw/virtio/virtio-net.h"
 #include "chardev/char-fe.h"
 #include "sysemu/kvm.h"
@@ -23,105 +24,6 @@
 #include <sys/un.h>
 #include <linux/vhost.h>
 
-#define VHOST_MEMORY_MAX_NREGIONS    8
-#define VHOST_USER_F_PROTOCOL_FEATURES 30
-
-enum VhostUserProtocolFeature {
-    VHOST_USER_PROTOCOL_F_MQ = 0,
-    VHOST_USER_PROTOCOL_F_LOG_SHMFD = 1,
-    VHOST_USER_PROTOCOL_F_RARP = 2,
-    VHOST_USER_PROTOCOL_F_REPLY_ACK = 3,
-    VHOST_USER_PROTOCOL_F_NET_MTU = 4,
-    VHOST_USER_PROTOCOL_F_SLAVE_REQ = 5,
-    VHOST_USER_PROTOCOL_F_CROSS_ENDIAN = 6,
-
-    VHOST_USER_PROTOCOL_F_MAX
-};
-
-#define VHOST_USER_PROTOCOL_FEATURE_MASK ((1 << VHOST_USER_PROTOCOL_F_MAX) - 1)
-
-typedef enum VhostUserRequest {
-    VHOST_USER_NONE = 0,
-    VHOST_USER_GET_FEATURES = 1,
-    VHOST_USER_SET_FEATURES = 2,
-    VHOST_USER_SET_OWNER = 3,
-    VHOST_USER_RESET_OWNER = 4,
-    VHOST_USER_SET_MEM_TABLE = 5,
-    VHOST_USER_SET_LOG_BASE = 6,
-    VHOST_USER_SET_LOG_FD = 7,
-    VHOST_USER_SET_VRING_NUM = 8,
-    VHOST_USER_SET_VRING_ADDR = 9,
-    VHOST_USER_SET_VRING_BASE = 10,
-    VHOST_USER_GET_VRING_BASE = 11,
-    VHOST_USER_SET_VRING_KICK = 12,
-    VHOST_USER_SET_VRING_CALL = 13,
-    VHOST_USER_SET_VRING_ERR = 14,
-    VHOST_USER_GET_PROTOCOL_FEATURES = 15,
-    VHOST_USER_SET_PROTOCOL_FEATURES = 16,
-    VHOST_USER_GET_QUEUE_NUM = 17,
-    VHOST_USER_SET_VRING_ENABLE = 18,
-    VHOST_USER_SEND_RARP = 19,
-    VHOST_USER_NET_SET_MTU = 20,
-    VHOST_USER_SET_SLAVE_REQ_FD = 21,
-    VHOST_USER_IOTLB_MSG = 22,
-    VHOST_USER_SET_VRING_ENDIAN = 23,
-    VHOST_USER_MAX
-} VhostUserRequest;
-
-typedef enum VhostUserSlaveRequest {
-    VHOST_USER_SLAVE_NONE = 0,
-    VHOST_USER_SLAVE_IOTLB_MSG = 1,
-    VHOST_USER_SLAVE_MAX
-}  VhostUserSlaveRequest;
-
-typedef struct VhostUserMemoryRegion {
-    uint64_t guest_phys_addr;
-    uint64_t memory_size;
-    uint64_t userspace_addr;
-    uint64_t mmap_offset;
-} VhostUserMemoryRegion;
-
-typedef struct VhostUserMemory {
-    uint32_t nregions;
-    uint32_t padding;
-    VhostUserMemoryRegion regions[VHOST_MEMORY_MAX_NREGIONS];
-} VhostUserMemory;
-
-typedef struct VhostUserLog {
-    uint64_t mmap_size;
-    uint64_t mmap_offset;
-} VhostUserLog;
-
-typedef struct VhostUserMsg {
-    VhostUserRequest request;
-
-#define VHOST_USER_VERSION_MASK     (0x3)
-#define VHOST_USER_REPLY_MASK       (0x1<<2)
-#define VHOST_USER_NEED_REPLY_MASK  (0x1 << 3)
-    uint32_t flags;
-    uint32_t size; /* the following payload size */
-    union {
-#define VHOST_USER_VRING_IDX_MASK   (0xff)
-#define VHOST_USER_VRING_NOFD_MASK  (0x1<<8)
-        uint64_t u64;
-        struct vhost_vring_state state;
-        struct vhost_vring_addr addr;
-        VhostUserMemory memory;
-        VhostUserLog log;
-        struct vhost_iotlb_msg iotlb;
-    } payload;
-} QEMU_PACKED VhostUserMsg;
-
-static VhostUserMsg m __attribute__ ((unused));
-#define VHOST_USER_HDR_SIZE (sizeof(m.request) \
-                            + sizeof(m.flags) \
-                            + sizeof(m.size))
-
-#define VHOST_USER_PAYLOAD_SIZE (sizeof(m) - VHOST_USER_HDR_SIZE)
-
-/* The version of the protocol we support */
-#define VHOST_USER_VERSION    (0x1)
-
 struct vhost_user {
     CharBackend *chr;
     int slave_fd;
-- 
2.14.3

[Qemu-devel] [RFC 2/2] virtio-vhost-user: add virtio-vhost-user device

[Stefan Hajnoczi]

The virtio-vhost-user device lets a guest act as a vhost device backend.
It works by tunneling vhost-user protocol messages into a guest.  The
new device syntax is as follows:

  -chardev socket,id=chardev0,path=vhost-user.sock,server,nowait \
  -device virtio-vhost-user-pci,chardev=chardev0

The VIRTIO device specification is here:
https://stefanha.github.io/virtio/vhost-user-slave.html#x1-2830007

For more information about virtio-vhost-user, see
https://wiki.qemu.org/Features/VirtioVhostUser.

Cc: Wei Wang <wei.w.wang@intel.com>
Cc: zhiyong.yang@intel.com
Cc: jasowang@redhat.com
Cc: Maxime Coquelin <maxime.coquelin@redhat.com>
Cc: mst@redhat.com
Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
---
 configure                                   |   18 +
 hw/virtio/Makefile.objs                     |    1 +
 hw/virtio/virtio-pci.h                      |   21 +
 include/hw/pci/pci.h                        |    1 +
 include/hw/virtio/virtio-vhost-user.h       |   88 +++
 include/standard-headers/linux/virtio_ids.h |    1 +
 hw/virtio/virtio-pci.c                      |   61 ++
 hw/virtio/virtio-vhost-user.c               | 1047 +++++++++++++++++++++++++++
 hw/virtio/trace-events                      |   22 +
 9 files changed, 1260 insertions(+)
 create mode 100644 include/hw/virtio/virtio-vhost-user.h
 create mode 100644 hw/virtio/virtio-vhost-user.c

diff --git a/configure b/configure
index 9c8aa5a98b..fa0cf4937d 100755
--- a/configure
+++ b/configure
@@ -323,6 +323,7 @@ vhost_net="no"
 vhost_scsi="no"
 vhost_vsock="no"
 vhost_user=""
+virtio_vhost_user=""
 kvm="no"
 hax="no"
 rdma=""
@@ -1299,6 +1300,10 @@ for opt do
           error_exit "vhost-user isn't available on win32"
       fi
   ;;
+  --disable-virtio-vhost-user) virtio_vhost_user="no"
+  ;;
+  --enable-virtio-vhost-user) virtio_vhost_user="yes"
+  ;;
   --disable-capstone) capstone="no"
   ;;
   --enable-capstone) capstone="yes"
@@ -1329,6 +1334,15 @@ if test "$vhost_user" = ""; then
     fi
 fi
 
+# UNIX domain sockets are required
+if test "$virtio_vhost_user" = ""; then
+    if test "$mingw32" = "yes"; then
+        virtio_vhost_user="no"
+    else
+        virtio_vhost_user="yes"
+    fi
+fi
+
 case "$cpu" in
     ppc)
            CPU_CFLAGS="-m32"
@@ -5516,6 +5530,7 @@ echo "vhost-net support $vhost_net"
 echo "vhost-scsi support $vhost_scsi"
 echo "vhost-vsock support $vhost_vsock"
 echo "vhost-user support $vhost_user"
+echo "virtio-vhost-user support $virtio_vhost_user"
 echo "Trace backends    $trace_backends"
 if have_backend "simple"; then
 echo "Trace output file $trace_file-<pid>"
@@ -5954,6 +5969,9 @@ fi
 if test "$vhost_user" = "yes" ; then
   echo "CONFIG_VHOST_USER=y" >> $config_host_mak
 fi
+if test "$virtio_vhost_user" = "yes" ; then
+  echo "CONFIG_VIRTIO_VHOST_USER=y" >> $config_host_mak
+fi
 if test "$blobs" = "yes" ; then
   echo "INSTALL_BLOBS=yes" >> $config_host_mak
 fi
diff --git a/hw/virtio/Makefile.objs b/hw/virtio/Makefile.objs
index 765d363c1f..4a666b9c7e 100644
--- a/hw/virtio/Makefile.objs
+++ b/hw/virtio/Makefile.objs
@@ -9,6 +9,7 @@ obj-$(CONFIG_LINUX) += vhost.o vhost-backend.o vhost-user.o
 obj-$(CONFIG_VHOST_VSOCK) += vhost-vsock.o
 obj-y += virtio-crypto.o
 obj-$(CONFIG_VIRTIO_PCI) += virtio-crypto-pci.o
+obj-$(CONFIG_VIRTIO_VHOST_USER) += virtio-vhost-user.o
 endif
 
 common-obj-$(call lnot,$(CONFIG_LINUX)) += vhost-stub.o
diff --git a/hw/virtio/virtio-pci.h b/hw/virtio/virtio-pci.h
index 12d3a90686..4e454d99ab 100644
--- a/hw/virtio/virtio-pci.h
+++ b/hw/virtio/virtio-pci.h
@@ -37,6 +37,9 @@
 #ifdef CONFIG_VHOST_VSOCK
 #include "hw/virtio/vhost-vsock.h"
 #endif
+#ifdef CONFIG_VIRTIO_VHOST_USER
+#include "hw/virtio/virtio-vhost-user.h"
+#endif
 
 typedef struct VirtIOPCIProxy VirtIOPCIProxy;
 typedef struct VirtIOBlkPCI VirtIOBlkPCI;
@@ -53,6 +56,7 @@ typedef struct VirtIOInputHostPCI VirtIOInputHostPCI;
 typedef struct VirtIOGPUPCI VirtIOGPUPCI;
 typedef struct VHostVSockPCI VHostVSockPCI;
 typedef struct VirtIOCryptoPCI VirtIOCryptoPCI;
+typedef struct VirtIOVhostUserPCI VirtIOVhostUserPCI;
 
 /* virtio-pci-bus */
 
@@ -396,6 +400,23 @@ struct VirtIOCryptoPCI {
     VirtIOCrypto vdev;
 };
 
+/*
+ * virtio-vhost-user-pci: This extends VirtioPCIProxy.
+ */
+
+#ifdef CONFIG_VIRTIO_VHOST_USER
+
+#define TYPE_VIRTIO_VHOST_USER_PCI "virtio-vhost-user-pci"
+#define VIRTIO_VHOST_USER_PCI(obj) \
+        OBJECT_CHECK(VirtIOVhostUserPCI, (obj), TYPE_VIRTIO_VHOST_USER_PCI)
+
+struct VirtIOVhostUserPCI {
+    VirtIOPCIProxy parent_obj;
+    VirtIOVhostUser vdev;
+};
+
+#endif /* CONFIG_VIRTIO_VHOST_USER */
+
 /* Virtio ABI version, if we increment this, we break the guest driver. */
 #define VIRTIO_PCI_ABI_VERSION          0
 
diff --git a/include/hw/pci/pci.h b/include/hw/pci/pci.h
index 8d02a0a383..e9fa33d973 100644
--- a/include/hw/pci/pci.h
+++ b/include/hw/pci/pci.h
@@ -85,6 +85,7 @@ extern bool pci_available;
 #define PCI_DEVICE_ID_VIRTIO_RNG         0x1005
 #define PCI_DEVICE_ID_VIRTIO_9P          0x1009
 #define PCI_DEVICE_ID_VIRTIO_VSOCK       0x1012
+#define PCI_DEVICE_ID_VIRTIO_VHOST_USER  0x1017
 
 #define PCI_VENDOR_ID_REDHAT             0x1b36
 #define PCI_DEVICE_ID_REDHAT_BRIDGE      0x0001
diff --git a/include/hw/virtio/virtio-vhost-user.h b/include/hw/virtio/virtio-vhost-user.h
new file mode 100644
index 0000000000..592d7d237a
--- /dev/null
+++ b/include/hw/virtio/virtio-vhost-user.h
@@ -0,0 +1,88 @@
+/*
+ * Virtio Vhost-user Device
+ *
+ * Copyright (C) 2017 Red Hat, Inc.
+ *
+ * Authors:
+ *  Stefan Hajnoczi   <stefanha@redhat.com>
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2 or later.
+ * See the COPYING file in the top-level directory.
+ *
+ */
+
+#ifndef QEMU_VIRTIO_VHOST_USER_H
+#define QEMU_VIRTIO_VHOST_USER_H
+
+#include "hw/virtio/virtio.h"
+#include "hw/virtio/vhost-user.h"
+#include "chardev/char-fe.h"
+
+#define TYPE_VIRTIO_VHOST_USER "virtio-vhost-user-device"
+#define VIRTIO_VHOST_USER(obj) \
+        OBJECT_CHECK(VirtIOVhostUser, (obj), TYPE_VIRTIO_VHOST_USER)
+
+/* The virtio configuration space fields */
+typedef struct {
+    uint32_t status;
+#define VIRTIO_VHOST_USER_STATUS_SLAVE_UP 0
+#define VIRTIO_VHOST_USER_STATUS_MASTER_UP 1
+    uint32_t max_vhost_queues;
+    uint8_t uuid[16];
+} QEMU_PACKED VirtIOVhostUserConfig;
+
+/* Keep track of the mmap for each memory table region */
+typedef struct {
+    MemoryRegion mr;
+    void *mmap_addr;
+    size_t total_size;
+} VirtIOVhostUserMemTableRegion;
+
+typedef struct VirtIOVhostUser VirtIOVhostUser;
+struct VirtIOVhostUser {
+    VirtIODevice parent_obj;
+
+    /* The vhost-user socket */
+    CharBackend chr;
+
+    /* TODO implement "Additional Device Resources over PCI" so that PCI
+     * details are hidden:
+     * https://stefanha.github.io/virtio/vhost-user-slave.html#x1-2920007
+     */
+    MemoryRegion additional_resources_bar;
+    MemoryRegion doorbell_region;
+
+    /* Eventfds from VHOST_USER_SET_VRING_CALL */
+    int callfds[VIRTIO_QUEUE_MAX];
+
+    /* Mapped memory regions from VHOST_USER_SET_MEM_TABLE */
+    VirtIOVhostUserMemTableRegion mem_table[VHOST_MEMORY_MAX_NREGIONS];
+
+    VirtIOVhostUserConfig config;
+
+    /* Connection establishment state */
+    int conn_state;
+
+    /* Device-to-driver message queue */
+    VirtQueue *rxq;
+
+    /* Driver-to-device message queue */
+    VirtQueue *txq;
+
+    /* Asynchronous read state */
+    int read_bytes_needed;
+    void *read_ptr;
+    void (*read_done)(VirtIOVhostUser *s);
+    VhostUserMsg read_msg;
+    bool read_waiting_on_rxq; /* need rx buffer? */
+    size_t read_msg_size;
+
+    /* Asynchronous write state */
+    int write_bytes_avail;
+    void *write_ptr;
+    void (*write_done)(VirtIOVhostUser *s);
+    VhostUserMsg write_msg;
+    guint write_watch_tag;
+};
+
+#endif /* QEMU_VIRTIO_VHOST_USER_H */
diff --git a/include/standard-headers/linux/virtio_ids.h b/include/standard-headers/linux/virtio_ids.h
index 6d5c3b2d4f..ab45ace2e3 100644
--- a/include/standard-headers/linux/virtio_ids.h
+++ b/include/standard-headers/linux/virtio_ids.h
@@ -43,5 +43,6 @@
 #define VIRTIO_ID_INPUT        18 /* virtio input */
 #define VIRTIO_ID_VSOCK        19 /* virtio vsock transport */
 #define VIRTIO_ID_CRYPTO       20 /* virtio crypto */
+#define VIRTIO_ID_VHOST_USER   24 /* virtio vhost-user */
 
 #endif /* _LINUX_VIRTIO_IDS_H */
diff --git a/hw/virtio/virtio-pci.c b/hw/virtio/virtio-pci.c
index e92837c42b..aff41ee3e1 100644
--- a/hw/virtio/virtio-pci.c
+++ b/hw/virtio/virtio-pci.c
@@ -25,6 +25,7 @@
 #include "hw/virtio/virtio-scsi.h"
 #include "hw/virtio/virtio-balloon.h"
 #include "hw/virtio/virtio-input.h"
+#include "hw/virtio/virtio-vhost-user.h"
 #include "hw/pci/pci.h"
 #include "qapi/error.h"
 #include "qemu/error-report.h"
@@ -2561,6 +2562,63 @@ static const TypeInfo virtio_host_pci_info = {
 };
 #endif
 
+/* virtio-vhost-user-pci */
+
+#ifdef CONFIG_VIRTIO_VHOST_USER
+static void virtio_vhost_user_pci_realize(VirtIOPCIProxy *vpci_dev,
+                                          Error **errp)
+{
+    VirtIOVhostUserPCI *vvup = VIRTIO_VHOST_USER_PCI(vpci_dev);
+    DeviceState *vdev = DEVICE(&vvup->vdev);
+    Error *err = NULL;
+
+    qdev_set_parent_bus(vdev, BUS(&vpci_dev->bus));
+    object_property_set_bool(OBJECT(vdev), true, "realized", &err);
+    if (err) {
+        error_propagate(errp, err);
+        return;
+    }
+}
+
+static Property virtio_vhost_user_pci_properties[] = {
+    DEFINE_PROP_UINT32("vectors", VirtIOPCIProxy, nvectors, 3),
+    DEFINE_PROP_END_OF_LIST(),
+};
+
+static void virtio_vhost_user_pci_class_init(ObjectClass *klass, void *data)
+{
+    DeviceClass *dc = DEVICE_CLASS(klass);
+    VirtioPCIClass *k = VIRTIO_PCI_CLASS(klass);
+    PCIDeviceClass *pcidev_k = PCI_DEVICE_CLASS(klass);
+
+    dc->props = virtio_vhost_user_pci_properties;
+    k->realize = virtio_vhost_user_pci_realize;
+    set_bit(DEVICE_CATEGORY_MISC, dc->categories);
+
+    pcidev_k->vendor_id = PCI_VENDOR_ID_REDHAT_QUMRANET;
+    pcidev_k->device_id = PCI_DEVICE_ID_VIRTIO_VHOST_USER;
+    pcidev_k->revision = VIRTIO_PCI_ABI_VERSION;
+    pcidev_k->class_id = PCI_CLASS_OTHERS;
+}
+
+static void virtio_vhost_user_pci_initfn(Object *obj)
+{
+    VirtIOVhostUserPCI *dev = VIRTIO_VHOST_USER_PCI(obj);
+
+    virtio_instance_init_common(obj, &dev->vdev, sizeof(dev->vdev),
+                                TYPE_VIRTIO_VHOST_USER);
+}
+
+static const TypeInfo virtio_vhost_user_pci_info = {
+    .name          = TYPE_VIRTIO_VHOST_USER_PCI,
+    .parent        = TYPE_VIRTIO_PCI,
+    .instance_size = sizeof(VirtIOVhostUserPCI),
+    .instance_init = virtio_vhost_user_pci_initfn,
+    .class_init    = virtio_vhost_user_pci_class_init,
+};
+#endif /* CONFIG_VIRTIO_VHOST_USER */
+
+
 /* virtio-pci-bus */
 
 static void virtio_pci_bus_new(VirtioBusState *bus, size_t bus_size,
@@ -2635,6 +2693,9 @@ static void virtio_pci_register_types(void)
 #ifdef CONFIG_VHOST_VSOCK
     type_register_static(&vhost_vsock_pci_info);
 #endif
+#ifdef CONFIG_VIRTIO_VHOST_USER
+    type_register_static(&virtio_vhost_user_pci_info);
+#endif
 }
 
 type_init(virtio_pci_register_types)
diff --git a/hw/virtio/virtio-vhost-user.c b/hw/virtio/virtio-vhost-user.c
new file mode 100644
index 0000000000..6a09bed879
--- /dev/null
+++ b/hw/virtio/virtio-vhost-user.c
@@ -0,0 +1,1047 @@
+/*
+ * Virtio Vhost-user Device
+ *
+ * Copyright (C) 2017-2018 Red Hat, Inc.
+ *
+ * Authors:
+ *  Stefan Hajnoczi   <stefanha@redhat.com>
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2 or later.
+ * See the COPYING file in the top-level directory.
+ *
+ */
+
+#include "qemu/osdep.h"
+#include "standard-headers/linux/virtio_ids.h"
+#include "qapi/error.h"
+#include "qemu/iov.h"
+#include "qemu/sockets.h"
+#include "hw/virtio/virtio-access.h"
+#include "hw/virtio/virtio-vhost-user.h"
+#include "virtio-pci.h" /* TODO remove, see virtio_vhost_user_init_bar() */
+#include "trace.h"
+
+/* vmstate migration version number */
+#define VIRTIO_VHOST_USER_VM_VERSION    0
+
+/* Descriptor ring size.  Only one vhost-user protocol message is processed at
+ * a time but later messages can be queued.
+ */
+#define VIRTIO_VHOST_USER_VIRTQUEUE_SIZE 128
+
+/* Protocol features that have been implemented */
+#define SUPPORTED_VHOST_USER_FEATURES \
+    (VHOST_USER_PROTOCOL_F_MQ | VHOST_USER_PROTOCOL_F_REPLY_ACK)
+
+enum {
+    /* TODO Doorbell register size in bytes.  Remove this, see
+     * virtio_vhost_user_init_bar() */
+    DOORBELLS_SIZE = (VIRTIO_QUEUE_MAX + 1 /* logfd */) * sizeof(uint16_t),
+};
+
+/* Connection state machine
+ *
+ * The vhost-user master might not always be connected and the driver might not
+ * always be ready either.  The device interface has a way to manage connection
+ * establishment:
+ *
+ * The driver indicates readiness with the VIRTIO_VHOST_USER_STATUS_SLAVE_UP
+ * status bit.  The device then begins establishing a connection with the
+ * vhost-user master.  The VIRTIO_VHOST_USER_STATUS_MASTER_UP status bit is set
+ * when connected.
+ *
+ * The driver may decide it wants to disconnect at any time.  Vhost-user
+ * protocol violations and other errors might cause the device to give up on
+ * the connection too.
+ *
+ * This state machine captures all transitions in one place.  This way the
+ * connection management code isn't sprinkled around many locations.
+ */
+typedef enum {
+    CONN_STATE_UNDEFINED,
+    CONN_STATE_INITIAL,     /* !SLAVE_UP + !CHR_OPENED */
+    CONN_STATE_SLAVE_UP,    /* SLAVE_UP + !CHR_OPENED */
+    CONN_STATE_CHR_OPENED,  /* !SLAVE_UP + CHR_OPENED */
+    CONN_STATE_CONNECTED,   /* SLAVE_UP + CHR_OPENED */
+    CONN_STATE_MAX
+} ConnectionState;
+
+typedef enum {
+    /* Driver sets VIRTIO_VHOST_USER_STATUS_SLAVE_UP */
+    CONN_EVENT_SLAVE_UP,
+
+    /* Driver clears VIRTIO_VHOST_USER_STATUS_SLAVE_UP */
+    CONN_EVENT_SLAVE_DOWN,
+
+    /* Socket connected and also each time we update chardev handlers */
+    CONN_EVENT_CHR_OPENED,
+
+    /* Socket disconnected */
+    CONN_EVENT_CHR_CLOSED,
+
+    /* Socket chardev was replaced */
+    CONN_EVENT_CHR_CHANGE,
+
+    /* Socket I/O error */
+    CONN_EVENT_SOCKET_ERROR,
+
+    /* Virtio device reset */
+    CONN_EVENT_DEVICE_RESET,
+
+    /* Vhost-user protocol violation by master */
+    CONN_EVENT_MASTER_EINVAL,
+
+    /* Vhost-user protocol violation by slave */
+    CONN_EVENT_SLAVE_EINVAL,
+
+    CONN_EVENT_MAX
+} ConnectionEvent;
+
+static void conn_state_transition(VirtIOVhostUser *s, ConnectionEvent evt);
+
+static void virtio_vhost_user_reset_async_state(VirtIOVhostUser *s)
+{
+    s->read_bytes_needed = 0;
+    s->read_ptr = NULL;
+    s->read_done = NULL;
+    s->read_waiting_on_rxq = false;
+    s->read_msg_size = 0;
+
+    s->write_bytes_avail = 0;
+    s->write_ptr = NULL;
+    s->write_done = NULL;
+    if (s->write_watch_tag) {
+        g_source_remove(s->write_watch_tag);
+    }
+    s->write_watch_tag = 0;
+}
+
+static void virtio_vhost_user_chr_event(void *opaque, int event)
+{
+    VirtIOVhostUser *s = opaque;
+
+    trace_virtio_vhost_user_chr_event(s, event);
+
+    switch (event) {
+    case CHR_EVENT_OPENED:
+        conn_state_transition(s, CONN_EVENT_CHR_OPENED);
+        break;
+    case CHR_EVENT_CLOSED:
+        conn_state_transition(s, CONN_EVENT_CHR_CLOSED);
+        break;
+    }
+}
+
+static int virtio_vhost_user_chr_change(void *opaque)
+{
+    VirtIOVhostUser *s = opaque;
+
+    trace_virtio_vhost_user_chr_change(s);
+
+    if (s->config.status & (1 << VIRTIO_VHOST_USER_STATUS_MASTER_UP)) {
+        conn_state_transition(s, CONN_EVENT_CHR_CHANGE);
+    }
+    return 0;
+}
+
+static int virtio_vhost_user_chr_can_read(void *opaque)
+{
+    VirtIOVhostUser *s = opaque;
+
+    return s->read_bytes_needed;
+}
+
+static void virtio_vhost_user_chr_read(void *opaque,
+                                       const uint8_t *buf, int size)
+{
+    VirtIOVhostUser *s = opaque;
+
+    assert(size <= s->read_bytes_needed);
+
+    memcpy(s->read_ptr, buf, size);
+    s->read_ptr += size;
+    s->read_bytes_needed -= size;
+
+    if (s->read_bytes_needed == 0) {
+        qemu_chr_fe_set_handlers(&s->chr, NULL, NULL,
+                virtio_vhost_user_chr_event,
+                virtio_vhost_user_chr_change,
+                s, NULL, false);
+
+        s->read_done(s);
+    }
+}
+
+/* Start reading from vhost-user socket */
+static void virtio_vhost_user_aio_read(VirtIOVhostUser *s,
+                                       void *buf, size_t len,
+                                       void (*done)(VirtIOVhostUser *s))
+{
+    assert(s->read_bytes_needed == 0);
+
+    s->read_ptr = buf;
+    s->read_bytes_needed = len;
+    s->read_done = done;
+
+    qemu_chr_fe_set_handlers(&s->chr,
+            virtio_vhost_user_chr_can_read,
+            virtio_vhost_user_chr_read,
+            virtio_vhost_user_chr_event,
+            virtio_vhost_user_chr_change,
+            s, NULL, false);
+}
+
+/* Called once with chan=NULL, cond=0 to begin and then called by event loop */
+static gboolean virtio_vhost_user_chr_write(GIOChannel *chan, GIOCondition cond,
+                                            void *opaque)
+{
+    VirtIOVhostUser *s = opaque;
+    int nwritten;
+    guint tag = s->write_watch_tag;
+
+    nwritten = qemu_chr_fe_write(&s->chr, s->write_ptr, s->write_bytes_avail);
+    if (nwritten < 0) {
+        if (errno == EAGAIN) {
+            nwritten = 0;
+        } else {
+            conn_state_transition(s, CONN_EVENT_SOCKET_ERROR);
+            return G_SOURCE_REMOVE;
+        }
+    }
+
+    s->write_bytes_avail -= nwritten;
+    if (s->write_bytes_avail == 0) {
+        s->write_done(s);
+        return G_SOURCE_REMOVE;
+    }
+
+    if (tag == 0) {
+        tag = qemu_chr_fe_add_watch(&s->chr, G_IO_OUT | G_IO_HUP,
+                                    virtio_vhost_user_chr_write, s);
+        if (!tag) {
+            conn_state_transition(s, CONN_EVENT_SOCKET_ERROR);
+            return G_SOURCE_REMOVE;
+        }
+
+        s->write_watch_tag = tag;
+    }
+
+    return G_SOURCE_CONTINUE;
+}
+
+/* Start writing to vhost-user socket */
+static void virtio_vhost_user_aio_write(VirtIOVhostUser *s,
+                                        void *buf, size_t len,
+                                        void (*done)(VirtIOVhostUser *s))
+{
+    assert(s->write_bytes_avail == 0);
+
+    s->write_ptr = buf;
+    s->write_bytes_avail = len;
+    s->write_done = done;
+
+    virtio_vhost_user_chr_write(NULL, 0, s);
+}
+
+static void virtio_vhost_user_cleanup_callfds(VirtIOVhostUser *s)
+{
+    size_t i;
+
+    for (i = 0; i < ARRAY_SIZE(s->callfds); i++) {
+        if (s->callfds[i] >= 0) {
+            close(s->callfds[i]);
+            s->callfds[i] = -1;
+        }
+    }
+}
+
+static void virtio_vhost_user_cleanup_mem_table(VirtIOVhostUser *s)
+{
+    int i;
+
+    for (i = 0; i < VHOST_MEMORY_MAX_NREGIONS; i++) {
+        VirtIOVhostUserMemTableRegion *region = &s->mem_table[i];
+
+        if (!region->mmap_addr) {
+            continue;
+        }
+
+        munmap(region->mmap_addr, region->total_size);
+        region->mmap_addr = NULL;
+
+        memory_region_del_subregion(&s->additional_resources_bar,
+                                    &region->mr);
+        object_unparent(OBJECT(&region->mr));
+    }
+}
+
+static void conn_action_set_slave_up(VirtIOVhostUser *s)
+{
+    /* Guest-initiated, no need for virtio_notify_config() */
+    s->config.status = (1 << VIRTIO_VHOST_USER_STATUS_SLAVE_UP);
+}
+
+static void conn_action_set_slave_down(VirtIOVhostUser *s)
+{
+    /* Guest-initiated, no need for virtio_notify_config() */
+    s->config.status = 0;
+}
+
+static void virtio_vhost_user_hdr_done(VirtIOVhostUser *s);
+
+static void conn_action_connect(VirtIOVhostUser *s)
+{
+    s->config.status = (1 << VIRTIO_VHOST_USER_STATUS_SLAVE_UP) |
+                       (1 << VIRTIO_VHOST_USER_STATUS_MASTER_UP);
+    virtio_notify_config(VIRTIO_DEVICE(s));
+
+    /* Begin servicing vhost-user messages */
+    virtio_vhost_user_aio_read(s, &s->read_msg, VHOST_USER_HDR_SIZE,
+                               virtio_vhost_user_hdr_done);
+}
+
+static void conn_action_disconnect_no_notify(VirtIOVhostUser *s)
+{
+    qemu_chr_fe_set_handlers(&s->chr,
+            NULL,
+            NULL,
+            virtio_vhost_user_chr_event,
+            virtio_vhost_user_chr_change,
+            s, NULL, false);
+    qemu_chr_fe_set_open(&s->chr, 0);
+
+    virtio_vhost_user_reset_async_state(s);
+
+    /* TODO drain txq? */
+
+    /* It is only safe to clean up resources where future accesses have no
+     * guest-visible effects.  Vcpus may still access resources if they haven't
+     * noticed the disconnect event yet.  Callfds are safe since writes to
+     * invalid indices are ignored.  Memory table regions cannot be unmapped
+     * since vring polling may still be running.
+     */
+    virtio_vhost_user_cleanup_callfds(s);
+
+    s->config.status = 0;
+}
+
+static void conn_action_disconnect(VirtIOVhostUser *s)
+{
+    conn_action_disconnect_no_notify(s);
+    virtio_notify_config(VIRTIO_DEVICE(s));
+}
+
+static const struct {
+    void (*action)(VirtIOVhostUser *s);
+    ConnectionState new_state;
+} conn_state_machine[CONN_STATE_MAX][CONN_EVENT_MAX] = {
+    [CONN_STATE_INITIAL] = {
+        [CONN_EVENT_SLAVE_UP] = {conn_action_set_slave_up,
+                                 CONN_STATE_SLAVE_UP},
+        [CONN_EVENT_CHR_OPENED] = {NULL, CONN_STATE_CHR_OPENED},
+        [CONN_EVENT_CHR_CLOSED] = {NULL, CONN_STATE_INITIAL},
+        [CONN_EVENT_CHR_CHANGE] = {NULL, CONN_STATE_INITIAL},
+        [CONN_EVENT_DEVICE_RESET] = {NULL, CONN_STATE_INITIAL},
+    },
+    [CONN_STATE_SLAVE_UP] = {
+        [CONN_EVENT_SLAVE_DOWN] = {conn_action_set_slave_down,
+                                   CONN_STATE_INITIAL},
+        [CONN_EVENT_CHR_OPENED] = {conn_action_connect, CONN_STATE_CONNECTED},
+        [CONN_EVENT_CHR_CLOSED] = {NULL, CONN_STATE_SLAVE_UP},
+        [CONN_EVENT_CHR_CHANGE] = {NULL, CONN_STATE_SLAVE_UP},
+        [CONN_EVENT_DEVICE_RESET] = {conn_action_set_slave_down,
+                                     CONN_STATE_INITIAL},
+    },
+    [CONN_STATE_CHR_OPENED] = {
+        [CONN_EVENT_SLAVE_UP] = {conn_action_connect, CONN_STATE_CONNECTED},
+        [CONN_EVENT_CHR_OPENED] = {NULL, CONN_STATE_CHR_OPENED},
+        [CONN_EVENT_CHR_CLOSED] = {NULL, CONN_STATE_INITIAL},
+        [CONN_EVENT_CHR_CHANGE] = {NULL, CONN_STATE_CHR_OPENED},
+        [CONN_EVENT_DEVICE_RESET] = {NULL, CONN_STATE_INITIAL},
+    },
+    [CONN_STATE_CONNECTED] = {
+        [CONN_EVENT_SLAVE_DOWN] = {conn_action_disconnect_no_notify,
+                                   CONN_STATE_INITIAL},
+        [CONN_EVENT_CHR_OPENED] = {NULL, CONN_STATE_CONNECTED},
+        [CONN_EVENT_CHR_CLOSED] = {conn_action_disconnect,
+                                   CONN_STATE_INITIAL},
+        [CONN_EVENT_CHR_CHANGE] = {conn_action_disconnect, CONN_STATE_INITIAL},
+        [CONN_EVENT_SOCKET_ERROR] = {conn_action_disconnect,
+                                     CONN_STATE_INITIAL},
+        [CONN_EVENT_DEVICE_RESET] = {conn_action_disconnect_no_notify,
+                                     CONN_STATE_INITIAL},
+        [CONN_EVENT_MASTER_EINVAL] = {conn_action_disconnect,
+                                      CONN_STATE_INITIAL},
+        [CONN_EVENT_SLAVE_EINVAL] = {conn_action_disconnect,
+                                     CONN_STATE_INITIAL},
+    },
+};
+
+static void conn_state_transition(VirtIOVhostUser *s, ConnectionEvent evt)
+{
+    ConnectionState old_state = s->conn_state;
+    ConnectionState new_state = conn_state_machine[old_state][evt].new_state;
+
+    trace_virtio_vhost_user_conn_state_transition(s, old_state, evt,
+                                                  new_state);
+    assert(new_state != CONN_STATE_UNDEFINED);
+
+    s->conn_state = new_state;
+
+    if (conn_state_machine[old_state][evt].action) {
+        conn_state_machine[old_state][evt].action(s);
+    }
+}
+
+/* Master-to-slave message processing
+ *
+ * Messages are read from the vhost-user socket into s->read_msg.  They are
+ * then parsed and may be modified.  Finally they are put onto the rxq for the
+ * driver to read.
+ *
+ * Functions with "m2s" in their name handle the master-to-slave code path.
+ */
+
+/* Put s->read_msg onto the rxq */
+static void virtio_vhost_user_deliver_m2s(VirtIOVhostUser *s)
+{
+    VirtQueueElement *elem;
+    size_t copied;
+
+    elem = virtqueue_pop(s->rxq, sizeof(*elem));
+    if (!elem) {
+        /* Leave message in s->read_msg and wait for rxq */
+        trace_virtio_vhost_user_rxq_empty(s);
+        s->read_waiting_on_rxq = true;
+        return;
+    }
+
+    s->read_waiting_on_rxq = false;
+
+    copied = iov_from_buf(elem->in_sg, elem->in_num, 0, &s->read_msg,
+                          s->read_msg_size);
+    if (copied != s->read_msg_size) {
+        g_free(elem);
+        virtio_error(VIRTIO_DEVICE(s),
+                     "rxq buffer too small, got %zu, needed %zu",
+                     copied, s->read_msg_size);
+        return;
+    }
+
+    virtqueue_push(s->rxq, elem, copied);
+    g_free(elem);
+
+    virtio_notify(VIRTIO_DEVICE(s), s->rxq);
+
+    /* Next message, please */
+    virtio_vhost_user_aio_read(s, &s->read_msg, VHOST_USER_HDR_SIZE,
+                               virtio_vhost_user_hdr_done);
+}
+
+static void m2s_set_vring_kick(VirtIOVhostUser *s)
+{
+    if (s->read_msg.payload.u64 & VHOST_USER_VRING_NOFD_MASK) {
+        return;
+    }
+
+    /* TODO implement an interrupt.  For now we force polling mode. */
+    close(qemu_chr_fe_get_msgfd(&s->chr));
+    s->read_msg.payload.u64 |= VHOST_USER_VRING_NOFD_MASK;
+}
+
+static void m2s_set_vring_call(VirtIOVhostUser *s)
+{
+    uint8_t vq_idx;
+    int fd;
+
+    vq_idx = s->read_msg.payload.u64 & VHOST_USER_VRING_IDX_MASK;
+
+    /* We should always have a large enough array */
+    QEMU_BUILD_BUG_ON(0xff >= ARRAY_SIZE(s->callfds));
+
+    if (s->read_msg.payload.u64 & VHOST_USER_VRING_NOFD_MASK) {
+        fd = -1;
+    } else {
+        fd = qemu_chr_fe_get_msgfd(&s->chr);
+
+        /* Must not block when reach max eventfd counter value */
+        qemu_set_nonblock(fd);
+    }
+
+    if (s->callfds[vq_idx] >= 0) {
+        close(s->callfds[vq_idx]);
+    }
+
+    s->callfds[vq_idx] = fd;
+}
+
+static void m2s_set_mem_table(VirtIOVhostUser *s)
+{
+    VhostUserMemory *memory = &s->read_msg.payload.memory;
+    hwaddr subregion_offset;
+    int fds[VHOST_MEMORY_MAX_NREGIONS];
+    int num_fds;
+    uint32_t i;
+
+    if (memory->nregions > VHOST_MEMORY_MAX_NREGIONS) {
+        conn_state_transition(s, CONN_EVENT_MASTER_EINVAL);
+        return;
+    }
+
+    num_fds = qemu_chr_fe_get_msgfds(&s->chr, fds, ARRAY_SIZE(fds));
+    if (num_fds != memory->nregions) {
+        conn_state_transition(s, CONN_EVENT_MASTER_EINVAL);
+        return;
+    }
+
+    virtio_vhost_user_cleanup_mem_table(s);
+
+    /* Start after the doorbell registers */
+    subregion_offset = QEMU_ALIGN_UP(DOORBELLS_SIZE, 4096);
+
+    for (i = 0; i < memory->nregions; i++) {
+        VhostUserMemoryRegion *input = &memory->regions[i];
+        VirtIOVhostUserMemTableRegion *region = &s->mem_table[i];
+        void *mmap_addr;
+
+        region->total_size = input->mmap_offset + input->memory_size;
+        if (region->total_size < input->mmap_offset ||
+            region->total_size < input->memory_size) {
+            goto err;
+        }
+
+        mmap_addr = mmap(0, region->total_size, PROT_READ | PROT_WRITE,
+                         MAP_SHARED, fds[i], 0);
+        close(fds[i]);
+        fds[i] = -1;
+        if (mmap_addr == MAP_FAILED) {
+            goto err;
+        }
+        region->mmap_addr = mmap_addr;
+
+        trace_virtio_vhost_user_memory_region(s,
+                memory->regions[i].guest_phys_addr,
+                memory->regions[i].memory_size,
+                memory->regions[i].userspace_addr,
+                memory->regions[i].mmap_offset,
+                region->mmap_addr);
+
+        memory_region_init_ram_ptr(&region->mr, OBJECT(s),
+                "virtio-vhost-user-mem-table-region",
+                region->total_size, region->mmap_addr);
+        memory_region_add_subregion(&s->additional_resources_bar,
+                                    subregion_offset, &region->mr);
+
+        subregion_offset += region->total_size;
+    }
+
+    return;
+
+err:
+    for (i = 0; i < memory->nregions; i++) {
+        if (fds[i] >= 0) {
+            close(fds[i]);
+        }
+    }
+    conn_state_transition(s, CONN_EVENT_MASTER_EINVAL);
+}
+
+static void m2s_set_protocol_features(VirtIOVhostUser *s)
+{
+    /* Only allow features we support too */
+    s->read_msg.payload.u64 &= SUPPORTED_VHOST_USER_FEATURES;
+}
+
+/* Parse s->read_msg from master */
+static void virtio_vhost_user_parse_m2s(VirtIOVhostUser *s)
+{
+    uint32_t version = s->read_msg.flags & VHOST_USER_VERSION_MASK;
+
+    if (version != VHOST_USER_VERSION) {
+        trace_virtio_vhost_user_m2s_bad_version(s, version);
+        conn_state_transition(s, CONN_EVENT_MASTER_EINVAL);
+        return;
+    }
+
+    if (s->read_msg.flags & VHOST_USER_REPLY_MASK) {
+        trace_virtio_vhost_user_m2s_unexpected_reply(s);
+        conn_state_transition(s, CONN_EVENT_MASTER_EINVAL);
+        return;
+    }
+
+    if (s->read_msg.request >= VHOST_USER_MAX) {
+        trace_virtio_vhost_user_m2s_bad_request(s, s->read_msg.request);
+        conn_state_transition(s, CONN_EVENT_MASTER_EINVAL);
+        return;
+    }
+
+    trace_virtio_vhost_user_m2s_request(s, s->read_msg.request);
+
+    /* Most messages are passed through but a few need to be handled */
+    switch (s->read_msg.request) {
+    case VHOST_USER_GET_FEATURES:
+        break;
+    case VHOST_USER_SET_FEATURES:
+        break;
+    case VHOST_USER_SET_OWNER:
+        break;
+    case VHOST_USER_RESET_OWNER:
+        break;
+    case VHOST_USER_SET_MEM_TABLE:
+        m2s_set_mem_table(s);
+        break;
+    case VHOST_USER_SET_VRING_NUM:
+        break;
+    case VHOST_USER_SET_VRING_ADDR:
+        break;
+    case VHOST_USER_SET_VRING_BASE:
+        break;
+    case VHOST_USER_GET_VRING_BASE:
+        break;
+    case VHOST_USER_SET_VRING_KICK:
+        m2s_set_vring_kick(s);
+        break;
+    case VHOST_USER_SET_VRING_CALL:
+        m2s_set_vring_call(s);
+        break;
+    case VHOST_USER_GET_PROTOCOL_FEATURES:
+        break;
+    case VHOST_USER_SET_PROTOCOL_FEATURES:
+        m2s_set_protocol_features(s);
+        break;
+    case VHOST_USER_GET_QUEUE_NUM:
+        break;
+    case VHOST_USER_SET_VRING_ENABLE:
+        break;
+    default:
+        trace_virtio_vhost_user_m2s_unknown_request(s, s->read_msg.request);
+        conn_state_transition(s, CONN_EVENT_MASTER_EINVAL);
+        return;
+    }
+
+    /* Bail if a handler function reset the connection */
+    if (s->conn_state != CONN_STATE_CONNECTED) {
+        return;
+    }
+
+    /* Stash size before we endian-convert s->read_msg */
+    s->read_msg_size = VHOST_USER_HDR_SIZE + s->read_msg.size;
+
+    /* TODO convert read_msg to little-endian for cross-endian support */
+
+    virtio_vhost_user_deliver_m2s(s);
+}
+
+static void virtio_vhost_user_hdr_done(VirtIOVhostUser *s)
+{
+    if (s->read_msg.size > VHOST_USER_PAYLOAD_SIZE) {
+        trace_virtio_vhost_user_m2s_bad_payload_size(s, s->read_msg.size);
+        conn_state_transition(s, CONN_EVENT_MASTER_EINVAL);
+        return;
+    }
+
+    /* Clear out unused payload bytes */
+    memset(&s->read_msg.payload, 0, VHOST_USER_PAYLOAD_SIZE);
+
+    if (s->read_msg.size > 0) {
+        virtio_vhost_user_aio_read(s, &s->read_msg.payload, s->read_msg.size,
+                                   virtio_vhost_user_parse_m2s);
+    } else {
+        virtio_vhost_user_parse_m2s(s);
+    }
+}
+
+static void virtio_vhost_user_rxq(VirtIODevice *vdev, VirtQueue *vq)
+{
+    VirtIOVhostUser *s = VIRTIO_VHOST_USER(vdev);
+
+    if (s->read_waiting_on_rxq) {
+        virtio_vhost_user_deliver_m2s(s);
+    }
+}
+
+/* Slave-to-master message processing
+ *
+ * Messages are read from the txq into s->write_msg.  They are then parsed and
+ * may be modified.  Finally they are written to the vhost-user socket.
+ *
+ * Functions with "s2m" in their name handle the slave-to-master code path.
+ */
+
+static void s2m_get_protocol_features(VirtIOVhostUser *s)
+{
+    /* Only allow features we support too */
+    s->write_msg.payload.u64 &= SUPPORTED_VHOST_USER_FEATURES;
+}
+
+static void virtio_vhost_user_tx_done(VirtIOVhostUser *s);
+
+/* Parse s->write_msg from slave */
+static void virtio_vhost_user_parse_s2m(VirtIOVhostUser *s)
+{
+    uint32_t version = s->write_msg.flags & VHOST_USER_VERSION_MASK;
+
+    if (version != VHOST_USER_VERSION) {
+        trace_virtio_vhost_user_s2m_bad_version(s, version);
+        conn_state_transition(s, CONN_EVENT_SLAVE_EINVAL);
+        return;
+    }
+
+    if (!(s->write_msg.flags & VHOST_USER_REPLY_MASK)) {
+        trace_virtio_vhost_user_s2m_expected_reply(s);
+        conn_state_transition(s, CONN_EVENT_SLAVE_EINVAL);
+        return;
+    }
+
+    if (s->write_msg.request >= VHOST_USER_MAX) {
+        trace_virtio_vhost_user_s2m_bad_request(s, s->write_msg.request);
+        conn_state_transition(s, CONN_EVENT_SLAVE_EINVAL);
+        return;
+    }
+
+    trace_virtio_vhost_user_s2m_request(s, s->write_msg.request);
+
+    /* Very few messages need to be touched */
+    switch (s->write_msg.request) {
+    case VHOST_USER_GET_FEATURES:
+        break;
+    case VHOST_USER_SET_FEATURES:
+        break;
+    case VHOST_USER_SET_OWNER:
+        break;
+    case VHOST_USER_RESET_OWNER:
+        break;
+    case VHOST_USER_SET_MEM_TABLE:
+        break;
+    case VHOST_USER_SET_VRING_NUM:
+        break;
+    case VHOST_USER_SET_VRING_ADDR:
+        break;
+    case VHOST_USER_SET_VRING_BASE:
+        break;
+    case VHOST_USER_GET_VRING_BASE:
+        break;
+    case VHOST_USER_SET_VRING_KICK:
+        break;
+    case VHOST_USER_SET_VRING_CALL:
+        break;
+    case VHOST_USER_GET_PROTOCOL_FEATURES:
+        s2m_get_protocol_features(s);
+        break;
+    case VHOST_USER_SET_PROTOCOL_FEATURES:
+        break;
+    case VHOST_USER_GET_QUEUE_NUM:
+        break;
+    case VHOST_USER_SET_VRING_ENABLE:
+        break;
+    default:
+        trace_virtio_vhost_user_s2m_unknown_request(s, s->write_msg.request);
+        conn_state_transition(s, CONN_EVENT_SLAVE_EINVAL);
+        return;
+    }
+
+    /* Bail if a handler function reset the connection */
+    if (s->conn_state != CONN_STATE_CONNECTED) {
+        return;
+    }
+
+    virtio_vhost_user_aio_write(s, &s->write_msg,
+                                VHOST_USER_HDR_SIZE + s->write_msg.size,
+                                virtio_vhost_user_tx_done);
+}
+
+static void virtio_vhost_user_txq(VirtIODevice *vdev, VirtQueue *vq)
+{
+    VirtIOVhostUser *s = VIRTIO_VHOST_USER(vdev);
+    VirtQueueElement *elem;
+    size_t msgsize;
+    size_t copied;
+
+    /* If the last message is still being transferred we'll come back later */
+    if (s->write_bytes_avail != 0) {
+        return;
+    }
+
+    elem = virtqueue_pop(s->txq, sizeof(*elem));
+    if (!elem) {
+        return; /* no elements left on virtqueue */
+    }
+
+    msgsize = iov_size(elem->out_sg, elem->out_num);
+    if (msgsize < VHOST_USER_HDR_SIZE || msgsize > sizeof(s->write_msg)) {
+        g_free(elem);
+        virtio_error(VIRTIO_DEVICE(s),
+                     "invalid txq buffer size, got %zu", msgsize);
+        return;
+    }
+
+    /* Clear out unused payload bytes */
+    memset(&s->write_msg.payload, 0, VHOST_USER_PAYLOAD_SIZE);
+
+    copied = iov_to_buf(elem->out_sg, elem->out_num, 0,
+                        &s->write_msg, msgsize);
+    if (copied != VHOST_USER_HDR_SIZE + s->write_msg.size ||
+        copied != msgsize) {
+        g_free(elem);
+        virtio_error(VIRTIO_DEVICE(s),
+                     "invalid txq buffer size, got %zu", msgsize);
+        return;
+    }
+
+    virtqueue_push(s->txq, elem, copied);
+    g_free(elem);
+
+    virtio_notify(VIRTIO_DEVICE(s), s->txq);
+
+    /* TODO convert from little-endian */
+
+    virtio_vhost_user_parse_s2m(s);
+}
+
+static void virtio_vhost_user_tx_done(VirtIOVhostUser *s)
+{
+    VirtIODevice *vdev = VIRTIO_DEVICE(s);
+    VirtQueue *vq = s->txq;
+
+    trace_virtio_vhost_user_tx_done(s);
+
+    /* Try to process more messages from the driver */
+    virtio_vhost_user_txq(vdev, vq);
+}
+
+static uint64_t
+virtio_vhost_user_get_features(VirtIODevice *vdev,
+                               uint64_t requested_features,
+                               Error **errp)
+{
+    return requested_features;
+}
+
+static void virtio_vhost_user_get_config(VirtIODevice *vdev, uint8_t *config)
+{
+    VirtIOVhostUserConfig *vvuconfig = (VirtIOVhostUserConfig *)config;
+    VirtIOVhostUser *s = VIRTIO_VHOST_USER(vdev);
+
+    virtio_stl_p(vdev, &vvuconfig->status, s->config.status);
+    virtio_stl_p(vdev, &vvuconfig->max_vhost_queues,
+                 s->config.max_vhost_queues);
+    memcpy(vvuconfig->uuid, s->config.uuid, sizeof(vvuconfig->uuid));
+}
+
+static void virtio_vhost_user_set_config(VirtIODevice *vdev,
+                                         const uint8_t *config)
+{
+    VirtIOVhostUserConfig *vvuconfig = (VirtIOVhostUserConfig *)config;
+    VirtIOVhostUser *s = VIRTIO_VHOST_USER(vdev);
+    uint32_t status;
+    bool old_slave_up;
+    bool new_slave_up;
+
+    status = virtio_ldl_p(vdev, &vvuconfig->status);
+    trace_virtio_vhost_user_set_config(s, s->config.status, status);
+    if (status & ~((1 << VIRTIO_VHOST_USER_STATUS_SLAVE_UP) |
+                   (1 << VIRTIO_VHOST_USER_STATUS_MASTER_UP))) {
+        virtio_error(vdev, "undefined virtio-vhost-user status bit set "
+                           "(%#x)", status);
+        return;
+    }
+
+    old_slave_up = s->config.status & (1 << VIRTIO_VHOST_USER_STATUS_SLAVE_UP);
+    new_slave_up = status & (1 << VIRTIO_VHOST_USER_STATUS_SLAVE_UP);
+
+    if (!old_slave_up && new_slave_up) {
+        conn_state_transition(s, CONN_EVENT_SLAVE_UP);
+    } else if (old_slave_up && !new_slave_up) {
+        conn_state_transition(s, CONN_EVENT_SLAVE_DOWN);
+    }
+}
+
+static void virtio_vhost_user_reset(VirtIODevice *vdev)
+{
+    VirtIOVhostUser *s = VIRTIO_VHOST_USER(vdev);
+
+    conn_state_transition(s, CONN_EVENT_DEVICE_RESET);
+
+    virtio_vhost_user_reset_async_state(s);
+}
+
+static uint64_t virtio_vhost_user_doorbells_read(void *opaque, hwaddr addr,
+                                                 unsigned size)
+{
+    return 0;
+}
+
+static void virtio_vhost_user_doorbells_write(void *opaque, hwaddr addr,
+                                              uint64_t val, unsigned size)
+{
+    VirtIOVhostUser *s = opaque;
+    unsigned idx = addr / sizeof(uint16_t);
+
+    if (idx < VIRTIO_QUEUE_MAX) {
+        /* TODO use memory_region_add_eventfd() to avoid entering QEMU */
+
+        if (s->callfds[idx] >= 0) {
+            uint64_t val = 1;
+            ssize_t nwritten;
+
+            nwritten = write(s->callfds[idx], &val, sizeof(val));
+            trace_virtio_vhost_user_doorbell_write(s, idx, nwritten);
+        }
+    } else if (idx == VIRTIO_QUEUE_MAX) {
+        /* TODO log doorbell */
+    }
+}
+
+/* TODO implement "5.7.7 Additional Device Resources over PCI" in
+ * hw/virtio/virtio-pci.c instead of adding PCI BARs here
+ * https://stefanha.github.io/virtio/vhost-user-slave.html#x1-2920007
+ */
+static void virtio_vhost_user_init_bar(VirtIOVhostUser *s)
+{
+    static const MemoryRegionOps virtio_vhost_user_doorbells_ops = {
+        .read = virtio_vhost_user_doorbells_read,
+        .write = virtio_vhost_user_doorbells_write,
+        .valid = {
+            .min_access_size = 1,
+            .max_access_size = 4,
+        },
+        .endianness = DEVICE_LITTLE_ENDIAN,
+    };
+
+    /* virtio-pci doesn't use BAR 2 & 3, so we use it */
+    const int bar_index = 2;
+
+    /* TODO If the BAR is too large the guest won't have address space to map
+     * it!
+     */
+    const uint64_t bar_size = 1ULL << 36;
+
+    VirtIOVhostUserPCI *vvup = container_of(s, struct VirtIOVhostUserPCI, vdev);
+
+    memory_region_init(&s->additional_resources_bar, OBJECT(s),
+                       "virtio-vhost-user", bar_size);
+
+    memory_region_init_io(&s->doorbell_region, OBJECT(s),
+                          &virtio_vhost_user_doorbells_ops,
+                          s, "virtio-vhost-user-doorbells",
+                          DOORBELLS_SIZE);
+    memory_region_add_subregion(&s->additional_resources_bar, 0,
+                                &s->doorbell_region);
+
+    pci_register_bar(&vvup->parent_obj.pci_dev, bar_index,
+                     PCI_BASE_ADDRESS_SPACE_MEMORY |
+                     PCI_BASE_ADDRESS_MEM_PREFETCH |
+                     PCI_BASE_ADDRESS_MEM_TYPE_64,
+                     &s->additional_resources_bar);
+}
+
+static void virtio_vhost_user_cleanup_bar(VirtIOVhostUser *s)
+{
+    memory_region_del_subregion(&s->additional_resources_bar,
+                                &s->doorbell_region);
+}
+
+static void virtio_vhost_user_device_realize(DeviceState *dev, Error **errp)
+{
+    VirtIODevice *vdev = VIRTIO_DEVICE(dev);
+    VirtIOVhostUser *s = VIRTIO_VHOST_USER(dev);
+    size_t i;
+
+    if (!qemu_chr_fe_backend_connected(&s->chr)) {
+        error_setg(errp, "Missing chardev");
+        return;
+    }
+
+    for (i = 0; i < ARRAY_SIZE(s->callfds); i++) {
+        s->callfds[i] = -1;
+    }
+
+    virtio_vhost_user_init_bar(s);
+
+    virtio_init(vdev, "virtio-vhost-user", VIRTIO_ID_VHOST_USER,
+                sizeof(VirtIOVhostUserConfig));
+
+    s->rxq = virtio_add_queue(vdev, VIRTIO_VHOST_USER_VIRTQUEUE_SIZE,
+                                 virtio_vhost_user_rxq);
+    s->txq = virtio_add_queue(vdev, VIRTIO_VHOST_USER_VIRTQUEUE_SIZE,
+
+                                 virtio_vhost_user_txq);
+    /* Each vhost-user queue uses doorbells and a notification resources */
+    s->config.max_vhost_queues = 1024;
+
+    /* TODO uuid */
+
+    virtio_vhost_user_reset_async_state(s);
+
+    s->conn_state = CONN_STATE_INITIAL;
+    qemu_chr_fe_set_handlers(&s->chr, NULL, NULL, virtio_vhost_user_chr_event,
+            virtio_vhost_user_chr_change, s, NULL, false);
+}
+
+static void virtio_vhost_user_device_unrealize(DeviceState *dev, Error **errp)
+{
+    VirtIODevice *vdev = VIRTIO_DEVICE(dev);
+    VirtIOVhostUser *s = VIRTIO_VHOST_USER(vdev);
+
+    qemu_chr_fe_set_handlers(&s->chr, NULL, NULL, NULL,
+                             NULL, NULL, NULL, false);
+    virtio_cleanup(vdev);
+    virtio_vhost_user_cleanup_bar(s);
+    virtio_vhost_user_cleanup_mem_table(s);
+    virtio_vhost_user_cleanup_callfds(s);
+}
+
+static const VMStateDescription vmstate_virtio_vhost_user_device = {
+    .name = "virtio-vhost-user-device",
+    .version_id = VIRTIO_VHOST_USER_VM_VERSION,
+    .minimum_version_id = VIRTIO_VHOST_USER_VM_VERSION,
+    .fields = (VMStateField[]) {
+        VMSTATE_END_OF_LIST()
+    },
+};
+
+static const VMStateDescription vmstate_virtio_vhost_user = {
+    .name = "virtio-vhost-user",
+    .minimum_version_id = VIRTIO_VHOST_USER_VM_VERSION,
+    .version_id = VIRTIO_VHOST_USER_VM_VERSION,
+    .fields = (VMStateField[]) {
+        VMSTATE_VIRTIO_DEVICE,
+        VMSTATE_END_OF_LIST()
+    },
+};
+
+static Property virtio_vhost_user_properties[] = {
+    DEFINE_PROP_CHR("chardev", VirtIOVhostUser, chr),
+    DEFINE_PROP_END_OF_LIST(),
+};
+
+static void virtio_vhost_user_class_init(ObjectClass *klass, void *data)
+{
+    DeviceClass *dc = DEVICE_CLASS(klass);
+    VirtioDeviceClass *vdc = VIRTIO_DEVICE_CLASS(klass);
+
+    dc->props = virtio_vhost_user_properties;
+    dc->vmsd = &vmstate_virtio_vhost_user;
+    set_bit(DEVICE_CATEGORY_MISC, dc->categories);
+    vdc->realize = virtio_vhost_user_device_realize;
+    vdc->unrealize = virtio_vhost_user_device_unrealize;
+    vdc->get_config = virtio_vhost_user_get_config;
+    vdc->set_config = virtio_vhost_user_set_config;
+    vdc->get_features = virtio_vhost_user_get_features;
+    vdc->reset = virtio_vhost_user_reset;
+    vdc->vmsd = &vmstate_virtio_vhost_user_device;
+}
+
+static const TypeInfo virtio_vhost_user_info = {
+    .name = TYPE_VIRTIO_VHOST_USER,
+    .parent = TYPE_VIRTIO_DEVICE,
+    .instance_size = sizeof(VirtIOVhostUser),
+    .class_init = virtio_vhost_user_class_init,
+};
+
+static void virtio_register_types(void)
+{
+    type_register_static(&virtio_vhost_user_info);
+}
+
+type_init(virtio_register_types)
diff --git a/hw/virtio/trace-events b/hw/virtio/trace-events
index 775461ae98..a5849ba5af 100644
--- a/hw/virtio/trace-events
+++ b/hw/virtio/trace-events
@@ -25,3 +25,25 @@ virtio_balloon_handle_output(const char *name, uint64_t gpa) "section name: %s g
 virtio_balloon_get_config(uint32_t num_pages, uint32_t actual) "num_pages: %d actual: %d"
 virtio_balloon_set_config(uint32_t actual, uint32_t oldactual) "actual: %d oldactual: %d"
 virtio_balloon_to_target(uint64_t target, uint32_t num_pages) "balloon target: 0x%"PRIx64" num_pages: %d"
+
+# hw/virtio/virtio-vhost-user.c
+virtio_vhost_user_m2s_bad_version(void *s, unsigned int version) "s %p version %u"
+virtio_vhost_user_m2s_unexpected_reply(void *s) "s %p"
+virtio_vhost_user_m2s_bad_payload_size(void *s, unsigned int size) "s %p size %u"
+virtio_vhost_user_m2s_bad_request(void *s, unsigned request) "s %p request %u"
+virtio_vhost_user_m2s_request(void *s, unsigned int request) "s %p request %u"
+virtio_vhost_user_m2s_unknown_request(void *s, unsigned int request) "s %p request %u"
+virtio_vhost_user_s2m_bad_version(void *s, unsigned int version) "s %p version %u"
+virtio_vhost_user_s2m_expected_reply(void *s) "s %p"
+virtio_vhost_user_s2m_bad_payload_size(void *s, unsigned int size) "s %p size %u"
+virtio_vhost_user_s2m_bad_request(void *s, unsigned request) "s %p request %u"
+virtio_vhost_user_s2m_request(void *s, unsigned int request) "s %p request %u"
+virtio_vhost_user_s2m_unknown_request(void *s, unsigned int request) "s %p request %u"
+virtio_vhost_user_rxq_empty(void *s) "s %p"
+virtio_vhost_user_tx_done(void *s) "s %p"
+virtio_vhost_user_chr_event(void *s, int event) "s %p event %d"
+virtio_vhost_user_chr_change(void *s) "s %p"
+virtio_vhost_user_conn_state_transition(void *s, int old_state, int event, int new_state) "s %p old_state %d event %d new_state %d"
+virtio_vhost_user_set_config(void *s, unsigned int old_status, unsigned int new_status) "s %p old_status %u new_status %u"
+virtio_vhost_user_doorbell_write(void *s, unsigned int vq_idx, ssize_t nwritten) "s %p vq_idx %u nwritten %zd"
+virtio_vhost_user_memory_region(void *s, uint64_t guest_phys_addr, uint64_t memory_size, uint64_t userspace_addr, uint64_t mmap_offset, void *mmap_addr) "s %p guest_phys_addr 0x%"PRIx64" memory_size 0x%"PRIx64" userspace_addr 0x%"PRIx64" mmap_offset 0x%"PRIx64" mmap_addr %p"
-- 
2.14.3

Re: [Qemu-devel] [RFC 0/2] virtio-vhost-user: add virtio-vhost-user device

[Stefan Hajnoczi]

The DPDK patch series is here:
http://dpdk.org/ml/archives/dev/2018-January/088155.html

Re: [Qemu-devel] [RFC 0/2] virtio-vhost-user: add virtio-vhost-user device

[Jason Wang]

On 2018年01月19日 21:06, Stefan Hajnoczi wrote:
> These patches implement the virtio-vhost-user device design that I have
> described here:
> https://stefanha.github.io/virtio/vhost-user-slave.html#x1-2830007

Thanks for the patches, looks rather interesting and similar to split 
device model used by Xen.

>
> The goal is to let the guest act as the vhost device backend for other guests.
> This allows virtual networking and storage appliances to run inside guests.

So the question still, what kind of protocol do you want to run on top? 
If it was ethernet based, virtio-net work pretty well and it can even do 
migration.

> This device is particularly interesting for poll mode drivers where exitless
> VM-to-VM communication is possible, completely bypassing the hypervisor in the
> data path.

It's better to clarify the reason of hypervisor bypassing. (performance, 
security or scalability).

Probably not for the following cases:

1) kick/call
2) device IOTLB / IOMMU transaction (or any other case that backends 
needs metadata from qemu).

>
> The DPDK driver is here:
> https://github.com/stefanha/dpdk/tree/virtio-vhost-user
>
> For more information, see
> https://wiki.qemu.org/Features/VirtioVhostUser.
>
> virtio-vhost-user is inspired by Wei Wang and Zhiyong Yang's vhost-pci work.
> It differs from vhost-pci in that it has:
> 1. Vhost-user protocol message tunneling, allowing existing vhost-user
>     slave software to be reused inside the guest.
> 2. Support for all vhost device types.
> 3. Disconnected operation and reconnection support.
> 4. Asynchronous vhost-user socket implementation that avoids blocking.
>
> I have written this code to demonstrate how the virtio-vhost-user approach
> works and that it is more maintainable than vhost-pci because vhost-user slave
> software can use both AF_UNIX and virtio-vhost-user without significant code
> changes to the vhost device backends.

Yes, this looks cleaner than vhost-pci.

>
> One of the main concerns about virtio-vhost-user was that the QEMU
> virtio-vhost-user device implementation could be complex because it needs to
> parse all messages.  I hope this patch series shows that it's actually very
> simple because most messages are passed through.
>
> After this patch series has been reviewed, we need to decide whether to follow
> the original vhost-pci approach or to use this one.  Either way, both patch
> series still require improvements before they can be merged.  Here are my todos
> for this series:
>
>   * Implement "Additional Device Resources over PCI" for shared memory,
>     doorbells, and notifications instead of hardcoding a BAR with magic
>     offsets into virtio-vhost-user:
>     https://stefanha.github.io/virtio/vhost-user-slave.html#x1-2920007

Does this mean we need to standardize vhost-user protocol first?

>   * Implement the VRING_KICK eventfd - currently vhost-user slaves must be poll
>     mode drivers.
>   * Optimize VRING_CALL doorbell with ioeventfd to avoid QEMU exit.

The performance implication needs to be measured. It looks to me both 
kick and call will introduce more latency form the point of guest.

>   * vhost-user log feature
>   * UUID config field for stable device identification regardless of PCI
>     bus addresses.
>   * vhost-user IOMMU and SLAVE_REQ_FD feature

So an assumption is the VM that implements vhost backends should be at 
least as secure as vhost-user backend process on host. Could we have 
this conclusion?

>   * VhostUserMsg little-endian conversion for cross-endian support
>   * Chardev disconnect using qemu_chr_fe_set_watch() since CHR_CLOSED is
>     only emitted while a read callback is registered.  We don't keep a
>     read callback registered all the time.
>   * Drain txq on reconnection to discard stale messages.
>
> Stefan Hajnoczi (1):
>    virtio-vhost-user: add virtio-vhost-user device
>
> Wei Wang (1):
>    vhost-user: share the vhost-user protocol related structures
>
>   configure                                   |   18 +
>   hw/virtio/Makefile.objs                     |    1 +
>   hw/virtio/virtio-pci.h                      |   21 +
>   include/hw/pci/pci.h                        |    1 +
>   include/hw/virtio/vhost-user.h              |  106 +++
>   include/hw/virtio/virtio-vhost-user.h       |   88 +++
>   include/standard-headers/linux/virtio_ids.h |    1 +
>   hw/virtio/vhost-user.c                      |  100 +--
>   hw/virtio/virtio-pci.c                      |   61 ++
>   hw/virtio/virtio-vhost-user.c               | 1047 +++++++++++++++++++++++++++
>   hw/virtio/trace-events                      |   22 +
>   11 files changed, 1367 insertions(+), 99 deletions(-)
>   create mode 100644 include/hw/virtio/vhost-user.h
>   create mode 100644 include/hw/virtio/virtio-vhost-user.h
>   create mode 100644 hw/virtio/virtio-vhost-user.c
>

Btw, it's better to have some early numbers, e.g what testpmd reports 
during forwarding.

Thanks

Re: [Qemu-devel] [RFC 0/2] virtio-vhost-user: add virtio-vhost-user device

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 4688 bytes --]

On Mon, Jan 22, 2018 at 11:33:46AM +0800, Jason Wang wrote:
> On 2018年01月19日 21:06, Stefan Hajnoczi wrote:
> > These patches implement the virtio-vhost-user device design that I have
> > described here:
> > https://stefanha.github.io/virtio/vhost-user-slave.html#x1-2830007
> 
> Thanks for the patches, looks rather interesting and similar to split device
> model used by Xen.
> 
> > 
> > The goal is to let the guest act as the vhost device backend for other guests.
> > This allows virtual networking and storage appliances to run inside guests.
> 
> So the question still, what kind of protocol do you want to run on top? If
> it was ethernet based, virtio-net work pretty well and it can even do
> migration.
> 
> > This device is particularly interesting for poll mode drivers where exitless
> > VM-to-VM communication is possible, completely bypassing the hypervisor in the
> > data path.
> 
> It's better to clarify the reason of hypervisor bypassing. (performance,
> security or scalability).

Performance - yes, definitely.  Exitless VM-to-VM is the fastest
possible way to communicate between VMs.  Today it can only be done
using ivshmem.  This patch series allows virtio devices to take
advantage of it and will encourage people to use virtio instead of
non-standard ivshmem devices.

Security - I don't think this feature is a security improvement.  It
reduces isolation because VM1 has full shared memory access to VM2.  In
fact, this is a reason for users to consider carefully whether they
even want to use this feature.

Scalability - much for the same reasons as the Performance section
above.  Bypassing the hypervisor eliminates scalability bottlenecks
(e.g. host network stack and bridge).

> Probably not for the following cases:
> 
> 1) kick/call

I disagree here because kick/call is actually very efficient!

VM1's irqfd is the ioeventfd for VM2.  When VM2 writes to the ioeventfd
there is a single lightweight vmexit which injects an interrupt into
VM1.  QEMU is not involved and the host kernel scheduler is not involved
so this is a low-latency operation.

I haven't tested this yet but the ioeventfd code looks like this will
work.

> 2) device IOTLB / IOMMU transaction (or any other case that backends needs
> metadata from qemu).

Yes, this is the big weakness of vhost-user in general.  The IOMMU
feature doesn't offer good isolation and even when it does, performance
will be an issue.

> >   * Implement "Additional Device Resources over PCI" for shared memory,
> >     doorbells, and notifications instead of hardcoding a BAR with magic
> >     offsets into virtio-vhost-user:
> >     https://stefanha.github.io/virtio/vhost-user-slave.html#x1-2920007
> 
> Does this mean we need to standardize vhost-user protocol first?

Currently the draft spec says:

  This section relies on definitions from the Vhost-user Protocol [1].

  [1] https://git.qemu.org/?p=qemu.git;a=blob_plain;f=docs/interop/vhost-user.txt;hb=HEAD

Michael: Is it okay to simply include this link?

> >   * Implement the VRING_KICK eventfd - currently vhost-user slaves must be poll
> >     mode drivers.
> >   * Optimize VRING_CALL doorbell with ioeventfd to avoid QEMU exit.
> 
> The performance implication needs to be measured. It looks to me both kick
> and call will introduce more latency form the point of guest.

I described the irqfd + ioeventfd approach above.  It should be faster
than virtio-net + bridge today.

> >   * vhost-user log feature
> >   * UUID config field for stable device identification regardless of PCI
> >     bus addresses.
> >   * vhost-user IOMMU and SLAVE_REQ_FD feature
> 
> So an assumption is the VM that implements vhost backends should be at least
> as secure as vhost-user backend process on host. Could we have this
> conclusion?

Yes.

Sadly the vhost-user IOMMU protocol feature does not provide isolation.
At the moment IOMMU is basically a layer of indirection (mapping) but
the vhost-user backend process still has full access to guest RAM :(.

> Btw, it's better to have some early numbers, e.g what testpmd reports during
> forwarding.

I need to rely on others to do this (and many other things!) because
virtio-vhost-user isn't the focus of my work.

These patches were written to demonstrate my suggestions for vhost-pci.
They were written at work but also on weekends, early mornings, and late
nights to avoid delaying Wei and Zhiyong's vhost-pci work too much.

If this approach has merit then I hope others will take over and I'll
play a smaller role addressing some of the todo items and cleanups.

Stefan

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 455 bytes --]

Re: [Qemu-devel] [RFC 0/2] virtio-vhost-user: add virtio-vhost-user device

[Michael S. Tsirkin]

On Mon, Jan 22, 2018 at 12:17:51PM +0000, Stefan Hajnoczi wrote:
> On Mon, Jan 22, 2018 at 11:33:46AM +0800, Jason Wang wrote:
> > On 2018年01月19日 21:06, Stefan Hajnoczi wrote:
> > > These patches implement the virtio-vhost-user device design that I have
> > > described here:
> > > https://stefanha.github.io/virtio/vhost-user-slave.html#x1-2830007
> > 
> > Thanks for the patches, looks rather interesting and similar to split device
> > model used by Xen.
> > 
> > > 
> > > The goal is to let the guest act as the vhost device backend for other guests.
> > > This allows virtual networking and storage appliances to run inside guests.
> > 
> > So the question still, what kind of protocol do you want to run on top? If
> > it was ethernet based, virtio-net work pretty well and it can even do
> > migration.
> > 
> > > This device is particularly interesting for poll mode drivers where exitless
> > > VM-to-VM communication is possible, completely bypassing the hypervisor in the
> > > data path.
> > 
> > It's better to clarify the reason of hypervisor bypassing. (performance,
> > security or scalability).
> 
> Performance - yes, definitely.  Exitless VM-to-VM is the fastest
> possible way to communicate between VMs.  Today it can only be done
> using ivshmem.  This patch series allows virtio devices to take
> advantage of it and will encourage people to use virtio instead of
> non-standard ivshmem devices.
> 
> Security - I don't think this feature is a security improvement.  It
> reduces isolation because VM1 has full shared memory access to VM2.  In
> fact, this is a reason for users to consider carefully whether they
> even want to use this feature.

True without an IOMMU, however using a vIOMMU within VM2
can protect the VM2, can't it?

> Scalability - much for the same reasons as the Performance section
> above.  Bypassing the hypervisor eliminates scalability bottlenecks
> (e.g. host network stack and bridge).
> 
> > Probably not for the following cases:
> > 
> > 1) kick/call
> 
> I disagree here because kick/call is actually very efficient!
> 
> VM1's irqfd is the ioeventfd for VM2.  When VM2 writes to the ioeventfd
> there is a single lightweight vmexit which injects an interrupt into
> VM1.  QEMU is not involved and the host kernel scheduler is not involved
> so this is a low-latency operation.
> 
> I haven't tested this yet but the ioeventfd code looks like this will
> work.
> 
> > 2) device IOTLB / IOMMU transaction (or any other case that backends needs
> > metadata from qemu).
> 
> Yes, this is the big weakness of vhost-user in general.  The IOMMU
> feature doesn't offer good isolation

I think that's an implementation issue, not a protocol issue.


> and even when it does, performance
> will be an issue.

If the IOMMU mappings are dynamic - but they are mostly
static with e.g. dpdk, right?


> > >   * Implement "Additional Device Resources over PCI" for shared memory,
> > >     doorbells, and notifications instead of hardcoding a BAR with magic
> > >     offsets into virtio-vhost-user:
> > >     https://stefanha.github.io/virtio/vhost-user-slave.html#x1-2920007
> > 
> > Does this mean we need to standardize vhost-user protocol first?
> 
> Currently the draft spec says:
> 
>   This section relies on definitions from the Vhost-user Protocol [1].
> 
>   [1] https://git.qemu.org/?p=qemu.git;a=blob_plain;f=docs/interop/vhost-user.txt;hb=HEAD
> 
> Michael: Is it okay to simply include this link?


It is OK to include normative and non-normative references,
they go in the introduction and then you refer to them
anywhere in the document.


I'm still reviewing the draft.  At some level, this is a general tunnel
feature, it can tunnel any protocol. That would be one way to
isolate it.

> > >   * Implement the VRING_KICK eventfd - currently vhost-user slaves must be poll
> > >     mode drivers.
> > >   * Optimize VRING_CALL doorbell with ioeventfd to avoid QEMU exit.
> > 
> > The performance implication needs to be measured. It looks to me both kick
> > and call will introduce more latency form the point of guest.
> 
> I described the irqfd + ioeventfd approach above.  It should be faster
> than virtio-net + bridge today.
> 
> > >   * vhost-user log feature
> > >   * UUID config field for stable device identification regardless of PCI
> > >     bus addresses.
> > >   * vhost-user IOMMU and SLAVE_REQ_FD feature
> > 
> > So an assumption is the VM that implements vhost backends should be at least
> > as secure as vhost-user backend process on host. Could we have this
> > conclusion?
> 
> Yes.
> 
> Sadly the vhost-user IOMMU protocol feature does not provide isolation.
> At the moment IOMMU is basically a layer of indirection (mapping) but
> the vhost-user backend process still has full access to guest RAM :(.

An important feature would be to do the isolation in the qemu.
So trust the qemu running VM2 but not VM2 itself.


> > Btw, it's better to have some early numbers, e.g what testpmd reports during
> > forwarding.
> 
> I need to rely on others to do this (and many other things!) because
> virtio-vhost-user isn't the focus of my work.
> 
> These patches were written to demonstrate my suggestions for vhost-pci.
> They were written at work but also on weekends, early mornings, and late
> nights to avoid delaying Wei and Zhiyong's vhost-pci work too much.
> 
> If this approach has merit then I hope others will take over and I'll
> play a smaller role addressing some of the todo items and cleanups.
> 
> Stefan

Re: [Qemu-devel] [RFC 0/2] virtio-vhost-user: add virtio-vhost-user device

[Jason Wang]

On 2018年01月23日 04:04, Michael S. Tsirkin wrote:
> On Mon, Jan 22, 2018 at 12:17:51PM +0000, Stefan Hajnoczi wrote:
>> On Mon, Jan 22, 2018 at 11:33:46AM +0800, Jason Wang wrote:
>>> On 2018年01月19日 21:06, Stefan Hajnoczi wrote:
>>>> These patches implement the virtio-vhost-user device design that I have
>>>> described here:
>>>> https://stefanha.github.io/virtio/vhost-user-slave.html#x1-2830007
>>> Thanks for the patches, looks rather interesting and similar to split device
>>> model used by Xen.
>>>
>>>> The goal is to let the guest act as the vhost device backend for other guests.
>>>> This allows virtual networking and storage appliances to run inside guests.
>>> So the question still, what kind of protocol do you want to run on top? If
>>> it was ethernet based, virtio-net work pretty well and it can even do
>>> migration.
>>>
>>>> This device is particularly interesting for poll mode drivers where exitless
>>>> VM-to-VM communication is possible, completely bypassing the hypervisor in the
>>>> data path.
>>> It's better to clarify the reason of hypervisor bypassing. (performance,
>>> security or scalability).
>> Performance - yes, definitely.  Exitless VM-to-VM is the fastest
>> possible way to communicate between VMs.  Today it can only be done
>> using ivshmem.  This patch series allows virtio devices to take
>> advantage of it and will encourage people to use virtio instead of
>> non-standard ivshmem devices.
>>
>> Security - I don't think this feature is a security improvement.  It
>> reduces isolation because VM1 has full shared memory access to VM2.  In
>> fact, this is a reason for users to consider carefully whether they
>> even want to use this feature.
> True without an IOMMU, however using a vIOMMU within VM2
> can protect the VM2, can't it?

It's not clear to me how to do this. E.g need a way to report failure to 
VM2 or #PF?

>
>> Scalability - much for the same reasons as the Performance section
>> above.  Bypassing the hypervisor eliminates scalability bottlenecks
>> (e.g. host network stack and bridge).
>>
>>> Probably not for the following cases:
>>>
>>> 1) kick/call
>> I disagree here because kick/call is actually very efficient!
>>
>> VM1's irqfd is the ioeventfd for VM2.  When VM2 writes to the ioeventfd
>> there is a single lightweight vmexit which injects an interrupt into
>> VM1.  QEMU is not involved and the host kernel scheduler is not involved
>> so this is a low-latency operation.

Right, looks like I was wrong. But consider irqfd may do wakup which 
means scheduler is still needed.

>> I haven't tested this yet but the ioeventfd code looks like this will
>> work.
>>
>>> 2) device IOTLB / IOMMU transaction (or any other case that backends needs
>>> metadata from qemu).
>> Yes, this is the big weakness of vhost-user in general.  The IOMMU
>> feature doesn't offer good isolation
> I think that's an implementation issue, not a protocol issue.
>
>
>> and even when it does, performance
>> will be an issue.
> If the IOMMU mappings are dynamic - but they are mostly
> static with e.g. dpdk, right?
>
>
>>>>    * Implement "Additional Device Resources over PCI" for shared memory,
>>>>      doorbells, and notifications instead of hardcoding a BAR with magic
>>>>      offsets into virtio-vhost-user:
>>>>      https://stefanha.github.io/virtio/vhost-user-slave.html#x1-2920007
>>> Does this mean we need to standardize vhost-user protocol first?
>> Currently the draft spec says:
>>
>>    This section relies on definitions from the Vhost-user Protocol [1].
>>
>>    [1] https://git.qemu.org/?p=qemu.git;a=blob_plain;f=docs/interop/vhost-user.txt;hb=HEAD
>>
>> Michael: Is it okay to simply include this link?
>
> It is OK to include normative and non-normative references,
> they go in the introduction and then you refer to them
> anywhere in the document.
>
>
> I'm still reviewing the draft.  At some level, this is a general tunnel
> feature, it can tunnel any protocol. That would be one way to
> isolate it.

Right, but it should not be the main motivation, consider we can tunnel 
any protocol on top of ethernet too.

>
>>>>    * Implement the VRING_KICK eventfd - currently vhost-user slaves must be poll
>>>>      mode drivers.
>>>>    * Optimize VRING_CALL doorbell with ioeventfd to avoid QEMU exit.
>>> The performance implication needs to be measured. It looks to me both kick
>>> and call will introduce more latency form the point of guest.
>> I described the irqfd + ioeventfd approach above.  It should be faster
>> than virtio-net + bridge today.
>>
>>>>    * vhost-user log feature
>>>>    * UUID config field for stable device identification regardless of PCI
>>>>      bus addresses.
>>>>    * vhost-user IOMMU and SLAVE_REQ_FD feature
>>> So an assumption is the VM that implements vhost backends should be at least
>>> as secure as vhost-user backend process on host. Could we have this
>>> conclusion?
>> Yes.
>>
>> Sadly the vhost-user IOMMU protocol feature does not provide isolation.
>> At the moment IOMMU is basically a layer of indirection (mapping) but
>> the vhost-user backend process still has full access to guest RAM :(.
> An important feature would be to do the isolation in the qemu.
> So trust the qemu running VM2 but not VM2 itself.

Agree, we'd better not consider VM is as secure as qemu.

>
>
>>> Btw, it's better to have some early numbers, e.g what testpmd reports during
>>> forwarding.
>> I need to rely on others to do this (and many other things!) because
>> virtio-vhost-user isn't the focus of my work.
>>
>> These patches were written to demonstrate my suggestions for vhost-pci.
>> They were written at work but also on weekends, early mornings, and late
>> nights to avoid delaying Wei and Zhiyong's vhost-pci work too much.

Thanks a lot for the effort! If anyone want to benchmark, I would expect 
compare the following three solutions:

1) vhost-pci
2) virtio-vhost-user
3) testpmd with two vhost-user ports


Performance number is really important to show the advantages of new ideas.

>>
>> If this approach has merit then I hope others will take over and I'll
>> play a smaller role addressing some of the todo items and cleanups.

It looks to me the advantages are 1) generic virtio layer (vhost-pci can 
achieve this too if necessary) 2) some kind of code reusing (vhost pmd). 
And I'd expect they have similar performance result consider no major 
differences between them.

Thanks

>> Stefan
>

Re: [Qemu-devel] [RFC 0/2] virtio-vhost-user: add virtio-vhost-user device

[Michael S. Tsirkin]

On Tue, Jan 23, 2018 at 06:01:15PM +0800, Jason Wang wrote:
> 
> 
> On 2018年01月23日 04:04, Michael S. Tsirkin wrote:
> > On Mon, Jan 22, 2018 at 12:17:51PM +0000, Stefan Hajnoczi wrote:
> > > On Mon, Jan 22, 2018 at 11:33:46AM +0800, Jason Wang wrote:
> > > > On 2018年01月19日 21:06, Stefan Hajnoczi wrote:
> > > > > These patches implement the virtio-vhost-user device design that I have
> > > > > described here:
> > > > > https://stefanha.github.io/virtio/vhost-user-slave.html#x1-2830007
> > > > Thanks for the patches, looks rather interesting and similar to split device
> > > > model used by Xen.
> > > > 
> > > > > The goal is to let the guest act as the vhost device backend for other guests.
> > > > > This allows virtual networking and storage appliances to run inside guests.
> > > > So the question still, what kind of protocol do you want to run on top? If
> > > > it was ethernet based, virtio-net work pretty well and it can even do
> > > > migration.
> > > > 
> > > > > This device is particularly interesting for poll mode drivers where exitless
> > > > > VM-to-VM communication is possible, completely bypassing the hypervisor in the
> > > > > data path.
> > > > It's better to clarify the reason of hypervisor bypassing. (performance,
> > > > security or scalability).
> > > Performance - yes, definitely.  Exitless VM-to-VM is the fastest
> > > possible way to communicate between VMs.  Today it can only be done
> > > using ivshmem.  This patch series allows virtio devices to take
> > > advantage of it and will encourage people to use virtio instead of
> > > non-standard ivshmem devices.
> > > 
> > > Security - I don't think this feature is a security improvement.  It
> > > reduces isolation because VM1 has full shared memory access to VM2.  In
> > > fact, this is a reason for users to consider carefully whether they
> > > even want to use this feature.
> > True without an IOMMU, however using a vIOMMU within VM2
> > can protect the VM2, can't it?
> 
> It's not clear to me how to do this. E.g need a way to report failure to VM2
> or #PF?

Why would there be a failure? qemu running vm1 would be responsible for
preventing access to vm2's memory not mapped through an IOMMU.
Basically munmap these.

> > 
> > > Scalability - much for the same reasons as the Performance section
> > > above.  Bypassing the hypervisor eliminates scalability bottlenecks
> > > (e.g. host network stack and bridge).
> > > 
> > > > Probably not for the following cases:
> > > > 
> > > > 1) kick/call
> > > I disagree here because kick/call is actually very efficient!
> > > 
> > > VM1's irqfd is the ioeventfd for VM2.  When VM2 writes to the ioeventfd
> > > there is a single lightweight vmexit which injects an interrupt into
> > > VM1.  QEMU is not involved and the host kernel scheduler is not involved
> > > so this is a low-latency operation.
> 
> Right, looks like I was wrong. But consider irqfd may do wakup which means
> scheduler is still needed.
> 
> > > I haven't tested this yet but the ioeventfd code looks like this will
> > > work.
> > > 
> > > > 2) device IOTLB / IOMMU transaction (or any other case that backends needs
> > > > metadata from qemu).
> > > Yes, this is the big weakness of vhost-user in general.  The IOMMU
> > > feature doesn't offer good isolation
> > I think that's an implementation issue, not a protocol issue.
> > 
> > 
> > > and even when it does, performance
> > > will be an issue.
> > If the IOMMU mappings are dynamic - but they are mostly
> > static with e.g. dpdk, right?
> > 
> > 
> > > > >    * Implement "Additional Device Resources over PCI" for shared memory,
> > > > >      doorbells, and notifications instead of hardcoding a BAR with magic
> > > > >      offsets into virtio-vhost-user:
> > > > >      https://stefanha.github.io/virtio/vhost-user-slave.html#x1-2920007
> > > > Does this mean we need to standardize vhost-user protocol first?
> > > Currently the draft spec says:
> > > 
> > >    This section relies on definitions from the Vhost-user Protocol [1].
> > > 
> > >    [1] https://git.qemu.org/?p=qemu.git;a=blob_plain;f=docs/interop/vhost-user.txt;hb=HEAD
> > > 
> > > Michael: Is it okay to simply include this link?
> > 
> > It is OK to include normative and non-normative references,
> > they go in the introduction and then you refer to them
> > anywhere in the document.
> > 
> > 
> > I'm still reviewing the draft.  At some level, this is a general tunnel
> > feature, it can tunnel any protocol. That would be one way to
> > isolate it.
> 
> Right, but it should not be the main motivation, consider we can tunnel any
> protocol on top of ethernet too.
> 
> > 
> > > > >    * Implement the VRING_KICK eventfd - currently vhost-user slaves must be poll
> > > > >      mode drivers.
> > > > >    * Optimize VRING_CALL doorbell with ioeventfd to avoid QEMU exit.
> > > > The performance implication needs to be measured. It looks to me both kick
> > > > and call will introduce more latency form the point of guest.
> > > I described the irqfd + ioeventfd approach above.  It should be faster
> > > than virtio-net + bridge today.
> > > 
> > > > >    * vhost-user log feature
> > > > >    * UUID config field for stable device identification regardless of PCI
> > > > >      bus addresses.
> > > > >    * vhost-user IOMMU and SLAVE_REQ_FD feature
> > > > So an assumption is the VM that implements vhost backends should be at least
> > > > as secure as vhost-user backend process on host. Could we have this
> > > > conclusion?
> > > Yes.
> > > 
> > > Sadly the vhost-user IOMMU protocol feature does not provide isolation.
> > > At the moment IOMMU is basically a layer of indirection (mapping) but
> > > the vhost-user backend process still has full access to guest RAM :(.
> > An important feature would be to do the isolation in the qemu.
> > So trust the qemu running VM2 but not VM2 itself.
> 
> Agree, we'd better not consider VM is as secure as qemu.
> 
> > 
> > 
> > > > Btw, it's better to have some early numbers, e.g what testpmd reports during
> > > > forwarding.
> > > I need to rely on others to do this (and many other things!) because
> > > virtio-vhost-user isn't the focus of my work.
> > > 
> > > These patches were written to demonstrate my suggestions for vhost-pci.
> > > They were written at work but also on weekends, early mornings, and late
> > > nights to avoid delaying Wei and Zhiyong's vhost-pci work too much.
> 
> Thanks a lot for the effort! If anyone want to benchmark, I would expect
> compare the following three solutions:
> 
> 1) vhost-pci
> 2) virtio-vhost-user
> 3) testpmd with two vhost-user ports
> 
> 
> Performance number is really important to show the advantages of new ideas.
> 
> > > 
> > > If this approach has merit then I hope others will take over and I'll
> > > play a smaller role addressing some of the todo items and cleanups.
> 
> It looks to me the advantages are 1) generic virtio layer (vhost-pci can
> achieve this too if necessary) 2) some kind of code reusing (vhost pmd). And
> I'd expect they have similar performance result consider no major
> differences between them.
> 
> Thanks
> 
> > > Stefan
> > 

Re: [Qemu-devel] [RFC 0/2] virtio-vhost-user: add virtio-vhost-user device

[Paolo Bonzini]

On 23/01/2018 17:07, Michael S. Tsirkin wrote:
>> It's not clear to me how to do this. E.g need a way to report failure to VM2
>> or #PF?
> 
> Why would there be a failure? qemu running vm1 would be responsible for
> preventing access to vm2's memory not mapped through an IOMMU.
> Basically munmap these.

Access to VM2's memory would use VM2's configured IOVAs for VM1's
requester id.  VM2's QEMU send device IOTLB messages to VM1's QEMU,
which would remap VM2's memory on the fly into VM1's BAR2.

It's not trivial to do it efficiently, but it's possible.  The important
thing is that, if VM2 has an IOMMU, QEMU must *not* connect to a
virtio-vhost-user device that lacks IOTLB support.  But that would be a
vhost-user bug, not a virtio-vhost-user bug---and that's the beauty of
Stefan's approach. :)

Paolo

Re: [Qemu-devel] [RFC 0/2] virtio-vhost-user: add virtio-vhost-user device

[Michael S. Tsirkin]

On Thu, Jan 25, 2018 at 03:07:23PM +0100, Paolo Bonzini wrote:
> On 23/01/2018 17:07, Michael S. Tsirkin wrote:
> >> It's not clear to me how to do this. E.g need a way to report failure to VM2
> >> or #PF?
> > 
> > Why would there be a failure? qemu running vm1 would be responsible for
> > preventing access to vm2's memory not mapped through an IOMMU.
> > Basically munmap these.
> 
> Access to VM2's memory would use VM2's configured IOVAs for VM1's
> requester id.  VM2's QEMU send device IOTLB messages to VM1's QEMU,
> which would remap VM2's memory on the fly into VM1's BAR2.

Right. Almost.

One problem is that IOVA range is bigger than RAM range,
so you will have trouble making arbitrary virtual addresses
fit in a BAR.

This is why I suggested a hybrid approach where
translation happens within guest, qemu only does protection.

Another problem with it is that IOMMU has page granularity
while with hugetlbfs we might not be able to remap at that
granularity.

Not sure what to do about it - teach host to break
up pages? Pass limitation to guest through virtio-iommu?

Ideas?

> It's not trivial to do it efficiently, but it's possible.  The important
> thing is that, if VM2 has an IOMMU, QEMU must *not* connect to a
> virtio-vhost-user device that lacks IOTLB support.  But that would be a
> vhost-user bug, not a virtio-vhost-user bug---and that's the beauty of
> Stefan's approach. :)
> 
> Paolo

Re: [Qemu-devel] [RFC 0/2] virtio-vhost-user: add virtio-vhost-user device

[Jason Wang]

On 2018年01月25日 22:48, Michael S. Tsirkin wrote:
> On Thu, Jan 25, 2018 at 03:07:23PM +0100, Paolo Bonzini wrote:
>> On 23/01/2018 17:07, Michael S. Tsirkin wrote:
>>>> It's not clear to me how to do this. E.g need a way to report failure to VM2
>>>> or #PF?
>>> Why would there be a failure? qemu running vm1 would be responsible for
>>> preventing access to vm2's memory not mapped through an IOMMU.
>>> Basically munmap these.
>> Access to VM2's memory would use VM2's configured IOVAs for VM1's
>> requester id.  VM2's QEMU send device IOTLB messages to VM1's QEMU,
>> which would remap VM2's memory on the fly into VM1's BAR2.
> Right. Almost.

That would be extremely slow for dynamic mappings.

>
> One problem is that IOVA range is bigger than RAM range,
> so you will have trouble making arbitrary virtual addresses
> fit in a BAR.
>
> This is why I suggested a hybrid approach where
> translation happens within guest, qemu only does protection.
>
> Another problem with it is that IOMMU has page granularity
> while with hugetlbfs we might not be able to remap at that
> granularity.
>
> Not sure what to do about it - teach host to break
> up pages? Pass limitation to guest through virtio-iommu?

If we decide to go virtio IOMMU, maybe device can limit its IOVA range too.

Thanks

>
> Ideas?
>
>> It's not trivial to do it efficiently, but it's possible.  The important
>> thing is that, if VM2 has an IOMMU, QEMU must *not* connect to a
>> virtio-vhost-user device that lacks IOTLB support.  But that would be a
>> vhost-user bug, not a virtio-vhost-user bug---and that's the beauty of
>> Stefan's approach. :)
>>
>> Paolo

Re: [Qemu-devel] [RFC 0/2] virtio-vhost-user: add virtio-vhost-user device

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 4627 bytes --]

On Mon, Jan 22, 2018 at 10:04:18PM +0200, Michael S. Tsirkin wrote:
> On Mon, Jan 22, 2018 at 12:17:51PM +0000, Stefan Hajnoczi wrote:
> > On Mon, Jan 22, 2018 at 11:33:46AM +0800, Jason Wang wrote:
> > > On 2018年01月19日 21:06, Stefan Hajnoczi wrote:
> > > > These patches implement the virtio-vhost-user device design that I have
> > > > described here:
> > > > https://stefanha.github.io/virtio/vhost-user-slave.html#x1-2830007
> > > 
> > > Thanks for the patches, looks rather interesting and similar to split device
> > > model used by Xen.
> > > 
> > > > 
> > > > The goal is to let the guest act as the vhost device backend for other guests.
> > > > This allows virtual networking and storage appliances to run inside guests.
> > > 
> > > So the question still, what kind of protocol do you want to run on top? If
> > > it was ethernet based, virtio-net work pretty well and it can even do
> > > migration.
> > > 
> > > > This device is particularly interesting for poll mode drivers where exitless
> > > > VM-to-VM communication is possible, completely bypassing the hypervisor in the
> > > > data path.
> > > 
> > > It's better to clarify the reason of hypervisor bypassing. (performance,
> > > security or scalability).
> > 
> > Performance - yes, definitely.  Exitless VM-to-VM is the fastest
> > possible way to communicate between VMs.  Today it can only be done
> > using ivshmem.  This patch series allows virtio devices to take
> > advantage of it and will encourage people to use virtio instead of
> > non-standard ivshmem devices.
> > 
> > Security - I don't think this feature is a security improvement.  It
> > reduces isolation because VM1 has full shared memory access to VM2.  In
> > fact, this is a reason for users to consider carefully whether they
> > even want to use this feature.
> 
> True without an IOMMU, however using a vIOMMU within VM2
> can protect the VM2, can't it?
[...]
> > Yes, this is the big weakness of vhost-user in general.  The IOMMU
> > feature doesn't offer good isolation
> 
> I think that's an implementation issue, not a protocol issue.

The IOMMU feature in the vhost-user protocol adds address translation
and permissions but the underlying memory region file descriptors still
expose all of guest RAM.

An implementation that offers true isolation has to enforce the IOMMU
state on the shared memory (e.g. read & write access).

> > and even when it does, performance
> > will be an issue.
> 
> If the IOMMU mappings are dynamic - but they are mostly
> static with e.g. dpdk, right?

Excellent idea!  My understanding is that DPDK's memory area for mbufs
is static.

The question is whether this area contains only packet payload buffers?
Then it would be feasible to map only this memory (plus vrings) and
achieve isolation.

If other data is intermingled with the packet payload buffers then it's
difficult to guarantee security since an evil VM can modify the
non-payload data to trigger race conditions, out-of-bounds memory
accesses, and other bugs.

Anyway, what you've suggested sounds like the most realistic option for
achieving isolation without losing performance.

> > > >   * Implement the VRING_KICK eventfd - currently vhost-user slaves must be poll
> > > >     mode drivers.
> > > >   * Optimize VRING_CALL doorbell with ioeventfd to avoid QEMU exit.
> > > 
> > > The performance implication needs to be measured. It looks to me both kick
> > > and call will introduce more latency form the point of guest.
> > 
> > I described the irqfd + ioeventfd approach above.  It should be faster
> > than virtio-net + bridge today.
> > 
> > > >   * vhost-user log feature
> > > >   * UUID config field for stable device identification regardless of PCI
> > > >     bus addresses.
> > > >   * vhost-user IOMMU and SLAVE_REQ_FD feature
> > > 
> > > So an assumption is the VM that implements vhost backends should be at least
> > > as secure as vhost-user backend process on host. Could we have this
> > > conclusion?
> > 
> > Yes.
> > 
> > Sadly the vhost-user IOMMU protocol feature does not provide isolation.
> > At the moment IOMMU is basically a layer of indirection (mapping) but
> > the vhost-user backend process still has full access to guest RAM :(.
> 
> An important feature would be to do the isolation in the qemu.
> So trust the qemu running VM2 but not VM2 itself.

I think what you suggested above about DPDK's static mappings is even
better.  It does not require trusting VM2's QEMU.  It's also simpler
than implementing isolation in QEMU.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 455 bytes --]

Re: [Qemu-devel] [RFC 0/2] virtio-vhost-user: add virtio-vhost-user device

[Wei Wang]

On 01/22/2018 08:17 PM, Stefan Hajnoczi wrote:
> On Mon, Jan 22, 2018 at 11:33:46AM +0800, Jason Wang wrote:
>> On 2018年01月19日 21:06, Stefan Hajnoczi wrote:
>>
>> Probably not for the following cases:
>>
>> 1) kick/call
> I disagree here because kick/call is actually very efficient!
>
> VM1's irqfd is the ioeventfd for VM2.  When VM2 writes to the ioeventfd
> there is a single lightweight vmexit which injects an interrupt into
> VM1.  QEMU is not involved and the host kernel scheduler is not involved
> so this is a low-latency operation.
>
> I haven't tested this yet but the ioeventfd code looks like this will
> work.


This have been tested in vhost-pci v2 patches which worked with with a 
kernel driver. It worked pretty well.

>> Btw, it's better to have some early numbers, e.g what testpmd reports during
>> forwarding.
> I need to rely on others to do this (and many other things!) because
> virtio-vhost-user isn't the focus of my work.
>
> These patches were written to demonstrate my suggestions for vhost-pci.
> They were written at work but also on weekends, early mornings, and late
> nights to avoid delaying Wei and Zhiyong's vhost-pci work too much.
>
> If this approach has merit then I hope others will take over and I'll
> play a smaller role addressing some of the todo items and cleanups.

Thanks again for the great effort, your implementation looks nice.

If we finally decide to go with the virtio-vhost-user approach, I think 
zhiyong and I can help take over the work to continue, too.

I'm still thinking about solutions to the two issues that I shared 
yesterday - it should be like a normal PCI device, and if we unbind its 
driver, and bind back, it should also work.


Best,
Wei

Re: [Qemu-devel] [RFC 0/2] virtio-vhost-user: add virtio-vhost-user device

[Wei Wang]

On 01/19/2018 09:06 PM, Stefan Hajnoczi wrote:
> These patches implement the virtio-vhost-user device design that I have
> described here:
> https://stefanha.github.io/virtio/vhost-user-slave.html#x1-2830007
>
>
>   configure                                   |   18 +
>   hw/virtio/Makefile.objs                     |    1 +
>   hw/virtio/virtio-pci.h                      |   21 +
>   include/hw/pci/pci.h                        |    1 +
>   include/hw/virtio/vhost-user.h              |  106 +++
>   include/hw/virtio/virtio-vhost-user.h       |   88 +++
>   include/standard-headers/linux/virtio_ids.h |    1 +
>   hw/virtio/vhost-user.c                      |  100 +--
>   hw/virtio/virtio-pci.c                      |   61 ++
>   hw/virtio/virtio-vhost-user.c               | 1047 +++++++++++++++++++++++++++
>   hw/virtio/trace-events                      |   22 +
>   11 files changed, 1367 insertions(+), 99 deletions(-)
>   create mode 100644 include/hw/virtio/vhost-user.h
>   create mode 100644 include/hw/virtio/virtio-vhost-user.h
>   create mode 100644 hw/virtio/virtio-vhost-user.c
>

Thanks for the quick implementation. Not sure if the following issues 
could be solved with this approach:
  - After we boot the slave VM, if we don't run the virtio-vhost-user 
driver (i.e. testpmd), then the master VM couldn't boot, because the 
booting of the virtio-net device relies on a negotiation with the 
virtio-vhost-user driver.
  - Suppose in the future there is also a kernel virtio-vhost-user 
driver as other PCI devices, can we unbind the kernel driver first, and 
then bind the device to the dpdk driver? A normal PCI device should be 
able to smoothly switch between the kernel driver and dpdk driver.

Best,
Wei

Re: [Qemu-devel] [RFC 0/2] virtio-vhost-user: add virtio-vhost-user device

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 2694 bytes --]

On Mon, Jan 22, 2018 at 07:09:06PM +0800, Wei Wang wrote:
> On 01/19/2018 09:06 PM, Stefan Hajnoczi wrote:
> Thanks for the quick implementation. Not sure if the following issues could
> be solved with this approach:
>  - After we boot the slave VM, if we don't run the virtio-vhost-user driver
> (i.e. testpmd), then the master VM couldn't boot, because the booting of the
> virtio-net device relies on a negotiation with the virtio-vhost-user driver.

This is a limitation of QEMU's vhost-user master implementation which
also affects AF_UNIX.  It cannot be solved by this patch series since
this is the slave side.

Here is what I suggest.  Introduce a new VIRTIO feature bit:

  #define VIRTIO_F_DEVICE_READY 34

When the device supports this bit we extend the device initialization
procedure.  If the device is not yet ready, the FEATURES_OK status bit
is not accepted by the device.  Device initialization fails temporarily
but the device may raise the configuration change interrupt to indicate
that device initialization should be retried.

Using a feature bit guarantees that existing device and driver behavior
remains unchanged.

On the QEMU side the changes are:

1. Virtio hardware registers and configuration space are available even
   when vhost-user is disconnected.  This shouldn't be difficult to
   implement because QEMU always has a virtio device model for each
   vhost-user device.  We just need to put dummy values in the
   registers/configuration space until vhost-user has connected.

2. When vhost-user connects, raise the configuration change interrupt
   and allow vhost-user to process.

On the guest side the changes are:

1. virtio_pci.ko must set VIRTIO_F_DEVICE_READY and handle !FEATURES_OK
   by waiting for the configuration change interrupt and retrying.

This doesn't fully solve the problem because it assumes that a connected
slave always responds to VIRTIO hardware register accesses (e.g. get/set
features).  If the slave crashes but leaves the virtio-vhost-user PCI
adapter connected then vhost-user requests from the master go
unanswered and cause hangs...

>  - Suppose in the future there is also a kernel virtio-vhost-user driver as
> other PCI devices, can we unbind the kernel driver first, and then bind the
> device to the dpdk driver? A normal PCI device should be able to smoothly
> switch between the kernel driver and dpdk driver.

It depends what you mean by "smoothly switch".

If you mean whether it's possible to go from a kernel driver to
vfio-pci, then the answer is yes.

But if the kernel driver has an established vhost-user connection then
it will be closed.  This is the same as reconnecting with AF_UNIX
vhost-user.

Stefan

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 455 bytes --]

Re: [Qemu-devel] [RFC 0/2] virtio-vhost-user: add virtio-vhost-user device

[Wei Wang]

On 01/23/2018 07:12 PM, Stefan Hajnoczi wrote:
> On Mon, Jan 22, 2018 at 07:09:06PM +0800, Wei Wang wrote:
>> On 01/19/2018 09:06 PM, Stefan Hajnoczi wrote:
>>
>>
>>   - Suppose in the future there is also a kernel virtio-vhost-user driver as
>> other PCI devices, can we unbind the kernel driver first, and then bind the
>> device to the dpdk driver? A normal PCI device should be able to smoothly
>> switch between the kernel driver and dpdk driver.
> It depends what you mean by "smoothly switch".
>
> If you mean whether it's possible to go from a kernel driver to
> vfio-pci, then the answer is yes.
>
> But if the kernel driver has an established vhost-user connection then
> it will be closed.  This is the same as reconnecting with AF_UNIX
> vhost-user.
>

Actually not only the case of switching to testpmd after kernel 
establishes the connection, but also for several runs of testpmd. That 
is, if we run testpmd, then exit testpmd. I think the second run of 
testpmd won't work. I'm thinking about caching the received master msgs 
in QEMU when virtio_vhost_user_parse_m2s().

Btw, I'm trying to run the code, but couldn't bind the virito-vhost-user 
device to vfio-pci (reports Unknown device), not sure if it is because 
the device type is "Unclassified device".

Best,
Wei

Re: [Qemu-devel] [RFC 0/2] virtio-vhost-user: add virtio-vhost-user device

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 1880 bytes --]

On Tue, Jan 23, 2018 at 09:06:49PM +0800, Wei Wang wrote:
> On 01/23/2018 07:12 PM, Stefan Hajnoczi wrote:
> > On Mon, Jan 22, 2018 at 07:09:06PM +0800, Wei Wang wrote:
> > > On 01/19/2018 09:06 PM, Stefan Hajnoczi wrote:
> > > 
> > > 
> > >   - Suppose in the future there is also a kernel virtio-vhost-user driver as
> > > other PCI devices, can we unbind the kernel driver first, and then bind the
> > > device to the dpdk driver? A normal PCI device should be able to smoothly
> > > switch between the kernel driver and dpdk driver.
> > It depends what you mean by "smoothly switch".
> > 
> > If you mean whether it's possible to go from a kernel driver to
> > vfio-pci, then the answer is yes.
> > 
> > But if the kernel driver has an established vhost-user connection then
> > it will be closed.  This is the same as reconnecting with AF_UNIX
> > vhost-user.
> > 
> 
> Actually not only the case of switching to testpmd after kernel establishes
> the connection, but also for several runs of testpmd. That is, if we run
> testpmd, then exit testpmd. I think the second run of testpmd won't work.

The vhost-user master must reconnect and initialize again (SET_FEATURES,
SET_MEM_TABLE, etc).  Is your master reconnecting after the AF_UNIX
connection is closed?

> I'm thinking about caching the received master msgs in QEMU when
> virtio_vhost_user_parse_m2s().

Why is that necessary and how does QEMU know they are still up-to-date
when a new connection is made?

> Btw, I'm trying to run the code, but couldn't bind the virito-vhost-user
> device to vfio-pci (reports Unknown device), not sure if it is because the
> device type is "Unclassified device".

You need to use the modified usertools/dpdk-devbind.py from my patch
series inside the guest.  Please see:
https://dpdk.org/ml/archives/dev/2018-January/088177.html

Stefan

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 455 bytes --]

Re: [Qemu-devel] [RFC 0/2] virtio-vhost-user: add virtio-vhost-user device

[Wei Wang]

On 01/24/2018 07:40 PM, Stefan Hajnoczi wrote:
> On Tue, Jan 23, 2018 at 09:06:49PM +0800, Wei Wang wrote:
>> On 01/23/2018 07:12 PM, Stefan Hajnoczi wrote:
>>> On Mon, Jan 22, 2018 at 07:09:06PM +0800, Wei Wang wrote:
>>>> On 01/19/2018 09:06 PM, Stefan Hajnoczi wrote:
>>>>
>>>>
>>>>    - Suppose in the future there is also a kernel virtio-vhost-user driver as
>>>> other PCI devices, can we unbind the kernel driver first, and then bind the
>>>> device to the dpdk driver? A normal PCI device should be able to smoothly
>>>> switch between the kernel driver and dpdk driver.
>>> It depends what you mean by "smoothly switch".
>>>
>>> If you mean whether it's possible to go from a kernel driver to
>>> vfio-pci, then the answer is yes.
>>>
>>> But if the kernel driver has an established vhost-user connection then
>>> it will be closed.  This is the same as reconnecting with AF_UNIX
>>> vhost-user.
>>>
>> Actually not only the case of switching to testpmd after kernel establishes
>> the connection, but also for several runs of testpmd. That is, if we run
>> testpmd, then exit testpmd. I think the second run of testpmd won't work.
> The vhost-user master must reconnect and initialize again (SET_FEATURES,
> SET_MEM_TABLE, etc).  Is your master reconnecting after the AF_UNIX
> connection is closed?

Is this an explicit qmp operation to make the master re-connect?

>
>> I'm thinking about caching the received master msgs in QEMU when
>> virtio_vhost_user_parse_m2s().
> Why is that necessary and how does QEMU know they are still up-to-date
> when a new connection is made?

OK. I think that's not needed if the master can reconnect.

>
>> Btw, I'm trying to run the code, but couldn't bind the virito-vhost-user
>> device to vfio-pci (reports Unknown device), not sure if it is because the
>> device type is "Unclassified device".
> You need to use the modified usertools/dpdk-devbind.py from my patch
> series inside the guest.  Please see:
> https://dpdk.org/ml/archives/dev/2018-January/088177.html

Thanks. I haven't got time on this yet due to some more urgent things. 
I'll try.

Best,
Wei

Re: [Qemu-devel] [RFC 0/2] virtio-vhost-user: add virtio-vhost-user device

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 2234 bytes --]

On Thu, Jan 25, 2018 at 06:19:13PM +0800, Wei Wang wrote:
> On 01/24/2018 07:40 PM, Stefan Hajnoczi wrote:
> > On Tue, Jan 23, 2018 at 09:06:49PM +0800, Wei Wang wrote:
> > > On 01/23/2018 07:12 PM, Stefan Hajnoczi wrote:
> > > > On Mon, Jan 22, 2018 at 07:09:06PM +0800, Wei Wang wrote:
> > > > > On 01/19/2018 09:06 PM, Stefan Hajnoczi wrote:
> > > > > 
> > > > > 
> > > > >    - Suppose in the future there is also a kernel virtio-vhost-user driver as
> > > > > other PCI devices, can we unbind the kernel driver first, and then bind the
> > > > > device to the dpdk driver? A normal PCI device should be able to smoothly
> > > > > switch between the kernel driver and dpdk driver.
> > > > It depends what you mean by "smoothly switch".
> > > > 
> > > > If you mean whether it's possible to go from a kernel driver to
> > > > vfio-pci, then the answer is yes.
> > > > 
> > > > But if the kernel driver has an established vhost-user connection then
> > > > it will be closed.  This is the same as reconnecting with AF_UNIX
> > > > vhost-user.
> > > > 
> > > Actually not only the case of switching to testpmd after kernel establishes
> > > the connection, but also for several runs of testpmd. That is, if we run
> > > testpmd, then exit testpmd. I think the second run of testpmd won't work.
> > The vhost-user master must reconnect and initialize again (SET_FEATURES,
> > SET_MEM_TABLE, etc).  Is your master reconnecting after the AF_UNIX
> > connection is closed?
> 
> Is this an explicit qmp operation to make the master re-connect?

I haven't tested it myself but I'm aware of two modes of operation:

1. -chardev socket,id=chardev0,...,server
   -netdev vhost-user,chardev=chardev0

   When the vhost-user socket is disconnected the peer needs to
   reconnect.  In this case no special commands are necessary.

   Here we're relying on DPDK librte_vhost's reconnection behavior.

Or

2. -chardev socket,id=chardev0,...,reconnect=3
   -netdev vhost-user,chardev=chardev0

   When the vhost-user socket is disconnected a new connection attempt
   will be made after 3 seconds.

In both cases vhost-user negotiation will resume when the new connection
is established.

Stefan

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 455 bytes --]

Re: [Qemu-devel] [RFC 0/2] virtio-vhost-user: add virtio-vhost-user device

[Wei Wang]

On 01/26/2018 10:44 PM, Stefan Hajnoczi wrote:
> On Thu, Jan 25, 2018 at 06:19:13PM +0800, Wei Wang wrote:
>> On 01/24/2018 07:40 PM, Stefan Hajnoczi wrote:
>>> On Tue, Jan 23, 2018 at 09:06:49PM +0800, Wei Wang wrote:
>>>> On 01/23/2018 07:12 PM, Stefan Hajnoczi wrote:
>>>>> On Mon, Jan 22, 2018 at 07:09:06PM +0800, Wei Wang wrote:
>>>>>> On 01/19/2018 09:06 PM, Stefan Hajnoczi wrote:
>>>>>>
>>>>>>
>>>>>>     - Suppose in the future there is also a kernel virtio-vhost-user driver as
>>>>>> other PCI devices, can we unbind the kernel driver first, and then bind the
>>>>>> device to the dpdk driver? A normal PCI device should be able to smoothly
>>>>>> switch between the kernel driver and dpdk driver.
>>>>> It depends what you mean by "smoothly switch".
>>>>>
>>>>> If you mean whether it's possible to go from a kernel driver to
>>>>> vfio-pci, then the answer is yes.
>>>>>
>>>>> But if the kernel driver has an established vhost-user connection then
>>>>> it will be closed.  This is the same as reconnecting with AF_UNIX
>>>>> vhost-user.
>>>>>
>>>> Actually not only the case of switching to testpmd after kernel establishes
>>>> the connection, but also for several runs of testpmd. That is, if we run
>>>> testpmd, then exit testpmd. I think the second run of testpmd won't work.
>>> The vhost-user master must reconnect and initialize again (SET_FEATURES,
>>> SET_MEM_TABLE, etc).  Is your master reconnecting after the AF_UNIX
>>> connection is closed?
>> Is this an explicit qmp operation to make the master re-connect?
> I haven't tested it myself but I'm aware of two modes of operation:
>
> 1. -chardev socket,id=chardev0,...,server
>     -netdev vhost-user,chardev=chardev0
>
>     When the vhost-user socket is disconnected the peer needs to
>     reconnect.  In this case no special commands are necessary.
>
>     Here we're relying on DPDK librte_vhost's reconnection behavior.
>
> Or
>
> 2. -chardev socket,id=chardev0,...,reconnect=3
>     -netdev vhost-user,chardev=chardev0
>
>     When the vhost-user socket is disconnected a new connection attempt
>     will be made after 3 seconds.
>
> In both cases vhost-user negotiation will resume when the new connection
> is established.
>
> Stefan

I've been thinking about the issues, and it looks vhost-pci outperforms 
in this aspect.
Vhost-pci is like using a mail box. messages are just dropped into the 
box, and whenever vhost-pci pmd gets booted, it can always get the 
messages from the box, the negotiation between vhost-pci pmd and 
virtio-net is asynchronous.
Virtio-vhost-user is like a phone call, which is a synchronous 
communication. If one side is absent, then the other side will hang on 
without knowing when it could get connected or hang up with messages not 
passed (lost).

I also think the above solutions won't help. Please see below:

Background:
The vhost-user negotiation is split into 2 phases currently. The 1st 
phase happens when the connection is established, and we can find what's 
done in the 1st phase in vhost_user_init(). The 2nd phase happens when 
the master driver is loaded (e.g. run of virtio-net pmd) and set status 
to the device, and we can find what's done in the 2nd phase in 
vhost_dev_start(), which includes sending the memory info and virtqueue 
info. The socket is connected, till one of the QEMU devices exits, so 
pmd exiting won't end the QEMU side socket connection.

Issues:
Suppose we have both the vhost and virtio-net set up, and vhost pmd <-> 
virtio-net pmd communication works well. Now, vhost pmd exits 
(virtio-net pmd is still there). Some time later, we re-run vhost pmd, 
the vhost pmd doesn't know the virtqueue addresses of the virtio-net 
pmd, unless the virtio-net pmd reloads to start the 2nd phase of the 
vhost-user protocol. So the second run of the vhost pmd won't work.

Any thoughts?

Best,
Wei

Re: [Qemu-devel] [RFC 0/2] virtio-vhost-user: add virtio-vhost-user device

[Michael S. Tsirkin]

On Tue, Jan 30, 2018 at 08:09:19PM +0800, Wei Wang wrote:
> Issues:
> Suppose we have both the vhost and virtio-net set up, and vhost pmd <->
> virtio-net pmd communication works well. Now, vhost pmd exits (virtio-net
> pmd is still there). Some time later, we re-run vhost pmd, the vhost pmd
> doesn't know the virtqueue addresses of the virtio-net pmd, unless the
> virtio-net pmd reloads to start the 2nd phase of the vhost-user protocol. So
> the second run of the vhost pmd won't work.
> 
> Any thoughts?
> 
> Best,
> Wei

So vhost in qemu must resend all configuration on reconnect.
Does this address the issues?

-- 
MST

Re: [Qemu-devel] [RFC 0/2] virtio-vhost-user: add virtio-vhost-user device

[Wei Wang]

On 02/02/2018 01:08 AM, Michael S. Tsirkin wrote:
> On Tue, Jan 30, 2018 at 08:09:19PM +0800, Wei Wang wrote:
>> Issues:
>> Suppose we have both the vhost and virtio-net set up, and vhost pmd <->
>> virtio-net pmd communication works well. Now, vhost pmd exits (virtio-net
>> pmd is still there). Some time later, we re-run vhost pmd, the vhost pmd
>> doesn't know the virtqueue addresses of the virtio-net pmd, unless the
>> virtio-net pmd reloads to start the 2nd phase of the vhost-user protocol. So
>> the second run of the vhost pmd won't work.
>>
>> Any thoughts?
>>
>> Best,
>> Wei
> So vhost in qemu must resend all configuration on reconnect.
> Does this address the issues?
>

Yes, but the issues are
1) there is no reconnecting when a pmd exits (the socket connection 
seems still on at the device layer);
2) If we find a way to break the QEMU layer socket connection when pmd 
exits and get it reconnect, virtio-net device still won't send all the 
configure when reconnecting, because socket connecting only triggers 
phase 1 of vhost-user negotiation (i.e. vhost_user_init). Phase 2 is 
triggered after the driver loads (i.e. vhost_net_start). If the 
virtio-net pmd doesn't reload, there are no phase 2 messages (like 
virtqueue addresses which are allocated by the pmd). I think we need to 
think more about this before moving forward.

Best,
Wei

Re: [Qemu-devel] [RFC 0/2] virtio-vhost-user: add virtio-vhost-user device

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 2024 bytes --]

On Fri, Feb 02, 2018 at 09:08:44PM +0800, Wei Wang wrote:
> On 02/02/2018 01:08 AM, Michael S. Tsirkin wrote:
> > On Tue, Jan 30, 2018 at 08:09:19PM +0800, Wei Wang wrote:
> > > Issues:
> > > Suppose we have both the vhost and virtio-net set up, and vhost pmd <->
> > > virtio-net pmd communication works well. Now, vhost pmd exits (virtio-net
> > > pmd is still there). Some time later, we re-run vhost pmd, the vhost pmd
> > > doesn't know the virtqueue addresses of the virtio-net pmd, unless the
> > > virtio-net pmd reloads to start the 2nd phase of the vhost-user protocol. So
> > > the second run of the vhost pmd won't work.
> > > 
> > > Any thoughts?
> > > 
> > > Best,
> > > Wei
> > So vhost in qemu must resend all configuration on reconnect.
> > Does this address the issues?
> > 
> 
> Yes, but the issues are
> 1) there is no reconnecting when a pmd exits (the socket connection seems
> still on at the device layer);

This is how real hardware works too.  If the driver suddenly stops
running then the device remains operational.  When the driver is started
again it resets the device and initializes it.

> 2) If we find a way to break the QEMU layer socket connection when pmd exits
> and get it reconnect, virtio-net device still won't send all the configure
> when reconnecting, because socket connecting only triggers phase 1 of
> vhost-user negotiation (i.e. vhost_user_init). Phase 2 is triggered after
> the driver loads (i.e. vhost_net_start). If the virtio-net pmd doesn't
> reload, there are no phase 2 messages (like virtqueue addresses which are
> allocated by the pmd). I think we need to think more about this before
> moving forward.

Marc-André: How does vhost-user reconnect work when the master goes away
and a new master comes online?  Wei found that the QEMU slave
implementation only does partial vhost-user initialization upon
reconnect, so the new master doesn't get the virtqueue address and
related information.  Is this a QEMU bug?

Stefan

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 455 bytes --]

Re: [Qemu-devel] [RFC 0/2] virtio-vhost-user: add virtio-vhost-user device

[Wang, Wei W]

On Tuesday, February 6, 2018 12:26 AM, Stefan Hajnoczi wrote:
> On Fri, Feb 02, 2018 at 09:08:44PM +0800, Wei Wang wrote:
> > On 02/02/2018 01:08 AM, Michael S. Tsirkin wrote:
> > > On Tue, Jan 30, 2018 at 08:09:19PM +0800, Wei Wang wrote:
> > > > Issues:
> > > > Suppose we have both the vhost and virtio-net set up, and vhost
> > > > pmd <-> virtio-net pmd communication works well. Now, vhost pmd
> > > > exits (virtio-net pmd is still there). Some time later, we re-run
> > > > vhost pmd, the vhost pmd doesn't know the virtqueue addresses of
> > > > the virtio-net pmd, unless the virtio-net pmd reloads to start the
> > > > 2nd phase of the vhost-user protocol. So the second run of the vhost
> pmd won't work.
> > > >
> > > > Any thoughts?
> > > >
> > > > Best,
> > > > Wei
> > > So vhost in qemu must resend all configuration on reconnect.
> > > Does this address the issues?
> > >
> >
> > Yes, but the issues are
> > 1) there is no reconnecting when a pmd exits (the socket connection
> > seems still on at the device layer);
> 
> This is how real hardware works too.  If the driver suddenly stops running
> then the device remains operational.  When the driver is started again it
> resets the device and initializes it.
> 
> > 2) If we find a way to break the QEMU layer socket connection when pmd
> > exits and get it reconnect, virtio-net device still won't send all the
> > configure when reconnecting, because socket connecting only triggers
> > phase 1 of vhost-user negotiation (i.e. vhost_user_init). Phase 2 is
> > triggered after the driver loads (i.e. vhost_net_start). If the
> > virtio-net pmd doesn't reload, there are no phase 2 messages (like
> > virtqueue addresses which are allocated by the pmd). I think we need
> > to think more about this before moving forward.
> 
> Marc-André: How does vhost-user reconnect work when the master goes
> away and a new master comes online?  Wei found that the QEMU slave
> implementation only does partial vhost-user initialization upon reconnect, so
> the new master doesn't get the virtqueue address and related information.
> Is this a QEMU bug?

Actually we are discussing the slave (vhost is the slave, right?) going away. When a slave exits and some moment later a new slave runs, the master (virtio-net) won't send the virtqueue addresses to the new vhost slave.

Best,
Wei

Re: [Qemu-devel] [RFC 0/2] virtio-vhost-user: add virtio-vhost-user device

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 2926 bytes --]

On Tue, Feb 06, 2018 at 01:28:25AM +0000, Wang, Wei W wrote:
> On Tuesday, February 6, 2018 12:26 AM, Stefan Hajnoczi wrote:
> > On Fri, Feb 02, 2018 at 09:08:44PM +0800, Wei Wang wrote:
> > > On 02/02/2018 01:08 AM, Michael S. Tsirkin wrote:
> > > > On Tue, Jan 30, 2018 at 08:09:19PM +0800, Wei Wang wrote:
> > > > > Issues:
> > > > > Suppose we have both the vhost and virtio-net set up, and vhost
> > > > > pmd <-> virtio-net pmd communication works well. Now, vhost pmd
> > > > > exits (virtio-net pmd is still there). Some time later, we re-run
> > > > > vhost pmd, the vhost pmd doesn't know the virtqueue addresses of
> > > > > the virtio-net pmd, unless the virtio-net pmd reloads to start the
> > > > > 2nd phase of the vhost-user protocol. So the second run of the vhost
> > pmd won't work.
> > > > >
> > > > > Any thoughts?
> > > > >
> > > > > Best,
> > > > > Wei
> > > > So vhost in qemu must resend all configuration on reconnect.
> > > > Does this address the issues?
> > > >
> > >
> > > Yes, but the issues are
> > > 1) there is no reconnecting when a pmd exits (the socket connection
> > > seems still on at the device layer);
> > 
> > This is how real hardware works too.  If the driver suddenly stops running
> > then the device remains operational.  When the driver is started again it
> > resets the device and initializes it.
> > 
> > > 2) If we find a way to break the QEMU layer socket connection when pmd
> > > exits and get it reconnect, virtio-net device still won't send all the
> > > configure when reconnecting, because socket connecting only triggers
> > > phase 1 of vhost-user negotiation (i.e. vhost_user_init). Phase 2 is
> > > triggered after the driver loads (i.e. vhost_net_start). If the
> > > virtio-net pmd doesn't reload, there are no phase 2 messages (like
> > > virtqueue addresses which are allocated by the pmd). I think we need
> > > to think more about this before moving forward.
> > 
> > Marc-André: How does vhost-user reconnect work when the master goes
> > away and a new master comes online?  Wei found that the QEMU slave
> > implementation only does partial vhost-user initialization upon reconnect, so
> > the new master doesn't get the virtqueue address and related information.
> > Is this a QEMU bug?
> 
> Actually we are discussing the slave (vhost is the slave, right?) going away. When a slave exits and some moment later a new slave runs, the master (virtio-net) won't send the virtqueue addresses to the new vhost slave.

Yes, apologies for the typo.  s/QEMU slave/QEMU master/

Yesterday I asked Marc-André for help on IRC and we found the code path
where the QEMU master performs phase 2 negotiation upon reconnect.  It's
not obvious but the qmp_set_link() calls in net_vhost_user_event() will
do it.

I'm going to try to reproduce the issue you're seeing now.  Will let you
know what I find.

Stefan

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 455 bytes --]

Re: [Qemu-devel] [RFC 0/2] virtio-vhost-user: add virtio-vhost-user device

[Wang, Wei W]

On Tuesday, February 6, 2018 5:32 PM, Stefan Hajnoczi wrote:
> On Tue, Feb 06, 2018 at 01:28:25AM +0000, Wang, Wei W wrote:
> > On Tuesday, February 6, 2018 12:26 AM, Stefan Hajnoczi wrote:
> > > On Fri, Feb 02, 2018 at 09:08:44PM +0800, Wei Wang wrote:
> > > > On 02/02/2018 01:08 AM, Michael S. Tsirkin wrote:
> > > > > On Tue, Jan 30, 2018 at 08:09:19PM +0800, Wei Wang wrote:
> > > > > > Issues:
> > > > > > Suppose we have both the vhost and virtio-net set up, and 
> > > > > > vhost pmd <-> virtio-net pmd communication works well. Now, 
> > > > > > vhost pmd exits (virtio-net pmd is still there). Some time 
> > > > > > later, we re-run vhost pmd, the vhost pmd doesn't know the 
> > > > > > virtqueue addresses of the virtio-net pmd, unless the 
> > > > > > virtio-net pmd reloads to start the 2nd phase of the 
> > > > > > vhost-user protocol. So the second run of the vhost
> > > pmd won't work.
> > > > > >
> > > > > > Any thoughts?
> > > > > >
> > > > > > Best,
> > > > > > Wei
> > > > > So vhost in qemu must resend all configuration on reconnect.
> > > > > Does this address the issues?
> > > > >
> > > >
> > > > Yes, but the issues are
> > > > 1) there is no reconnecting when a pmd exits (the socket 
> > > > connection seems still on at the device layer);
> > >
> > > This is how real hardware works too.  If the driver suddenly stops 
> > > running then the device remains operational.  When the driver is 
> > > started again it resets the device and initializes it.
> > >
> > > > 2) If we find a way to break the QEMU layer socket connection 
> > > > when pmd exits and get it reconnect, virtio-net device still 
> > > > won't send all the configure when reconnecting, because socket 
> > > > connecting only triggers phase 1 of vhost-user negotiation (i.e.
> > > > vhost_user_init). Phase 2 is triggered after the driver loads 
> > > > (i.e. vhost_net_start). If the virtio-net pmd doesn't reload, 
> > > > there are no phase 2 messages (like virtqueue addresses which 
> > > > are allocated by the pmd). I think we need to think more about 
> > > > this before
> moving forward.
> > >
> > > Marc-André: How does vhost-user reconnect work when the master 
> > > goes away and a new master comes online?  Wei found that the QEMU 
> > > slave implementation only does partial vhost-user initialization 
> > > upon reconnect, so the new master doesn't get the virtqueue 
> > > address and
> related information.
> > > Is this a QEMU bug?
> >
> > Actually we are discussing the slave (vhost is the slave, right?) going away.
> When a slave exits and some moment later a new slave runs, the master
> (virtio-net) won't send the virtqueue addresses to the new vhost slave.
> 
> Yes, apologies for the typo.  s/QEMU slave/QEMU master/
> 
> Yesterday I asked Marc-André for help on IRC and we found the code 
> path where the QEMU master performs phase 2 negotiation upon 
> reconnect.  It's not obvious but the qmp_set_link() calls in net_vhost_user_event() will do it.
> 
> I'm going to try to reproduce the issue you're seeing now.  Will let 
> you know what I find.
> 

OK. Thanks. I observed no messages after re-run virtio-vhost-user pmd, and found there is no re-connection event happening in the device side. 

I also tried to switch the role of client/server - virtio-net to run a server socket, and virtio-vhost-user to run the client, and it seems the current code fails to run that way. The reason is the virtio-net side vhost_user_get_features() doesn't return. On the vhost side, I don't see virtio_vhost_user_deliver_m2s being invoked to deliver the GET_FEATURES message. I'll come back to continue later.

Best,
Wei

Re: [Qemu-devel] [RFC 0/2] virtio-vhost-user: add virtio-vhost-user device

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 5455 bytes --]

On Tue, Feb 06, 2018 at 12:42:36PM +0000, Wang, Wei W wrote:
> On Tuesday, February 6, 2018 5:32 PM, Stefan Hajnoczi wrote:
> > On Tue, Feb 06, 2018 at 01:28:25AM +0000, Wang, Wei W wrote:
> > > On Tuesday, February 6, 2018 12:26 AM, Stefan Hajnoczi wrote:
> > > > On Fri, Feb 02, 2018 at 09:08:44PM +0800, Wei Wang wrote:
> > > > > On 02/02/2018 01:08 AM, Michael S. Tsirkin wrote:
> > > > > > On Tue, Jan 30, 2018 at 08:09:19PM +0800, Wei Wang wrote:
> > > > > > > Issues:
> > > > > > > Suppose we have both the vhost and virtio-net set up, and 
> > > > > > > vhost pmd <-> virtio-net pmd communication works well. Now, 
> > > > > > > vhost pmd exits (virtio-net pmd is still there). Some time 
> > > > > > > later, we re-run vhost pmd, the vhost pmd doesn't know the 
> > > > > > > virtqueue addresses of the virtio-net pmd, unless the 
> > > > > > > virtio-net pmd reloads to start the 2nd phase of the 
> > > > > > > vhost-user protocol. So the second run of the vhost
> > > > pmd won't work.
> > > > > > >
> > > > > > > Any thoughts?
> > > > > > >
> > > > > > > Best,
> > > > > > > Wei
> > > > > > So vhost in qemu must resend all configuration on reconnect.
> > > > > > Does this address the issues?
> > > > > >
> > > > >
> > > > > Yes, but the issues are
> > > > > 1) there is no reconnecting when a pmd exits (the socket 
> > > > > connection seems still on at the device layer);
> > > >
> > > > This is how real hardware works too.  If the driver suddenly stops 
> > > > running then the device remains operational.  When the driver is 
> > > > started again it resets the device and initializes it.
> > > >
> > > > > 2) If we find a way to break the QEMU layer socket connection 
> > > > > when pmd exits and get it reconnect, virtio-net device still 
> > > > > won't send all the configure when reconnecting, because socket 
> > > > > connecting only triggers phase 1 of vhost-user negotiation (i.e.
> > > > > vhost_user_init). Phase 2 is triggered after the driver loads 
> > > > > (i.e. vhost_net_start). If the virtio-net pmd doesn't reload, 
> > > > > there are no phase 2 messages (like virtqueue addresses which 
> > > > > are allocated by the pmd). I think we need to think more about 
> > > > > this before
> > moving forward.
> > > >
> > > > Marc-André: How does vhost-user reconnect work when the master 
> > > > goes away and a new master comes online?  Wei found that the QEMU 
> > > > slave implementation only does partial vhost-user initialization 
> > > > upon reconnect, so the new master doesn't get the virtqueue 
> > > > address and
> > related information.
> > > > Is this a QEMU bug?
> > >
> > > Actually we are discussing the slave (vhost is the slave, right?) going away.
> > When a slave exits and some moment later a new slave runs, the master
> > (virtio-net) won't send the virtqueue addresses to the new vhost slave.
> > 
> > Yes, apologies for the typo.  s/QEMU slave/QEMU master/
> > 
> > Yesterday I asked Marc-André for help on IRC and we found the code 
> > path where the QEMU master performs phase 2 negotiation upon 
> > reconnect.  It's not obvious but the qmp_set_link() calls in net_vhost_user_event() will do it.
> > 
> > I'm going to try to reproduce the issue you're seeing now.  Will let 
> > you know what I find.
> > 
> 
> OK. Thanks. I observed no messages after re-run virtio-vhost-user pmd, and found there is no re-connection event happening in the device side. 
> 
> I also tried to switch the role of client/server - virtio-net to run a server socket, and virtio-vhost-user to run the client, and it seems the current code fails to run that way. The reason is the virtio-net side vhost_user_get_features() doesn't return. On the vhost side, I don't see virtio_vhost_user_deliver_m2s being invoked to deliver the GET_FEATURES message. I'll come back to continue later.

This morning I reached the conclusion that reconnection is currently
broken in the QEMU vhost-user master.  It's a bug in the QEMU vhost-user
master implementation, not a design or protocol problem.

On my machine the following QEMU command-line does not launch because
vhost-user.c gets stuck while trying to connect/negotiate:

  qemu -M accel=kvm -cpu host -m 1G \
       -object memory-backend-file,id=mem0,mem-path=/var/tmp/foo,size=1G,share=on \
       -numa node,memdev=mem0 \
       -drive if=virtio,file=test.img,format=raw \
       -chardev socket,id=chardev0,path=vhost-user.sock,reconnect=1 \
       -netdev vhost-user,chardev=chardev0,id=netdev0 \
       -device virtio-net-pci,netdev=netdev0

Commit c89804d674e4e3804bd3ac1fe79650896044b4e8 ("vhost-user: wait until
backend init is completed") broke reconnect by introducing a call to
qemu_chr_fe_wait_connected().

qemu_chr_fe_wait_connected() doesn't work together with -chardev
...,reconnect=1.  This is because reconnect=1 connects asynchronously
and then qemu_chr_fe_wait_connect() connects synchronously (if the async
connect hasn't completed yet).  This means there will be 2 sockets
connecting to the vhost-user slave!

The virtio-vhost-user slave accepts the first connection but never
receives any data because the QEMU master is trying to use the 2nd
socket instead.

Reconnection probably worked when Marc-André implemented it since QEMU
wasn't using qemu_chr_fe_wait_connected().

Marc-André: How do you think this should be fixed?

Stefan

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 455 bytes --]

Re: [Qemu-devel] [RFC 0/2] virtio-vhost-user: add virtio-vhost-user device

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 4421 bytes --]

On Tue, Jan 30, 2018 at 08:09:19PM +0800, Wei Wang wrote:
> On 01/26/2018 10:44 PM, Stefan Hajnoczi wrote:
> > On Thu, Jan 25, 2018 at 06:19:13PM +0800, Wei Wang wrote:
> > > On 01/24/2018 07:40 PM, Stefan Hajnoczi wrote:
> > > > On Tue, Jan 23, 2018 at 09:06:49PM +0800, Wei Wang wrote:
> > > > > On 01/23/2018 07:12 PM, Stefan Hajnoczi wrote:
> > > > > > On Mon, Jan 22, 2018 at 07:09:06PM +0800, Wei Wang wrote:
> > > > > > > On 01/19/2018 09:06 PM, Stefan Hajnoczi wrote:
> > > > > > > 
> > > > > > > 
> > > > > > >     - Suppose in the future there is also a kernel virtio-vhost-user driver as
> > > > > > > other PCI devices, can we unbind the kernel driver first, and then bind the
> > > > > > > device to the dpdk driver? A normal PCI device should be able to smoothly
> > > > > > > switch between the kernel driver and dpdk driver.
> > > > > > It depends what you mean by "smoothly switch".
> > > > > > 
> > > > > > If you mean whether it's possible to go from a kernel driver to
> > > > > > vfio-pci, then the answer is yes.
> > > > > > 
> > > > > > But if the kernel driver has an established vhost-user connection then
> > > > > > it will be closed.  This is the same as reconnecting with AF_UNIX
> > > > > > vhost-user.
> > > > > > 
> > > > > Actually not only the case of switching to testpmd after kernel establishes
> > > > > the connection, but also for several runs of testpmd. That is, if we run
> > > > > testpmd, then exit testpmd. I think the second run of testpmd won't work.
> > > > The vhost-user master must reconnect and initialize again (SET_FEATURES,
> > > > SET_MEM_TABLE, etc).  Is your master reconnecting after the AF_UNIX
> > > > connection is closed?
> > > Is this an explicit qmp operation to make the master re-connect?
> > I haven't tested it myself but I'm aware of two modes of operation:
> > 
> > 1. -chardev socket,id=chardev0,...,server
> >     -netdev vhost-user,chardev=chardev0
> > 
> >     When the vhost-user socket is disconnected the peer needs to
> >     reconnect.  In this case no special commands are necessary.
> > 
> >     Here we're relying on DPDK librte_vhost's reconnection behavior.
> > 
> > Or
> > 
> > 2. -chardev socket,id=chardev0,...,reconnect=3
> >     -netdev vhost-user,chardev=chardev0
> > 
> >     When the vhost-user socket is disconnected a new connection attempt
> >     will be made after 3 seconds.
> > 
> > In both cases vhost-user negotiation will resume when the new connection
> > is established.
> > 
> > Stefan
> 
> I've been thinking about the issues, and it looks vhost-pci outperforms in
> this aspect.
> Vhost-pci is like using a mail box. messages are just dropped into the box,
> and whenever vhost-pci pmd gets booted, it can always get the messages from
> the box, the negotiation between vhost-pci pmd and virtio-net is
> asynchronous.
> Virtio-vhost-user is like a phone call, which is a synchronous
> communication. If one side is absent, then the other side will hang on
> without knowing when it could get connected or hang up with messages not
> passed (lost).
> 
> I also think the above solutions won't help. Please see below:
> 
> Background:
> The vhost-user negotiation is split into 2 phases currently. The 1st phase
> happens when the connection is established, and we can find what's done in
> the 1st phase in vhost_user_init(). The 2nd phase happens when the master
> driver is loaded (e.g. run of virtio-net pmd) and set status to the device,
> and we can find what's done in the 2nd phase in vhost_dev_start(), which
> includes sending the memory info and virtqueue info. The socket is
> connected, till one of the QEMU devices exits, so pmd exiting won't end the
> QEMU side socket connection.
>
> Issues:
> Suppose we have both the vhost and virtio-net set up, and vhost pmd <->
> virtio-net pmd communication works well. Now, vhost pmd exits (virtio-net
> pmd is still there). Some time later, we re-run vhost pmd, the vhost pmd
> doesn't know the virtqueue addresses of the virtio-net pmd, unless the
> virtio-net pmd reloads to start the 2nd phase of the vhost-user protocol. So
> the second run of the vhost pmd won't work.

This isn't a problem for virtio-vhost-user since the vhost-pmd resets
the virtio-vhost-user device when it restarts.  The vhost-user AF_UNIX
socket reconnects and negotiation restarts.

Stefan

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 455 bytes --]

Re: [Qemu-devel] [RFC 0/2] virtio-vhost-user: add virtio-vhost-user device

[Wang, Wei W]

On Friday, February 2, 2018 11:26 PM, Stefan Hajnoczi wrote:
> On Tue, Jan 30, 2018 at 08:09:19PM +0800, Wei Wang wrote:
> > Background:
> > The vhost-user negotiation is split into 2 phases currently. The 1st
> > phase happens when the connection is established, and we can find
> > what's done in the 1st phase in vhost_user_init(). The 2nd phase
> > happens when the master driver is loaded (e.g. run of virtio-net pmd)
> > and set status to the device, and we can find what's done in the 2nd
> > phase in vhost_dev_start(), which includes sending the memory info and
> > virtqueue info. The socket is connected, till one of the QEMU devices
> > exits, so pmd exiting won't end the QEMU side socket connection.
> >
> > Issues:
> > Suppose we have both the vhost and virtio-net set up, and vhost pmd
> > <-> virtio-net pmd communication works well. Now, vhost pmd exits
> > (virtio-net pmd is still there). Some time later, we re-run vhost pmd,
> > the vhost pmd doesn't know the virtqueue addresses of the virtio-net
> > pmd, unless the virtio-net pmd reloads to start the 2nd phase of the
> > vhost-user protocol. So the second run of the vhost pmd won't work.
> 
> This isn't a problem for virtio-vhost-user since the vhost-pmd resets the
> virtio-vhost-user device when it restarts.  The vhost-user AF_UNIX socket
> reconnects and negotiation restarts.

I'm not sure if you've agreed that vhost-user negotiation is split into two phases as described above. If not, it's also not difficult to check, thanks to the RTE_LOG put at the vhost_user_msg_handler:
RTE_LOG(INFO, VHOST_CONFIG, "read message %s\n",
                        vhost_message_str[msg.request.master]);
It tells us what messages are received and when they are received. 

Before trying the virtio-vhost-user setup, please make sure the virtio-net side VM doesn't have the virtio-net kernel driver loaded (blacklist the module or disable it in .config).
VM1: the VM with virtio-vhost-user
VM2: the VM with virtio-net

1) After we boot VM1 and the virtio-vhost-user pmd, we boot VM2, and we will see the following log in VM1:
VHOST_CONFIG: new device, handle is 0
VHOST_CONFIG: read message VHOST_USER_GET_FEATURES
VHOST_CONFIG: read message VHOST_USER_GET_PROTOCOL_FEATURES
VHOST_CONFIG: read message VHOST_USER_SET_PROTOCOL_FEATURES
VHOST_CONFIG: read message VHOST_USER_GET_QUEUE_NUM
VHOST_CONFIG: read message VHOST_USER_SET_OWNER
VHOST_CONFIG: read message VHOST_USER_GET_FEATURES
VHOST_CONFIG: read message VHOST_USER_SET_VRING_CALL
VHOST_CONFIG: vring call idx:0 file:-1
VHOST_CONFIG: read message VHOST_USER_SET_VRING_CALL
VHOST_CONFIG: vring call idx:1 file:-1

Those messages are what I called phase1 negotiation. They are negotiated when the AF_UNIX socket connects.

2) Then in VM2 we load the virtio-net pmd, we will see phase2 messages showing up in VM1, like:
VHOST_CONFIG: read message VHOST_USER_SET_VRING_NUM
VHOST_CONFIG: read message VHOST_USER_SET_VRING_BASE
VHOST_CONFIG: read message VHOST_USER_SET_VRING_ADDR
...
Those messages are sent to virtio-vhost-user only when the VM2's virtio-net pmd loads.

AF_UNIX socket reconnection only triggers phase1 negotiation. If virtio-net pmd in VM2 doesn't reload, virtio-vhost-user pmd won't get the above phase2 messages. Do you agree with the issue?

Best,
Wei 

Re: [Qemu-devel] [RFC 0/2] virtio-vhost-user: add virtio-vhost-user device

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 3608 bytes --]

On Mon, Feb 05, 2018 at 09:57:23AM +0000, Wang, Wei W wrote:
> On Friday, February 2, 2018 11:26 PM, Stefan Hajnoczi wrote:
> > On Tue, Jan 30, 2018 at 08:09:19PM +0800, Wei Wang wrote:
> > > Background:
> > > The vhost-user negotiation is split into 2 phases currently. The 1st
> > > phase happens when the connection is established, and we can find
> > > what's done in the 1st phase in vhost_user_init(). The 2nd phase
> > > happens when the master driver is loaded (e.g. run of virtio-net pmd)
> > > and set status to the device, and we can find what's done in the 2nd
> > > phase in vhost_dev_start(), which includes sending the memory info and
> > > virtqueue info. The socket is connected, till one of the QEMU devices
> > > exits, so pmd exiting won't end the QEMU side socket connection.
> > >
> > > Issues:
> > > Suppose we have both the vhost and virtio-net set up, and vhost pmd
> > > <-> virtio-net pmd communication works well. Now, vhost pmd exits
> > > (virtio-net pmd is still there). Some time later, we re-run vhost pmd,
> > > the vhost pmd doesn't know the virtqueue addresses of the virtio-net
> > > pmd, unless the virtio-net pmd reloads to start the 2nd phase of the
> > > vhost-user protocol. So the second run of the vhost pmd won't work.
> > 
> > This isn't a problem for virtio-vhost-user since the vhost-pmd resets the
> > virtio-vhost-user device when it restarts.  The vhost-user AF_UNIX socket
> > reconnects and negotiation restarts.
> 
> I'm not sure if you've agreed that vhost-user negotiation is split into two phases as described above. If not, it's also not difficult to check, thanks to the RTE_LOG put at the vhost_user_msg_handler:
> RTE_LOG(INFO, VHOST_CONFIG, "read message %s\n",
>                         vhost_message_str[msg.request.master]);
> It tells us what messages are received and when they are received. 
> 
> Before trying the virtio-vhost-user setup, please make sure the virtio-net side VM doesn't have the virtio-net kernel driver loaded (blacklist the module or disable it in .config).
> VM1: the VM with virtio-vhost-user
> VM2: the VM with virtio-net
> 
> 1) After we boot VM1 and the virtio-vhost-user pmd, we boot VM2, and we will see the following log in VM1:
> VHOST_CONFIG: new device, handle is 0
> VHOST_CONFIG: read message VHOST_USER_GET_FEATURES
> VHOST_CONFIG: read message VHOST_USER_GET_PROTOCOL_FEATURES
> VHOST_CONFIG: read message VHOST_USER_SET_PROTOCOL_FEATURES
> VHOST_CONFIG: read message VHOST_USER_GET_QUEUE_NUM
> VHOST_CONFIG: read message VHOST_USER_SET_OWNER
> VHOST_CONFIG: read message VHOST_USER_GET_FEATURES
> VHOST_CONFIG: read message VHOST_USER_SET_VRING_CALL
> VHOST_CONFIG: vring call idx:0 file:-1
> VHOST_CONFIG: read message VHOST_USER_SET_VRING_CALL
> VHOST_CONFIG: vring call idx:1 file:-1
> 
> Those messages are what I called phase1 negotiation. They are negotiated when the AF_UNIX socket connects.
> 
> 2) Then in VM2 we load the virtio-net pmd, we will see phase2 messages showing up in VM1, like:
> VHOST_CONFIG: read message VHOST_USER_SET_VRING_NUM
> VHOST_CONFIG: read message VHOST_USER_SET_VRING_BASE
> VHOST_CONFIG: read message VHOST_USER_SET_VRING_ADDR
> ...
> Those messages are sent to virtio-vhost-user only when the VM2's virtio-net pmd loads.
> 
> AF_UNIX socket reconnection only triggers phase1 negotiation. If virtio-net pmd in VM2 doesn't reload, virtio-vhost-user pmd won't get the above phase2 messages. Do you agree with the issue?

I'll reply on the subthread with Michael where you posted a similar
response.

Stefan

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 455 bytes --]