[Qemu-devel] [PATCH v2 0/4] *** vhost-user spec extension for vhost-pci ***

[Qemu-devel] [PATCH v2 0/4] *** vhost-user spec extension for vhost-pci ***

[Wei Wang]

This spec patch series extends the vhost-user protocol to support the vhost-pci
based inter-VM communiaction.

v1->v2 changes:
1) start from the simpler case - change "1-slave-N-master" to "1-slave-1-master"
configuration plane. Accordingly, the previous "uuid", "conn_id" are removed;
2) add the _CREATE_ and _DESTROY_ comands to the VHOST_USER_SET_PEER_CONNECTION
message; and
3) fix the VHOST_USER prefix.

Wei Wang (4):
  spec/vhost-user: fix the VHOST_USER prefix
  spec/vhost-user: extend vhost-user to support the vhost-pci based    
    inter-vm communiaction
  spec/vhost-user: add the VHOST_USER_SET_PEER_CONNECTION message
  spec/vhost-user: add VHOST_USER_PROTOCOL_F_VERSATILE_SLAVE

 docs/specs/vhost-user.txt | 93 +++++++++++++++++++++++++++++++++++++----------
 1 file changed, 74 insertions(+), 19 deletions(-)

-- 
2.7.4

[Qemu-devel] [PATCH v2 1/4] spec/vhost-user: fix the VHOST_USER prefix

[Wei Wang]

Signed-off-by: Wei Wang <wei.w.wang@intel.com>
---
 docs/specs/vhost-user.txt | 20 ++++++++++----------
 1 file changed, 10 insertions(+), 10 deletions(-)

diff --git a/docs/specs/vhost-user.txt b/docs/specs/vhost-user.txt
index 7890d71..d70bd83 100644
--- a/docs/specs/vhost-user.txt
+++ b/docs/specs/vhost-user.txt
@@ -123,22 +123,22 @@ The communication consists of master sending message requests and slave sending
 message replies. Most of the requests don't require replies. Here is a list of
 the ones that do:
 
- * VHOST_GET_FEATURES
- * VHOST_GET_PROTOCOL_FEATURES
- * VHOST_GET_VRING_BASE
- * VHOST_SET_LOG_BASE (if VHOST_USER_PROTOCOL_F_LOG_SHMFD)
+ * VHOST_USER_GET_FEATURES
+ * VHOST_USER_GET_PROTOCOL_FEATURES
+ * VHOST_USER_GET_VRING_BASE
+ * VHOST_USER_SET_LOG_BASE (if VHOST_USER_PROTOCOL_F_LOG_SHMFD)
 
 [ Also see the section on REPLY_ACK protocol extension. ]
 
 There are several messages that the master sends with file descriptors passed
 in the ancillary data:
 
- * VHOST_SET_MEM_TABLE
- * VHOST_SET_LOG_BASE (if VHOST_USER_PROTOCOL_F_LOG_SHMFD)
- * VHOST_SET_LOG_FD
- * VHOST_SET_VRING_KICK
- * VHOST_SET_VRING_CALL
- * VHOST_SET_VRING_ERR
+ * VHOST_USER_SET_MEM_TABLE
+ * VHOST_USER_SET_LOG_BASE (if VHOST_USER_PROTOCOL_F_LOG_SHMFD)
+ * VHOST_USER_SET_LOG_FD
+ * VHOST_USER_SET_VRING_KICK
+ * VHOST_USER_SET_VRING_CALL
+ * VHOST_USER_SET_VRING_ERR
 
 If Master is unable to send the full message or receives a wrong reply it will
 close the connection. An optional reconnection mechanism can be implemented.
-- 
2.7.4

[Qemu-devel] [PATCH v2 2/4] spec/vhost-user: extend vhost-user to support the vhost-pci based inter-vm communiaction

[Wei Wang]

The protocol feature, VHOST_USER_PROTOCOL_F_VHOST_PCI, indicates the
support of vhost-pci. With the vhost-pci extension, the slave may
actively sending meesages to the master. VHOST_USER_SET_FEATURES is one
of the examples.

To understand this example, let's first understand how
the device feature bits are negotiated between the slave
device/driver and the master device/driver:
1) the master device (e.g. virtio-net) GET_FEATURES from the slave
(assume the feature bits are "f1");
2) the master device negotiates the feature bits with its driver (assume
the device gets "f2" after the negotiation);
3) the master device SET_FEATURES("f2") to the slave;
4) the slave creates a slave device (e.g. vhost-pci-net) with "f2" and
the slave device negotiates "f2" with its driver (assume the device gets
"f3" after the negotiation);
5) the slave _actively_ SET_FEATURES("f3") to the master device;
6) if "f3" != "f2", the master device needs to perform a device reset
and re-negotiate the feature bits with its driver using "f3".

Signed-off-by: Wei Wang <wei.w.wang@intel.com>
---
 docs/specs/vhost-user.txt | 19 ++++++++++++++-----
 1 file changed, 14 insertions(+), 5 deletions(-)

diff --git a/docs/specs/vhost-user.txt b/docs/specs/vhost-user.txt
index d70bd83..3bbe641 100644
--- a/docs/specs/vhost-user.txt
+++ b/docs/specs/vhost-user.txt
@@ -17,12 +17,15 @@ The protocol defines 2 sides of the communication, master and slave. Master is
 the application that shares its virtqueues, in our case QEMU. Slave is the
 consumer of the virtqueues.
 
-In the current implementation QEMU is the Master, and the Slave is intended to
+In the traditional implementation QEMU is the master, and the slave is intended to
 be a software Ethernet switch running in user space, such as Snabbswitch.
 
 Master and slave can be either a client (i.e. connecting) or server (listening)
 in the socket communication.
 
+The current vhost-user protocol is extended to support the vhost-pci based inter-VM
+communication. In this case, both the slave and master are QEMU instances.
+
 Message Specification
 ---------------------
 
@@ -36,7 +39,7 @@ consists of 3 header fields and a payload:
  * Request: 32-bit type of the request
  * Flags: 32-bit bit field:
    - Lower 2 bits are the version (currently 0x01)
-   - Bit 2 is the reply flag - needs to be sent on each reply from the slave
+   - Bit 2 is the reply flag - needs to be sent on each reply
    - Bit 3 is the need_reply flag - see VHOST_USER_PROTOCOL_F_REPLY_ACK for
      details.
  * Size - 32-bit size of the payload
@@ -119,9 +122,9 @@ The protocol for vhost-user is based on the existing implementation of vhost
 for the Linux Kernel. Most messages that can be sent via the Unix domain socket
 implementing vhost-user have an equivalent ioctl to the kernel implementation.
 
-The communication consists of master sending message requests and slave sending
-message replies. Most of the requests don't require replies. Here is a list of
-the ones that do:
+Traditionally, the communication consists of master sending message requests and
+slave sending message replies. Most of the requests don't require replies. Here
+is a list of the ones that do:
 
  * VHOST_USER_GET_FEATURES
  * VHOST_USER_GET_PROTOCOL_FEATURES
@@ -130,6 +133,10 @@ the ones that do:
 
 [ Also see the section on REPLY_ACK protocol extension. ]
 
+Currently, the communication also supports the slave actively sending messages
+to the master. Here is a list of them:
+ * VHOST_USER_SET_FEATURES
+
 There are several messages that the master sends with file descriptors passed
 in the ancillary data:
 
@@ -259,6 +266,7 @@ Protocol features
 #define VHOST_USER_PROTOCOL_F_LOG_SHMFD      1
 #define VHOST_USER_PROTOCOL_F_RARP           2
 #define VHOST_USER_PROTOCOL_F_REPLY_ACK      3
+#define VHOST_USER_PROTOCOL_F_VHOST_PCI      4
 
 Message types
 -------------
@@ -279,6 +287,7 @@ Message types
       Id: 2
       Ioctl: VHOST_SET_FEATURES
       Master payload: u64
+      Slave payload: u64
 
       Enable features in the underlying vhost implementation using a bitmask.
       Feature bit VHOST_USER_F_PROTOCOL_FEATURES signals slave support for
-- 
2.7.4

[Qemu-devel] [PATCH v2 3/4] spec/vhost-user: add the VHOST_USER_SET_PEER_CONNECTION message

[Wei Wang]

The VHOST_USER_SET_PEER_CONNECTION message is introduced to manage the
vhost-pci dataplane connection status. The slave can use the vhost-pci
dataplane to transmit/receive packets to/from the master when the
connection is turned ON, and stops using it when the connection is
turned OFF.

Signed-off-by: Wei Wang <wei.w.wang@intel.com>
---
 docs/specs/vhost-user.txt | 35 +++++++++++++++++++++++++++++++++++
 1 file changed, 35 insertions(+)

diff --git a/docs/specs/vhost-user.txt b/docs/specs/vhost-user.txt
index 3bbe641..fdc99ea 100644
--- a/docs/specs/vhost-user.txt
+++ b/docs/specs/vhost-user.txt
@@ -130,12 +130,14 @@ is a list of the ones that do:
  * VHOST_USER_GET_PROTOCOL_FEATURES
  * VHOST_USER_GET_VRING_BASE
  * VHOST_USER_SET_LOG_BASE (if VHOST_USER_PROTOCOL_F_LOG_SHMFD)
+ * VHOST_USER_SET_PEER_CONNECTION
 
 [ Also see the section on REPLY_ACK protocol extension. ]
 
 Currently, the communication also supports the slave actively sending messages
 to the master. Here is a list of them:
  * VHOST_USER_SET_FEATURES
+ * VHOST_USER_SET_PEER_CONNECTION
 
 There are several messages that the master sends with file descriptors passed
 in the ancillary data:
@@ -479,6 +481,39 @@ Message types
       The first 6 bytes of the payload contain the mac address of the guest to
       allow the vhost user backend to construct and broadcast the fake RARP.
 
+ * VHOST_USER_SET_PEER_CONNECTION
+
+      Id: 20
+      Equivalent ioctl: N/A
+      Master payload: u64
+      Slave payload: u64
+
+      The slave device requests to connect or disconnect to the master device.
+      The master device may request to disconnect to the slave device.
+      This request should be sent only when VHOST_USER_PROTOCOL_F_VHOST_PCI has
+      been negotiated.
+
+      Connection command (ON): If the reply message indicates "success", the
+      connection status is "active", and the slave can start to transmit and
+      receive packets to the master through the vhost-pci dataplane. Replying of
+      this message is asynchronous, because the device needs to talk to the
+      driver first.
+      Disconnection command (OFF): If the reply message indicates "success", the
+      connection status is "inactive". The slave cannot use the vhost-pci
+      dataplane when the connection is "inactive". Replying of this message is
+      asynchronous, because the device needs to talk to the driver first.
+      Creation command (CREATE): Sent by the master to the slave to request for
+      the creation of a slave device. If the reply messages indicates "success",
+      it means that the slave is able to create a slave device for the master.
+      Destroy command (DESTROY): Sent by the master to the slave to request for
+      the destruction of the slave device. This command should only be sent when
+      the connection is "inactive". No reply is needed for this command.
+
+      #define VHOST_USER_SET_PEER_CONNECTION_F_OFF       0
+      #define VHOST_USER_SET_PEER_CONNECTION_F_ON        1
+      #define VHOST_USER_SET_PEER_CONNECTION_F_CREATE    2
+      #define VHOST_USER_SET_PEER_CONNECTION_F_DESTROY   3
+
 VHOST_USER_PROTOCOL_F_REPLY_ACK:
 -------------------------------
 The original vhost-user specification only demands replies for certain
-- 
2.7.4

[Qemu-devel] [PATCH v2 4/4] spec/vhost-user: add VHOST_USER_PROTOCOL_F_VERSATILE_SLAVE

[Wei Wang]

The VHOST_USER_PROTOCOL_F_VERSATILE_SLAVE protocol feature indicates
that the slave side implementation supports different types of devices.
The master tells the slave what type of device to create by sending
the VHOST_USER_SET_DEV_INFO message.

Signed-off-by: Wei Wang <wei.w.wang@intel.com>
---
 docs/specs/vhost-user.txt | 21 ++++++++++++++++-----
 1 file changed, 16 insertions(+), 5 deletions(-)

diff --git a/docs/specs/vhost-user.txt b/docs/specs/vhost-user.txt
index fdc99ea..da1314d 100644
--- a/docs/specs/vhost-user.txt
+++ b/docs/specs/vhost-user.txt
@@ -264,11 +264,12 @@ restarted.
 Protocol features
 -----------------
 
-#define VHOST_USER_PROTOCOL_F_MQ             0
-#define VHOST_USER_PROTOCOL_F_LOG_SHMFD      1
-#define VHOST_USER_PROTOCOL_F_RARP           2
-#define VHOST_USER_PROTOCOL_F_REPLY_ACK      3
-#define VHOST_USER_PROTOCOL_F_VHOST_PCI      4
+#define VHOST_USER_PROTOCOL_F_MQ               0
+#define VHOST_USER_PROTOCOL_F_LOG_SHMFD        1
+#define VHOST_USER_PROTOCOL_F_RARP             2
+#define VHOST_USER_PROTOCOL_F_REPLY_ACK        3
+#define VHOST_USER_PROTOCOL_F_VHOST_PCI        4
+#define VHOST_USER_PROTOCOL_F_VERSATILE_SLAVE  5
 
 Message types
 -------------
@@ -514,6 +515,16 @@ Message types
       #define VHOST_USER_SET_PEER_CONNECTION_F_CREATE    2
       #define VHOST_USER_SET_PEER_CONNECTION_F_DESTROY   3
 
+ * VHOST_USER_SET_DEV_INFO
+
+      Id: 21
+      Equivalent ioctl: N/A
+      Master payload: u64
+
+      The master sends the device type info to the slave.
+      This request should be sent only when VHOST_USER_PROTOCOL_F_VERSATILE_SLAVE
+      has been negotiated.
+
 VHOST_USER_PROTOCOL_F_REPLY_ACK:
 -------------------------------
 The original vhost-user specification only demands replies for certain
-- 
2.7.4

Re: [Qemu-devel] [PATCH v2 1/4] spec/vhost-user: fix the VHOST_USER prefix

[Marc-André Lureau]

On Thu, Nov 24, 2016 at 7:20 AM Wei Wang <wei.w.wang@intel.com> wrote:

> Signed-off-by: Wei Wang <wei.w.wang@intel.com>
>


Reviewed-by: Marc-André Lureau <marcandre.lureau@redhat.com>



> ---
>  docs/specs/vhost-user.txt | 20 ++++++++++----------
>  1 file changed, 10 insertions(+), 10 deletions(-)
>
> diff --git a/docs/specs/vhost-user.txt b/docs/specs/vhost-user.txt
> index 7890d71..d70bd83 100644
> --- a/docs/specs/vhost-user.txt
> +++ b/docs/specs/vhost-user.txt
> @@ -123,22 +123,22 @@ The communication consists of master sending message
> requests and slave sending
>  message replies. Most of the requests don't require replies. Here is a
> list of
>  the ones that do:
>
> - * VHOST_GET_FEATURES
> - * VHOST_GET_PROTOCOL_FEATURES
> - * VHOST_GET_VRING_BASE
> - * VHOST_SET_LOG_BASE (if VHOST_USER_PROTOCOL_F_LOG_SHMFD)
> + * VHOST_USER_GET_FEATURES
> + * VHOST_USER_GET_PROTOCOL_FEATURES
> + * VHOST_USER_GET_VRING_BASE
> + * VHOST_USER_SET_LOG_BASE (if VHOST_USER_PROTOCOL_F_LOG_SHMFD)
>
>  [ Also see the section on REPLY_ACK protocol extension. ]
>
>  There are several messages that the master sends with file descriptors
> passed
>  in the ancillary data:
>
> - * VHOST_SET_MEM_TABLE
> - * VHOST_SET_LOG_BASE (if VHOST_USER_PROTOCOL_F_LOG_SHMFD)
> - * VHOST_SET_LOG_FD
> - * VHOST_SET_VRING_KICK
> - * VHOST_SET_VRING_CALL
> - * VHOST_SET_VRING_ERR
> + * VHOST_USER_SET_MEM_TABLE
> + * VHOST_USER_SET_LOG_BASE (if VHOST_USER_PROTOCOL_F_LOG_SHMFD)
> + * VHOST_USER_SET_LOG_FD
> + * VHOST_USER_SET_VRING_KICK
> + * VHOST_USER_SET_VRING_CALL
> + * VHOST_USER_SET_VRING_ERR


>  If Master is unable to send the full message or receives a wrong reply it
> will
>  close the connection. An optional reconnection mechanism can be
> implemented.
> --
> 2.7.4
>
> --
Marc-André Lureau

Re: [Qemu-devel] [PATCH v2 4/4] spec/vhost-user: add VHOST_USER_PROTOCOL_F_VERSATILE_SLAVE

[Marc-André Lureau]

Hi

On Thu, Nov 24, 2016 at 7:20 AM Wei Wang <wei.w.wang@intel.com> wrote:

> The VHOST_USER_PROTOCOL_F_VERSATILE_SLAVE protocol feature indicates
> that the slave side implementation supports different types of devices.
> The master tells the slave what type of device to create by sending
> the VHOST_USER_SET_DEV_INFO message.
>
> Signed-off-by: Wei Wang <wei.w.wang@intel.com>
> ---
>  docs/specs/vhost-user.txt | 21 ++++++++++++++++-----
>  1 file changed, 16 insertions(+), 5 deletions(-)
>
> diff --git a/docs/specs/vhost-user.txt b/docs/specs/vhost-user.txt
> index fdc99ea..da1314d 100644
> --- a/docs/specs/vhost-user.txt
> +++ b/docs/specs/vhost-user.txt
> @@ -264,11 +264,12 @@ restarted.
>  Protocol features
>  -----------------
>
> -#define VHOST_USER_PROTOCOL_F_MQ             0
> -#define VHOST_USER_PROTOCOL_F_LOG_SHMFD      1
> -#define VHOST_USER_PROTOCOL_F_RARP           2
> -#define VHOST_USER_PROTOCOL_F_REPLY_ACK      3
> -#define VHOST_USER_PROTOCOL_F_VHOST_PCI      4
> +#define VHOST_USER_PROTOCOL_F_MQ               0
> +#define VHOST_USER_PROTOCOL_F_LOG_SHMFD        1
> +#define VHOST_USER_PROTOCOL_F_RARP             2
> +#define VHOST_USER_PROTOCOL_F_REPLY_ACK        3
> +#define VHOST_USER_PROTOCOL_F_VHOST_PCI        4
> +#define VHOST_USER_PROTOCOL_F_VERSATILE_SLAVE  5
>

 I would rather name it after the message,
VHOST_USER_PROTOCOL_F_SET_DEV_INFO

>
>  Message types
>  -------------
> @@ -514,6 +515,16 @@ Message types
>        #define VHOST_USER_SET_PEER_CONNECTION_F_CREATE    2
>        #define VHOST_USER_SET_PEER_CONNECTION_F_DESTROY   3
>
> + * VHOST_USER_SET_DEV_INFO
> +
> +      Id: 21
> +      Equivalent ioctl: N/A
> +      Master payload: u64
>
+
> +      The master sends the device type info to the slave.
>

What is the meaning of this payload? Is it the virtio device id? better be
explicit about it.

If it is the case, I would name the message "VHOST_USER_SET_DEVICE_ID".



> +      This request should be sent only when
> VHOST_USER_PROTOCOL_F_VERSATILE_SLAVE
> +      has been negotiated.
> +
>  VHOST_USER_PROTOCOL_F_REPLY_ACK:
>  -------------------------------
>  The original vhost-user specification only demands replies for certain
> --
> 2.7.4
>
> --
Marc-André Lureau

Re: [Qemu-devel] [virtio-comment] Re: [PATCH v2 4/4] spec/vhost-user: add VHOST_USER_PROTOCOL_F_VERSATILE_SLAVE

[Wei Wang]

On 11/28/2016 08:41 PM, Marc-André Lureau wrote:
> Hi
>
> On Thu, Nov 24, 2016 at 7:20 AM Wei Wang <wei.w.wang@intel.com 
> <mailto:wei.w.wang@intel.com>> wrote:
>
>     The VHOST_USER_PROTOCOL_F_VERSATILE_SLAVE protocol feature indicates
>     that the slave side implementation supports different types of
>     devices.
>     The master tells the slave what type of device to create by sending
>     the VHOST_USER_SET_DEV_INFO message.
>
>     Signed-off-by: Wei Wang <wei.w.wang@intel.com
>     <mailto:wei.w.wang@intel.com>>
>     ---
>      docs/specs/vhost-user.txt | 21 ++++++++++++++++-----
>      1 file changed, 16 insertions(+), 5 deletions(-)
>
>     diff --git a/docs/specs/vhost-user.txt b/docs/specs/vhost-user.txt
>     index fdc99ea..da1314d 100644
>     --- a/docs/specs/vhost-user.txt
>     +++ b/docs/specs/vhost-user.txt
>     @@ -264,11 +264,12 @@ restarted.
>      Protocol features
>      -----------------
>
>     -#define VHOST_USER_PROTOCOL_F_MQ             0
>     -#define VHOST_USER_PROTOCOL_F_LOG_SHMFD      1
>     -#define VHOST_USER_PROTOCOL_F_RARP           2
>     -#define VHOST_USER_PROTOCOL_F_REPLY_ACK      3
>     -#define VHOST_USER_PROTOCOL_F_VHOST_PCI      4
>     +#define VHOST_USER_PROTOCOL_F_MQ               0
>     +#define VHOST_USER_PROTOCOL_F_LOG_SHMFD        1
>     +#define VHOST_USER_PROTOCOL_F_RARP             2
>     +#define VHOST_USER_PROTOCOL_F_REPLY_ACK        3
>     +#define VHOST_USER_PROTOCOL_F_VHOST_PCI        4
>     +#define VHOST_USER_PROTOCOL_F_VERSATILE_SLAVE  5
>
>
>  I would rather name it after the message, 
> VHOST_USER_PROTOCOL_F_SET_DEV_INFO

OK, I will take this suggestion. Thanks. <*v2-AR1*>
>
>
>      Message types
>      -------------
>     @@ -514,6 +515,16 @@ Message types
>            #define VHOST_USER_SET_PEER_CONNECTION_F_CREATE 2
>            #define VHOST_USER_SET_PEER_CONNECTION_F_DESTROY  3
>
>     + * VHOST_USER_SET_DEV_INFO
>     +
>     +      Id: 21
>     +      Equivalent ioctl: N/A
>     +      Master payload: u64
>
>     +
>     +      The master sends the device type info to the slave.
>
>
> What is the meaning of this payload? Is it the virtio device id? 
> better be explicit about it.
>
> If it is the case, I would name the message "VHOST_USER_SET_DEVICE_ID".

Yes, currently we only have the virtio device id as the payload. I was 
thinking that in the future we would have other more info about the 
device. But no problem, I will change to use "VHOST_USER_SET_DEVICE_ID" 
for now. <*v2-AR2*>


Best,
Wei