* [Qemu-devel] [PATCH 0/6] *** Vhost-pci RFC ***
@ 2016-05-28 23:36 ` Wei Wang
0 siblings, 0 replies; 17+ messages in thread
From: Wei Wang @ 2016-05-28 23:36 UTC (permalink / raw)
To: kvm, qemu-devel, virtio-comment, virtio-dev, mst, stefanha, pbonzini
Cc: Wei Wang
This RFC proposes a design of vhost-pci, which is a new virtio device type.
The vhost-pci device is used for inter-VM communication. Please read the RFC
patches for details.
Wei Wang (6):
Vhost-pci RFC: Introduction
Vhost-pci RFC: Modification Scope
Vhost-pci RFC: Benefits to KVM
Vhost-pci RFC: Detailed Description in the Virtio Specification Format
Vhost-pci RFC: Future Security Enhancement
Vhost-pci RFC: Experimental Results
Benefits | 8 ++
Details | 324 ++++++++++++++++++++++++++++++++++++++++++++++++++++++
FutureWorks | 21 ++++
Introduction | 31 ++++++
ModificationScope | 3 +
Results | 18 +++
6 files changed, 405 insertions(+)
create mode 100644 Benefits
create mode 100644 Details
create mode 100644 FutureWorks
create mode 100644 Introduction
create mode 100644 ModificationScope
create mode 100644 Results
--
1.8.3.1
^ permalink raw reply [flat|nested] 17+ messages in thread
* [PATCH 1/6] Vhost-pci RFC: Introduction
2016-05-28 23:36 ` [Qemu-devel] " Wei Wang
@ 2016-05-28 23:36 ` Wei Wang
-1 siblings, 0 replies; 17+ messages in thread
From: Wei Wang @ 2016-05-28 23:36 UTC (permalink / raw)
To: kvm, qemu-devel, virtio-comment, virtio-dev, mst, stefanha, pbonzini
Cc: Wei Wang
Signed-off-by: Wei Wang <wei.w.wang@intel.com>
---
Introduction | 31 +++++++++++++++++++++++++++++++
1 file changed, 31 insertions(+)
create mode 100644 Introduction
diff --git a/Introduction b/Introduction
new file mode 100644
index 0000000..8774676
--- /dev/null
+++ b/Introduction
@@ -0,0 +1,31 @@
+Current vhost-user based backend designs for virtio-net devices present scaling
+challenges, as communication intensive applications (e.g. virtual network
+functions) running in VMs start to stress this centralized design and resources
+assigned to it.
+
+Vhost-pci was initially proposed by Michael S. Tsirkin with vIOMMU support to
+offer a protected and point-to-point based inter-VM communication. In many
+cases, such as network function virtualization (NFV) and software defined
+networking (SDN) usages, VMs in an isolated network trust each other and they
+may be chained together to complete a task. In these use cases, people care
+more about performance than security. In this RFC we present a comprehensive
+design of vhost-pci without vIOMMU support. A VM with such a vhost-pci device
+is able to copy data to another VM's memory directly. The advantages of using
+vhost-pci over vhost-user are: 1) one less packet copy per packet transfer; and
+2) better scalability.
+
+To follow the naming conventions in the virtio specification, we call the VM
+who sends packets to the destination VM the device VM, and the VM who provides
+the vring and receives packets the driver VM. The vhost-pci device/driver works
+independently in the device VM. It can be considered as a simple device mapping
+the entire memory of a driver VM. But a lot of times, it may be used to
+communicate to a virtio device in the driver VM. That is, it usually plays the
+role of a backend part of a virtio device of a driver VM.
+
+The vhost-pci design is not limited to networking usages. The design presented
+in this RFC is quite fundamental, and it is potentially useful for other
+inter-VM data moving usages. For the convenience of descriptions, we will
+simply use "virtio device" to refer to the device on a driver VM that is backed
+by a vhost-pci device. The figures of this RFC are shown in this link:
+https://etherpad.opnfv.org/p/vhost-pci_RFC
+
--
1.8.3.1
^ permalink raw reply related [flat|nested] 17+ messages in thread
* [Qemu-devel] [PATCH 1/6] Vhost-pci RFC: Introduction
@ 2016-05-28 23:36 ` Wei Wang
0 siblings, 0 replies; 17+ messages in thread
From: Wei Wang @ 2016-05-28 23:36 UTC (permalink / raw)
To: kvm, qemu-devel, virtio-comment, virtio-dev, mst, stefanha, pbonzini
Cc: Wei Wang
Signed-off-by: Wei Wang <wei.w.wang@intel.com>
---
Introduction | 31 +++++++++++++++++++++++++++++++
1 file changed, 31 insertions(+)
create mode 100644 Introduction
diff --git a/Introduction b/Introduction
new file mode 100644
index 0000000..8774676
--- /dev/null
+++ b/Introduction
@@ -0,0 +1,31 @@
+Current vhost-user based backend designs for virtio-net devices present scaling
+challenges, as communication intensive applications (e.g. virtual network
+functions) running in VMs start to stress this centralized design and resources
+assigned to it.
+
+Vhost-pci was initially proposed by Michael S. Tsirkin with vIOMMU support to
+offer a protected and point-to-point based inter-VM communication. In many
+cases, such as network function virtualization (NFV) and software defined
+networking (SDN) usages, VMs in an isolated network trust each other and they
+may be chained together to complete a task. In these use cases, people care
+more about performance than security. In this RFC we present a comprehensive
+design of vhost-pci without vIOMMU support. A VM with such a vhost-pci device
+is able to copy data to another VM's memory directly. The advantages of using
+vhost-pci over vhost-user are: 1) one less packet copy per packet transfer; and
+2) better scalability.
+
+To follow the naming conventions in the virtio specification, we call the VM
+who sends packets to the destination VM the device VM, and the VM who provides
+the vring and receives packets the driver VM. The vhost-pci device/driver works
+independently in the device VM. It can be considered as a simple device mapping
+the entire memory of a driver VM. But a lot of times, it may be used to
+communicate to a virtio device in the driver VM. That is, it usually plays the
+role of a backend part of a virtio device of a driver VM.
+
+The vhost-pci design is not limited to networking usages. The design presented
+in this RFC is quite fundamental, and it is potentially useful for other
+inter-VM data moving usages. For the convenience of descriptions, we will
+simply use "virtio device" to refer to the device on a driver VM that is backed
+by a vhost-pci device. The figures of this RFC are shown in this link:
+https://etherpad.opnfv.org/p/vhost-pci_RFC
+
--
1.8.3.1
^ permalink raw reply related [flat|nested] 17+ messages in thread
* [PATCH 2/6] Vhost-pci RFC: Modification Scope
2016-05-28 23:36 ` [Qemu-devel] " Wei Wang
@ 2016-05-28 23:36 ` Wei Wang
-1 siblings, 0 replies; 17+ messages in thread
From: Wei Wang @ 2016-05-28 23:36 UTC (permalink / raw)
To: kvm, qemu-devel, virtio-comment, virtio-dev, mst, stefanha, pbonzini
Cc: Wei Wang
Signed-off-by: Wei Wang <wei.w.wang@intel.com>
---
ModificationScope | 3 +++
1 file changed, 3 insertions(+)
create mode 100644 ModificationScope
diff --git a/ModificationScope b/ModificationScope
new file mode 100644
index 0000000..ea949ee
--- /dev/null
+++ b/ModificationScope
@@ -0,0 +1,3 @@
+Changes to: QEMU, Guest kernel
+Affects specific archs: x86
+Affects specific guests: Linux
--
1.8.3.1
^ permalink raw reply related [flat|nested] 17+ messages in thread
* [Qemu-devel] [PATCH 2/6] Vhost-pci RFC: Modification Scope
@ 2016-05-28 23:36 ` Wei Wang
0 siblings, 0 replies; 17+ messages in thread
From: Wei Wang @ 2016-05-28 23:36 UTC (permalink / raw)
To: kvm, qemu-devel, virtio-comment, virtio-dev, mst, stefanha, pbonzini
Cc: Wei Wang
Signed-off-by: Wei Wang <wei.w.wang@intel.com>
---
ModificationScope | 3 +++
1 file changed, 3 insertions(+)
create mode 100644 ModificationScope
diff --git a/ModificationScope b/ModificationScope
new file mode 100644
index 0000000..ea949ee
--- /dev/null
+++ b/ModificationScope
@@ -0,0 +1,3 @@
+Changes to: QEMU, Guest kernel
+Affects specific archs: x86
+Affects specific guests: Linux
--
1.8.3.1
^ permalink raw reply related [flat|nested] 17+ messages in thread
* [PATCH 3/6] Vhost-pci RFC: Benefits to KVM
2016-05-28 23:36 ` [Qemu-devel] " Wei Wang
@ 2016-05-28 23:36 ` Wei Wang
-1 siblings, 0 replies; 17+ messages in thread
From: Wei Wang @ 2016-05-28 23:36 UTC (permalink / raw)
To: kvm, qemu-devel, virtio-comment, virtio-dev, mst, stefanha, pbonzini
Cc: Wei Wang
Signed-off-by: Wei Wang <wei.w.wang@intel.com>
---
Benefits | 8 ++++++++
1 file changed, 8 insertions(+)
create mode 100644 Benefits
diff --git a/Benefits b/Benefits
new file mode 100644
index 0000000..7d10485
--- /dev/null
+++ b/Benefits
@@ -0,0 +1,8 @@
+The vhost-pci design provides KVM with a faster and more scalable inter-VM
+communication mechanism. In NFV usages and today's virtualized datacenters,
+where multiple VMs connect to a centralized vswitch to communicate, the more
+the VMs connect to (or chained via) the vswitch, the lower the inter-VM
+communication throughput will be. The vhost-pci design successfully resolves
+this scalability issue and makes KVM a better hypervisor for NFV and
+datacenters.
+
--
1.8.3.1
^ permalink raw reply related [flat|nested] 17+ messages in thread
* [Qemu-devel] [PATCH 3/6] Vhost-pci RFC: Benefits to KVM
@ 2016-05-28 23:36 ` Wei Wang
0 siblings, 0 replies; 17+ messages in thread
From: Wei Wang @ 2016-05-28 23:36 UTC (permalink / raw)
To: kvm, qemu-devel, virtio-comment, virtio-dev, mst, stefanha, pbonzini
Cc: Wei Wang
Signed-off-by: Wei Wang <wei.w.wang@intel.com>
---
Benefits | 8 ++++++++
1 file changed, 8 insertions(+)
create mode 100644 Benefits
diff --git a/Benefits b/Benefits
new file mode 100644
index 0000000..7d10485
--- /dev/null
+++ b/Benefits
@@ -0,0 +1,8 @@
+The vhost-pci design provides KVM with a faster and more scalable inter-VM
+communication mechanism. In NFV usages and today's virtualized datacenters,
+where multiple VMs connect to a centralized vswitch to communicate, the more
+the VMs connect to (or chained via) the vswitch, the lower the inter-VM
+communication throughput will be. The vhost-pci design successfully resolves
+this scalability issue and makes KVM a better hypervisor for NFV and
+datacenters.
+
--
1.8.3.1
^ permalink raw reply related [flat|nested] 17+ messages in thread
* [PATCH 4/6] Vhost-pci RFC: Detailed Description in the Virtio Specification Format
2016-05-28 23:36 ` [Qemu-devel] " Wei Wang
@ 2016-05-28 23:36 ` Wei Wang
-1 siblings, 0 replies; 17+ messages in thread
From: Wei Wang @ 2016-05-28 23:36 UTC (permalink / raw)
To: kvm, qemu-devel, virtio-comment, virtio-dev, mst, stefanha, pbonzini
Cc: Wei Wang
Signed-off-by: Wei Wang <wei.w.wang@intel.com>
---
Details | 324 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
1 file changed, 324 insertions(+)
create mode 100644 Details
diff --git a/Details b/Details
new file mode 100644
index 0000000..4ea2252
--- /dev/null
+++ b/Details
@@ -0,0 +1,324 @@
+1 Device ID
+TBD
+
+2 Virtqueues
+0 controlq
+
+3 Feature Bits
+3.1 Local Feature Bits
+Currently no local feature bits are defined, so the standard virtio feature
+bits negation will always be successful and complete.
+
+3.2 Remote Feature Bits
+The remote feature bits are obtained from the frontend virtio device and
+negotiated with the vhost-pci driver via the controlq. The negotiation steps
+are described in 4.5 Device Initialization.
+
+4 Device Configuration Layout
+struct vhost_pci_config {
+ #define VHOST_PCI_CONTROLQ_MEMORY_INFO_ACK 0
+ #define VHOST_PCI_CONTROLQ_DEVICE_INFO_ACK 1
+ #define VHOST_PCI_CONTROLQ_FEATURE_BITS_ACK 2
+ u32 ack_type;
+ u32 ack_device_type;
+ u64 ack_device_id;
+ union {
+ #define VHOST_PCI_CONTROLQ_ACK_ADD_DONE 0
+ #define VHOST_PCI_CONTROLQ_ACK_ADD_FAIL 1
+ #define VHOST_PCI_CONTROLQ_ACK_DEL_DONE 2
+ #define VHOST_PCI_CONTROLQ_ACK_DEL_FAIL 3
+ u64 ack_memory_info;
+ u64 ack_device_info;
+ u64 ack_feature_bits;
+ };
+};
+
+The configuration fields are currently used for the vhost-pci driver to
+acknowledge to the vhost-pci device after it receives controlq messages.
+
+4.5 Device Initialization
+When a device VM boots, it creates a vhost-pci server socket.
+
+When a virtio device on the driver VM is created with specifying the use of a
+vhost-pci device as a backend, a client socket is created and connected to the
+corresponding vhost-pci server for message exchanges.
+
+The messages passed to the vhost-pci server is proceeded by the following
+header:
+struct vhost_pci_socket_hdr {
+ #define VHOST_PCI_SOCKET_MEMORY_INFO 0
+ #define VHOST_PCI_SOCKET_MEMORY_INFO_ACK 1
+ #define VHOST_PCI_SOCKET_DEVICE_INFO 2
+ #define VHOST_PCI_SOCKET_DEVICE_INFO_ACK 3
+ #define VHOST_PCI_SOCKET_FEATURE_BITS 4
+ #define VHOST_PCI_SOCKET_FEATURE_BITS_ACK 5
+ u16 msg_type;
+ u16 msg_version;
+ u32 msg_len;
+ u64 qemu_pid;
+};
+
+The payload of the above message types can be constructed using the structures
+below:
+/* VHOST_PCI_SOCKET_MEMORY_INFO message */
+struct vhost_pci_socket_memory_info {
+ #define VHOST_PCI_ADD_MEMORY 0
+ #define VHOST_PCI_DEL_MEMORY 1
+ u16 ops;
+ u32 nregions;
+ struct vhost_pci_memory_region {
+ int fd;
+ u64 guest_phys_addr;
+ u64 memory_size;
+ u64 mmap_offset;
+ } regions[VHOST_PCI_MAX_NREGIONS];
+};
+
+/* VHOST_PCI_SOCKET_DEVICE_INFO message */
+struct vhost_pci_device_info {
+ #define VHOST_PCI_ADD_FRONTEND_DEVICE 0
+ #define VHOST_PCI_DEL_FRONTEND_DEVICE 1
+ u16 ops;
+ u32 nvirtq;
+ #define VHOST_PCI_FRONTEND_DEVICE_NET 1
+ #define VHOST_PCI_FRONTEND_DEVICE_BLK 2
+ #define VHOST_PCI_FRONTEND_DEVICE_CONSOLE 3
+ #define VHOST_PCI_FRONTEND_DEVICE_ENTROPY 4
+ #define VHOST_PCI_FRONTEND_DEVICE_BALLOON 5
+ #define VHOST_PCI_FRONTEND_DEVICE_SCSI 8
+ u32 device_type;
+ u64 device_id;
+ struct virtq exotic_virtq[VHOST_PCI_MAX_NVIRTQ];
+};
+The device_id field identifies the device. For example, it can be used to
+store a MAC address if the device_type is VHOST_PCI_FRONTEND_DEVICE_NET.
+
+/* VHOST_PCI_SOCKET_FEATURE_BITS message*/
+struct vhost_pci_feature_bits {
+ u64 feature_bits;
+};
+
+/* VHOST_PCI_SOCKET_xx_ACK messages */
+struct vhost_pci_socket_ack {
+ #define VHOST_PCI_SOCKET_ACK_ADD_DONE 0
+ #define VHOST_PCI_SOCKET_ACK_ADD_FAIL 1
+ #define VHOST_PCI_SOCKET_ACK_DEL_DONE 2
+ #define VHOST_PCI_SOCKET_ACK_DEL_FAIL 3
+ u64 ack;
+};
+
+The driver update message passed via the controlq is preceded by the following
+header:
+struct vhost_pci_controlq_hdr {
+ #define VHOST_PCI_CONTROLQ_MEMORY_INFO 0
+ #define VHOST_PCI_CONTROLQ_DEVICE_INFO 1
+ #define VHOST_PCI_CONTROLQ_FEATURE_BITS 2
+ #define VHOST_PCI_CONTROLQ_UPDATE_DONE 3
+ u16 msg_type;
+ u16 msg_version;
+ u32 msg_len;
+};
+
+The payload of a VHOST_PCI_CONTROLQ_MEMORY_INFO message can be constructed
+using the following structure:
+/* VHOST_PCI_CONTROLQ_MEMORY_INFO message */
+struct vhost_pci_controlq_memory_info {
+ #define VHOST_PCI_ADD_MEMORY 0
+ #define VHOST_PCI_DEL_MEMORY 1
+ u16 ops;
+ u32 nregion;
+ struct exotic_memory_region {
+ u64 region_base_xgpa;
+ u64 size;
+ u64 offset_in_bar_area;
+ } region[VHOST_PCI_MAX_NREGIONS];
+};
+
+The payload of VHOST_PCI_CONTROLQ_DEVICE_INFO and
+VHOST_PCI_CONTROLQ_FEATURE_BITS messages can be constructed using the
+vhost_pci_device_info structure and the vhost_pci_feature_bits structure
+respectively.
+
+The payload of a VHOST_PCI_CONTROLQ_UPDATE_DONE message can be constructed
+using the structure below:
+struct vhost_pci_controlq_update_done {
+ u32 device_type;
+ u64 device_id;
+};
+
+Fig. 1 shows the initialization steps.
+
+When the vhost-pci server receives a VHOST_PCI_SOCKET_MEMORY_INFO(ADD) message,
+it checks if a vhost-pci device has been created for the requesting VM whose
+QEMU process id is qemu_pid. If yes, it will simply update the subsequent
+received messages to the vhost-pci driver via the controlq. Otherwise, the
+server creates a new vhost-pci device, and continues the following
+initialization steps.
+
+The vhost-pci server adds up all the memory region size, and uses a 64-bit
+device bar for the mapping of all the memory regions obtained from the socket
+message. To better support memory hot-plugging of the driver VM, the bar is
+configured with a double size of the driver VM's memory. The server maps the
+received memory info via the QEMU MemoryRegion mechanism, and then the new
+created vhost-pci device is hot-plugged to the VM.
+
+When the device status is updated with DRIVER_OK, a
+VHOST_PCI_CONTROLQ_MEMORY_INFO(ADD) message, which is stemed from the memory
+info socket message, is put on the controlq and a controlq interrupt is injected
+to the VM.
+
+When the vhost-pci server receives a
+VHOST_PCI_CONTROLQ_MEMORY_INFO_ACK(ADD_DONE) acknowledgement from the driver,
+it sends a VHOST_PCI_SOCKET_MEMORY_INFO_ACK(ADD_DONE) message to the client
+that is identified by the ack_device_type and ack_device_id fields.
+
+When the vhost-pci server receives a
+VHOST_PCI_SOCKET_FEATURE_BITS(feature bits) message, a
+VHOST_PCI_CONTROLQ_FEATURE_BITS(feature bits) message is put on the controlq
+and a controlq interrupt is injected to the VM.
+
+If the vhost-pci server notices that the driver fully accepted the offered
+feature bits, it sends a VHOST_PCI_SOCKET_FEATURE_BITS_ACK(ADD_DONE) message
+to the client. If the vhost-pci server notices that the vhost-pci driver only
+accepted a subset of the offered feature bits, it sends a
+VHOST_PCI_SOCKET_FEATURE_BITS(accepted feature bits) message back to the
+client. The client side virtio device re-negotiates the new feature bits with
+its driver, and sends back a VHOST_PCI_SOCKET_FEATURE_BITS_ACK(ADD_DONE)
+message to the server.
+
+Either when the vhost-pci driver fully accepted the offered feature bits or a
+VHOST_PCI_SOCKET_FEATURE_BITS_ACK(ADD_DONE) message is received from the
+client, the vhost-pci server puts a VHOST_PCI_CONTROLQ_UPDATE_DONE message on
+the controlq, and a controlq interrupt is injected to the VM.
+
+When the vhost-pci server receives a VHOST_PCI_SOCKET_DEVICE_INFO(ADD) message,
+a VHOST_PCI_CONTROLQ_DEVICE_INFO(ADD) message is put on the controlq and a
+controlq interrupt is injected to the VM.
+
+When the vhost-pci server receives a
+VHOST_PCI_CONTROLQ_DEVICE_INFO_ACK(ADD_DONE) acknowledgement from the driver,
+it sends a VHOST_PCI_SOCKET_DEVICE_INFO_ACK(ADD_DONE) message to the
+corresponding client.
+
+4.5.1 Device Requirements: Device Initialization
+To let a VM be capable of creating vhost-pci devices, a vhost-pci server MUST
+be created when it boots.
+
+The vhost-pci server socket path SHOULD be provided to a virtio client socket
+for the connection to the vhost-pci server.
+
+The virtio device MUST finish the feature bits negotiation with its driver
+before negotiating them with the vhost-pci device.
+
+If the client receives a VHOST_PCI_SOCKET_FEATURE_BITS(feature bits) message,
+it MUST reset the device to go into backwards capability mode, re-negotiate
+the received feature bits with its driver, and send back a
+VHOST_PCI_SOCKET_FEATURE_BITS_ACK(ADD_DONE) message to the server.
+
+In any cases that an acknowledgement from the vhost-pci driver indicates a
+FAIL, the vhost-pci server SHOULD send a FAIL socket message to the client.
+
+In any cases that the msg_type is different between the sender and the
+receiver, the receiver SHOULD acknowledge a FAIL to the sender or convert the
+message to its version if the converted version is still functionally usable.
+
+4.5.2 Driver Requirements: Device Initialization
+The vhost-pci driver MUST NOT accept any feature bits that are not offered by
+the remote feature bits, and SHOULD acknowledge to the device of the accepted
+feature bits by writing them to the vhost_pci_config fields.
+
+When the vhost-pci driver receives a VHOST_PCI_CONTROLQ_UPDATE_DONE message
+from the controlq, the vhost-pci driver MUST initialize the corresponding
+driver interface of the device_type if it has not been initialized, and add
+the device_id to the frontend device list that records all the frontend virtio
+devices being supported by vhost-pci for inter-VM communications.
+
+The vhost-pci driver SHOULD acknowledge to the device that the device and
+memory info update (add or delete) is DONE or FAIL by writing the
+acknowledgement (DONE or FAIL) to the vhost_pci_config fields.
+
+The vhost-pci driver MUST ensure that writing to the vhost_pci_config fields
+to be atomic.
+
+4.6 Device Operation
+4.6.1 Device Requirements: Device Operation
+4.6.1.1 Frontend Device Info Update
+When the frontend virtio device changes any info (e.g. device_id, virtq
+address) that it has sent to the vhost-pci device, it SHOULD send a
+VHOST_PCI_SOCKET_DEVICE_INFO(ADD) message, which contains the new device info,
+to the vhost-pci server. The vhost-pci device SHOULD insert a
+VHOST_PCI_CONTROLQ_DEVICE_INFO(ADD) to the controlq and inject a contrlq
+interrupt to the VM.
+
+When the vhost-pci device receives a
+VHOST_PCI_CONTROLQ_DEVICE_INFO_ACK(ADD_DONE) acknowledgement from the driver,
+it SHOULD send a VHOST_PCI_SOCKET_DEVICE_INFO_ACK(ADD_DONE) message to the
+client that is identified by the ack_device_type and ack_device_id fields, to
+indicate that the vhost-pci driver has finished the handling of the device
+info update.
+
+4.6.1.2 Frontend Device Remove
+When the frontend virtio device is removed (e.g. hot-plug out), the client
+SHOULD send a VHOST_PCI_SOCKET_DEVICE_INFO(DEL) message to the vhost-pci
+server. The vhost-pci device SHOULD put a VHOST_PCI_CONTROLQ_DEVICE_INFO(DEL)
+message on the controlq and inject a contrlq interrupt to the VM.
+
+When the vhost-pci receives a VHOST_PCI_CONTROLQ_DEVICE_INFO_ACK(DEL_DONE), it
+SHOULD send a VHOST_PCI_SOCKET_DEVICE_INFO_ACK(DEL_DONE) message to the
+corresponding client to indicate that the vhost-pci driver has removed the
+vhost-pci based inter-VM communication support for the requesting virtio
+device.
+
+4.6.1.3 Driver VM Shutdown and Migration
+Before the driver VM is destroyed or migrated, all the clients that connect to
+the vhost-pci server SHOULD send a VHOST_PCI_SOCKET_DEVICE_INFO(DEL) message to
+the vhost-pci server. The destroying or migrating activity MUST wait until all
+the VHOST_PCI_SOCKET_DEL_CONNECTION_ACK(DEL_DONE) messages are received.
+
+When a vhost-pci device has no frontend devices, the vhost-pci device SHOULD be
+destroyed.
+
+4.6.1.4 Driver VM Memory Hot-plug
+When the vhost-pci server receives a VHOST_PCI_SOCKET_MEMORY_INFO(DEL) message,
+a VHOST_PCI_CONTROLQ_MEMORY_INFO(DEL) message SHOULD be put on the controlq and
+a controlq interrupt is injected to the VM. When the vhost-pci server receives
+a VHOST_PCI_CONTROLQ_MEMORY_INFO_ACK(DEL_DONE) acknowledgement from the driver,
+it SHOULD unmap that memory region and send a
+VHOST_PCI_SOCKET_MEMORY_INFO_ACK(DEL_DONE) message to the client.
+
+When the vhost-pci server receives a VHOST_PCI_SOCKET_MEMORY_INFO(ADD) message,
+and the received memory info is new to what has already been mapped, it
+calculates the total received memory size.
+
+If the new memory size plus the mapped memory size is smaller than the address
+space size reserved by the bar, the server SHOULD map the new memory and expose
+it to the VM via the QEMU MemoryRegion mechanism. Then it SHOULD put the new
+memory info on the controlq, and injects a controlq interrupt to the VM.
+
+If the new memory size plus the mapped memory size is larger than the address
+space size reserved by the bar, the server clones out a new vhost-pci device,
+configures the bar size to be double of the current memory, hot-plugs out the
+old vhost-pci device, and hot-plugs in the new vhost-pci device to the VM. The
+initialization steps SHOULD follow 4.5 Device Initialization, except the
+interaction between the server and client is not needed.
+
+When the vhost-pci server receives a
+VHOST_PCI_CONTROLQ_MEMORY_INFO_ACK(ADD_DONE) acknowledgement from the driver,
+it SHOULD send a VHOST_PCI_SOCKET_MEMORY_INFO_ACK(ADD_DONE) message to the
+client.
+
+4.6.2 Driver Requirements: Device Operation
+The vhost-pci driver SHOULD acknowledge to the vhost-pci device by writing
+VHOST_PCI_CONTROLQ_DEVICE_INFO_ACK(ADD_DONE) to the vhost_pci_config fields
+when it finishes handling the device info update.
+
+The vhost-pci driver SHOULD ensure that all the CPUs are noticed about the
+device info update before acknowledging to the vhost-pci device.
+
+The vhost-pci driver SHOULD acknowledge to the vhost-pci device by writing
+VHOST_PCI_CONTROLQ_DEVICE_INFO_ACK(DEL_DONE) to vhost_pci_config fields when
+it finishes removing the vhost-pci support for the requesting virtio device.
+
+The vhost-pci driver SHOULD ensure that all the CPUs are noticed about the
+removing of the vhost-pci support for the requesting virtio device before
+acknowledging to the vhost-pci device.
--
1.8.3.1
^ permalink raw reply related [flat|nested] 17+ messages in thread* [Qemu-devel] [PATCH 4/6] Vhost-pci RFC: Detailed Description in the Virtio Specification Format
@ 2016-05-28 23:36 ` Wei Wang
0 siblings, 0 replies; 17+ messages in thread
From: Wei Wang @ 2016-05-28 23:36 UTC (permalink / raw)
To: kvm, qemu-devel, virtio-comment, virtio-dev, mst, stefanha, pbonzini
Cc: Wei Wang
Signed-off-by: Wei Wang <wei.w.wang@intel.com>
---
Details | 324 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
1 file changed, 324 insertions(+)
create mode 100644 Details
diff --git a/Details b/Details
new file mode 100644
index 0000000..4ea2252
--- /dev/null
+++ b/Details
@@ -0,0 +1,324 @@
+1 Device ID
+TBD
+
+2 Virtqueues
+0 controlq
+
+3 Feature Bits
+3.1 Local Feature Bits
+Currently no local feature bits are defined, so the standard virtio feature
+bits negation will always be successful and complete.
+
+3.2 Remote Feature Bits
+The remote feature bits are obtained from the frontend virtio device and
+negotiated with the vhost-pci driver via the controlq. The negotiation steps
+are described in 4.5 Device Initialization.
+
+4 Device Configuration Layout
+struct vhost_pci_config {
+ #define VHOST_PCI_CONTROLQ_MEMORY_INFO_ACK 0
+ #define VHOST_PCI_CONTROLQ_DEVICE_INFO_ACK 1
+ #define VHOST_PCI_CONTROLQ_FEATURE_BITS_ACK 2
+ u32 ack_type;
+ u32 ack_device_type;
+ u64 ack_device_id;
+ union {
+ #define VHOST_PCI_CONTROLQ_ACK_ADD_DONE 0
+ #define VHOST_PCI_CONTROLQ_ACK_ADD_FAIL 1
+ #define VHOST_PCI_CONTROLQ_ACK_DEL_DONE 2
+ #define VHOST_PCI_CONTROLQ_ACK_DEL_FAIL 3
+ u64 ack_memory_info;
+ u64 ack_device_info;
+ u64 ack_feature_bits;
+ };
+};
+
+The configuration fields are currently used for the vhost-pci driver to
+acknowledge to the vhost-pci device after it receives controlq messages.
+
+4.5 Device Initialization
+When a device VM boots, it creates a vhost-pci server socket.
+
+When a virtio device on the driver VM is created with specifying the use of a
+vhost-pci device as a backend, a client socket is created and connected to the
+corresponding vhost-pci server for message exchanges.
+
+The messages passed to the vhost-pci server is proceeded by the following
+header:
+struct vhost_pci_socket_hdr {
+ #define VHOST_PCI_SOCKET_MEMORY_INFO 0
+ #define VHOST_PCI_SOCKET_MEMORY_INFO_ACK 1
+ #define VHOST_PCI_SOCKET_DEVICE_INFO 2
+ #define VHOST_PCI_SOCKET_DEVICE_INFO_ACK 3
+ #define VHOST_PCI_SOCKET_FEATURE_BITS 4
+ #define VHOST_PCI_SOCKET_FEATURE_BITS_ACK 5
+ u16 msg_type;
+ u16 msg_version;
+ u32 msg_len;
+ u64 qemu_pid;
+};
+
+The payload of the above message types can be constructed using the structures
+below:
+/* VHOST_PCI_SOCKET_MEMORY_INFO message */
+struct vhost_pci_socket_memory_info {
+ #define VHOST_PCI_ADD_MEMORY 0
+ #define VHOST_PCI_DEL_MEMORY 1
+ u16 ops;
+ u32 nregions;
+ struct vhost_pci_memory_region {
+ int fd;
+ u64 guest_phys_addr;
+ u64 memory_size;
+ u64 mmap_offset;
+ } regions[VHOST_PCI_MAX_NREGIONS];
+};
+
+/* VHOST_PCI_SOCKET_DEVICE_INFO message */
+struct vhost_pci_device_info {
+ #define VHOST_PCI_ADD_FRONTEND_DEVICE 0
+ #define VHOST_PCI_DEL_FRONTEND_DEVICE 1
+ u16 ops;
+ u32 nvirtq;
+ #define VHOST_PCI_FRONTEND_DEVICE_NET 1
+ #define VHOST_PCI_FRONTEND_DEVICE_BLK 2
+ #define VHOST_PCI_FRONTEND_DEVICE_CONSOLE 3
+ #define VHOST_PCI_FRONTEND_DEVICE_ENTROPY 4
+ #define VHOST_PCI_FRONTEND_DEVICE_BALLOON 5
+ #define VHOST_PCI_FRONTEND_DEVICE_SCSI 8
+ u32 device_type;
+ u64 device_id;
+ struct virtq exotic_virtq[VHOST_PCI_MAX_NVIRTQ];
+};
+The device_id field identifies the device. For example, it can be used to
+store a MAC address if the device_type is VHOST_PCI_FRONTEND_DEVICE_NET.
+
+/* VHOST_PCI_SOCKET_FEATURE_BITS message*/
+struct vhost_pci_feature_bits {
+ u64 feature_bits;
+};
+
+/* VHOST_PCI_SOCKET_xx_ACK messages */
+struct vhost_pci_socket_ack {
+ #define VHOST_PCI_SOCKET_ACK_ADD_DONE 0
+ #define VHOST_PCI_SOCKET_ACK_ADD_FAIL 1
+ #define VHOST_PCI_SOCKET_ACK_DEL_DONE 2
+ #define VHOST_PCI_SOCKET_ACK_DEL_FAIL 3
+ u64 ack;
+};
+
+The driver update message passed via the controlq is preceded by the following
+header:
+struct vhost_pci_controlq_hdr {
+ #define VHOST_PCI_CONTROLQ_MEMORY_INFO 0
+ #define VHOST_PCI_CONTROLQ_DEVICE_INFO 1
+ #define VHOST_PCI_CONTROLQ_FEATURE_BITS 2
+ #define VHOST_PCI_CONTROLQ_UPDATE_DONE 3
+ u16 msg_type;
+ u16 msg_version;
+ u32 msg_len;
+};
+
+The payload of a VHOST_PCI_CONTROLQ_MEMORY_INFO message can be constructed
+using the following structure:
+/* VHOST_PCI_CONTROLQ_MEMORY_INFO message */
+struct vhost_pci_controlq_memory_info {
+ #define VHOST_PCI_ADD_MEMORY 0
+ #define VHOST_PCI_DEL_MEMORY 1
+ u16 ops;
+ u32 nregion;
+ struct exotic_memory_region {
+ u64 region_base_xgpa;
+ u64 size;
+ u64 offset_in_bar_area;
+ } region[VHOST_PCI_MAX_NREGIONS];
+};
+
+The payload of VHOST_PCI_CONTROLQ_DEVICE_INFO and
+VHOST_PCI_CONTROLQ_FEATURE_BITS messages can be constructed using the
+vhost_pci_device_info structure and the vhost_pci_feature_bits structure
+respectively.
+
+The payload of a VHOST_PCI_CONTROLQ_UPDATE_DONE message can be constructed
+using the structure below:
+struct vhost_pci_controlq_update_done {
+ u32 device_type;
+ u64 device_id;
+};
+
+Fig. 1 shows the initialization steps.
+
+When the vhost-pci server receives a VHOST_PCI_SOCKET_MEMORY_INFO(ADD) message,
+it checks if a vhost-pci device has been created for the requesting VM whose
+QEMU process id is qemu_pid. If yes, it will simply update the subsequent
+received messages to the vhost-pci driver via the controlq. Otherwise, the
+server creates a new vhost-pci device, and continues the following
+initialization steps.
+
+The vhost-pci server adds up all the memory region size, and uses a 64-bit
+device bar for the mapping of all the memory regions obtained from the socket
+message. To better support memory hot-plugging of the driver VM, the bar is
+configured with a double size of the driver VM's memory. The server maps the
+received memory info via the QEMU MemoryRegion mechanism, and then the new
+created vhost-pci device is hot-plugged to the VM.
+
+When the device status is updated with DRIVER_OK, a
+VHOST_PCI_CONTROLQ_MEMORY_INFO(ADD) message, which is stemed from the memory
+info socket message, is put on the controlq and a controlq interrupt is injected
+to the VM.
+
+When the vhost-pci server receives a
+VHOST_PCI_CONTROLQ_MEMORY_INFO_ACK(ADD_DONE) acknowledgement from the driver,
+it sends a VHOST_PCI_SOCKET_MEMORY_INFO_ACK(ADD_DONE) message to the client
+that is identified by the ack_device_type and ack_device_id fields.
+
+When the vhost-pci server receives a
+VHOST_PCI_SOCKET_FEATURE_BITS(feature bits) message, a
+VHOST_PCI_CONTROLQ_FEATURE_BITS(feature bits) message is put on the controlq
+and a controlq interrupt is injected to the VM.
+
+If the vhost-pci server notices that the driver fully accepted the offered
+feature bits, it sends a VHOST_PCI_SOCKET_FEATURE_BITS_ACK(ADD_DONE) message
+to the client. If the vhost-pci server notices that the vhost-pci driver only
+accepted a subset of the offered feature bits, it sends a
+VHOST_PCI_SOCKET_FEATURE_BITS(accepted feature bits) message back to the
+client. The client side virtio device re-negotiates the new feature bits with
+its driver, and sends back a VHOST_PCI_SOCKET_FEATURE_BITS_ACK(ADD_DONE)
+message to the server.
+
+Either when the vhost-pci driver fully accepted the offered feature bits or a
+VHOST_PCI_SOCKET_FEATURE_BITS_ACK(ADD_DONE) message is received from the
+client, the vhost-pci server puts a VHOST_PCI_CONTROLQ_UPDATE_DONE message on
+the controlq, and a controlq interrupt is injected to the VM.
+
+When the vhost-pci server receives a VHOST_PCI_SOCKET_DEVICE_INFO(ADD) message,
+a VHOST_PCI_CONTROLQ_DEVICE_INFO(ADD) message is put on the controlq and a
+controlq interrupt is injected to the VM.
+
+When the vhost-pci server receives a
+VHOST_PCI_CONTROLQ_DEVICE_INFO_ACK(ADD_DONE) acknowledgement from the driver,
+it sends a VHOST_PCI_SOCKET_DEVICE_INFO_ACK(ADD_DONE) message to the
+corresponding client.
+
+4.5.1 Device Requirements: Device Initialization
+To let a VM be capable of creating vhost-pci devices, a vhost-pci server MUST
+be created when it boots.
+
+The vhost-pci server socket path SHOULD be provided to a virtio client socket
+for the connection to the vhost-pci server.
+
+The virtio device MUST finish the feature bits negotiation with its driver
+before negotiating them with the vhost-pci device.
+
+If the client receives a VHOST_PCI_SOCKET_FEATURE_BITS(feature bits) message,
+it MUST reset the device to go into backwards capability mode, re-negotiate
+the received feature bits with its driver, and send back a
+VHOST_PCI_SOCKET_FEATURE_BITS_ACK(ADD_DONE) message to the server.
+
+In any cases that an acknowledgement from the vhost-pci driver indicates a
+FAIL, the vhost-pci server SHOULD send a FAIL socket message to the client.
+
+In any cases that the msg_type is different between the sender and the
+receiver, the receiver SHOULD acknowledge a FAIL to the sender or convert the
+message to its version if the converted version is still functionally usable.
+
+4.5.2 Driver Requirements: Device Initialization
+The vhost-pci driver MUST NOT accept any feature bits that are not offered by
+the remote feature bits, and SHOULD acknowledge to the device of the accepted
+feature bits by writing them to the vhost_pci_config fields.
+
+When the vhost-pci driver receives a VHOST_PCI_CONTROLQ_UPDATE_DONE message
+from the controlq, the vhost-pci driver MUST initialize the corresponding
+driver interface of the device_type if it has not been initialized, and add
+the device_id to the frontend device list that records all the frontend virtio
+devices being supported by vhost-pci for inter-VM communications.
+
+The vhost-pci driver SHOULD acknowledge to the device that the device and
+memory info update (add or delete) is DONE or FAIL by writing the
+acknowledgement (DONE or FAIL) to the vhost_pci_config fields.
+
+The vhost-pci driver MUST ensure that writing to the vhost_pci_config fields
+to be atomic.
+
+4.6 Device Operation
+4.6.1 Device Requirements: Device Operation
+4.6.1.1 Frontend Device Info Update
+When the frontend virtio device changes any info (e.g. device_id, virtq
+address) that it has sent to the vhost-pci device, it SHOULD send a
+VHOST_PCI_SOCKET_DEVICE_INFO(ADD) message, which contains the new device info,
+to the vhost-pci server. The vhost-pci device SHOULD insert a
+VHOST_PCI_CONTROLQ_DEVICE_INFO(ADD) to the controlq and inject a contrlq
+interrupt to the VM.
+
+When the vhost-pci device receives a
+VHOST_PCI_CONTROLQ_DEVICE_INFO_ACK(ADD_DONE) acknowledgement from the driver,
+it SHOULD send a VHOST_PCI_SOCKET_DEVICE_INFO_ACK(ADD_DONE) message to the
+client that is identified by the ack_device_type and ack_device_id fields, to
+indicate that the vhost-pci driver has finished the handling of the device
+info update.
+
+4.6.1.2 Frontend Device Remove
+When the frontend virtio device is removed (e.g. hot-plug out), the client
+SHOULD send a VHOST_PCI_SOCKET_DEVICE_INFO(DEL) message to the vhost-pci
+server. The vhost-pci device SHOULD put a VHOST_PCI_CONTROLQ_DEVICE_INFO(DEL)
+message on the controlq and inject a contrlq interrupt to the VM.
+
+When the vhost-pci receives a VHOST_PCI_CONTROLQ_DEVICE_INFO_ACK(DEL_DONE), it
+SHOULD send a VHOST_PCI_SOCKET_DEVICE_INFO_ACK(DEL_DONE) message to the
+corresponding client to indicate that the vhost-pci driver has removed the
+vhost-pci based inter-VM communication support for the requesting virtio
+device.
+
+4.6.1.3 Driver VM Shutdown and Migration
+Before the driver VM is destroyed or migrated, all the clients that connect to
+the vhost-pci server SHOULD send a VHOST_PCI_SOCKET_DEVICE_INFO(DEL) message to
+the vhost-pci server. The destroying or migrating activity MUST wait until all
+the VHOST_PCI_SOCKET_DEL_CONNECTION_ACK(DEL_DONE) messages are received.
+
+When a vhost-pci device has no frontend devices, the vhost-pci device SHOULD be
+destroyed.
+
+4.6.1.4 Driver VM Memory Hot-plug
+When the vhost-pci server receives a VHOST_PCI_SOCKET_MEMORY_INFO(DEL) message,
+a VHOST_PCI_CONTROLQ_MEMORY_INFO(DEL) message SHOULD be put on the controlq and
+a controlq interrupt is injected to the VM. When the vhost-pci server receives
+a VHOST_PCI_CONTROLQ_MEMORY_INFO_ACK(DEL_DONE) acknowledgement from the driver,
+it SHOULD unmap that memory region and send a
+VHOST_PCI_SOCKET_MEMORY_INFO_ACK(DEL_DONE) message to the client.
+
+When the vhost-pci server receives a VHOST_PCI_SOCKET_MEMORY_INFO(ADD) message,
+and the received memory info is new to what has already been mapped, it
+calculates the total received memory size.
+
+If the new memory size plus the mapped memory size is smaller than the address
+space size reserved by the bar, the server SHOULD map the new memory and expose
+it to the VM via the QEMU MemoryRegion mechanism. Then it SHOULD put the new
+memory info on the controlq, and injects a controlq interrupt to the VM.
+
+If the new memory size plus the mapped memory size is larger than the address
+space size reserved by the bar, the server clones out a new vhost-pci device,
+configures the bar size to be double of the current memory, hot-plugs out the
+old vhost-pci device, and hot-plugs in the new vhost-pci device to the VM. The
+initialization steps SHOULD follow 4.5 Device Initialization, except the
+interaction between the server and client is not needed.
+
+When the vhost-pci server receives a
+VHOST_PCI_CONTROLQ_MEMORY_INFO_ACK(ADD_DONE) acknowledgement from the driver,
+it SHOULD send a VHOST_PCI_SOCKET_MEMORY_INFO_ACK(ADD_DONE) message to the
+client.
+
+4.6.2 Driver Requirements: Device Operation
+The vhost-pci driver SHOULD acknowledge to the vhost-pci device by writing
+VHOST_PCI_CONTROLQ_DEVICE_INFO_ACK(ADD_DONE) to the vhost_pci_config fields
+when it finishes handling the device info update.
+
+The vhost-pci driver SHOULD ensure that all the CPUs are noticed about the
+device info update before acknowledging to the vhost-pci device.
+
+The vhost-pci driver SHOULD acknowledge to the vhost-pci device by writing
+VHOST_PCI_CONTROLQ_DEVICE_INFO_ACK(DEL_DONE) to vhost_pci_config fields when
+it finishes removing the vhost-pci support for the requesting virtio device.
+
+The vhost-pci driver SHOULD ensure that all the CPUs are noticed about the
+removing of the vhost-pci support for the requesting virtio device before
+acknowledging to the vhost-pci device.
--
1.8.3.1
^ permalink raw reply related [flat|nested] 17+ messages in thread
* [PATCH 5/6] Vhost-pci RFC: Future Security Enhancement
2016-05-28 23:36 ` [Qemu-devel] " Wei Wang
@ 2016-05-28 23:36 ` Wei Wang
-1 siblings, 0 replies; 17+ messages in thread
From: Wei Wang @ 2016-05-28 23:36 UTC (permalink / raw)
To: kvm, qemu-devel, virtio-comment, virtio-dev, mst, stefanha, pbonzini
Cc: Wei Wang
Signed-off-by: Wei Wang <wei.w.wang@intel.com>
---
FutureWorks | 21 +++++++++++++++++++++
1 file changed, 21 insertions(+)
create mode 100644 FutureWorks
diff --git a/FutureWorks b/FutureWorks
new file mode 100644
index 0000000..210edcd
--- /dev/null
+++ b/FutureWorks
@@ -0,0 +1,21 @@
+The vhost-pci design is currently suitable for a group of VMs who trust each
+other. To extend it to a more general use case, two security features can be
+added in the future.
+
+1 vIOMMU
+vIOMMU provides the driver VM with the ability to restrict the device VM to
+transiently access a specified portion of its memory. The vhost-pci design
+proposed in this RFC can be extended to access the driver VM's memory with
+vIOMMU. Precisely, the vIOMMU engine in the driver VM configures access
+permissions (R/W) for the vhost-pci device to access its memory. More details
+can be found at https://wiki.opnfv.org/display/kvm/Vm2vm+Mst and
+https://lists.gnu.org/archive/html/qemu-devel/2015-08/msg03993.html
+
+2 eptp switching
+The idea of eptp swithing allows a vhost-pci device driver to access the mapped
+driver VM's memory in an alternative view, where only a piece of trusted code
+can access the driver VM's memory. More details can be found at
+http://events.linuxfoundation.org/sites/events/files/slides/
+Jun_Nakajima_NFV_KVM%202015_final.pdf
+
+
--
1.8.3.1
^ permalink raw reply related [flat|nested] 17+ messages in thread
* [Qemu-devel] [PATCH 5/6] Vhost-pci RFC: Future Security Enhancement
@ 2016-05-28 23:36 ` Wei Wang
0 siblings, 0 replies; 17+ messages in thread
From: Wei Wang @ 2016-05-28 23:36 UTC (permalink / raw)
To: kvm, qemu-devel, virtio-comment, virtio-dev, mst, stefanha, pbonzini
Cc: Wei Wang
Signed-off-by: Wei Wang <wei.w.wang@intel.com>
---
FutureWorks | 21 +++++++++++++++++++++
1 file changed, 21 insertions(+)
create mode 100644 FutureWorks
diff --git a/FutureWorks b/FutureWorks
new file mode 100644
index 0000000..210edcd
--- /dev/null
+++ b/FutureWorks
@@ -0,0 +1,21 @@
+The vhost-pci design is currently suitable for a group of VMs who trust each
+other. To extend it to a more general use case, two security features can be
+added in the future.
+
+1 vIOMMU
+vIOMMU provides the driver VM with the ability to restrict the device VM to
+transiently access a specified portion of its memory. The vhost-pci design
+proposed in this RFC can be extended to access the driver VM's memory with
+vIOMMU. Precisely, the vIOMMU engine in the driver VM configures access
+permissions (R/W) for the vhost-pci device to access its memory. More details
+can be found at https://wiki.opnfv.org/display/kvm/Vm2vm+Mst and
+https://lists.gnu.org/archive/html/qemu-devel/2015-08/msg03993.html
+
+2 eptp switching
+The idea of eptp swithing allows a vhost-pci device driver to access the mapped
+driver VM's memory in an alternative view, where only a piece of trusted code
+can access the driver VM's memory. More details can be found at
+http://events.linuxfoundation.org/sites/events/files/slides/
+Jun_Nakajima_NFV_KVM%202015_final.pdf
+
+
--
1.8.3.1
^ permalink raw reply related [flat|nested] 17+ messages in thread
* [PATCH 6/6] Vhost-pci RFC: Experimental Results
2016-05-28 23:36 ` [Qemu-devel] " Wei Wang
@ 2016-05-28 23:36 ` Wei Wang
-1 siblings, 0 replies; 17+ messages in thread
From: Wei Wang @ 2016-05-28 23:36 UTC (permalink / raw)
To: kvm, qemu-devel, virtio-comment, virtio-dev, mst, stefanha, pbonzini
Cc: Wei Wang
Signed-off-by: Wei Wang <wei.w.wang@intel.com>
---
Results | 18 ++++++++++++++++++
1 file changed, 18 insertions(+)
create mode 100644 Results
diff --git a/Results b/Results
new file mode 100644
index 0000000..7402826
--- /dev/null
+++ b/Results
@@ -0,0 +1,18 @@
+We have built a fundamental vhost-pci based inter-VM communication framework
+for network packet transmission. To test the throughput affected by scaling
+with more VMs to stream out packets, we chain 2 to 5 VMs, and follow the vsperf
+test methodology proposed by OPNFV, as shown in Fig. 2. The first VM is
+passthrough-ed with a physical NIC to inject packets from an external packet
+generator, and the last VM is passthrough-ed with a physical NIC to eject
+packets back to the external generator. A layer2 forwarding module in each VM
+is responsible for forwarding incoming packets from NIC1 (the injection NIC) to
+NIC2 (the ejection NIC). In the traditional way, NIC2 is a virtio-net device
+connected to the vhost-user backend in OVS. With our proposed solution, NIC2 is
+a vhost-pci device, which directly copies packets to the next VM. The packet
+generator implements the RFC2544 standard, which keeps running at a 0 packet
+loss rate.
+
+Fig. 3 shows the scalability test results. In the vhost-user case, a
+significant performance drop (40%~55%) occurs when 4 and 5 VMs are chained
+together. The vhost-pci based inter-VM communication scales well (no
+significant throughput drop) with more VMs are chained together.
--
1.8.3.1
^ permalink raw reply related [flat|nested] 17+ messages in thread
* [Qemu-devel] [PATCH 6/6] Vhost-pci RFC: Experimental Results
@ 2016-05-28 23:36 ` Wei Wang
0 siblings, 0 replies; 17+ messages in thread
From: Wei Wang @ 2016-05-28 23:36 UTC (permalink / raw)
To: kvm, qemu-devel, virtio-comment, virtio-dev, mst, stefanha, pbonzini
Cc: Wei Wang
Signed-off-by: Wei Wang <wei.w.wang@intel.com>
---
Results | 18 ++++++++++++++++++
1 file changed, 18 insertions(+)
create mode 100644 Results
diff --git a/Results b/Results
new file mode 100644
index 0000000..7402826
--- /dev/null
+++ b/Results
@@ -0,0 +1,18 @@
+We have built a fundamental vhost-pci based inter-VM communication framework
+for network packet transmission. To test the throughput affected by scaling
+with more VMs to stream out packets, we chain 2 to 5 VMs, and follow the vsperf
+test methodology proposed by OPNFV, as shown in Fig. 2. The first VM is
+passthrough-ed with a physical NIC to inject packets from an external packet
+generator, and the last VM is passthrough-ed with a physical NIC to eject
+packets back to the external generator. A layer2 forwarding module in each VM
+is responsible for forwarding incoming packets from NIC1 (the injection NIC) to
+NIC2 (the ejection NIC). In the traditional way, NIC2 is a virtio-net device
+connected to the vhost-user backend in OVS. With our proposed solution, NIC2 is
+a vhost-pci device, which directly copies packets to the next VM. The packet
+generator implements the RFC2544 standard, which keeps running at a 0 packet
+loss rate.
+
+Fig. 3 shows the scalability test results. In the vhost-user case, a
+significant performance drop (40%~55%) occurs when 4 and 5 VMs are chained
+together. The vhost-pci based inter-VM communication scales well (no
+significant throughput drop) with more VMs are chained together.
--
1.8.3.1
^ permalink raw reply related [flat|nested] 17+ messages in thread
* Re: [Qemu-devel] [PATCH 0/6] *** Vhost-pci RFC ***
2016-05-28 23:36 ` [Qemu-devel] " Wei Wang
` (6 preceding siblings ...)
(?)
@ 2016-05-31 18:21 ` Eric Blake
2016-06-01 2:15 ` Wang, Wei W
-1 siblings, 1 reply; 17+ messages in thread
From: Eric Blake @ 2016-05-31 18:21 UTC (permalink / raw)
To: Wei Wang, kvm, qemu-devel, virtio-comment, virtio-dev, mst,
stefanha, pbonzini
[-- Attachment #1: Type: text/plain, Size: 1266 bytes --]
On 05/28/2016 05:36 PM, Wei Wang wrote:
> This RFC proposes a design of vhost-pci, which is a new virtio device type.
> The vhost-pci device is used for inter-VM communication. Please read the RFC
> patches for details.
>
>
> Wei Wang (6):
> Vhost-pci RFC: Introduction
> Vhost-pci RFC: Modification Scope
> Vhost-pci RFC: Benefits to KVM
> Vhost-pci RFC: Detailed Description in the Virtio Specification Format
> Vhost-pci RFC: Future Security Enhancement
> Vhost-pci RFC: Experimental Results
>
> Benefits | 8 ++
> Details | 324 ++++++++++++++++++++++++++++++++++++++++++++++++++++++
> FutureWorks | 21 ++++
> Introduction | 31 ++++++
> ModificationScope | 3 +
> Results | 18 +++
Umm, are you really creating 6 new files? Shouldn't this just be a
single patch, as a single file, under the docs/ subdirectory?
> 6 files changed, 405 insertions(+)
> create mode 100644 Benefits
> create mode 100644 Details
> create mode 100644 FutureWorks
> create mode 100644 Introduction
> create mode 100644 ModificationScope
> create mode 100644 Results
>
--
Eric Blake eblake redhat com +1-919-301-3266
Libvirt virtualization library http://libvirt.org
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 604 bytes --]
^ permalink raw reply [flat|nested] 17+ messages in thread* [virtio-comment] RE: [Qemu-devel] [PATCH 0/6] *** Vhost-pci RFC ***
2016-05-31 18:21 ` [Qemu-devel] [PATCH 0/6] *** Vhost-pci RFC *** Eric Blake
@ 2016-06-01 2:15 ` Wang, Wei W
0 siblings, 0 replies; 17+ messages in thread
From: Wang, Wei W @ 2016-06-01 2:15 UTC (permalink / raw)
To: Eric Blake, kvm, qemu-devel, virtio-comment, virtio-dev, mst,
stefanha, pbonzini
On Wed 6/1/2016 2:21 AM, Eric Blake wrote:
> On 05/28/2016 05:36 PM, Wei Wang wrote:
> > This RFC proposes a design of vhost-pci, which is a new virtio device type.
> > The vhost-pci device is used for inter-VM communication. Please read
> > the RFC patches for details.
> >
> >
> > Wei Wang (6):
> > Vhost-pci RFC: Introduction
> > Vhost-pci RFC: Modification Scope
> > Vhost-pci RFC: Benefits to KVM
> > Vhost-pci RFC: Detailed Description in the Virtio Specification Format
> > Vhost-pci RFC: Future Security Enhancement
> > Vhost-pci RFC: Experimental Results
> >
> > Benefits | 8 ++
> > Details | 324
> ++++++++++++++++++++++++++++++++++++++++++++++++++++++
> > FutureWorks | 21 ++++
> > Introduction | 31 ++++++
> > ModificationScope | 3 +
> > Results | 18 +++
>
> Umm, are you really creating 6 new files? Shouldn't this just be a single patch,
> as a single file, under the docs/ subdirectory?
Yeah, I actually split it into 6 files. I think it's more convenient to review and discuss them.
Best,
Wei
>
> > 6 files changed, 405 insertions(+)
> > create mode 100644 Benefits
> > create mode 100644 Details
> > create mode 100644 FutureWorks
> > create mode 100644 Introduction
> > create mode 100644 ModificationScope
> > create mode 100644 Results
> >
>
> --
> Eric Blake eblake redhat com +1-919-301-3266
> Libvirt virtualization library http://libvirt.org
^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: [Qemu-devel] [PATCH 0/6] *** Vhost-pci RFC ***
@ 2016-06-01 2:15 ` Wang, Wei W
0 siblings, 0 replies; 17+ messages in thread
From: Wang, Wei W @ 2016-06-01 2:15 UTC (permalink / raw)
To: Eric Blake, kvm, qemu-devel, virtio-comment, virtio-dev, mst,
stefanha, pbonzini
On Wed 6/1/2016 2:21 AM, Eric Blake wrote:
> On 05/28/2016 05:36 PM, Wei Wang wrote:
> > This RFC proposes a design of vhost-pci, which is a new virtio device type.
> > The vhost-pci device is used for inter-VM communication. Please read
> > the RFC patches for details.
> >
> >
> > Wei Wang (6):
> > Vhost-pci RFC: Introduction
> > Vhost-pci RFC: Modification Scope
> > Vhost-pci RFC: Benefits to KVM
> > Vhost-pci RFC: Detailed Description in the Virtio Specification Format
> > Vhost-pci RFC: Future Security Enhancement
> > Vhost-pci RFC: Experimental Results
> >
> > Benefits | 8 ++
> > Details | 324
> ++++++++++++++++++++++++++++++++++++++++++++++++++++++
> > FutureWorks | 21 ++++
> > Introduction | 31 ++++++
> > ModificationScope | 3 +
> > Results | 18 +++
>
> Umm, are you really creating 6 new files? Shouldn't this just be a single patch,
> as a single file, under the docs/ subdirectory?
Yeah, I actually split it into 6 files. I think it's more convenient to review and discuss them.
Best,
Wei
>
> > 6 files changed, 405 insertions(+)
> > create mode 100644 Benefits
> > create mode 100644 Details
> > create mode 100644 FutureWorks
> > create mode 100644 Introduction
> > create mode 100644 ModificationScope
> > create mode 100644 Results
> >
>
> --
> Eric Blake eblake redhat com +1-919-301-3266
> Libvirt virtualization library http://libvirt.org
^ permalink raw reply [flat|nested] 17+ messages in thread