From: "Michael S. Tsirkin" <mst@redhat.com>
To: qemu-devel@nongnu.org
Cc: "Peter Maydell" <peter.maydell@linaro.org>,
"Jens Freimann" <jfreimann@redhat.com>,
"Marc-André Lureau" <marcandre.lureau@redhat.com>
Subject: [Qemu-devel] [PULL 03/37] docs: reST-ify vhost-user documentation
Date: Thu, 16 May 2019 08:17:57 -0400 [thread overview]
Message-ID: <20190315180735.13096-1-marcandre.lureau@redhat.com> (raw)
In-Reply-To: <20190515121146.7248-1-mst@redhat.com>
From: Marc-André Lureau <marcandre.lureau@redhat.com>
Signed-off-by: Marc-André Lureau <marcandre.lureau@redhat.com>
Message-Id: <20190315180735.13096-1-marcandre.lureau@redhat.com>
Reviewed-by: Jens Freimann <jfreimann@redhat.com>
Reviewed-by: Michael S. Tsirkin <mst@redhat.com>
Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
---
docs/interop/vhost-user.txt | 1219 -------------------------------
MAINTAINERS | 2 +-
docs/interop/index.rst | 2 +-
docs/interop/vhost-user.rst | 1351 +++++++++++++++++++++++++++++++++++
4 files changed, 1353 insertions(+), 1221 deletions(-)
delete mode 100644 docs/interop/vhost-user.txt
create mode 100644 docs/interop/vhost-user.rst
diff --git a/docs/interop/vhost-user.txt b/docs/interop/vhost-user.txt
deleted file mode 100644
index 4dbd530cb9..0000000000
--- a/docs/interop/vhost-user.txt
+++ /dev/null
@@ -1,1219 +0,0 @@
-Vhost-user Protocol
-===================
-
-Copyright (c) 2014 Virtual Open Systems Sarl.
-
-This work is licensed under the terms of the GNU GPL, version 2 or later.
-See the COPYING file in the top-level directory.
-===================
-
-This protocol is aiming to complement the ioctl interface used to control the
-vhost implementation in the Linux kernel. It implements the control plane needed
-to establish virtqueue sharing with a user space process on the same host. It
-uses communication over a Unix domain socket to share file descriptors in the
-ancillary data of the message.
-
-The protocol defines 2 sides of the communication, master and slave. Master is
-the application that shares its virtqueues, in our case QEMU. Slave is the
-consumer of the virtqueues.
-
-In the current implementation QEMU is the Master, and the Slave is the
-external process consuming the virtio queues, for example a software
-Ethernet switch running in user space, such as Snabbswitch, or a block
-device backend processing read & write to a virtual disk. In order to
-facilitate interoperability between various backend implementations,
-it is recommended to follow the "Backend program conventions"
-described in this document.
-
-Master and slave can be either a client (i.e. connecting) or server (listening)
-in the socket communication.
-
-Message Specification
----------------------
-
-Note that all numbers are in the machine native byte order. A vhost-user message
-consists of 3 header fields and a payload:
-
-------------------------------------
-| request | flags | size | payload |
-------------------------------------
-
- * Request: 32-bit type of the request
- * Flags: 32-bit bit field:
- - Lower 2 bits are the version (currently 0x01)
- - Bit 2 is the reply flag - needs to be sent on each reply from the slave
- - Bit 3 is the need_reply flag - see VHOST_USER_PROTOCOL_F_REPLY_ACK for
- details.
- * Size - 32-bit size of the payload
-
-
-Depending on the request type, payload can be:
-
- * A single 64-bit integer
- -------
- | u64 |
- -------
-
- u64: a 64-bit unsigned integer
-
- * A vring state description
- ---------------
- | index | num |
- ---------------
-
- Index: a 32-bit index
- Num: a 32-bit number
-
- * A vring address description
- --------------------------------------------------------------
- | index | flags | size | descriptor | used | available | log |
- --------------------------------------------------------------
-
- Index: a 32-bit vring index
- Flags: a 32-bit vring flags
- Descriptor: a 64-bit ring address of the vring descriptor table
- Used: a 64-bit ring address of the vring used ring
- Available: a 64-bit ring address of the vring available ring
- Log: a 64-bit guest address for logging
-
- Note that a ring address is an IOVA if VIRTIO_F_IOMMU_PLATFORM has been
- negotiated. Otherwise it is a user address.
-
- * Memory regions description
- ---------------------------------------------------
- | num regions | padding | region0 | ... | region7 |
- ---------------------------------------------------
-
- Num regions: a 32-bit number of regions
- Padding: 32-bit
-
- A region is:
- -----------------------------------------------------
- | guest address | size | user address | mmap offset |
- -----------------------------------------------------
-
- Guest address: a 64-bit guest address of the region
- Size: a 64-bit size
- User address: a 64-bit user address
- mmap offset: 64-bit offset where region starts in the mapped memory
-
-* Log description
- ---------------------------
- | log size | log offset |
- ---------------------------
- log size: size of area used for logging
- log offset: offset from start of supplied file descriptor
- where logging starts (i.e. where guest address 0 would be logged)
-
- * An IOTLB message
- ---------------------------------------------------------
- | iova | size | user address | permissions flags | type |
- ---------------------------------------------------------
-
- IOVA: a 64-bit I/O virtual address programmed by the guest
- Size: a 64-bit size
- User address: a 64-bit user address
- Permissions: an 8-bit value:
- - 0: No access
- - 1: Read access
- - 2: Write access
- - 3: Read/Write access
- Type: an 8-bit IOTLB message type:
- - 1: IOTLB miss
- - 2: IOTLB update
- - 3: IOTLB invalidate
- - 4: IOTLB access fail
-
- * Virtio device config space
- -----------------------------------
- | offset | size | flags | payload |
- -----------------------------------
-
- Offset: a 32-bit offset of virtio device's configuration space
- Size: a 32-bit configuration space access size in bytes
- Flags: a 32-bit value:
- - 0: Vhost master messages used for writeable fields
- - 1: Vhost master messages used for live migration
- Payload: Size bytes array holding the contents of the virtio
- device's configuration space
-
- * Vring area description
- -----------------------
- | u64 | size | offset |
- -----------------------
-
- u64: a 64-bit integer contains vring index and flags
- Size: a 64-bit size of this area
- Offset: a 64-bit offset of this area from the start of the
- supplied file descriptor
-
- * Inflight description
- -----------------------------------------------------
- | mmap size | mmap offset | num queues | queue size |
- -----------------------------------------------------
-
- mmap size: a 64-bit size of area to track inflight I/O
- mmap offset: a 64-bit offset of this area from the start
- of the supplied file descriptor
- num queues: a 16-bit number of virtqueues
- queue size: a 16-bit size of virtqueues
-
-In QEMU the vhost-user message is implemented with the following struct:
-
-typedef struct VhostUserMsg {
- VhostUserRequest request;
- uint32_t flags;
- uint32_t size;
- union {
- uint64_t u64;
- struct vhost_vring_state state;
- struct vhost_vring_addr addr;
- VhostUserMemory memory;
- VhostUserLog log;
- struct vhost_iotlb_msg iotlb;
- VhostUserConfig config;
- VhostUserVringArea area;
- VhostUserInflight inflight;
- };
-} QEMU_PACKED VhostUserMsg;
-
-Communication
--------------
-
-The protocol for vhost-user is based on the existing implementation of vhost
-for the Linux Kernel. Most messages that can be sent via the Unix domain socket
-implementing vhost-user have an equivalent ioctl to the kernel implementation.
-
-The communication consists of master sending message requests and slave sending
-message replies. Most of the requests don't require replies. Here is a list of
-the ones that do:
-
- * VHOST_USER_GET_FEATURES
- * VHOST_USER_GET_PROTOCOL_FEATURES
- * VHOST_USER_GET_VRING_BASE
- * VHOST_USER_SET_LOG_BASE (if VHOST_USER_PROTOCOL_F_LOG_SHMFD)
- * VHOST_USER_GET_INFLIGHT_FD (if VHOST_USER_PROTOCOL_F_INFLIGHT_SHMFD)
-
-[ Also see the section on REPLY_ACK protocol extension. ]
-
-There are several messages that the master sends with file descriptors passed
-in the ancillary data:
-
- * VHOST_USER_SET_MEM_TABLE
- * VHOST_USER_SET_LOG_BASE (if VHOST_USER_PROTOCOL_F_LOG_SHMFD)
- * VHOST_USER_SET_LOG_FD
- * VHOST_USER_SET_VRING_KICK
- * VHOST_USER_SET_VRING_CALL
- * VHOST_USER_SET_VRING_ERR
- * VHOST_USER_SET_SLAVE_REQ_FD
- * VHOST_USER_SET_INFLIGHT_FD (if VHOST_USER_PROTOCOL_F_INFLIGHT_SHMFD)
-
-If Master is unable to send the full message or receives a wrong reply it will
-close the connection. An optional reconnection mechanism can be implemented.
-
-Any protocol extensions are gated by protocol feature bits,
-which allows full backwards compatibility on both master
-and slave.
-As older slaves don't support negotiating protocol features,
-a feature bit was dedicated for this purpose:
-#define VHOST_USER_F_PROTOCOL_FEATURES 30
-
-Starting and stopping rings
-----------------------
-Client must only process each ring when it is started.
-
-Client must only pass data between the ring and the
-backend, when the ring is enabled.
-
-If ring is started but disabled, client must process the
-ring without talking to the backend.
-
-For example, for a networking device, in the disabled state
-client must not supply any new RX packets, but must process
-and discard any TX packets.
-
-If VHOST_USER_F_PROTOCOL_FEATURES has not been negotiated, the ring is initialized
-in an enabled state.
-
-If VHOST_USER_F_PROTOCOL_FEATURES has been negotiated, the ring is initialized
-in a disabled state. Client must not pass data to/from the backend until ring is enabled by
-VHOST_USER_SET_VRING_ENABLE with parameter 1, or after it has been disabled by
-VHOST_USER_SET_VRING_ENABLE with parameter 0.
-
-Each ring is initialized in a stopped state, client must not process it until
-ring is started, or after it has been stopped.
-
-Client must start ring upon receiving a kick (that is, detecting that file
-descriptor is readable) on the descriptor specified by
-VHOST_USER_SET_VRING_KICK, and stop ring upon receiving
-VHOST_USER_GET_VRING_BASE.
-
-While processing the rings (whether they are enabled or not), client must
-support changing some configuration aspects on the fly.
-
-Multiple queue support
-----------------------
-
-Multiple queue is treated as a protocol extension, hence the slave has to
-implement protocol features first. The multiple queues feature is supported
-only when the protocol feature VHOST_USER_PROTOCOL_F_MQ (bit 0) is set.
-
-The max number of queue pairs the slave supports can be queried with message
-VHOST_USER_GET_QUEUE_NUM. Master should stop when the number of
-requested queues is bigger than that.
-
-As all queues share one connection, the master uses a unique index for each
-queue in the sent message to identify a specified queue. One queue pair
-is enabled initially. More queues are enabled dynamically, by sending
-message VHOST_USER_SET_VRING_ENABLE.
-
-Migration
----------
-
-During live migration, the master may need to track the modifications
-the slave makes to the memory mapped regions. The client should mark
-the dirty pages in a log. Once it complies to this logging, it may
-declare the VHOST_F_LOG_ALL vhost feature.
-
-To start/stop logging of data/used ring writes, server may send messages
-VHOST_USER_SET_FEATURES with VHOST_F_LOG_ALL and VHOST_USER_SET_VRING_ADDR with
-VHOST_VRING_F_LOG in ring's flags set to 1/0, respectively.
-
-All the modifications to memory pointed by vring "descriptor" should
-be marked. Modifications to "used" vring should be marked if
-VHOST_VRING_F_LOG is part of ring's flags.
-
-Dirty pages are of size:
-#define VHOST_LOG_PAGE 0x1000
-
-The log memory fd is provided in the ancillary data of
-VHOST_USER_SET_LOG_BASE message when the slave has
-VHOST_USER_PROTOCOL_F_LOG_SHMFD protocol feature.
-
-The size of the log is supplied as part of VhostUserMsg
-which should be large enough to cover all known guest
-addresses. Log starts at the supplied offset in the
-supplied file descriptor.
-The log covers from address 0 to the maximum of guest
-regions. In pseudo-code, to mark page at "addr" as dirty:
-
-page = addr / VHOST_LOG_PAGE
-log[page / 8] |= 1 << page % 8
-
-Where addr is the guest physical address.
-
-Use atomic operations, as the log may be concurrently manipulated.
-
-Note that when logging modifications to the used ring (when VHOST_VRING_F_LOG
-is set for this ring), log_guest_addr should be used to calculate the log
-offset: the write to first byte of the used ring is logged at this offset from
-log start. Also note that this value might be outside the legal guest physical
-address range (i.e. does not have to be covered by the VhostUserMemory table),
-but the bit offset of the last byte of the ring must fall within
-the size supplied by VhostUserLog.
-
-VHOST_USER_SET_LOG_FD is an optional message with an eventfd in
-ancillary data, it may be used to inform the master that the log has
-been modified.
-
-Once the source has finished migration, rings will be stopped by
-the source. No further update must be done before rings are
-restarted.
-
-In postcopy migration the slave is started before all the memory has been
-received from the source host, and care must be taken to avoid accessing pages
-that have yet to be received. The slave opens a 'userfault'-fd and registers
-the memory with it; this fd is then passed back over to the master.
-The master services requests on the userfaultfd for pages that are accessed
-and when the page is available it performs WAKE ioctl's on the userfaultfd
-to wake the stalled slave. The client indicates support for this via the
-VHOST_USER_PROTOCOL_F_PAGEFAULT feature.
-
-Memory access
--------------
-
-The master sends a list of vhost memory regions to the slave using the
-VHOST_USER_SET_MEM_TABLE message. Each region has two base addresses: a guest
-address and a user address.
-
-Messages contain guest addresses and/or user addresses to reference locations
-within the shared memory. The mapping of these addresses works as follows.
-
-User addresses map to the vhost memory region containing that user address.
-
-When the VIRTIO_F_IOMMU_PLATFORM feature has not been negotiated:
-
- * Guest addresses map to the vhost memory region containing that guest
- address.
-
-When the VIRTIO_F_IOMMU_PLATFORM feature has been negotiated:
-
- * Guest addresses are also called I/O virtual addresses (IOVAs). They are
- translated to user addresses via the IOTLB.
-
- * The vhost memory region guest address is not used.
-
-IOMMU support
--------------
-
-When the VIRTIO_F_IOMMU_PLATFORM feature has been negotiated, the master
-sends IOTLB entries update & invalidation by sending VHOST_USER_IOTLB_MSG
-requests to the slave with a struct vhost_iotlb_msg as payload. For update
-events, the iotlb payload has to be filled with the update message type (2),
-the I/O virtual address, the size, the user virtual address, and the
-permissions flags. Addresses and size must be within vhost memory regions set
-via the VHOST_USER_SET_MEM_TABLE request. For invalidation events, the iotlb
-payload has to be filled with the invalidation message type (3), the I/O virtual
-address and the size. On success, the slave is expected to reply with a zero
-payload, non-zero otherwise.
-
-The slave relies on the slave communcation channel (see "Slave communication"
-section below) to send IOTLB miss and access failure events, by sending
-VHOST_USER_SLAVE_IOTLB_MSG requests to the master with a struct vhost_iotlb_msg
-as payload. For miss events, the iotlb payload has to be filled with the miss
-message type (1), the I/O virtual address and the permissions flags. For access
-failure event, the iotlb payload has to be filled with the access failure
-message type (4), the I/O virtual address and the permissions flags.
-For synchronization purpose, the slave may rely on the reply-ack feature,
-so the master may send a reply when operation is completed if the reply-ack
-feature is negotiated and slaves requests a reply. For miss events, completed
-operation means either master sent an update message containing the IOTLB entry
-containing requested address and permission, or master sent nothing if the IOTLB
-miss message is invalid (invalid IOVA or permission).
-
-The master isn't expected to take the initiative to send IOTLB update messages,
-as the slave sends IOTLB miss messages for the guest virtual memory areas it
-needs to access.
-
-Slave communication
--------------------
-
-An optional communication channel is provided if the slave declares
-VHOST_USER_PROTOCOL_F_SLAVE_REQ protocol feature, to allow the slave to make
-requests to the master.
-
-The fd is provided via VHOST_USER_SET_SLAVE_REQ_FD ancillary data.
-
-A slave may then send VHOST_USER_SLAVE_* messages to the master
-using this fd communication channel.
-
-If VHOST_USER_PROTOCOL_F_SLAVE_SEND_FD protocol feature is negotiated,
-slave can send file descriptors (at most 8 descriptors in each message)
-to master via ancillary data using this fd communication channel.
-
-Inflight I/O tracking
----------------------
-
-To support reconnecting after restart or crash, slave may need to resubmit
-inflight I/Os. If virtqueue is processed in order, we can easily achieve
-that by getting the inflight descriptors from descriptor table (split virtqueue)
-or descriptor ring (packed virtqueue). However, it can't work when we process
-descriptors out-of-order because some entries which store the information of
-inflight descriptors in available ring (split virtqueue) or descriptor
-ring (packed virtqueue) might be overrided by new entries. To solve this
-problem, slave need to allocate an extra buffer to store this information of inflight
-descriptors and share it with master for persistent. VHOST_USER_GET_INFLIGHT_FD and
-VHOST_USER_SET_INFLIGHT_FD are used to transfer this buffer between master
-and slave. And the format of this buffer is described below:
-
--------------------------------------------------------
-| queue0 region | queue1 region | ... | queueN region |
--------------------------------------------------------
-
-N is the number of available virtqueues. Slave could get it from num queues
-field of VhostUserInflight.
-
-For split virtqueue, queue region can be implemented as:
-
-typedef struct DescStateSplit {
- /* Indicate whether this descriptor is inflight or not.
- * Only available for head-descriptor. */
- uint8_t inflight;
-
- /* Padding */
- uint8_t padding[5];
-
- /* Maintain a list for the last batch of used descriptors.
- * Only available when batching is used for submitting */
- uint16_t next;
-
- /* Used to preserve the order of fetching available descriptors.
- * Only available for head-descriptor. */
- uint64_t counter;
-} DescStateSplit;
-
-typedef struct QueueRegionSplit {
- /* The feature flags of this region. Now it's initialized to 0. */
- uint64_t features;
-
- /* The version of this region. It's 1 currently.
- * Zero value indicates an uninitialized buffer */
- uint16_t version;
-
- /* The size of DescStateSplit array. It's equal to the virtqueue
- * size. Slave could get it from queue size field of VhostUserInflight. */
- uint16_t desc_num;
-
- /* The head of list that track the last batch of used descriptors. */
- uint16_t last_batch_head;
-
- /* Store the idx value of used ring */
- uint16_t used_idx;
-
- /* Used to track the state of each descriptor in descriptor table */
- DescStateSplit desc[0];
-} QueueRegionSplit;
-
-To track inflight I/O, the queue region should be processed as follows:
-
-When receiving available buffers from the driver:
-
- 1. Get the next available head-descriptor index from available ring, i
-
- 2. Set desc[i].counter to the value of global counter
-
- 3. Increase global counter by 1
-
- 4. Set desc[i].inflight to 1
-
-When supplying used buffers to the driver:
-
- 1. Get corresponding used head-descriptor index, i
-
- 2. Set desc[i].next to last_batch_head
-
- 3. Set last_batch_head to i
-
- 4. Steps 1,2,3 may be performed repeatedly if batching is possible
-
- 5. Increase the idx value of used ring by the size of the batch
-
- 6. Set the inflight field of each DescStateSplit entry in the batch to 0
-
- 7. Set used_idx to the idx value of used ring
-
-When reconnecting:
-
- 1. If the value of used_idx does not match the idx value of used ring (means
- the inflight field of DescStateSplit entries in last batch may be incorrect),
-
- (a) Subtract the value of used_idx from the idx value of used ring to get
- last batch size of DescStateSplit entries
-
- (b) Set the inflight field of each DescStateSplit entry to 0 in last batch
- list which starts from last_batch_head
-
- (c) Set used_idx to the idx value of used ring
-
- 2. Resubmit inflight DescStateSplit entries in order of their counter value
-
-For packed virtqueue, queue region can be implemented as:
-
-typedef struct DescStatePacked {
- /* Indicate whether this descriptor is inflight or not.
- * Only available for head-descriptor. */
- uint8_t inflight;
-
- /* Padding */
- uint8_t padding;
-
- /* Link to the next free entry */
- uint16_t next;
-
- /* Link to the last entry of descriptor list.
- * Only available for head-descriptor. */
- uint16_t last;
-
- /* The length of descriptor list.
- * Only available for head-descriptor. */
- uint16_t num;
-
- /* Used to preserve the order of fetching available descriptors.
- * Only available for head-descriptor. */
- uint64_t counter;
-
- /* The buffer id */
- uint16_t id;
-
- /* The descriptor flags */
- uint16_t flags;
-
- /* The buffer length */
- uint32_t len;
-
- /* The buffer address */
- uint64_t addr;
-} DescStatePacked;
-
-typedef struct QueueRegionPacked {
- /* The feature flags of this region. Now it's initialized to 0. */
- uint64_t features;
-
- /* The version of this region. It's 1 currently.
- * Zero value indicates an uninitialized buffer */
- uint16_t version;
-
- /* The size of DescStatePacked array. It's equal to the virtqueue
- * size. Slave could get it from queue size field of VhostUserInflight. */
- uint16_t desc_num;
-
- /* The head of free DescStatePacked entry list */
- uint16_t free_head;
-
- /* The old head of free DescStatePacked entry list */
- uint16_t old_free_head;
-
- /* The used index of descriptor ring */
- uint16_t used_idx;
-
- /* The old used index of descriptor ring */
- uint16_t old_used_idx;
-
- /* Device ring wrap counter */
- uint8_t used_wrap_counter;
-
- /* The old device ring wrap counter */
- uint8_t old_used_wrap_counter;
-
- /* Padding */
- uint8_t padding[7];
-
- /* Used to track the state of each descriptor fetched from descriptor ring */
- DescStatePacked desc[0];
-} QueueRegionPacked;
-
-To track inflight I/O, the queue region should be processed as follows:
-
-When receiving available buffers from the driver:
-
- 1. Get the next available descriptor entry from descriptor ring, d
-
- 2. If d is head descriptor,
-
- (a) Set desc[old_free_head].num to 0
-
- (b) Set desc[old_free_head].counter to the value of global counter
-
- (c) Increase global counter by 1
-
- (d) Set desc[old_free_head].inflight to 1
-
- 3. If d is last descriptor, set desc[old_free_head].last to free_head
-
- 4. Increase desc[old_free_head].num by 1
-
- 5. Set desc[free_head].addr, desc[free_head].len, desc[free_head].flags,
- desc[free_head].id to d.addr, d.len, d.flags, d.id
-
- 6. Set free_head to desc[free_head].next
-
- 7. If d is last descriptor, set old_free_head to free_head
-
-When supplying used buffers to the driver:
-
- 1. Get corresponding used head-descriptor entry from descriptor ring, d
-
- 2. Get corresponding DescStatePacked entry, e
-
- 3. Set desc[e.last].next to free_head
-
- 4. Set free_head to the index of e
-
- 5. Steps 1,2,3,4 may be performed repeatedly if batching is possible
-
- 6. Increase used_idx by the size of the batch and update used_wrap_counter if needed
-
- 7. Update d.flags
-
- 8. Set the inflight field of each head DescStatePacked entry in the batch to 0
-
- 9. Set old_free_head, old_used_idx, old_used_wrap_counter to free_head, used_idx,
- used_wrap_counter
-
-When reconnecting:
-
- 1. If used_idx does not match old_used_idx (means the inflight field of DescStatePacked
- entries in last batch may be incorrect),
-
- (a) Get the next descriptor ring entry through old_used_idx, d
-
- (b) Use old_used_wrap_counter to calculate the available flags
-
- (c) If d.flags is not equal to the calculated flags value (means slave has
- submitted the buffer to guest driver before crash, so it has to commit the
- in-progres update), set old_free_head, old_used_idx, old_used_wrap_counter
- to free_head, used_idx, used_wrap_counter
-
- 2. Set free_head, used_idx, used_wrap_counter to old_free_head, old_used_idx,
- old_used_wrap_counter (roll back any in-progress update)
-
- 3. Set the inflight field of each DescStatePacked entry in free list to 0
-
- 4. Resubmit inflight DescStatePacked entries in order of their counter value
-
-Protocol features
------------------
-
-#define VHOST_USER_PROTOCOL_F_MQ 0
-#define VHOST_USER_PROTOCOL_F_LOG_SHMFD 1
-#define VHOST_USER_PROTOCOL_F_RARP 2
-#define VHOST_USER_PROTOCOL_F_REPLY_ACK 3
-#define VHOST_USER_PROTOCOL_F_MTU 4
-#define VHOST_USER_PROTOCOL_F_SLAVE_REQ 5
-#define VHOST_USER_PROTOCOL_F_CROSS_ENDIAN 6
-#define VHOST_USER_PROTOCOL_F_CRYPTO_SESSION 7
-#define VHOST_USER_PROTOCOL_F_PAGEFAULT 8
-#define VHOST_USER_PROTOCOL_F_CONFIG 9
-#define VHOST_USER_PROTOCOL_F_SLAVE_SEND_FD 10
-#define VHOST_USER_PROTOCOL_F_HOST_NOTIFIER 11
-#define VHOST_USER_PROTOCOL_F_INFLIGHT_SHMFD 12
-
-Master message types
---------------------
-
- * VHOST_USER_GET_FEATURES
-
- Id: 1
- Equivalent ioctl: VHOST_GET_FEATURES
- Master payload: N/A
- Slave payload: u64
-
- Get from the underlying vhost implementation the features bitmask.
- Feature bit VHOST_USER_F_PROTOCOL_FEATURES signals slave support for
- VHOST_USER_GET_PROTOCOL_FEATURES and VHOST_USER_SET_PROTOCOL_FEATURES.
-
- * VHOST_USER_SET_FEATURES
-
- Id: 2
- Ioctl: VHOST_SET_FEATURES
- Master payload: u64
-
- Enable features in the underlying vhost implementation using a bitmask.
- Feature bit VHOST_USER_F_PROTOCOL_FEATURES signals slave support for
- VHOST_USER_GET_PROTOCOL_FEATURES and VHOST_USER_SET_PROTOCOL_FEATURES.
-
- * VHOST_USER_GET_PROTOCOL_FEATURES
-
- Id: 15
- Equivalent ioctl: VHOST_GET_FEATURES
- Master payload: N/A
- Slave payload: u64
-
- Get the protocol feature bitmask from the underlying vhost implementation.
- Only legal if feature bit VHOST_USER_F_PROTOCOL_FEATURES is present in
- VHOST_USER_GET_FEATURES.
- Note: slave that reported VHOST_USER_F_PROTOCOL_FEATURES must support
- this message even before VHOST_USER_SET_FEATURES was called.
-
- * VHOST_USER_SET_PROTOCOL_FEATURES
-
- Id: 16
- Ioctl: VHOST_SET_FEATURES
- Master payload: u64
-
- Enable protocol features in the underlying vhost implementation.
- Only legal if feature bit VHOST_USER_F_PROTOCOL_FEATURES is present in
- VHOST_USER_GET_FEATURES.
- Note: slave that reported VHOST_USER_F_PROTOCOL_FEATURES must support
- this message even before VHOST_USER_SET_FEATURES was called.
-
- * VHOST_USER_SET_OWNER
-
- Id: 3
- Equivalent ioctl: VHOST_SET_OWNER
- Master payload: N/A
-
- Issued when a new connection is established. It sets the current Master
- as an owner of the session. This can be used on the Slave as a
- "session start" flag.
-
- * VHOST_USER_RESET_OWNER
-
- Id: 4
- Master payload: N/A
-
- This is no longer used. Used to be sent to request disabling
- all rings, but some clients interpreted it to also discard
- connection state (this interpretation would lead to bugs).
- It is recommended that clients either ignore this message,
- or use it to disable all rings.
-
- * VHOST_USER_SET_MEM_TABLE
-
- Id: 5
- Equivalent ioctl: VHOST_SET_MEM_TABLE
- Master payload: memory regions description
- Slave payload: (postcopy only) memory regions description
-
- Sets the memory map regions on the slave so it can translate the vring
- addresses. In the ancillary data there is an array of file descriptors
- for each memory mapped region. The size and ordering of the fds matches
- the number and ordering of memory regions.
-
- When VHOST_USER_POSTCOPY_LISTEN has been received, SET_MEM_TABLE replies with
- the bases of the memory mapped regions to the master. The slave must
- have mmap'd the regions but not yet accessed them and should not yet generate
- a userfault event. Note NEED_REPLY_MASK is not set in this case.
- QEMU will then reply back to the list of mappings with an empty
- VHOST_USER_SET_MEM_TABLE as an acknowledgment; only upon reception of this
- message may the guest start accessing the memory and generating faults.
-
- * VHOST_USER_SET_LOG_BASE
-
- Id: 6
- Equivalent ioctl: VHOST_SET_LOG_BASE
- Master payload: u64
- Slave payload: N/A
-
- Sets logging shared memory space.
- When slave has VHOST_USER_PROTOCOL_F_LOG_SHMFD protocol
- feature, the log memory fd is provided in the ancillary data of
- VHOST_USER_SET_LOG_BASE message, the size and offset of shared
- memory area provided in the message.
-
-
- * VHOST_USER_SET_LOG_FD
-
- Id: 7
- Equivalent ioctl: VHOST_SET_LOG_FD
- Master payload: N/A
-
- Sets the logging file descriptor, which is passed as ancillary data.
-
- * VHOST_USER_SET_VRING_NUM
-
- Id: 8
- Equivalent ioctl: VHOST_SET_VRING_NUM
- Master payload: vring state description
-
- Set the size of the queue.
-
- * VHOST_USER_SET_VRING_ADDR
-
- Id: 9
- Equivalent ioctl: VHOST_SET_VRING_ADDR
- Master payload: vring address description
- Slave payload: N/A
-
- Sets the addresses of the different aspects of the vring.
-
- * VHOST_USER_SET_VRING_BASE
-
- Id: 10
- Equivalent ioctl: VHOST_SET_VRING_BASE
- Master payload: vring state description
-
- Sets the base offset in the available vring.
-
- * VHOST_USER_GET_VRING_BASE
-
- Id: 11
- Equivalent ioctl: VHOST_USER_GET_VRING_BASE
- Master payload: vring state description
- Slave payload: vring state description
-
- Get the available vring base offset.
-
- * VHOST_USER_SET_VRING_KICK
-
- Id: 12
- Equivalent ioctl: VHOST_SET_VRING_KICK
- Master payload: u64
-
- Set the event file descriptor for adding buffers to the vring. It
- is passed in the ancillary data.
- Bits (0-7) of the payload contain the vring index. Bit 8 is the
- invalid FD flag. This flag is set when there is no file descriptor
- in the ancillary data. This signals that polling should be used
- instead of waiting for a kick.
-
- * VHOST_USER_SET_VRING_CALL
-
- Id: 13
- Equivalent ioctl: VHOST_SET_VRING_CALL
- Master payload: u64
-
- Set the event file descriptor to signal when buffers are used. It
- is passed in the ancillary data.
- Bits (0-7) of the payload contain the vring index. Bit 8 is the
- invalid FD flag. This flag is set when there is no file descriptor
- in the ancillary data. This signals that polling will be used
- instead of waiting for the call.
-
- * VHOST_USER_SET_VRING_ERR
-
- Id: 14
- Equivalent ioctl: VHOST_SET_VRING_ERR
- Master payload: u64
-
- Set the event file descriptor to signal when error occurs. It
- is passed in the ancillary data.
- Bits (0-7) of the payload contain the vring index. Bit 8 is the
- invalid FD flag. This flag is set when there is no file descriptor
- in the ancillary data.
-
- * VHOST_USER_GET_QUEUE_NUM
-
- Id: 17
- Equivalent ioctl: N/A
- Master payload: N/A
- Slave payload: u64
-
- Query how many queues the backend supports. This request should be
- sent only when VHOST_USER_PROTOCOL_F_MQ is set in queried protocol
- features by VHOST_USER_GET_PROTOCOL_FEATURES.
-
- * VHOST_USER_SET_VRING_ENABLE
-
- Id: 18
- Equivalent ioctl: N/A
- Master payload: vring state description
-
- Signal slave to enable or disable corresponding vring.
- This request should be sent only when VHOST_USER_F_PROTOCOL_FEATURES
- has been negotiated.
-
- * VHOST_USER_SEND_RARP
-
- Id: 19
- Equivalent ioctl: N/A
- Master payload: u64
-
- Ask vhost user backend to broadcast a fake RARP to notify the migration
- is terminated for guest that does not support GUEST_ANNOUNCE.
- Only legal if feature bit VHOST_USER_F_PROTOCOL_FEATURES is present in
- VHOST_USER_GET_FEATURES and protocol feature bit VHOST_USER_PROTOCOL_F_RARP
- is present in VHOST_USER_GET_PROTOCOL_FEATURES.
- The first 6 bytes of the payload contain the mac address of the guest to
- allow the vhost user backend to construct and broadcast the fake RARP.
-
- * VHOST_USER_NET_SET_MTU
-
- Id: 20
- Equivalent ioctl: N/A
- Master payload: u64
-
- Set host MTU value exposed to the guest.
- This request should be sent only when VIRTIO_NET_F_MTU feature has been
- successfully negotiated, VHOST_USER_F_PROTOCOL_FEATURES is present in
- VHOST_USER_GET_FEATURES and protocol feature bit
- VHOST_USER_PROTOCOL_F_NET_MTU is present in
- VHOST_USER_GET_PROTOCOL_FEATURES.
- If VHOST_USER_PROTOCOL_F_REPLY_ACK is negotiated, slave must respond
- with zero in case the specified MTU is valid, or non-zero otherwise.
-
- * VHOST_USER_SET_SLAVE_REQ_FD
-
- Id: 21
- Equivalent ioctl: N/A
- Master payload: N/A
-
- Set the socket file descriptor for slave initiated requests. It is passed
- in the ancillary data.
- This request should be sent only when VHOST_USER_F_PROTOCOL_FEATURES
- has been negotiated, and protocol feature bit VHOST_USER_PROTOCOL_F_SLAVE_REQ
- bit is present in VHOST_USER_GET_PROTOCOL_FEATURES.
- If VHOST_USER_PROTOCOL_F_REPLY_ACK is negotiated, slave must respond
- with zero for success, non-zero otherwise.
-
- * VHOST_USER_IOTLB_MSG
-
- Id: 22
- Equivalent ioctl: N/A (equivalent to VHOST_IOTLB_MSG message type)
- Master payload: struct vhost_iotlb_msg
- Slave payload: u64
-
- Send IOTLB messages with struct vhost_iotlb_msg as payload.
- Master sends such requests to update and invalidate entries in the device
- IOTLB. The slave has to acknowledge the request with sending zero as u64
- payload for success, non-zero otherwise.
- This request should be send only when VIRTIO_F_IOMMU_PLATFORM feature
- has been successfully negotiated.
-
- * VHOST_USER_SET_VRING_ENDIAN
-
- Id: 23
- Equivalent ioctl: VHOST_SET_VRING_ENDIAN
- Master payload: vring state description
-
- Set the endianness of a VQ for legacy devices. Little-endian is indicated
- with state.num set to 0 and big-endian is indicated with state.num set
- to 1. Other values are invalid.
- This request should be sent only when VHOST_USER_PROTOCOL_F_CROSS_ENDIAN
- has been negotiated.
- Backends that negotiated this feature should handle both endiannesses
- and expect this message once (per VQ) during device configuration
- (ie. before the master starts the VQ).
-
- * VHOST_USER_GET_CONFIG
-
- Id: 24
- Equivalent ioctl: N/A
- Master payload: virtio device config space
- Slave payload: virtio device config space
-
- When VHOST_USER_PROTOCOL_F_CONFIG is negotiated, this message is
- submitted by the vhost-user master to fetch the contents of the virtio
- device configuration space, vhost-user slave's payload size MUST match
- master's request, vhost-user slave uses zero length of payload to
- indicate an error to vhost-user master. The vhost-user master may
- cache the contents to avoid repeated VHOST_USER_GET_CONFIG calls.
-
-* VHOST_USER_SET_CONFIG
-
- Id: 25
- Equivalent ioctl: N/A
- Master payload: virtio device config space
- Slave payload: N/A
-
- When VHOST_USER_PROTOCOL_F_CONFIG is negotiated, this message is
- submitted by the vhost-user master when the Guest changes the virtio
- device configuration space and also can be used for live migration
- on the destination host. The vhost-user slave must check the flags
- field, and slaves MUST NOT accept SET_CONFIG for read-only
- configuration space fields unless the live migration bit is set.
-
-* VHOST_USER_CREATE_CRYPTO_SESSION
-
- Id: 26
- Equivalent ioctl: N/A
- Master payload: crypto session description
- Slave payload: crypto session description
-
- Create a session for crypto operation. The server side must return the
- session id, 0 or positive for success, negative for failure.
- This request should be sent only when VHOST_USER_PROTOCOL_F_CRYPTO_SESSION
- feature has been successfully negotiated.
- It's a required feature for crypto devices.
-
-* VHOST_USER_CLOSE_CRYPTO_SESSION
-
- Id: 27
- Equivalent ioctl: N/A
- Master payload: u64
-
- Close a session for crypto operation which was previously
- created by VHOST_USER_CREATE_CRYPTO_SESSION.
- This request should be sent only when VHOST_USER_PROTOCOL_F_CRYPTO_SESSION
- feature has been successfully negotiated.
- It's a required feature for crypto devices.
-
- * VHOST_USER_POSTCOPY_ADVISE
- Id: 28
- Master payload: N/A
- Slave payload: userfault fd
-
- When VHOST_USER_PROTOCOL_F_PAGEFAULT is supported, the
- master advises slave that a migration with postcopy enabled is underway,
- the slave must open a userfaultfd for later use.
- Note that at this stage the migration is still in precopy mode.
-
- * VHOST_USER_POSTCOPY_LISTEN
- Id: 29
- Master payload: N/A
-
- Master advises slave that a transition to postcopy mode has happened.
- The slave must ensure that shared memory is registered with userfaultfd
- to cause faulting of non-present pages.
-
- This is always sent sometime after a VHOST_USER_POSTCOPY_ADVISE, and
- thus only when VHOST_USER_PROTOCOL_F_PAGEFAULT is supported.
-
- * VHOST_USER_POSTCOPY_END
- Id: 30
- Slave payload: u64
-
- Master advises that postcopy migration has now completed. The
- slave must disable the userfaultfd. The response is an acknowledgement
- only.
- When VHOST_USER_PROTOCOL_F_PAGEFAULT is supported, this message
- is sent at the end of the migration, after VHOST_USER_POSTCOPY_LISTEN
- was previously sent.
- The value returned is an error indication; 0 is success.
-
- * VHOST_USER_GET_INFLIGHT_FD
- Id: 31
- Equivalent ioctl: N/A
- Master payload: inflight description
-
- When VHOST_USER_PROTOCOL_F_INFLIGHT_SHMFD protocol feature has been
- successfully negotiated, this message is submitted by master to get
- a shared buffer from slave. The shared buffer will be used to track
- inflight I/O by slave. QEMU should retrieve a new one when vm reset.
-
- * VHOST_USER_SET_INFLIGHT_FD
- Id: 32
- Equivalent ioctl: N/A
- Master payload: inflight description
-
- When VHOST_USER_PROTOCOL_F_INFLIGHT_SHMFD protocol feature has been
- successfully negotiated, this message is submitted by master to send
- the shared inflight buffer back to slave so that slave could get
- inflight I/O after a crash or restart.
-
-Slave message types
--------------------
-
- * VHOST_USER_SLAVE_IOTLB_MSG
-
- Id: 1
- Equivalent ioctl: N/A (equivalent to VHOST_IOTLB_MSG message type)
- Slave payload: struct vhost_iotlb_msg
- Master payload: N/A
-
- Send IOTLB messages with struct vhost_iotlb_msg as payload.
- Slave sends such requests to notify of an IOTLB miss, or an IOTLB
- access failure. If VHOST_USER_PROTOCOL_F_REPLY_ACK is negotiated,
- and slave set the VHOST_USER_NEED_REPLY flag, master must respond with
- zero when operation is successfully completed, or non-zero otherwise.
- This request should be send only when VIRTIO_F_IOMMU_PLATFORM feature
- has been successfully negotiated.
-
-* VHOST_USER_SLAVE_CONFIG_CHANGE_MSG
-
- Id: 2
- Equivalent ioctl: N/A
- Slave payload: N/A
- Master payload: N/A
-
- When VHOST_USER_PROTOCOL_F_CONFIG is negotiated, vhost-user slave sends
- such messages to notify that the virtio device's configuration space has
- changed, for those host devices which can support such feature, host
- driver can send VHOST_USER_GET_CONFIG message to slave to get the latest
- content. If VHOST_USER_PROTOCOL_F_REPLY_ACK is negotiated, and slave set
- the VHOST_USER_NEED_REPLY flag, master must respond with zero when
- operation is successfully completed, or non-zero otherwise.
-
- * VHOST_USER_SLAVE_VRING_HOST_NOTIFIER_MSG
-
- Id: 3
- Equivalent ioctl: N/A
- Slave payload: vring area description
- Master payload: N/A
-
- Sets host notifier for a specified queue. The queue index is contained
- in the u64 field of the vring area description. The host notifier is
- described by the file descriptor (typically it's a VFIO device fd) which
- is passed as ancillary data and the size (which is mmap size and should
- be the same as host page size) and offset (which is mmap offset) carried
- in the vring area description. QEMU can mmap the file descriptor based
- on the size and offset to get a memory range. Registering a host notifier
- means mapping this memory range to the VM as the specified queue's notify
- MMIO region. Slave sends this request to tell QEMU to de-register the
- existing notifier if any and register the new notifier if the request is
- sent with a file descriptor.
- This request should be sent only when VHOST_USER_PROTOCOL_F_HOST_NOTIFIER
- protocol feature has been successfully negotiated.
-
-VHOST_USER_PROTOCOL_F_REPLY_ACK:
--------------------------------
-The original vhost-user specification only demands replies for certain
-commands. This differs from the vhost protocol implementation where commands
-are sent over an ioctl() call and block until the client has completed.
-
-With this protocol extension negotiated, the sender (QEMU) can set the
-"need_reply" [Bit 3] flag to any command. This indicates that
-the client MUST respond with a Payload VhostUserMsg indicating success or
-failure. The payload should be set to zero on success or non-zero on failure,
-unless the message already has an explicit reply body.
-
-The response payload gives QEMU a deterministic indication of the result
-of the command. Today, QEMU is expected to terminate the main vhost-user
-loop upon receiving such errors. In future, qemu could be taught to be more
-resilient for selective requests.
-
-For the message types that already solicit a reply from the client, the
-presence of VHOST_USER_PROTOCOL_F_REPLY_ACK or need_reply bit being set brings
-no behavioural change. (See the 'Communication' section for details.)
-
-Backend program conventions
----------------------------
-
-vhost-user backends can provide various devices & services and may
-need to be configured manually depending on the use case. However, it
-is a good idea to follow the conventions listed here when
-possible. Users, QEMU or libvirt, can then rely on some common
-behaviour to avoid heterogenous configuration and management of the
-backend programs and facilitate interoperability.
-
-Each backend installed on a host system should come with at least one
-JSON file that conforms to the vhost-user.json schema. Each file
-informs the management applications about the backend type, and binary
-location. In addition, it defines rules for management apps for
-picking the highest priority backend when multiple match the search
-criteria (see @VhostUserBackend documentation in the schema file).
-
-If the backend is not capable of enabling a requested feature on the
-host (such as 3D acceleration with virgl), or the initialization
-failed, the backend should fail to start early and exit with a status
-!= 0. It may also print a message to stderr for further details.
-
-The backend program must not daemonize itself, but it may be
-daemonized by the management layer. It may also have a restricted
-access to the system.
-
-File descriptors 0, 1 and 2 will exist, and have regular
-stdin/stdout/stderr usage (they may have been redirected to /dev/null
-by the management layer, or to a log handler).
-
-The backend program must end (as quickly and cleanly as possible) when
-the SIGTERM signal is received. Eventually, it may receive SIGKILL by
-the management layer after a few seconds.
-
-The following command line options have an expected behaviour. They
-are mandatory, unless explicitly said differently:
-
-* --socket-path=PATH
-
-This option specify the location of the vhost-user Unix domain socket.
-It is incompatible with --fd.
-
-* --fd=FDNUM
-
-When this argument is given, the backend program is started with the
-vhost-user socket as file descriptor FDNUM. It is incompatible with
---socket-path.
-
-* --print-capabilities
-
-Output to stdout the backend capabilities in JSON format, and then
-exit successfully. Other options and arguments should be ignored, and
-the backend program should not perform its normal function. The
-capabilities can be reported dynamically depending on the host
-capabilities.
-
-The JSON output is described in the vhost-user.json schema, by
-@VHostUserBackendCapabilities. Example:
-{
- "type": "foo",
- "features": [
- "feature-a",
- "feature-b"
- ]
-}
-
-vhost-user-input
-----------------
-
-Command line options:
-
-* --evdev-path=PATH (optional)
-
-Specify the linux input device.
-
-* --no-grab (optional)
-
-Do no request exclusive access to the input device.
-
-vhost-user-gpu
---------------
-
-Command line options:
-
-* --render-node=PATH (optional)
-
-Specify the GPU DRM render node.
-
-* --virgl (optional)
-
-Enable virgl rendering support.
diff --git a/MAINTAINERS b/MAINTAINERS
index 66ddbda9c9..562d0ee0c6 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -1476,7 +1476,7 @@ M: Michael S. Tsirkin <mst@redhat.com>
S: Supported
F: hw/*/*vhost*
F: docs/interop/vhost-user.json
-F: docs/interop/vhost-user.txt
+F: docs/interop/vhost-user.rst
F: contrib/vhost-user-*/
virtio
diff --git a/docs/interop/index.rst b/docs/interop/index.rst
index 2df977dd52..a037bd67ec 100644
--- a/docs/interop/index.rst
+++ b/docs/interop/index.rst
@@ -15,4 +15,4 @@ Contents:
bitmaps
live-block-operations
pr-helper
-
+ vhost-user
diff --git a/docs/interop/vhost-user.rst b/docs/interop/vhost-user.rst
new file mode 100644
index 0000000000..7f3232c798
--- /dev/null
+++ b/docs/interop/vhost-user.rst
@@ -0,0 +1,1351 @@
+===================
+Vhost-user Protocol
+===================
+:Copyright: 2014 Virtual Open Systems Sarl.
+:Licence: This work is licensed under the terms of the GNU GPL,
+ version 2 or later. See the COPYING file in the top-level
+ directory.
+
+.. contents:: Table of Contents
+
+Introduction
+============
+
+This protocol is aiming to complement the ``ioctl`` interface used to
+control the vhost implementation in the Linux kernel. It implements
+the control plane needed to establish virtqueue sharing with a user
+space process on the same host. It uses communication over a Unix
+domain socket to share file descriptors in the ancillary data of the
+message.
+
+The protocol defines 2 sides of the communication, *master* and
+*slave*. *Master* is the application that shares its virtqueues, in
+our case QEMU. *Slave* is the consumer of the virtqueues.
+
+In the current implementation QEMU is the *master*, and the *slave* is
+the external process consuming the virtio queues, for example a
+software Ethernet switch running in user space, such as Snabbswitch,
+or a block device backend processing read & write to a virtual
+disk. In order to facilitate interoperability between various backend
+implementations, it is recommended to follow the :ref:`Backend program
+conventions <backend_conventions>`.
+
+*Master* and *slave* can be either a client (i.e. connecting) or
+server (listening) in the socket communication.
+
+Message Specification
+=====================
+
+.. Note:: All numbers are in the machine native byte order.
+
+A vhost-user message consists of 3 header fields and a payload.
+
++---------+-------+------+---------+
+| request | flags | size | payload |
++---------+-------+------+---------+
+
+Header
+------
+
+:request: 32-bit type of the request
+
+:flags: 32-bit bit field
+
+- Lower 2 bits are the version (currently 0x01)
+- Bit 2 is the reply flag - needs to be sent on each reply from the slave
+- Bit 3 is the need_reply flag - see :ref:`REPLY_ACK <reply_ack>` for
+ details.
+
+:size: 32-bit size of the payload
+
+Payload
+-------
+
+Depending on the request type, **payload** can be:
+
+A single 64-bit integer
+^^^^^^^^^^^^^^^^^^^^^^^
+
++-----+
+| u64 |
++-----+
+
+:u64: a 64-bit unsigned integer
+
+A vring state description
+^^^^^^^^^^^^^^^^^^^^^^^^^
+
++-------+-----+
+| index | num |
++-------+-----+
+
+:index: a 32-bit index
+
+:num: a 32-bit number
+
+A vring address description
+^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
++-------+-------+------+------------+------+-----------+-----+
+| index | flags | size | descriptor | used | available | log |
++-------+-------+------+------------+------+-----------+-----+
+
+:index: a 32-bit vring index
+
+:flags: a 32-bit vring flags
+
+:descriptor: a 64-bit ring address of the vring descriptor table
+
+:used: a 64-bit ring address of the vring used ring
+
+:available: a 64-bit ring address of the vring available ring
+
+:log: a 64-bit guest address for logging
+
+Note that a ring address is an IOVA if ``VIRTIO_F_IOMMU_PLATFORM`` has
+been negotiated. Otherwise it is a user address.
+
+Memory regions description
+^^^^^^^^^^^^^^^^^^^^^^^^^^
+
++-------------+---------+---------+-----+---------+
+| num regions | padding | region0 | ... | region7 |
++-------------+---------+---------+-----+---------+
+
+:num regions: a 32-bit number of regions
+
+:padding: 32-bit
+
+A region is:
+
++---------------+------+--------------+-------------+
+| guest address | size | user address | mmap offset |
++---------------+------+--------------+-------------+
+
+:guest address: a 64-bit guest address of the region
+
+:size: a 64-bit size
+
+:user address: a 64-bit user address
+
+:mmap offset: 64-bit offset where region starts in the mapped memory
+
+Log description
+^^^^^^^^^^^^^^^
+
++----------+------------+
+| log size | log offset |
++----------+------------+
+
+:log size: size of area used for logging
+
+:log offset: offset from start of supplied file descriptor where
+ logging starts (i.e. where guest address 0 would be
+ logged)
+
+An IOTLB message
+^^^^^^^^^^^^^^^^
+
++------+------+--------------+-------------------+------+
+| iova | size | user address | permissions flags | type |
++------+------+--------------+-------------------+------+
+
+:iova: a 64-bit I/O virtual address programmed by the guest
+
+:size: a 64-bit size
+
+:user address: a 64-bit user address
+
+:permissions flags: an 8-bit value:
+ - 0: No access
+ - 1: Read access
+ - 2: Write access
+ - 3: Read/Write access
+
+:type: an 8-bit IOTLB message type:
+ - 1: IOTLB miss
+ - 2: IOTLB update
+ - 3: IOTLB invalidate
+ - 4: IOTLB access fail
+
+Virtio device config space
+^^^^^^^^^^^^^^^^^^^^^^^^^^
+
++--------+------+-------+---------+
+| offset | size | flags | payload |
++--------+------+-------+---------+
+
+:offset: a 32-bit offset of virtio device's configuration space
+
+:size: a 32-bit configuration space access size in bytes
+
+:flags: a 32-bit value:
+ - 0: Vhost master messages used for writeable fields
+ - 1: Vhost master messages used for live migration
+
+:payload: Size bytes array holding the contents of the virtio
+ device's configuration space
+
+Vring area description
+^^^^^^^^^^^^^^^^^^^^^^
+
++-----+------+--------+
+| u64 | size | offset |
++-----+------+--------+
+
+:u64: a 64-bit integer contains vring index and flags
+
+:size: a 64-bit size of this area
+
+:offset: a 64-bit offset of this area from the start of the
+ supplied file descriptor
+
+Inflight description
+^^^^^^^^^^^^^^^^^^^^
+
++-----------+-------------+------------+------------+
+| mmap size | mmap offset | num queues | queue size |
++-----------+-------------+------------+------------+
+
+:mmap size: a 64-bit size of area to track inflight I/O
+
+:mmap offset: a 64-bit offset of this area from the start
+ of the supplied file descriptor
+
+:num queues: a 16-bit number of virtqueues
+
+:queue size: a 16-bit size of virtqueues
+
+C structure
+-----------
+
+In QEMU the vhost-user message is implemented with the following struct:
+
+.. code:: c
+
+ typedef struct VhostUserMsg {
+ VhostUserRequest request;
+ uint32_t flags;
+ uint32_t size;
+ union {
+ uint64_t u64;
+ struct vhost_vring_state state;
+ struct vhost_vring_addr addr;
+ VhostUserMemory memory;
+ VhostUserLog log;
+ struct vhost_iotlb_msg iotlb;
+ VhostUserConfig config;
+ VhostUserVringArea area;
+ VhostUserInflight inflight;
+ };
+ } QEMU_PACKED VhostUserMsg;
+
+Communication
+=============
+
+The protocol for vhost-user is based on the existing implementation of
+vhost for the Linux Kernel. Most messages that can be sent via the
+Unix domain socket implementing vhost-user have an equivalent ioctl to
+the kernel implementation.
+
+The communication consists of *master* sending message requests and
+*slave* sending message replies. Most of the requests don't require
+replies. Here is a list of the ones that do:
+
+* ``VHOST_USER_GET_FEATURES``
+* ``VHOST_USER_GET_PROTOCOL_FEATURES``
+* ``VHOST_USER_GET_VRING_BASE``
+* ``VHOST_USER_SET_LOG_BASE`` (if ``VHOST_USER_PROTOCOL_F_LOG_SHMFD``)
+* ``VHOST_USER_GET_INFLIGHT_FD`` (if ``VHOST_USER_PROTOCOL_F_INFLIGHT_SHMFD``)
+
+.. seealso::
+
+ :ref:`REPLY_ACK <reply_ack>`
+ The section on ``REPLY_ACK`` protocol extension.
+
+There are several messages that the master sends with file descriptors passed
+in the ancillary data:
+
+* ``VHOST_USER_SET_MEM_TABLE``
+* ``VHOST_USER_SET_LOG_BASE`` (if ``VHOST_USER_PROTOCOL_F_LOG_SHMFD``)
+* ``VHOST_USER_SET_LOG_FD``
+* ``VHOST_USER_SET_VRING_KICK``
+* ``VHOST_USER_SET_VRING_CALL``
+* ``VHOST_USER_SET_VRING_ERR``
+* ``VHOST_USER_SET_SLAVE_REQ_FD``
+* ``VHOST_USER_SET_INFLIGHT_FD`` (if ``VHOST_USER_PROTOCOL_F_INFLIGHT_SHMFD``)
+
+If *master* is unable to send the full message or receives a wrong
+reply it will close the connection. An optional reconnection mechanism
+can be implemented.
+
+Any protocol extensions are gated by protocol feature bits, which
+allows full backwards compatibility on both master and slave. As
+older slaves don't support negotiating protocol features, a feature
+bit was dedicated for this purpose::
+
+ #define VHOST_USER_F_PROTOCOL_FEATURES 30
+
+Starting and stopping rings
+---------------------------
+
+Client must only process each ring when it is started.
+
+Client must only pass data between the ring and the backend, when the
+ring is enabled.
+
+If ring is started but disabled, client must process the ring without
+talking to the backend.
+
+For example, for a networking device, in the disabled state client
+must not supply any new RX packets, but must process and discard any
+TX packets.
+
+If ``VHOST_USER_F_PROTOCOL_FEATURES`` has not been negotiated, the
+ring is initialized in an enabled state.
+
+If ``VHOST_USER_F_PROTOCOL_FEATURES`` has been negotiated, the ring is
+initialized in a disabled state. Client must not pass data to/from the
+backend until ring is enabled by ``VHOST_USER_SET_VRING_ENABLE`` with
+parameter 1, or after it has been disabled by
+``VHOST_USER_SET_VRING_ENABLE`` with parameter 0.
+
+Each ring is initialized in a stopped state, client must not process
+it until ring is started, or after it has been stopped.
+
+Client must start ring upon receiving a kick (that is, detecting that
+file descriptor is readable) on the descriptor specified by
+``VHOST_USER_SET_VRING_KICK``, and stop ring upon receiving
+``VHOST_USER_GET_VRING_BASE``.
+
+While processing the rings (whether they are enabled or not), client
+must support changing some configuration aspects on the fly.
+
+Multiple queue support
+----------------------
+
+Multiple queue is treated as a protocol extension, hence the slave has
+to implement protocol features first. The multiple queues feature is
+supported only when the protocol feature ``VHOST_USER_PROTOCOL_F_MQ``
+(bit 0) is set.
+
+The max number of queue pairs the slave supports can be queried with
+message ``VHOST_USER_GET_QUEUE_NUM``. Master should stop when the
+number of requested queues is bigger than that.
+
+As all queues share one connection, the master uses a unique index for each
+queue in the sent message to identify a specified queue. One queue pair
+is enabled initially. More queues are enabled dynamically, by sending
+message ``VHOST_USER_SET_VRING_ENABLE``.
+
+Migration
+---------
+
+During live migration, the master may need to track the modifications
+the slave makes to the memory mapped regions. The client should mark
+the dirty pages in a log. Once it complies to this logging, it may
+declare the ``VHOST_F_LOG_ALL`` vhost feature.
+
+To start/stop logging of data/used ring writes, server may send
+messages ``VHOST_USER_SET_FEATURES`` with ``VHOST_F_LOG_ALL`` and
+``VHOST_USER_SET_VRING_ADDR`` with ``VHOST_VRING_F_LOG`` in ring's
+flags set to 1/0, respectively.
+
+All the modifications to memory pointed by vring "descriptor" should
+be marked. Modifications to "used" vring should be marked if
+``VHOST_VRING_F_LOG`` is part of ring's flags.
+
+Dirty pages are of size::
+
+ #define VHOST_LOG_PAGE 0x1000
+
+The log memory fd is provided in the ancillary data of
+``VHOST_USER_SET_LOG_BASE`` message when the slave has
+``VHOST_USER_PROTOCOL_F_LOG_SHMFD`` protocol feature.
+
+The size of the log is supplied as part of ``VhostUserMsg`` which
+should be large enough to cover all known guest addresses. Log starts
+at the supplied offset in the supplied file descriptor. The log
+covers from address 0 to the maximum of guest regions. In pseudo-code,
+to mark page at ``addr`` as dirty::
+
+ page = addr / VHOST_LOG_PAGE
+ log[page / 8] |= 1 << page % 8
+
+Where ``addr`` is the guest physical address.
+
+Use atomic operations, as the log may be concurrently manipulated.
+
+Note that when logging modifications to the used ring (when
+``VHOST_VRING_F_LOG`` is set for this ring), ``log_guest_addr`` should
+be used to calculate the log offset: the write to first byte of the
+used ring is logged at this offset from log start. Also note that this
+value might be outside the legal guest physical address range
+(i.e. does not have to be covered by the ``VhostUserMemory`` table), but
+the bit offset of the last byte of the ring must fall within the size
+supplied by ``VhostUserLog``.
+
+``VHOST_USER_SET_LOG_FD`` is an optional message with an eventfd in
+ancillary data, it may be used to inform the master that the log has
+been modified.
+
+Once the source has finished migration, rings will be stopped by the
+source. No further update must be done before rings are restarted.
+
+In postcopy migration the slave is started before all the memory has
+been received from the source host, and care must be taken to avoid
+accessing pages that have yet to be received. The slave opens a
+'userfault'-fd and registers the memory with it; this fd is then
+passed back over to the master. The master services requests on the
+userfaultfd for pages that are accessed and when the page is available
+it performs WAKE ioctl's on the userfaultfd to wake the stalled
+slave. The client indicates support for this via the
+``VHOST_USER_PROTOCOL_F_PAGEFAULT`` feature.
+
+Memory access
+-------------
+
+The master sends a list of vhost memory regions to the slave using the
+``VHOST_USER_SET_MEM_TABLE`` message. Each region has two base
+addresses: a guest address and a user address.
+
+Messages contain guest addresses and/or user addresses to reference locations
+within the shared memory. The mapping of these addresses works as follows.
+
+User addresses map to the vhost memory region containing that user address.
+
+When the ``VIRTIO_F_IOMMU_PLATFORM`` feature has not been negotiated:
+
+* Guest addresses map to the vhost memory region containing that guest
+ address.
+
+When the ``VIRTIO_F_IOMMU_PLATFORM`` feature has been negotiated:
+
+* Guest addresses are also called I/O virtual addresses (IOVAs). They are
+ translated to user addresses via the IOTLB.
+
+* The vhost memory region guest address is not used.
+
+IOMMU support
+-------------
+
+When the ``VIRTIO_F_IOMMU_PLATFORM`` feature has been negotiated, the
+master sends IOTLB entries update & invalidation by sending
+``VHOST_USER_IOTLB_MSG`` requests to the slave with a ``struct
+vhost_iotlb_msg`` as payload. For update events, the ``iotlb`` payload
+has to be filled with the update message type (2), the I/O virtual
+address, the size, the user virtual address, and the permissions
+flags. Addresses and size must be within vhost memory regions set via
+the ``VHOST_USER_SET_MEM_TABLE`` request. For invalidation events, the
+``iotlb`` payload has to be filled with the invalidation message type
+(3), the I/O virtual address and the size. On success, the slave is
+expected to reply with a zero payload, non-zero otherwise.
+
+The slave relies on the slave communcation channel (see :ref:`Slave
+communication <slave_communication>` section below) to send IOTLB miss
+and access failure events, by sending ``VHOST_USER_SLAVE_IOTLB_MSG``
+requests to the master with a ``struct vhost_iotlb_msg`` as
+payload. For miss events, the iotlb payload has to be filled with the
+miss message type (1), the I/O virtual address and the permissions
+flags. For access failure event, the iotlb payload has to be filled
+with the access failure message type (4), the I/O virtual address and
+the permissions flags. For synchronization purpose, the slave may
+rely on the reply-ack feature, so the master may send a reply when
+operation is completed if the reply-ack feature is negotiated and
+slaves requests a reply. For miss events, completed operation means
+either master sent an update message containing the IOTLB entry
+containing requested address and permission, or master sent nothing if
+the IOTLB miss message is invalid (invalid IOVA or permission).
+
+The master isn't expected to take the initiative to send IOTLB update
+messages, as the slave sends IOTLB miss messages for the guest virtual
+memory areas it needs to access.
+
+.. _slave_communication:
+
+Slave communication
+-------------------
+
+An optional communication channel is provided if the slave declares
+``VHOST_USER_PROTOCOL_F_SLAVE_REQ`` protocol feature, to allow the
+slave to make requests to the master.
+
+The fd is provided via ``VHOST_USER_SET_SLAVE_REQ_FD`` ancillary data.
+
+A slave may then send ``VHOST_USER_SLAVE_*`` messages to the master
+using this fd communication channel.
+
+If ``VHOST_USER_PROTOCOL_F_SLAVE_SEND_FD`` protocol feature is
+negotiated, slave can send file descriptors (at most 8 descriptors in
+each message) to master via ancillary data using this fd communication
+channel.
+
+Inflight I/O tracking
+---------------------
+
+To support reconnecting after restart or crash, slave may need to
+resubmit inflight I/Os. If virtqueue is processed in order, we can
+easily achieve that by getting the inflight descriptors from
+descriptor table (split virtqueue) or descriptor ring (packed
+virtqueue). However, it can't work when we process descriptors
+out-of-order because some entries which store the information of
+inflight descriptors in available ring (split virtqueue) or descriptor
+ring (packed virtqueue) might be overrided by new entries. To solve
+this problem, slave need to allocate an extra buffer to store this
+information of inflight descriptors and share it with master for
+persistent. ``VHOST_USER_GET_INFLIGHT_FD`` and
+``VHOST_USER_SET_INFLIGHT_FD`` are used to transfer this buffer
+between master and slave. And the format of this buffer is described
+below:
+
++---------------+---------------+-----+---------------+
+| queue0 region | queue1 region | ... | queueN region |
++---------------+---------------+-----+---------------+
+
+N is the number of available virtqueues. Slave could get it from num
+queues field of ``VhostUserInflight``.
+
+For split virtqueue, queue region can be implemented as:
+
+.. code:: c
+
+ typedef struct DescStateSplit {
+ /* Indicate whether this descriptor is inflight or not.
+ * Only available for head-descriptor. */
+ uint8_t inflight;
+
+ /* Padding */
+ uint8_t padding[5];
+
+ /* Maintain a list for the last batch of used descriptors.
+ * Only available when batching is used for submitting */
+ uint16_t next;
+
+ /* Used to preserve the order of fetching available descriptors.
+ * Only available for head-descriptor. */
+ uint64_t counter;
+ } DescStateSplit;
+
+ typedef struct QueueRegionSplit {
+ /* The feature flags of this region. Now it's initialized to 0. */
+ uint64_t features;
+
+ /* The version of this region. It's 1 currently.
+ * Zero value indicates an uninitialized buffer */
+ uint16_t version;
+
+ /* The size of DescStateSplit array. It's equal to the virtqueue
+ * size. Slave could get it from queue size field of VhostUserInflight. */
+ uint16_t desc_num;
+
+ /* The head of list that track the last batch of used descriptors. */
+ uint16_t last_batch_head;
+
+ /* Store the idx value of used ring */
+ uint16_t used_idx;
+
+ /* Used to track the state of each descriptor in descriptor table */
+ DescStateSplit desc[0];
+ } QueueRegionSplit;
+
+To track inflight I/O, the queue region should be processed as follows:
+
+When receiving available buffers from the driver:
+
+#. Get the next available head-descriptor index from available ring, ``i``
+
+#. Set ``desc[i].counter`` to the value of global counter
+
+#. Increase global counter by 1
+
+#. Set ``desc[i].inflight`` to 1
+
+When supplying used buffers to the driver:
+
+1. Get corresponding used head-descriptor index, i
+
+2. Set ``desc[i].next`` to ``last_batch_head``
+
+3. Set ``last_batch_head`` to ``i``
+
+#. Steps 1,2,3 may be performed repeatedly if batching is possible
+
+#. Increase the ``idx`` value of used ring by the size of the batch
+
+#. Set the ``inflight`` field of each ``DescStateSplit`` entry in the batch to 0
+
+#. Set ``used_idx`` to the ``idx`` value of used ring
+
+When reconnecting:
+
+#. If the value of ``used_idx`` does not match the ``idx`` value of
+ used ring (means the inflight field of ``DescStateSplit`` entries in
+ last batch may be incorrect),
+
+ a. Subtract the value of ``used_idx`` from the ``idx`` value of
+ used ring to get last batch size of ``DescStateSplit`` entries
+
+ #. Set the ``inflight`` field of each ``DescStateSplit`` entry to 0 in last batch
+ list which starts from ``last_batch_head``
+
+ #. Set ``used_idx`` to the ``idx`` value of used ring
+
+#. Resubmit inflight ``DescStateSplit`` entries in order of their
+ counter value
+
+For packed virtqueue, queue region can be implemented as:
+
+.. code:: c
+
+ typedef struct DescStatePacked {
+ /* Indicate whether this descriptor is inflight or not.
+ * Only available for head-descriptor. */
+ uint8_t inflight;
+
+ /* Padding */
+ uint8_t padding;
+
+ /* Link to the next free entry */
+ uint16_t next;
+
+ /* Link to the last entry of descriptor list.
+ * Only available for head-descriptor. */
+ uint16_t last;
+
+ /* The length of descriptor list.
+ * Only available for head-descriptor. */
+ uint16_t num;
+
+ /* Used to preserve the order of fetching available descriptors.
+ * Only available for head-descriptor. */
+ uint64_t counter;
+
+ /* The buffer id */
+ uint16_t id;
+
+ /* The descriptor flags */
+ uint16_t flags;
+
+ /* The buffer length */
+ uint32_t len;
+
+ /* The buffer address */
+ uint64_t addr;
+ } DescStatePacked;
+
+ typedef struct QueueRegionPacked {
+ /* The feature flags of this region. Now it's initialized to 0. */
+ uint64_t features;
+
+ /* The version of this region. It's 1 currently.
+ * Zero value indicates an uninitialized buffer */
+ uint16_t version;
+
+ /* The size of DescStatePacked array. It's equal to the virtqueue
+ * size. Slave could get it from queue size field of VhostUserInflight. */
+ uint16_t desc_num;
+
+ /* The head of free DescStatePacked entry list */
+ uint16_t free_head;
+
+ /* The old head of free DescStatePacked entry list */
+ uint16_t old_free_head;
+
+ /* The used index of descriptor ring */
+ uint16_t used_idx;
+
+ /* The old used index of descriptor ring */
+ uint16_t old_used_idx;
+
+ /* Device ring wrap counter */
+ uint8_t used_wrap_counter;
+
+ /* The old device ring wrap counter */
+ uint8_t old_used_wrap_counter;
+
+ /* Padding */
+ uint8_t padding[7];
+
+ /* Used to track the state of each descriptor fetched from descriptor ring */
+ DescStatePacked desc[0];
+ } QueueRegionPacked;
+
+To track inflight I/O, the queue region should be processed as follows:
+
+When receiving available buffers from the driver:
+
+#. Get the next available descriptor entry from descriptor ring, ``d``
+
+#. If ``d`` is head descriptor,
+
+ a. Set ``desc[old_free_head].num`` to 0
+
+ #. Set ``desc[old_free_head].counter`` to the value of global counter
+
+ #. Increase global counter by 1
+
+ #. Set ``desc[old_free_head].inflight`` to 1
+
+#. If ``d`` is last descriptor, set ``desc[old_free_head].last`` to
+ ``free_head``
+
+#. Increase ``desc[old_free_head].num`` by 1
+
+#. Set ``desc[free_head].addr``, ``desc[free_head].len``,
+ ``desc[free_head].flags``, ``desc[free_head].id`` to ``d.addr``,
+ ``d.len``, ``d.flags``, ``d.id``
+
+#. Set ``free_head`` to ``desc[free_head].next``
+
+#. If ``d`` is last descriptor, set ``old_free_head`` to ``free_head``
+
+When supplying used buffers to the driver:
+
+1. Get corresponding used head-descriptor entry from descriptor ring,
+ ``d``
+
+2. Get corresponding ``DescStatePacked`` entry, ``e``
+
+3. Set ``desc[e.last].next`` to ``free_head``
+
+4. Set ``free_head`` to the index of ``e``
+
+#. Steps 1,2,3,4 may be performed repeatedly if batching is possible
+
+#. Increase ``used_idx`` by the size of the batch and update
+ ``used_wrap_counter`` if needed
+
+#. Update ``d.flags``
+
+#. Set the ``inflight`` field of each head ``DescStatePacked`` entry
+ in the batch to 0
+
+#. Set ``old_free_head``, ``old_used_idx``, ``old_used_wrap_counter``
+ to ``free_head``, ``used_idx``, ``used_wrap_counter``
+
+When reconnecting:
+
+#. If ``used_idx`` does not match ``old_used_idx`` (means the
+ ``inflight`` field of ``DescStatePacked`` entries in last batch may
+ be incorrect),
+
+ a. Get the next descriptor ring entry through ``old_used_idx``, ``d``
+
+ #. Use ``old_used_wrap_counter`` to calculate the available flags
+
+ #. If ``d.flags`` is not equal to the calculated flags value (means
+ slave has submitted the buffer to guest driver before crash, so
+ it has to commit the in-progres update), set ``old_free_head``,
+ ``old_used_idx``, ``old_used_wrap_counter`` to ``free_head``,
+ ``used_idx``, ``used_wrap_counter``
+
+#. Set ``free_head``, ``used_idx``, ``used_wrap_counter`` to
+ ``old_free_head``, ``old_used_idx``, ``old_used_wrap_counter``
+ (roll back any in-progress update)
+
+#. Set the ``inflight`` field of each ``DescStatePacked`` entry in
+ free list to 0
+
+#. Resubmit inflight ``DescStatePacked`` entries in order of their
+ counter value
+
+Protocol features
+-----------------
+
+.. code:: c
+
+ #define VHOST_USER_PROTOCOL_F_MQ 0
+ #define VHOST_USER_PROTOCOL_F_LOG_SHMFD 1
+ #define VHOST_USER_PROTOCOL_F_RARP 2
+ #define VHOST_USER_PROTOCOL_F_REPLY_ACK 3
+ #define VHOST_USER_PROTOCOL_F_MTU 4
+ #define VHOST_USER_PROTOCOL_F_SLAVE_REQ 5
+ #define VHOST_USER_PROTOCOL_F_CROSS_ENDIAN 6
+ #define VHOST_USER_PROTOCOL_F_CRYPTO_SESSION 7
+ #define VHOST_USER_PROTOCOL_F_PAGEFAULT 8
+ #define VHOST_USER_PROTOCOL_F_CONFIG 9
+ #define VHOST_USER_PROTOCOL_F_SLAVE_SEND_FD 10
+ #define VHOST_USER_PROTOCOL_F_HOST_NOTIFIER 11
+ #define VHOST_USER_PROTOCOL_F_INFLIGHT_SHMFD 12
+
+Master message types
+--------------------
+
+``VHOST_USER_GET_FEATURES``
+ :id: 1
+ :equivalent ioctl: ``VHOST_GET_FEATURES``
+ :master payload: N/A
+ :slave payload: ``u64``
+
+ Get from the underlying vhost implementation the features bitmask.
+ Feature bit ``VHOST_USER_F_PROTOCOL_FEATURES`` signals slave support
+ for ``VHOST_USER_GET_PROTOCOL_FEATURES`` and
+ ``VHOST_USER_SET_PROTOCOL_FEATURES``.
+
+``VHOST_USER_SET_FEATURES``
+ :id: 2
+ :equivalent ioctl: ``VHOST_SET_FEATURES``
+ :master payload: ``u64``
+
+ Enable features in the underlying vhost implementation using a
+ bitmask. Feature bit ``VHOST_USER_F_PROTOCOL_FEATURES`` signals
+ slave support for ``VHOST_USER_GET_PROTOCOL_FEATURES`` and
+ ``VHOST_USER_SET_PROTOCOL_FEATURES``.
+
+``VHOST_USER_GET_PROTOCOL_FEATURES``
+ :id: 15
+ :equivalent ioctl: ``VHOST_GET_FEATURES``
+ :master payload: N/A
+ :slave payload: ``u64``
+
+ Get the protocol feature bitmask from the underlying vhost
+ implementation. Only legal if feature bit
+ ``VHOST_USER_F_PROTOCOL_FEATURES`` is present in
+ ``VHOST_USER_GET_FEATURES``.
+
+.. Note::
+ Slave that reported ``VHOST_USER_F_PROTOCOL_FEATURES`` must
+ support this message even before ``VHOST_USER_SET_FEATURES`` was
+ called.
+
+``VHOST_USER_SET_PROTOCOL_FEATURES``
+ :id: 16
+ :equivalent ioctl: ``VHOST_SET_FEATURES``
+ :master payload: ``u64``
+
+ Enable protocol features in the underlying vhost implementation.
+
+ Only legal if feature bit ``VHOST_USER_F_PROTOCOL_FEATURES`` is present in
+ ``VHOST_USER_GET_FEATURES``.
+
+.. Note::
+ Slave that reported ``VHOST_USER_F_PROTOCOL_FEATURES`` must support
+ this message even before ``VHOST_USER_SET_FEATURES`` was called.
+
+``VHOST_USER_SET_OWNER``
+ :id: 3
+ :equivalent ioctl: ``VHOST_SET_OWNER``
+ :master payload: N/A
+
+ Issued when a new connection is established. It sets the current
+ *master* as an owner of the session. This can be used on the *slave*
+ as a "session start" flag.
+
+``VHOST_USER_RESET_OWNER``
+ :id: 4
+ :master payload: N/A
+
+.. admonition:: Deprecated
+
+ This is no longer used. Used to be sent to request disabling all
+ rings, but some clients interpreted it to also discard connection
+ state (this interpretation would lead to bugs). It is recommended
+ that clients either ignore this message, or use it to disable all
+ rings.
+
+``VHOST_USER_SET_MEM_TABLE``
+ :id: 5
+ :equivalent ioctl: ``VHOST_SET_MEM_TABLE``
+ :master payload: memory regions description
+ :slave payload: (postcopy only) memory regions description
+
+ Sets the memory map regions on the slave so it can translate the
+ vring addresses. In the ancillary data there is an array of file
+ descriptors for each memory mapped region. The size and ordering of
+ the fds matches the number and ordering of memory regions.
+
+ When ``VHOST_USER_POSTCOPY_LISTEN`` has been received,
+ ``SET_MEM_TABLE`` replies with the bases of the memory mapped
+ regions to the master. The slave must have mmap'd the regions but
+ not yet accessed them and should not yet generate a userfault
+ event.
+
+.. Note::
+ ``NEED_REPLY_MASK`` is not set in this case. QEMU will then
+ reply back to the list of mappings with an empty
+ ``VHOST_USER_SET_MEM_TABLE`` as an acknowledgement; only upon
+ reception of this message may the guest start accessing the memory
+ and generating faults.
+
+``VHOST_USER_SET_LOG_BASE``
+ :id: 6
+ :equivalent ioctl: ``VHOST_SET_LOG_BASE``
+ :master payload: u64
+ :slave payload: N/A
+
+ Sets logging shared memory space.
+
+ When slave has ``VHOST_USER_PROTOCOL_F_LOG_SHMFD`` protocol feature,
+ the log memory fd is provided in the ancillary data of
+ ``VHOST_USER_SET_LOG_BASE`` message, the size and offset of shared
+ memory area provided in the message.
+
+``VHOST_USER_SET_LOG_FD``
+ :id: 7
+ :equivalent ioctl: ``VHOST_SET_LOG_FD``
+ :master payload: N/A
+
+ Sets the logging file descriptor, which is passed as ancillary data.
+
+``VHOST_USER_SET_VRING_NUM``
+ :id: 8
+ :equivalent ioctl: ``VHOST_SET_VRING_NUM``
+ :master payload: vring state description
+
+ Set the size of the queue.
+
+``VHOST_USER_SET_VRING_ADDR``
+ :id: 9
+ :equivalent ioctl: ``VHOST_SET_VRING_ADDR``
+ :master payload: vring address description
+ :slave payload: N/A
+
+ Sets the addresses of the different aspects of the vring.
+
+``VHOST_USER_SET_VRING_BASE``
+ :id: 10
+ :equivalent ioctl: ``VHOST_SET_VRING_BASE``
+ :master payload: vring state description
+
+ Sets the base offset in the available vring.
+
+``VHOST_USER_GET_VRING_BASE``
+ :id: 11
+ :equivalent ioctl: ``VHOST_USER_GET_VRING_BASE``
+ :master payload: vring state description
+ :slave payload: vring state description
+
+ Get the available vring base offset.
+
+``VHOST_USER_SET_VRING_KICK``
+ :id: 12
+ :equivalent ioctl: ``VHOST_SET_VRING_KICK``
+ :master payload: ``u64``
+
+ Set the event file descriptor for adding buffers to the vring. It is
+ passed in the ancillary data.
+
+ Bits (0-7) of the payload contain the vring index. Bit 8 is the
+ invalid FD flag. This flag is set when there is no file descriptor
+ in the ancillary data. This signals that polling should be used
+ instead of waiting for a kick.
+
+``VHOST_USER_SET_VRING_CALL``
+ :id: 13
+ :equivalent ioctl: ``VHOST_SET_VRING_CALL``
+ :master payload: ``u64``
+
+ Set the event file descriptor to signal when buffers are used. It is
+ passed in the ancillary data.
+
+ Bits (0-7) of the payload contain the vring index. Bit 8 is the
+ invalid FD flag. This flag is set when there is no file descriptor
+ in the ancillary data. This signals that polling will be used
+ instead of waiting for the call.
+
+``VHOST_USER_SET_VRING_ERR``
+ :id: 14
+ :equivalent ioctl: ``VHOST_SET_VRING_ERR``
+ :master payload: ``u64``
+
+ Set the event file descriptor to signal when error occurs. It is
+ passed in the ancillary data.
+
+ Bits (0-7) of the payload contain the vring index. Bit 8 is the
+ invalid FD flag. This flag is set when there is no file descriptor
+ in the ancillary data.
+
+``VHOST_USER_GET_QUEUE_NUM``
+ :id: 17
+ :equivalent ioctl: N/A
+ :master payload: N/A
+ :slave payload: u64
+
+ Query how many queues the backend supports.
+
+ This request should be sent only when ``VHOST_USER_PROTOCOL_F_MQ``
+ is set in queried protocol features by
+ ``VHOST_USER_GET_PROTOCOL_FEATURES``.
+
+``VHOST_USER_SET_VRING_ENABLE``
+ :id: 18
+ :equivalent ioctl: N/A
+ :master payload: vring state description
+
+ Signal slave to enable or disable corresponding vring.
+
+ This request should be sent only when
+ ``VHOST_USER_F_PROTOCOL_FEATURES`` has been negotiated.
+
+``VHOST_USER_SEND_RARP``
+ :id: 19
+ :equivalent ioctl: N/A
+ :master payload: ``u64``
+
+ Ask vhost user backend to broadcast a fake RARP to notify the migration
+ is terminated for guest that does not support GUEST_ANNOUNCE.
+
+ Only legal if feature bit ``VHOST_USER_F_PROTOCOL_FEATURES`` is
+ present in ``VHOST_USER_GET_FEATURES`` and protocol feature bit
+ ``VHOST_USER_PROTOCOL_F_RARP`` is present in
+ ``VHOST_USER_GET_PROTOCOL_FEATURES``. The first 6 bytes of the
+ payload contain the mac address of the guest to allow the vhost user
+ backend to construct and broadcast the fake RARP.
+
+``VHOST_USER_NET_SET_MTU``
+ :id: 20
+ :equivalent ioctl: N/A
+ :master payload: ``u64``
+
+ Set host MTU value exposed to the guest.
+
+ This request should be sent only when ``VIRTIO_NET_F_MTU`` feature
+ has been successfully negotiated, ``VHOST_USER_F_PROTOCOL_FEATURES``
+ is present in ``VHOST_USER_GET_FEATURES`` and protocol feature bit
+ ``VHOST_USER_PROTOCOL_F_NET_MTU`` is present in
+ ``VHOST_USER_GET_PROTOCOL_FEATURES``.
+
+ If ``VHOST_USER_PROTOCOL_F_REPLY_ACK`` is negotiated, slave must
+ respond with zero in case the specified MTU is valid, or non-zero
+ otherwise.
+
+``VHOST_USER_SET_SLAVE_REQ_FD``
+ :id: 21
+ :equivalent ioctl: N/A
+ :master payload: N/A
+
+ Set the socket file descriptor for slave initiated requests. It is passed
+ in the ancillary data.
+
+ This request should be sent only when
+ ``VHOST_USER_F_PROTOCOL_FEATURES`` has been negotiated, and protocol
+ feature bit ``VHOST_USER_PROTOCOL_F_SLAVE_REQ`` bit is present in
+ ``VHOST_USER_GET_PROTOCOL_FEATURES``. If
+ ``VHOST_USER_PROTOCOL_F_REPLY_ACK`` is negotiated, slave must
+ respond with zero for success, non-zero otherwise.
+
+``VHOST_USER_IOTLB_MSG``
+ :id: 22
+ :equivalent ioctl: N/A (equivalent to ``VHOST_IOTLB_MSG`` message type)
+ :master payload: ``struct vhost_iotlb_msg``
+ :slave payload: ``u64``
+
+ Send IOTLB messages with ``struct vhost_iotlb_msg`` as payload.
+
+ Master sends such requests to update and invalidate entries in the
+ device IOTLB. The slave has to acknowledge the request with sending
+ zero as ``u64`` payload for success, non-zero otherwise.
+
+ This request should be send only when ``VIRTIO_F_IOMMU_PLATFORM``
+ feature has been successfully negotiated.
+
+``VHOST_USER_SET_VRING_ENDIAN``
+ :id: 23
+ :equivalent ioctl: ``VHOST_SET_VRING_ENDIAN``
+ :master payload: vring state description
+
+ Set the endianness of a VQ for legacy devices. Little-endian is
+ indicated with state.num set to 0 and big-endian is indicated with
+ state.num set to 1. Other values are invalid.
+
+ This request should be sent only when
+ ``VHOST_USER_PROTOCOL_F_CROSS_ENDIAN`` has been negotiated.
+ Backends that negotiated this feature should handle both
+ endiannesses and expect this message once (per VQ) during device
+ configuration (ie. before the master starts the VQ).
+
+``VHOST_USER_GET_CONFIG``
+ :id: 24
+ :equivalent ioctl: N/A
+ :master payload: virtio device config space
+ :slave payload: virtio device config space
+
+ When ``VHOST_USER_PROTOCOL_F_CONFIG`` is negotiated, this message is
+ submitted by the vhost-user master to fetch the contents of the
+ virtio device configuration space, vhost-user slave's payload size
+ MUST match master's request, vhost-user slave uses zero length of
+ payload to indicate an error to vhost-user master. The vhost-user
+ master may cache the contents to avoid repeated
+ ``VHOST_USER_GET_CONFIG`` calls.
+
+``VHOST_USER_SET_CONFIG``
+ :id: 25
+ :equivalent ioctl: N/A
+ :master payload: virtio device config space
+ :slave payload: N/A
+
+ When ``VHOST_USER_PROTOCOL_F_CONFIG`` is negotiated, this message is
+ submitted by the vhost-user master when the Guest changes the virtio
+ device configuration space and also can be used for live migration
+ on the destination host. The vhost-user slave must check the flags
+ field, and slaves MUST NOT accept SET_CONFIG for read-only
+ configuration space fields unless the live migration bit is set.
+
+``VHOST_USER_CREATE_CRYPTO_SESSION``
+ :id: 26
+ :equivalent ioctl: N/A
+ :master payload: crypto session description
+ :slave payload: crypto session description
+
+ Create a session for crypto operation. The server side must return
+ the session id, 0 or positive for success, negative for failure.
+ This request should be sent only when
+ ``VHOST_USER_PROTOCOL_F_CRYPTO_SESSION`` feature has been
+ successfully negotiated. It's a required feature for crypto
+ devices.
+
+``VHOST_USER_CLOSE_CRYPTO_SESSION``
+ :id: 27
+ :equivalent ioctl: N/A
+ :master payload: ``u64``
+
+ Close a session for crypto operation which was previously
+ created by ``VHOST_USER_CREATE_CRYPTO_SESSION``.
+
+ This request should be sent only when
+ ``VHOST_USER_PROTOCOL_F_CRYPTO_SESSION`` feature has been
+ successfully negotiated. It's a required feature for crypto
+ devices.
+
+``VHOST_USER_POSTCOPY_ADVISE``
+ :id: 28
+ :master payload: N/A
+ :slave payload: userfault fd
+
+ When ``VHOST_USER_PROTOCOL_F_PAGEFAULT`` is supported, the master
+ advises slave that a migration with postcopy enabled is underway,
+ the slave must open a userfaultfd for later use. Note that at this
+ stage the migration is still in precopy mode.
+
+``VHOST_USER_POSTCOPY_LISTEN``
+ :id: 29
+ :master payload: N/A
+
+ Master advises slave that a transition to postcopy mode has
+ happened. The slave must ensure that shared memory is registered
+ with userfaultfd to cause faulting of non-present pages.
+
+ This is always sent sometime after a ``VHOST_USER_POSTCOPY_ADVISE``,
+ and thus only when ``VHOST_USER_PROTOCOL_F_PAGEFAULT`` is supported.
+
+``VHOST_USER_POSTCOPY_END``
+ :id: 30
+ :slave payload: ``u64``
+
+ Master advises that postcopy migration has now completed. The slave
+ must disable the userfaultfd. The response is an acknowledgement
+ only.
+
+ When ``VHOST_USER_PROTOCOL_F_PAGEFAULT`` is supported, this message
+ is sent at the end of the migration, after
+ ``VHOST_USER_POSTCOPY_LISTEN`` was previously sent.
+
+ The value returned is an error indication; 0 is success.
+
+``VHOST_USER_GET_INFLIGHT_FD``
+ :id: 31
+ :equivalent ioctl: N/A
+ :master payload: inflight description
+
+ When ``VHOST_USER_PROTOCOL_F_INFLIGHT_SHMFD`` protocol feature has
+ been successfully negotiated, this message is submitted by master to
+ get a shared buffer from slave. The shared buffer will be used to
+ track inflight I/O by slave. QEMU should retrieve a new one when vm
+ reset.
+
+``VHOST_USER_SET_INFLIGHT_FD``
+ :id: 32
+ :equivalent ioctl: N/A
+ :master payload: inflight description
+
+ When ``VHOST_USER_PROTOCOL_F_INFLIGHT_SHMFD`` protocol feature has
+ been successfully negotiated, this message is submitted by master to
+ send the shared inflight buffer back to slave so that slave could
+ get inflight I/O after a crash or restart.
+
+Slave message types
+-------------------
+
+``VHOST_USER_SLAVE_IOTLB_MSG``
+ :id: 1
+ :equivalent ioctl: N/A (equivalent to ``VHOST_IOTLB_MSG`` message type)
+ :slave payload: ``struct vhost_iotlb_msg``
+ :master payload: N/A
+
+ Send IOTLB messages with ``struct vhost_iotlb_msg`` as payload.
+ Slave sends such requests to notify of an IOTLB miss, or an IOTLB
+ access failure. If ``VHOST_USER_PROTOCOL_F_REPLY_ACK`` is
+ negotiated, and slave set the ``VHOST_USER_NEED_REPLY`` flag, master
+ must respond with zero when operation is successfully completed, or
+ non-zero otherwise. This request should be send only when
+ ``VIRTIO_F_IOMMU_PLATFORM`` feature has been successfully
+ negotiated.
+
+``VHOST_USER_SLAVE_CONFIG_CHANGE_MSG``
+ :id: 2
+ :equivalent ioctl: N/A
+ :slave payload: N/A
+ :master payload: N/A
+
+ When ``VHOST_USER_PROTOCOL_F_CONFIG`` is negotiated, vhost-user
+ slave sends such messages to notify that the virtio device's
+ configuration space has changed, for those host devices which can
+ support such feature, host driver can send ``VHOST_USER_GET_CONFIG``
+ message to slave to get the latest content. If
+ ``VHOST_USER_PROTOCOL_F_REPLY_ACK`` is negotiated, and slave set the
+ ``VHOST_USER_NEED_REPLY`` flag, master must respond with zero when
+ operation is successfully completed, or non-zero otherwise.
+
+``VHOST_USER_SLAVE_VRING_HOST_NOTIFIER_MSG``
+ :id: 3
+ :equivalent ioctl: N/A
+ :slave payload: vring area description
+ :master payload: N/A
+
+ Sets host notifier for a specified queue. The queue index is
+ contained in the ``u64`` field of the vring area description. The
+ host notifier is described by the file descriptor (typically it's a
+ VFIO device fd) which is passed as ancillary data and the size
+ (which is mmap size and should be the same as host page size) and
+ offset (which is mmap offset) carried in the vring area
+ description. QEMU can mmap the file descriptor based on the size and
+ offset to get a memory range. Registering a host notifier means
+ mapping this memory range to the VM as the specified queue's notify
+ MMIO region. Slave sends this request to tell QEMU to de-register
+ the existing notifier if any and register the new notifier if the
+ request is sent with a file descriptor.
+
+ This request should be sent only when
+ ``VHOST_USER_PROTOCOL_F_HOST_NOTIFIER`` protocol feature has been
+ successfully negotiated.
+
+.. _reply_ack:
+
+VHOST_USER_PROTOCOL_F_REPLY_ACK
+-------------------------------
+
+The original vhost-user specification only demands replies for certain
+commands. This differs from the vhost protocol implementation where
+commands are sent over an ``ioctl()`` call and block until the client
+has completed.
+
+With this protocol extension negotiated, the sender (QEMU) can set the
+``need_reply`` [Bit 3] flag to any command. This indicates that the
+client MUST respond with a Payload ``VhostUserMsg`` indicating success
+or failure. The payload should be set to zero on success or non-zero
+on failure, unless the message already has an explicit reply body.
+
+The response payload gives QEMU a deterministic indication of the result
+of the command. Today, QEMU is expected to terminate the main vhost-user
+loop upon receiving such errors. In future, qemu could be taught to be more
+resilient for selective requests.
+
+For the message types that already solicit a reply from the client,
+the presence of ``VHOST_USER_PROTOCOL_F_REPLY_ACK`` or need_reply bit
+being set brings no behavioural change. (See the Communication_
+section for details.)
+
+.. _backend_conventions:
+
+Backend program conventions
+===========================
+
+vhost-user backends can provide various devices & services and may
+need to be configured manually depending on the use case. However, it
+is a good idea to follow the conventions listed here when
+possible. Users, QEMU or libvirt, can then rely on some common
+behaviour to avoid heterogenous configuration and management of the
+backend programs and facilitate interoperability.
+
+Each backend installed on a host system should come with at least one
+JSON file that conforms to the vhost-user.json schema. Each file
+informs the management applications about the backend type, and binary
+location. In addition, it defines rules for management apps for
+picking the highest priority backend when multiple match the search
+criteria (see ``@VhostUserBackend`` documentation in the schema file).
+
+If the backend is not capable of enabling a requested feature on the
+host (such as 3D acceleration with virgl), or the initialization
+failed, the backend should fail to start early and exit with a status
+!= 0. It may also print a message to stderr for further details.
+
+The backend program must not daemonize itself, but it may be
+daemonized by the management layer. It may also have a restricted
+access to the system.
+
+File descriptors 0, 1 and 2 will exist, and have regular
+stdin/stdout/stderr usage (they may have been redirected to /dev/null
+by the management layer, or to a log handler).
+
+The backend program must end (as quickly and cleanly as possible) when
+the SIGTERM signal is received. Eventually, it may receive SIGKILL by
+the management layer after a few seconds.
+
+The following command line options have an expected behaviour. They
+are mandatory, unless explicitly said differently:
+
+--socket-path=PATH
+
+ This option specify the location of the vhost-user Unix domain socket.
+ It is incompatible with --fd.
+
+--fd=FDNUM
+
+ When this argument is given, the backend program is started with the
+ vhost-user socket as file descriptor FDNUM. It is incompatible with
+ --socket-path.
+
+--print-capabilities
+
+ Output to stdout the backend capabilities in JSON format, and then
+ exit successfully. Other options and arguments should be ignored, and
+ the backend program should not perform its normal function. The
+ capabilities can be reported dynamically depending on the host
+ capabilities.
+
+The JSON output is described in the ``vhost-user.json`` schema, by
+```@VHostUserBackendCapabilities``. Example:
+
+.. code:: json
+
+ {
+ "type": "foo",
+ "features": [
+ "feature-a",
+ "feature-b"
+ ]
+ }
+
+vhost-user-input
+----------------
+
+Command line options:
+
+--evdev-path=PATH
+
+ Specify the linux input device.
+
+ (optional)
+
+--no-grab
+
+ Do no request exclusive access to the input device.
+
+ (optional)
+
+vhost-user-gpu
+--------------
+
+Command line options:
+
+--render-node=PATH
+
+ Specify the GPU DRM render node.
+
+ (optional)
+
+--virgl
+
+ Enable virgl rendering support.
+
+ (optional)
--
MST
WARNING: multiple messages have this Message-ID (diff)
From: "Michael S. Tsirkin" <mst@redhat.com>
To: qemu-devel@nongnu.org
Cc: "Peter Maydell" <peter.maydell@linaro.org>,
"Jens Freimann" <jfreimann@redhat.com>,
"Marc-André Lureau" <marcandre.lureau@redhat.com>
Subject: [Qemu-devel] [PULL v2 03/36] docs: reST-ify vhost-user documentation
Date: Mon, 20 May 2019 19:10:32 -0400 [thread overview]
Message-ID: <20190315180735.13096-1-marcandre.lureau@redhat.com> (raw)
Message-ID: <20190520231032.MwgPF6kWjniZm6F3SX15D7sazs5yLWipB7UTtM3vvV0@z> (raw)
In-Reply-To: <20190520231008.20140-1-mst@redhat.com>
From: Marc-André Lureau <marcandre.lureau@redhat.com>
Signed-off-by: Marc-André Lureau <marcandre.lureau@redhat.com>
Message-Id: <20190315180735.13096-1-marcandre.lureau@redhat.com>
Reviewed-by: Jens Freimann <jfreimann@redhat.com>
Reviewed-by: Michael S. Tsirkin <mst@redhat.com>
Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
---
docs/interop/vhost-user.txt | 1219 -------------------------------
MAINTAINERS | 2 +-
docs/interop/index.rst | 2 +-
docs/interop/vhost-user.rst | 1351 +++++++++++++++++++++++++++++++++++
4 files changed, 1353 insertions(+), 1221 deletions(-)
delete mode 100644 docs/interop/vhost-user.txt
create mode 100644 docs/interop/vhost-user.rst
diff --git a/docs/interop/vhost-user.txt b/docs/interop/vhost-user.txt
deleted file mode 100644
index 4dbd530cb9..0000000000
--- a/docs/interop/vhost-user.txt
+++ /dev/null
@@ -1,1219 +0,0 @@
-Vhost-user Protocol
-===================
-
-Copyright (c) 2014 Virtual Open Systems Sarl.
-
-This work is licensed under the terms of the GNU GPL, version 2 or later.
-See the COPYING file in the top-level directory.
-===================
-
-This protocol is aiming to complement the ioctl interface used to control the
-vhost implementation in the Linux kernel. It implements the control plane needed
-to establish virtqueue sharing with a user space process on the same host. It
-uses communication over a Unix domain socket to share file descriptors in the
-ancillary data of the message.
-
-The protocol defines 2 sides of the communication, master and slave. Master is
-the application that shares its virtqueues, in our case QEMU. Slave is the
-consumer of the virtqueues.
-
-In the current implementation QEMU is the Master, and the Slave is the
-external process consuming the virtio queues, for example a software
-Ethernet switch running in user space, such as Snabbswitch, or a block
-device backend processing read & write to a virtual disk. In order to
-facilitate interoperability between various backend implementations,
-it is recommended to follow the "Backend program conventions"
-described in this document.
-
-Master and slave can be either a client (i.e. connecting) or server (listening)
-in the socket communication.
-
-Message Specification
----------------------
-
-Note that all numbers are in the machine native byte order. A vhost-user message
-consists of 3 header fields and a payload:
-
-------------------------------------
-| request | flags | size | payload |
-------------------------------------
-
- * Request: 32-bit type of the request
- * Flags: 32-bit bit field:
- - Lower 2 bits are the version (currently 0x01)
- - Bit 2 is the reply flag - needs to be sent on each reply from the slave
- - Bit 3 is the need_reply flag - see VHOST_USER_PROTOCOL_F_REPLY_ACK for
- details.
- * Size - 32-bit size of the payload
-
-
-Depending on the request type, payload can be:
-
- * A single 64-bit integer
- -------
- | u64 |
- -------
-
- u64: a 64-bit unsigned integer
-
- * A vring state description
- ---------------
- | index | num |
- ---------------
-
- Index: a 32-bit index
- Num: a 32-bit number
-
- * A vring address description
- --------------------------------------------------------------
- | index | flags | size | descriptor | used | available | log |
- --------------------------------------------------------------
-
- Index: a 32-bit vring index
- Flags: a 32-bit vring flags
- Descriptor: a 64-bit ring address of the vring descriptor table
- Used: a 64-bit ring address of the vring used ring
- Available: a 64-bit ring address of the vring available ring
- Log: a 64-bit guest address for logging
-
- Note that a ring address is an IOVA if VIRTIO_F_IOMMU_PLATFORM has been
- negotiated. Otherwise it is a user address.
-
- * Memory regions description
- ---------------------------------------------------
- | num regions | padding | region0 | ... | region7 |
- ---------------------------------------------------
-
- Num regions: a 32-bit number of regions
- Padding: 32-bit
-
- A region is:
- -----------------------------------------------------
- | guest address | size | user address | mmap offset |
- -----------------------------------------------------
-
- Guest address: a 64-bit guest address of the region
- Size: a 64-bit size
- User address: a 64-bit user address
- mmap offset: 64-bit offset where region starts in the mapped memory
-
-* Log description
- ---------------------------
- | log size | log offset |
- ---------------------------
- log size: size of area used for logging
- log offset: offset from start of supplied file descriptor
- where logging starts (i.e. where guest address 0 would be logged)
-
- * An IOTLB message
- ---------------------------------------------------------
- | iova | size | user address | permissions flags | type |
- ---------------------------------------------------------
-
- IOVA: a 64-bit I/O virtual address programmed by the guest
- Size: a 64-bit size
- User address: a 64-bit user address
- Permissions: an 8-bit value:
- - 0: No access
- - 1: Read access
- - 2: Write access
- - 3: Read/Write access
- Type: an 8-bit IOTLB message type:
- - 1: IOTLB miss
- - 2: IOTLB update
- - 3: IOTLB invalidate
- - 4: IOTLB access fail
-
- * Virtio device config space
- -----------------------------------
- | offset | size | flags | payload |
- -----------------------------------
-
- Offset: a 32-bit offset of virtio device's configuration space
- Size: a 32-bit configuration space access size in bytes
- Flags: a 32-bit value:
- - 0: Vhost master messages used for writeable fields
- - 1: Vhost master messages used for live migration
- Payload: Size bytes array holding the contents of the virtio
- device's configuration space
-
- * Vring area description
- -----------------------
- | u64 | size | offset |
- -----------------------
-
- u64: a 64-bit integer contains vring index and flags
- Size: a 64-bit size of this area
- Offset: a 64-bit offset of this area from the start of the
- supplied file descriptor
-
- * Inflight description
- -----------------------------------------------------
- | mmap size | mmap offset | num queues | queue size |
- -----------------------------------------------------
-
- mmap size: a 64-bit size of area to track inflight I/O
- mmap offset: a 64-bit offset of this area from the start
- of the supplied file descriptor
- num queues: a 16-bit number of virtqueues
- queue size: a 16-bit size of virtqueues
-
-In QEMU the vhost-user message is implemented with the following struct:
-
-typedef struct VhostUserMsg {
- VhostUserRequest request;
- uint32_t flags;
- uint32_t size;
- union {
- uint64_t u64;
- struct vhost_vring_state state;
- struct vhost_vring_addr addr;
- VhostUserMemory memory;
- VhostUserLog log;
- struct vhost_iotlb_msg iotlb;
- VhostUserConfig config;
- VhostUserVringArea area;
- VhostUserInflight inflight;
- };
-} QEMU_PACKED VhostUserMsg;
-
-Communication
--------------
-
-The protocol for vhost-user is based on the existing implementation of vhost
-for the Linux Kernel. Most messages that can be sent via the Unix domain socket
-implementing vhost-user have an equivalent ioctl to the kernel implementation.
-
-The communication consists of master sending message requests and slave sending
-message replies. Most of the requests don't require replies. Here is a list of
-the ones that do:
-
- * VHOST_USER_GET_FEATURES
- * VHOST_USER_GET_PROTOCOL_FEATURES
- * VHOST_USER_GET_VRING_BASE
- * VHOST_USER_SET_LOG_BASE (if VHOST_USER_PROTOCOL_F_LOG_SHMFD)
- * VHOST_USER_GET_INFLIGHT_FD (if VHOST_USER_PROTOCOL_F_INFLIGHT_SHMFD)
-
-[ Also see the section on REPLY_ACK protocol extension. ]
-
-There are several messages that the master sends with file descriptors passed
-in the ancillary data:
-
- * VHOST_USER_SET_MEM_TABLE
- * VHOST_USER_SET_LOG_BASE (if VHOST_USER_PROTOCOL_F_LOG_SHMFD)
- * VHOST_USER_SET_LOG_FD
- * VHOST_USER_SET_VRING_KICK
- * VHOST_USER_SET_VRING_CALL
- * VHOST_USER_SET_VRING_ERR
- * VHOST_USER_SET_SLAVE_REQ_FD
- * VHOST_USER_SET_INFLIGHT_FD (if VHOST_USER_PROTOCOL_F_INFLIGHT_SHMFD)
-
-If Master is unable to send the full message or receives a wrong reply it will
-close the connection. An optional reconnection mechanism can be implemented.
-
-Any protocol extensions are gated by protocol feature bits,
-which allows full backwards compatibility on both master
-and slave.
-As older slaves don't support negotiating protocol features,
-a feature bit was dedicated for this purpose:
-#define VHOST_USER_F_PROTOCOL_FEATURES 30
-
-Starting and stopping rings
-----------------------
-Client must only process each ring when it is started.
-
-Client must only pass data between the ring and the
-backend, when the ring is enabled.
-
-If ring is started but disabled, client must process the
-ring without talking to the backend.
-
-For example, for a networking device, in the disabled state
-client must not supply any new RX packets, but must process
-and discard any TX packets.
-
-If VHOST_USER_F_PROTOCOL_FEATURES has not been negotiated, the ring is initialized
-in an enabled state.
-
-If VHOST_USER_F_PROTOCOL_FEATURES has been negotiated, the ring is initialized
-in a disabled state. Client must not pass data to/from the backend until ring is enabled by
-VHOST_USER_SET_VRING_ENABLE with parameter 1, or after it has been disabled by
-VHOST_USER_SET_VRING_ENABLE with parameter 0.
-
-Each ring is initialized in a stopped state, client must not process it until
-ring is started, or after it has been stopped.
-
-Client must start ring upon receiving a kick (that is, detecting that file
-descriptor is readable) on the descriptor specified by
-VHOST_USER_SET_VRING_KICK, and stop ring upon receiving
-VHOST_USER_GET_VRING_BASE.
-
-While processing the rings (whether they are enabled or not), client must
-support changing some configuration aspects on the fly.
-
-Multiple queue support
-----------------------
-
-Multiple queue is treated as a protocol extension, hence the slave has to
-implement protocol features first. The multiple queues feature is supported
-only when the protocol feature VHOST_USER_PROTOCOL_F_MQ (bit 0) is set.
-
-The max number of queue pairs the slave supports can be queried with message
-VHOST_USER_GET_QUEUE_NUM. Master should stop when the number of
-requested queues is bigger than that.
-
-As all queues share one connection, the master uses a unique index for each
-queue in the sent message to identify a specified queue. One queue pair
-is enabled initially. More queues are enabled dynamically, by sending
-message VHOST_USER_SET_VRING_ENABLE.
-
-Migration
----------
-
-During live migration, the master may need to track the modifications
-the slave makes to the memory mapped regions. The client should mark
-the dirty pages in a log. Once it complies to this logging, it may
-declare the VHOST_F_LOG_ALL vhost feature.
-
-To start/stop logging of data/used ring writes, server may send messages
-VHOST_USER_SET_FEATURES with VHOST_F_LOG_ALL and VHOST_USER_SET_VRING_ADDR with
-VHOST_VRING_F_LOG in ring's flags set to 1/0, respectively.
-
-All the modifications to memory pointed by vring "descriptor" should
-be marked. Modifications to "used" vring should be marked if
-VHOST_VRING_F_LOG is part of ring's flags.
-
-Dirty pages are of size:
-#define VHOST_LOG_PAGE 0x1000
-
-The log memory fd is provided in the ancillary data of
-VHOST_USER_SET_LOG_BASE message when the slave has
-VHOST_USER_PROTOCOL_F_LOG_SHMFD protocol feature.
-
-The size of the log is supplied as part of VhostUserMsg
-which should be large enough to cover all known guest
-addresses. Log starts at the supplied offset in the
-supplied file descriptor.
-The log covers from address 0 to the maximum of guest
-regions. In pseudo-code, to mark page at "addr" as dirty:
-
-page = addr / VHOST_LOG_PAGE
-log[page / 8] |= 1 << page % 8
-
-Where addr is the guest physical address.
-
-Use atomic operations, as the log may be concurrently manipulated.
-
-Note that when logging modifications to the used ring (when VHOST_VRING_F_LOG
-is set for this ring), log_guest_addr should be used to calculate the log
-offset: the write to first byte of the used ring is logged at this offset from
-log start. Also note that this value might be outside the legal guest physical
-address range (i.e. does not have to be covered by the VhostUserMemory table),
-but the bit offset of the last byte of the ring must fall within
-the size supplied by VhostUserLog.
-
-VHOST_USER_SET_LOG_FD is an optional message with an eventfd in
-ancillary data, it may be used to inform the master that the log has
-been modified.
-
-Once the source has finished migration, rings will be stopped by
-the source. No further update must be done before rings are
-restarted.
-
-In postcopy migration the slave is started before all the memory has been
-received from the source host, and care must be taken to avoid accessing pages
-that have yet to be received. The slave opens a 'userfault'-fd and registers
-the memory with it; this fd is then passed back over to the master.
-The master services requests on the userfaultfd for pages that are accessed
-and when the page is available it performs WAKE ioctl's on the userfaultfd
-to wake the stalled slave. The client indicates support for this via the
-VHOST_USER_PROTOCOL_F_PAGEFAULT feature.
-
-Memory access
--------------
-
-The master sends a list of vhost memory regions to the slave using the
-VHOST_USER_SET_MEM_TABLE message. Each region has two base addresses: a guest
-address and a user address.
-
-Messages contain guest addresses and/or user addresses to reference locations
-within the shared memory. The mapping of these addresses works as follows.
-
-User addresses map to the vhost memory region containing that user address.
-
-When the VIRTIO_F_IOMMU_PLATFORM feature has not been negotiated:
-
- * Guest addresses map to the vhost memory region containing that guest
- address.
-
-When the VIRTIO_F_IOMMU_PLATFORM feature has been negotiated:
-
- * Guest addresses are also called I/O virtual addresses (IOVAs). They are
- translated to user addresses via the IOTLB.
-
- * The vhost memory region guest address is not used.
-
-IOMMU support
--------------
-
-When the VIRTIO_F_IOMMU_PLATFORM feature has been negotiated, the master
-sends IOTLB entries update & invalidation by sending VHOST_USER_IOTLB_MSG
-requests to the slave with a struct vhost_iotlb_msg as payload. For update
-events, the iotlb payload has to be filled with the update message type (2),
-the I/O virtual address, the size, the user virtual address, and the
-permissions flags. Addresses and size must be within vhost memory regions set
-via the VHOST_USER_SET_MEM_TABLE request. For invalidation events, the iotlb
-payload has to be filled with the invalidation message type (3), the I/O virtual
-address and the size. On success, the slave is expected to reply with a zero
-payload, non-zero otherwise.
-
-The slave relies on the slave communcation channel (see "Slave communication"
-section below) to send IOTLB miss and access failure events, by sending
-VHOST_USER_SLAVE_IOTLB_MSG requests to the master with a struct vhost_iotlb_msg
-as payload. For miss events, the iotlb payload has to be filled with the miss
-message type (1), the I/O virtual address and the permissions flags. For access
-failure event, the iotlb payload has to be filled with the access failure
-message type (4), the I/O virtual address and the permissions flags.
-For synchronization purpose, the slave may rely on the reply-ack feature,
-so the master may send a reply when operation is completed if the reply-ack
-feature is negotiated and slaves requests a reply. For miss events, completed
-operation means either master sent an update message containing the IOTLB entry
-containing requested address and permission, or master sent nothing if the IOTLB
-miss message is invalid (invalid IOVA or permission).
-
-The master isn't expected to take the initiative to send IOTLB update messages,
-as the slave sends IOTLB miss messages for the guest virtual memory areas it
-needs to access.
-
-Slave communication
--------------------
-
-An optional communication channel is provided if the slave declares
-VHOST_USER_PROTOCOL_F_SLAVE_REQ protocol feature, to allow the slave to make
-requests to the master.
-
-The fd is provided via VHOST_USER_SET_SLAVE_REQ_FD ancillary data.
-
-A slave may then send VHOST_USER_SLAVE_* messages to the master
-using this fd communication channel.
-
-If VHOST_USER_PROTOCOL_F_SLAVE_SEND_FD protocol feature is negotiated,
-slave can send file descriptors (at most 8 descriptors in each message)
-to master via ancillary data using this fd communication channel.
-
-Inflight I/O tracking
----------------------
-
-To support reconnecting after restart or crash, slave may need to resubmit
-inflight I/Os. If virtqueue is processed in order, we can easily achieve
-that by getting the inflight descriptors from descriptor table (split virtqueue)
-or descriptor ring (packed virtqueue). However, it can't work when we process
-descriptors out-of-order because some entries which store the information of
-inflight descriptors in available ring (split virtqueue) or descriptor
-ring (packed virtqueue) might be overrided by new entries. To solve this
-problem, slave need to allocate an extra buffer to store this information of inflight
-descriptors and share it with master for persistent. VHOST_USER_GET_INFLIGHT_FD and
-VHOST_USER_SET_INFLIGHT_FD are used to transfer this buffer between master
-and slave. And the format of this buffer is described below:
-
--------------------------------------------------------
-| queue0 region | queue1 region | ... | queueN region |
--------------------------------------------------------
-
-N is the number of available virtqueues. Slave could get it from num queues
-field of VhostUserInflight.
-
-For split virtqueue, queue region can be implemented as:
-
-typedef struct DescStateSplit {
- /* Indicate whether this descriptor is inflight or not.
- * Only available for head-descriptor. */
- uint8_t inflight;
-
- /* Padding */
- uint8_t padding[5];
-
- /* Maintain a list for the last batch of used descriptors.
- * Only available when batching is used for submitting */
- uint16_t next;
-
- /* Used to preserve the order of fetching available descriptors.
- * Only available for head-descriptor. */
- uint64_t counter;
-} DescStateSplit;
-
-typedef struct QueueRegionSplit {
- /* The feature flags of this region. Now it's initialized to 0. */
- uint64_t features;
-
- /* The version of this region. It's 1 currently.
- * Zero value indicates an uninitialized buffer */
- uint16_t version;
-
- /* The size of DescStateSplit array. It's equal to the virtqueue
- * size. Slave could get it from queue size field of VhostUserInflight. */
- uint16_t desc_num;
-
- /* The head of list that track the last batch of used descriptors. */
- uint16_t last_batch_head;
-
- /* Store the idx value of used ring */
- uint16_t used_idx;
-
- /* Used to track the state of each descriptor in descriptor table */
- DescStateSplit desc[0];
-} QueueRegionSplit;
-
-To track inflight I/O, the queue region should be processed as follows:
-
-When receiving available buffers from the driver:
-
- 1. Get the next available head-descriptor index from available ring, i
-
- 2. Set desc[i].counter to the value of global counter
-
- 3. Increase global counter by 1
-
- 4. Set desc[i].inflight to 1
-
-When supplying used buffers to the driver:
-
- 1. Get corresponding used head-descriptor index, i
-
- 2. Set desc[i].next to last_batch_head
-
- 3. Set last_batch_head to i
-
- 4. Steps 1,2,3 may be performed repeatedly if batching is possible
-
- 5. Increase the idx value of used ring by the size of the batch
-
- 6. Set the inflight field of each DescStateSplit entry in the batch to 0
-
- 7. Set used_idx to the idx value of used ring
-
-When reconnecting:
-
- 1. If the value of used_idx does not match the idx value of used ring (means
- the inflight field of DescStateSplit entries in last batch may be incorrect),
-
- (a) Subtract the value of used_idx from the idx value of used ring to get
- last batch size of DescStateSplit entries
-
- (b) Set the inflight field of each DescStateSplit entry to 0 in last batch
- list which starts from last_batch_head
-
- (c) Set used_idx to the idx value of used ring
-
- 2. Resubmit inflight DescStateSplit entries in order of their counter value
-
-For packed virtqueue, queue region can be implemented as:
-
-typedef struct DescStatePacked {
- /* Indicate whether this descriptor is inflight or not.
- * Only available for head-descriptor. */
- uint8_t inflight;
-
- /* Padding */
- uint8_t padding;
-
- /* Link to the next free entry */
- uint16_t next;
-
- /* Link to the last entry of descriptor list.
- * Only available for head-descriptor. */
- uint16_t last;
-
- /* The length of descriptor list.
- * Only available for head-descriptor. */
- uint16_t num;
-
- /* Used to preserve the order of fetching available descriptors.
- * Only available for head-descriptor. */
- uint64_t counter;
-
- /* The buffer id */
- uint16_t id;
-
- /* The descriptor flags */
- uint16_t flags;
-
- /* The buffer length */
- uint32_t len;
-
- /* The buffer address */
- uint64_t addr;
-} DescStatePacked;
-
-typedef struct QueueRegionPacked {
- /* The feature flags of this region. Now it's initialized to 0. */
- uint64_t features;
-
- /* The version of this region. It's 1 currently.
- * Zero value indicates an uninitialized buffer */
- uint16_t version;
-
- /* The size of DescStatePacked array. It's equal to the virtqueue
- * size. Slave could get it from queue size field of VhostUserInflight. */
- uint16_t desc_num;
-
- /* The head of free DescStatePacked entry list */
- uint16_t free_head;
-
- /* The old head of free DescStatePacked entry list */
- uint16_t old_free_head;
-
- /* The used index of descriptor ring */
- uint16_t used_idx;
-
- /* The old used index of descriptor ring */
- uint16_t old_used_idx;
-
- /* Device ring wrap counter */
- uint8_t used_wrap_counter;
-
- /* The old device ring wrap counter */
- uint8_t old_used_wrap_counter;
-
- /* Padding */
- uint8_t padding[7];
-
- /* Used to track the state of each descriptor fetched from descriptor ring */
- DescStatePacked desc[0];
-} QueueRegionPacked;
-
-To track inflight I/O, the queue region should be processed as follows:
-
-When receiving available buffers from the driver:
-
- 1. Get the next available descriptor entry from descriptor ring, d
-
- 2. If d is head descriptor,
-
- (a) Set desc[old_free_head].num to 0
-
- (b) Set desc[old_free_head].counter to the value of global counter
-
- (c) Increase global counter by 1
-
- (d) Set desc[old_free_head].inflight to 1
-
- 3. If d is last descriptor, set desc[old_free_head].last to free_head
-
- 4. Increase desc[old_free_head].num by 1
-
- 5. Set desc[free_head].addr, desc[free_head].len, desc[free_head].flags,
- desc[free_head].id to d.addr, d.len, d.flags, d.id
-
- 6. Set free_head to desc[free_head].next
-
- 7. If d is last descriptor, set old_free_head to free_head
-
-When supplying used buffers to the driver:
-
- 1. Get corresponding used head-descriptor entry from descriptor ring, d
-
- 2. Get corresponding DescStatePacked entry, e
-
- 3. Set desc[e.last].next to free_head
-
- 4. Set free_head to the index of e
-
- 5. Steps 1,2,3,4 may be performed repeatedly if batching is possible
-
- 6. Increase used_idx by the size of the batch and update used_wrap_counter if needed
-
- 7. Update d.flags
-
- 8. Set the inflight field of each head DescStatePacked entry in the batch to 0
-
- 9. Set old_free_head, old_used_idx, old_used_wrap_counter to free_head, used_idx,
- used_wrap_counter
-
-When reconnecting:
-
- 1. If used_idx does not match old_used_idx (means the inflight field of DescStatePacked
- entries in last batch may be incorrect),
-
- (a) Get the next descriptor ring entry through old_used_idx, d
-
- (b) Use old_used_wrap_counter to calculate the available flags
-
- (c) If d.flags is not equal to the calculated flags value (means slave has
- submitted the buffer to guest driver before crash, so it has to commit the
- in-progres update), set old_free_head, old_used_idx, old_used_wrap_counter
- to free_head, used_idx, used_wrap_counter
-
- 2. Set free_head, used_idx, used_wrap_counter to old_free_head, old_used_idx,
- old_used_wrap_counter (roll back any in-progress update)
-
- 3. Set the inflight field of each DescStatePacked entry in free list to 0
-
- 4. Resubmit inflight DescStatePacked entries in order of their counter value
-
-Protocol features
------------------
-
-#define VHOST_USER_PROTOCOL_F_MQ 0
-#define VHOST_USER_PROTOCOL_F_LOG_SHMFD 1
-#define VHOST_USER_PROTOCOL_F_RARP 2
-#define VHOST_USER_PROTOCOL_F_REPLY_ACK 3
-#define VHOST_USER_PROTOCOL_F_MTU 4
-#define VHOST_USER_PROTOCOL_F_SLAVE_REQ 5
-#define VHOST_USER_PROTOCOL_F_CROSS_ENDIAN 6
-#define VHOST_USER_PROTOCOL_F_CRYPTO_SESSION 7
-#define VHOST_USER_PROTOCOL_F_PAGEFAULT 8
-#define VHOST_USER_PROTOCOL_F_CONFIG 9
-#define VHOST_USER_PROTOCOL_F_SLAVE_SEND_FD 10
-#define VHOST_USER_PROTOCOL_F_HOST_NOTIFIER 11
-#define VHOST_USER_PROTOCOL_F_INFLIGHT_SHMFD 12
-
-Master message types
---------------------
-
- * VHOST_USER_GET_FEATURES
-
- Id: 1
- Equivalent ioctl: VHOST_GET_FEATURES
- Master payload: N/A
- Slave payload: u64
-
- Get from the underlying vhost implementation the features bitmask.
- Feature bit VHOST_USER_F_PROTOCOL_FEATURES signals slave support for
- VHOST_USER_GET_PROTOCOL_FEATURES and VHOST_USER_SET_PROTOCOL_FEATURES.
-
- * VHOST_USER_SET_FEATURES
-
- Id: 2
- Ioctl: VHOST_SET_FEATURES
- Master payload: u64
-
- Enable features in the underlying vhost implementation using a bitmask.
- Feature bit VHOST_USER_F_PROTOCOL_FEATURES signals slave support for
- VHOST_USER_GET_PROTOCOL_FEATURES and VHOST_USER_SET_PROTOCOL_FEATURES.
-
- * VHOST_USER_GET_PROTOCOL_FEATURES
-
- Id: 15
- Equivalent ioctl: VHOST_GET_FEATURES
- Master payload: N/A
- Slave payload: u64
-
- Get the protocol feature bitmask from the underlying vhost implementation.
- Only legal if feature bit VHOST_USER_F_PROTOCOL_FEATURES is present in
- VHOST_USER_GET_FEATURES.
- Note: slave that reported VHOST_USER_F_PROTOCOL_FEATURES must support
- this message even before VHOST_USER_SET_FEATURES was called.
-
- * VHOST_USER_SET_PROTOCOL_FEATURES
-
- Id: 16
- Ioctl: VHOST_SET_FEATURES
- Master payload: u64
-
- Enable protocol features in the underlying vhost implementation.
- Only legal if feature bit VHOST_USER_F_PROTOCOL_FEATURES is present in
- VHOST_USER_GET_FEATURES.
- Note: slave that reported VHOST_USER_F_PROTOCOL_FEATURES must support
- this message even before VHOST_USER_SET_FEATURES was called.
-
- * VHOST_USER_SET_OWNER
-
- Id: 3
- Equivalent ioctl: VHOST_SET_OWNER
- Master payload: N/A
-
- Issued when a new connection is established. It sets the current Master
- as an owner of the session. This can be used on the Slave as a
- "session start" flag.
-
- * VHOST_USER_RESET_OWNER
-
- Id: 4
- Master payload: N/A
-
- This is no longer used. Used to be sent to request disabling
- all rings, but some clients interpreted it to also discard
- connection state (this interpretation would lead to bugs).
- It is recommended that clients either ignore this message,
- or use it to disable all rings.
-
- * VHOST_USER_SET_MEM_TABLE
-
- Id: 5
- Equivalent ioctl: VHOST_SET_MEM_TABLE
- Master payload: memory regions description
- Slave payload: (postcopy only) memory regions description
-
- Sets the memory map regions on the slave so it can translate the vring
- addresses. In the ancillary data there is an array of file descriptors
- for each memory mapped region. The size and ordering of the fds matches
- the number and ordering of memory regions.
-
- When VHOST_USER_POSTCOPY_LISTEN has been received, SET_MEM_TABLE replies with
- the bases of the memory mapped regions to the master. The slave must
- have mmap'd the regions but not yet accessed them and should not yet generate
- a userfault event. Note NEED_REPLY_MASK is not set in this case.
- QEMU will then reply back to the list of mappings with an empty
- VHOST_USER_SET_MEM_TABLE as an acknowledgment; only upon reception of this
- message may the guest start accessing the memory and generating faults.
-
- * VHOST_USER_SET_LOG_BASE
-
- Id: 6
- Equivalent ioctl: VHOST_SET_LOG_BASE
- Master payload: u64
- Slave payload: N/A
-
- Sets logging shared memory space.
- When slave has VHOST_USER_PROTOCOL_F_LOG_SHMFD protocol
- feature, the log memory fd is provided in the ancillary data of
- VHOST_USER_SET_LOG_BASE message, the size and offset of shared
- memory area provided in the message.
-
-
- * VHOST_USER_SET_LOG_FD
-
- Id: 7
- Equivalent ioctl: VHOST_SET_LOG_FD
- Master payload: N/A
-
- Sets the logging file descriptor, which is passed as ancillary data.
-
- * VHOST_USER_SET_VRING_NUM
-
- Id: 8
- Equivalent ioctl: VHOST_SET_VRING_NUM
- Master payload: vring state description
-
- Set the size of the queue.
-
- * VHOST_USER_SET_VRING_ADDR
-
- Id: 9
- Equivalent ioctl: VHOST_SET_VRING_ADDR
- Master payload: vring address description
- Slave payload: N/A
-
- Sets the addresses of the different aspects of the vring.
-
- * VHOST_USER_SET_VRING_BASE
-
- Id: 10
- Equivalent ioctl: VHOST_SET_VRING_BASE
- Master payload: vring state description
-
- Sets the base offset in the available vring.
-
- * VHOST_USER_GET_VRING_BASE
-
- Id: 11
- Equivalent ioctl: VHOST_USER_GET_VRING_BASE
- Master payload: vring state description
- Slave payload: vring state description
-
- Get the available vring base offset.
-
- * VHOST_USER_SET_VRING_KICK
-
- Id: 12
- Equivalent ioctl: VHOST_SET_VRING_KICK
- Master payload: u64
-
- Set the event file descriptor for adding buffers to the vring. It
- is passed in the ancillary data.
- Bits (0-7) of the payload contain the vring index. Bit 8 is the
- invalid FD flag. This flag is set when there is no file descriptor
- in the ancillary data. This signals that polling should be used
- instead of waiting for a kick.
-
- * VHOST_USER_SET_VRING_CALL
-
- Id: 13
- Equivalent ioctl: VHOST_SET_VRING_CALL
- Master payload: u64
-
- Set the event file descriptor to signal when buffers are used. It
- is passed in the ancillary data.
- Bits (0-7) of the payload contain the vring index. Bit 8 is the
- invalid FD flag. This flag is set when there is no file descriptor
- in the ancillary data. This signals that polling will be used
- instead of waiting for the call.
-
- * VHOST_USER_SET_VRING_ERR
-
- Id: 14
- Equivalent ioctl: VHOST_SET_VRING_ERR
- Master payload: u64
-
- Set the event file descriptor to signal when error occurs. It
- is passed in the ancillary data.
- Bits (0-7) of the payload contain the vring index. Bit 8 is the
- invalid FD flag. This flag is set when there is no file descriptor
- in the ancillary data.
-
- * VHOST_USER_GET_QUEUE_NUM
-
- Id: 17
- Equivalent ioctl: N/A
- Master payload: N/A
- Slave payload: u64
-
- Query how many queues the backend supports. This request should be
- sent only when VHOST_USER_PROTOCOL_F_MQ is set in queried protocol
- features by VHOST_USER_GET_PROTOCOL_FEATURES.
-
- * VHOST_USER_SET_VRING_ENABLE
-
- Id: 18
- Equivalent ioctl: N/A
- Master payload: vring state description
-
- Signal slave to enable or disable corresponding vring.
- This request should be sent only when VHOST_USER_F_PROTOCOL_FEATURES
- has been negotiated.
-
- * VHOST_USER_SEND_RARP
-
- Id: 19
- Equivalent ioctl: N/A
- Master payload: u64
-
- Ask vhost user backend to broadcast a fake RARP to notify the migration
- is terminated for guest that does not support GUEST_ANNOUNCE.
- Only legal if feature bit VHOST_USER_F_PROTOCOL_FEATURES is present in
- VHOST_USER_GET_FEATURES and protocol feature bit VHOST_USER_PROTOCOL_F_RARP
- is present in VHOST_USER_GET_PROTOCOL_FEATURES.
- The first 6 bytes of the payload contain the mac address of the guest to
- allow the vhost user backend to construct and broadcast the fake RARP.
-
- * VHOST_USER_NET_SET_MTU
-
- Id: 20
- Equivalent ioctl: N/A
- Master payload: u64
-
- Set host MTU value exposed to the guest.
- This request should be sent only when VIRTIO_NET_F_MTU feature has been
- successfully negotiated, VHOST_USER_F_PROTOCOL_FEATURES is present in
- VHOST_USER_GET_FEATURES and protocol feature bit
- VHOST_USER_PROTOCOL_F_NET_MTU is present in
- VHOST_USER_GET_PROTOCOL_FEATURES.
- If VHOST_USER_PROTOCOL_F_REPLY_ACK is negotiated, slave must respond
- with zero in case the specified MTU is valid, or non-zero otherwise.
-
- * VHOST_USER_SET_SLAVE_REQ_FD
-
- Id: 21
- Equivalent ioctl: N/A
- Master payload: N/A
-
- Set the socket file descriptor for slave initiated requests. It is passed
- in the ancillary data.
- This request should be sent only when VHOST_USER_F_PROTOCOL_FEATURES
- has been negotiated, and protocol feature bit VHOST_USER_PROTOCOL_F_SLAVE_REQ
- bit is present in VHOST_USER_GET_PROTOCOL_FEATURES.
- If VHOST_USER_PROTOCOL_F_REPLY_ACK is negotiated, slave must respond
- with zero for success, non-zero otherwise.
-
- * VHOST_USER_IOTLB_MSG
-
- Id: 22
- Equivalent ioctl: N/A (equivalent to VHOST_IOTLB_MSG message type)
- Master payload: struct vhost_iotlb_msg
- Slave payload: u64
-
- Send IOTLB messages with struct vhost_iotlb_msg as payload.
- Master sends such requests to update and invalidate entries in the device
- IOTLB. The slave has to acknowledge the request with sending zero as u64
- payload for success, non-zero otherwise.
- This request should be send only when VIRTIO_F_IOMMU_PLATFORM feature
- has been successfully negotiated.
-
- * VHOST_USER_SET_VRING_ENDIAN
-
- Id: 23
- Equivalent ioctl: VHOST_SET_VRING_ENDIAN
- Master payload: vring state description
-
- Set the endianness of a VQ for legacy devices. Little-endian is indicated
- with state.num set to 0 and big-endian is indicated with state.num set
- to 1. Other values are invalid.
- This request should be sent only when VHOST_USER_PROTOCOL_F_CROSS_ENDIAN
- has been negotiated.
- Backends that negotiated this feature should handle both endiannesses
- and expect this message once (per VQ) during device configuration
- (ie. before the master starts the VQ).
-
- * VHOST_USER_GET_CONFIG
-
- Id: 24
- Equivalent ioctl: N/A
- Master payload: virtio device config space
- Slave payload: virtio device config space
-
- When VHOST_USER_PROTOCOL_F_CONFIG is negotiated, this message is
- submitted by the vhost-user master to fetch the contents of the virtio
- device configuration space, vhost-user slave's payload size MUST match
- master's request, vhost-user slave uses zero length of payload to
- indicate an error to vhost-user master. The vhost-user master may
- cache the contents to avoid repeated VHOST_USER_GET_CONFIG calls.
-
-* VHOST_USER_SET_CONFIG
-
- Id: 25
- Equivalent ioctl: N/A
- Master payload: virtio device config space
- Slave payload: N/A
-
- When VHOST_USER_PROTOCOL_F_CONFIG is negotiated, this message is
- submitted by the vhost-user master when the Guest changes the virtio
- device configuration space and also can be used for live migration
- on the destination host. The vhost-user slave must check the flags
- field, and slaves MUST NOT accept SET_CONFIG for read-only
- configuration space fields unless the live migration bit is set.
-
-* VHOST_USER_CREATE_CRYPTO_SESSION
-
- Id: 26
- Equivalent ioctl: N/A
- Master payload: crypto session description
- Slave payload: crypto session description
-
- Create a session for crypto operation. The server side must return the
- session id, 0 or positive for success, negative for failure.
- This request should be sent only when VHOST_USER_PROTOCOL_F_CRYPTO_SESSION
- feature has been successfully negotiated.
- It's a required feature for crypto devices.
-
-* VHOST_USER_CLOSE_CRYPTO_SESSION
-
- Id: 27
- Equivalent ioctl: N/A
- Master payload: u64
-
- Close a session for crypto operation which was previously
- created by VHOST_USER_CREATE_CRYPTO_SESSION.
- This request should be sent only when VHOST_USER_PROTOCOL_F_CRYPTO_SESSION
- feature has been successfully negotiated.
- It's a required feature for crypto devices.
-
- * VHOST_USER_POSTCOPY_ADVISE
- Id: 28
- Master payload: N/A
- Slave payload: userfault fd
-
- When VHOST_USER_PROTOCOL_F_PAGEFAULT is supported, the
- master advises slave that a migration with postcopy enabled is underway,
- the slave must open a userfaultfd for later use.
- Note that at this stage the migration is still in precopy mode.
-
- * VHOST_USER_POSTCOPY_LISTEN
- Id: 29
- Master payload: N/A
-
- Master advises slave that a transition to postcopy mode has happened.
- The slave must ensure that shared memory is registered with userfaultfd
- to cause faulting of non-present pages.
-
- This is always sent sometime after a VHOST_USER_POSTCOPY_ADVISE, and
- thus only when VHOST_USER_PROTOCOL_F_PAGEFAULT is supported.
-
- * VHOST_USER_POSTCOPY_END
- Id: 30
- Slave payload: u64
-
- Master advises that postcopy migration has now completed. The
- slave must disable the userfaultfd. The response is an acknowledgement
- only.
- When VHOST_USER_PROTOCOL_F_PAGEFAULT is supported, this message
- is sent at the end of the migration, after VHOST_USER_POSTCOPY_LISTEN
- was previously sent.
- The value returned is an error indication; 0 is success.
-
- * VHOST_USER_GET_INFLIGHT_FD
- Id: 31
- Equivalent ioctl: N/A
- Master payload: inflight description
-
- When VHOST_USER_PROTOCOL_F_INFLIGHT_SHMFD protocol feature has been
- successfully negotiated, this message is submitted by master to get
- a shared buffer from slave. The shared buffer will be used to track
- inflight I/O by slave. QEMU should retrieve a new one when vm reset.
-
- * VHOST_USER_SET_INFLIGHT_FD
- Id: 32
- Equivalent ioctl: N/A
- Master payload: inflight description
-
- When VHOST_USER_PROTOCOL_F_INFLIGHT_SHMFD protocol feature has been
- successfully negotiated, this message is submitted by master to send
- the shared inflight buffer back to slave so that slave could get
- inflight I/O after a crash or restart.
-
-Slave message types
--------------------
-
- * VHOST_USER_SLAVE_IOTLB_MSG
-
- Id: 1
- Equivalent ioctl: N/A (equivalent to VHOST_IOTLB_MSG message type)
- Slave payload: struct vhost_iotlb_msg
- Master payload: N/A
-
- Send IOTLB messages with struct vhost_iotlb_msg as payload.
- Slave sends such requests to notify of an IOTLB miss, or an IOTLB
- access failure. If VHOST_USER_PROTOCOL_F_REPLY_ACK is negotiated,
- and slave set the VHOST_USER_NEED_REPLY flag, master must respond with
- zero when operation is successfully completed, or non-zero otherwise.
- This request should be send only when VIRTIO_F_IOMMU_PLATFORM feature
- has been successfully negotiated.
-
-* VHOST_USER_SLAVE_CONFIG_CHANGE_MSG
-
- Id: 2
- Equivalent ioctl: N/A
- Slave payload: N/A
- Master payload: N/A
-
- When VHOST_USER_PROTOCOL_F_CONFIG is negotiated, vhost-user slave sends
- such messages to notify that the virtio device's configuration space has
- changed, for those host devices which can support such feature, host
- driver can send VHOST_USER_GET_CONFIG message to slave to get the latest
- content. If VHOST_USER_PROTOCOL_F_REPLY_ACK is negotiated, and slave set
- the VHOST_USER_NEED_REPLY flag, master must respond with zero when
- operation is successfully completed, or non-zero otherwise.
-
- * VHOST_USER_SLAVE_VRING_HOST_NOTIFIER_MSG
-
- Id: 3
- Equivalent ioctl: N/A
- Slave payload: vring area description
- Master payload: N/A
-
- Sets host notifier for a specified queue. The queue index is contained
- in the u64 field of the vring area description. The host notifier is
- described by the file descriptor (typically it's a VFIO device fd) which
- is passed as ancillary data and the size (which is mmap size and should
- be the same as host page size) and offset (which is mmap offset) carried
- in the vring area description. QEMU can mmap the file descriptor based
- on the size and offset to get a memory range. Registering a host notifier
- means mapping this memory range to the VM as the specified queue's notify
- MMIO region. Slave sends this request to tell QEMU to de-register the
- existing notifier if any and register the new notifier if the request is
- sent with a file descriptor.
- This request should be sent only when VHOST_USER_PROTOCOL_F_HOST_NOTIFIER
- protocol feature has been successfully negotiated.
-
-VHOST_USER_PROTOCOL_F_REPLY_ACK:
--------------------------------
-The original vhost-user specification only demands replies for certain
-commands. This differs from the vhost protocol implementation where commands
-are sent over an ioctl() call and block until the client has completed.
-
-With this protocol extension negotiated, the sender (QEMU) can set the
-"need_reply" [Bit 3] flag to any command. This indicates that
-the client MUST respond with a Payload VhostUserMsg indicating success or
-failure. The payload should be set to zero on success or non-zero on failure,
-unless the message already has an explicit reply body.
-
-The response payload gives QEMU a deterministic indication of the result
-of the command. Today, QEMU is expected to terminate the main vhost-user
-loop upon receiving such errors. In future, qemu could be taught to be more
-resilient for selective requests.
-
-For the message types that already solicit a reply from the client, the
-presence of VHOST_USER_PROTOCOL_F_REPLY_ACK or need_reply bit being set brings
-no behavioural change. (See the 'Communication' section for details.)
-
-Backend program conventions
----------------------------
-
-vhost-user backends can provide various devices & services and may
-need to be configured manually depending on the use case. However, it
-is a good idea to follow the conventions listed here when
-possible. Users, QEMU or libvirt, can then rely on some common
-behaviour to avoid heterogenous configuration and management of the
-backend programs and facilitate interoperability.
-
-Each backend installed on a host system should come with at least one
-JSON file that conforms to the vhost-user.json schema. Each file
-informs the management applications about the backend type, and binary
-location. In addition, it defines rules for management apps for
-picking the highest priority backend when multiple match the search
-criteria (see @VhostUserBackend documentation in the schema file).
-
-If the backend is not capable of enabling a requested feature on the
-host (such as 3D acceleration with virgl), or the initialization
-failed, the backend should fail to start early and exit with a status
-!= 0. It may also print a message to stderr for further details.
-
-The backend program must not daemonize itself, but it may be
-daemonized by the management layer. It may also have a restricted
-access to the system.
-
-File descriptors 0, 1 and 2 will exist, and have regular
-stdin/stdout/stderr usage (they may have been redirected to /dev/null
-by the management layer, or to a log handler).
-
-The backend program must end (as quickly and cleanly as possible) when
-the SIGTERM signal is received. Eventually, it may receive SIGKILL by
-the management layer after a few seconds.
-
-The following command line options have an expected behaviour. They
-are mandatory, unless explicitly said differently:
-
-* --socket-path=PATH
-
-This option specify the location of the vhost-user Unix domain socket.
-It is incompatible with --fd.
-
-* --fd=FDNUM
-
-When this argument is given, the backend program is started with the
-vhost-user socket as file descriptor FDNUM. It is incompatible with
---socket-path.
-
-* --print-capabilities
-
-Output to stdout the backend capabilities in JSON format, and then
-exit successfully. Other options and arguments should be ignored, and
-the backend program should not perform its normal function. The
-capabilities can be reported dynamically depending on the host
-capabilities.
-
-The JSON output is described in the vhost-user.json schema, by
-@VHostUserBackendCapabilities. Example:
-{
- "type": "foo",
- "features": [
- "feature-a",
- "feature-b"
- ]
-}
-
-vhost-user-input
-----------------
-
-Command line options:
-
-* --evdev-path=PATH (optional)
-
-Specify the linux input device.
-
-* --no-grab (optional)
-
-Do no request exclusive access to the input device.
-
-vhost-user-gpu
---------------
-
-Command line options:
-
-* --render-node=PATH (optional)
-
-Specify the GPU DRM render node.
-
-* --virgl (optional)
-
-Enable virgl rendering support.
diff --git a/MAINTAINERS b/MAINTAINERS
index 9424a490d6..a6948ebc63 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -1484,7 +1484,7 @@ M: Michael S. Tsirkin <mst@redhat.com>
S: Supported
F: hw/*/*vhost*
F: docs/interop/vhost-user.json
-F: docs/interop/vhost-user.txt
+F: docs/interop/vhost-user.rst
F: contrib/vhost-user-*/
F: backends/vhost-user.c
F: include/sysemu/vhost-user-backend.h
diff --git a/docs/interop/index.rst b/docs/interop/index.rst
index 2df977dd52..a037bd67ec 100644
--- a/docs/interop/index.rst
+++ b/docs/interop/index.rst
@@ -15,4 +15,4 @@ Contents:
bitmaps
live-block-operations
pr-helper
-
+ vhost-user
diff --git a/docs/interop/vhost-user.rst b/docs/interop/vhost-user.rst
new file mode 100644
index 0000000000..7f3232c798
--- /dev/null
+++ b/docs/interop/vhost-user.rst
@@ -0,0 +1,1351 @@
+===================
+Vhost-user Protocol
+===================
+:Copyright: 2014 Virtual Open Systems Sarl.
+:Licence: This work is licensed under the terms of the GNU GPL,
+ version 2 or later. See the COPYING file in the top-level
+ directory.
+
+.. contents:: Table of Contents
+
+Introduction
+============
+
+This protocol is aiming to complement the ``ioctl`` interface used to
+control the vhost implementation in the Linux kernel. It implements
+the control plane needed to establish virtqueue sharing with a user
+space process on the same host. It uses communication over a Unix
+domain socket to share file descriptors in the ancillary data of the
+message.
+
+The protocol defines 2 sides of the communication, *master* and
+*slave*. *Master* is the application that shares its virtqueues, in
+our case QEMU. *Slave* is the consumer of the virtqueues.
+
+In the current implementation QEMU is the *master*, and the *slave* is
+the external process consuming the virtio queues, for example a
+software Ethernet switch running in user space, such as Snabbswitch,
+or a block device backend processing read & write to a virtual
+disk. In order to facilitate interoperability between various backend
+implementations, it is recommended to follow the :ref:`Backend program
+conventions <backend_conventions>`.
+
+*Master* and *slave* can be either a client (i.e. connecting) or
+server (listening) in the socket communication.
+
+Message Specification
+=====================
+
+.. Note:: All numbers are in the machine native byte order.
+
+A vhost-user message consists of 3 header fields and a payload.
+
++---------+-------+------+---------+
+| request | flags | size | payload |
++---------+-------+------+---------+
+
+Header
+------
+
+:request: 32-bit type of the request
+
+:flags: 32-bit bit field
+
+- Lower 2 bits are the version (currently 0x01)
+- Bit 2 is the reply flag - needs to be sent on each reply from the slave
+- Bit 3 is the need_reply flag - see :ref:`REPLY_ACK <reply_ack>` for
+ details.
+
+:size: 32-bit size of the payload
+
+Payload
+-------
+
+Depending on the request type, **payload** can be:
+
+A single 64-bit integer
+^^^^^^^^^^^^^^^^^^^^^^^
+
++-----+
+| u64 |
++-----+
+
+:u64: a 64-bit unsigned integer
+
+A vring state description
+^^^^^^^^^^^^^^^^^^^^^^^^^
+
++-------+-----+
+| index | num |
++-------+-----+
+
+:index: a 32-bit index
+
+:num: a 32-bit number
+
+A vring address description
+^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
++-------+-------+------+------------+------+-----------+-----+
+| index | flags | size | descriptor | used | available | log |
++-------+-------+------+------------+------+-----------+-----+
+
+:index: a 32-bit vring index
+
+:flags: a 32-bit vring flags
+
+:descriptor: a 64-bit ring address of the vring descriptor table
+
+:used: a 64-bit ring address of the vring used ring
+
+:available: a 64-bit ring address of the vring available ring
+
+:log: a 64-bit guest address for logging
+
+Note that a ring address is an IOVA if ``VIRTIO_F_IOMMU_PLATFORM`` has
+been negotiated. Otherwise it is a user address.
+
+Memory regions description
+^^^^^^^^^^^^^^^^^^^^^^^^^^
+
++-------------+---------+---------+-----+---------+
+| num regions | padding | region0 | ... | region7 |
++-------------+---------+---------+-----+---------+
+
+:num regions: a 32-bit number of regions
+
+:padding: 32-bit
+
+A region is:
+
++---------------+------+--------------+-------------+
+| guest address | size | user address | mmap offset |
++---------------+------+--------------+-------------+
+
+:guest address: a 64-bit guest address of the region
+
+:size: a 64-bit size
+
+:user address: a 64-bit user address
+
+:mmap offset: 64-bit offset where region starts in the mapped memory
+
+Log description
+^^^^^^^^^^^^^^^
+
++----------+------------+
+| log size | log offset |
++----------+------------+
+
+:log size: size of area used for logging
+
+:log offset: offset from start of supplied file descriptor where
+ logging starts (i.e. where guest address 0 would be
+ logged)
+
+An IOTLB message
+^^^^^^^^^^^^^^^^
+
++------+------+--------------+-------------------+------+
+| iova | size | user address | permissions flags | type |
++------+------+--------------+-------------------+------+
+
+:iova: a 64-bit I/O virtual address programmed by the guest
+
+:size: a 64-bit size
+
+:user address: a 64-bit user address
+
+:permissions flags: an 8-bit value:
+ - 0: No access
+ - 1: Read access
+ - 2: Write access
+ - 3: Read/Write access
+
+:type: an 8-bit IOTLB message type:
+ - 1: IOTLB miss
+ - 2: IOTLB update
+ - 3: IOTLB invalidate
+ - 4: IOTLB access fail
+
+Virtio device config space
+^^^^^^^^^^^^^^^^^^^^^^^^^^
+
++--------+------+-------+---------+
+| offset | size | flags | payload |
++--------+------+-------+---------+
+
+:offset: a 32-bit offset of virtio device's configuration space
+
+:size: a 32-bit configuration space access size in bytes
+
+:flags: a 32-bit value:
+ - 0: Vhost master messages used for writeable fields
+ - 1: Vhost master messages used for live migration
+
+:payload: Size bytes array holding the contents of the virtio
+ device's configuration space
+
+Vring area description
+^^^^^^^^^^^^^^^^^^^^^^
+
++-----+------+--------+
+| u64 | size | offset |
++-----+------+--------+
+
+:u64: a 64-bit integer contains vring index and flags
+
+:size: a 64-bit size of this area
+
+:offset: a 64-bit offset of this area from the start of the
+ supplied file descriptor
+
+Inflight description
+^^^^^^^^^^^^^^^^^^^^
+
++-----------+-------------+------------+------------+
+| mmap size | mmap offset | num queues | queue size |
++-----------+-------------+------------+------------+
+
+:mmap size: a 64-bit size of area to track inflight I/O
+
+:mmap offset: a 64-bit offset of this area from the start
+ of the supplied file descriptor
+
+:num queues: a 16-bit number of virtqueues
+
+:queue size: a 16-bit size of virtqueues
+
+C structure
+-----------
+
+In QEMU the vhost-user message is implemented with the following struct:
+
+.. code:: c
+
+ typedef struct VhostUserMsg {
+ VhostUserRequest request;
+ uint32_t flags;
+ uint32_t size;
+ union {
+ uint64_t u64;
+ struct vhost_vring_state state;
+ struct vhost_vring_addr addr;
+ VhostUserMemory memory;
+ VhostUserLog log;
+ struct vhost_iotlb_msg iotlb;
+ VhostUserConfig config;
+ VhostUserVringArea area;
+ VhostUserInflight inflight;
+ };
+ } QEMU_PACKED VhostUserMsg;
+
+Communication
+=============
+
+The protocol for vhost-user is based on the existing implementation of
+vhost for the Linux Kernel. Most messages that can be sent via the
+Unix domain socket implementing vhost-user have an equivalent ioctl to
+the kernel implementation.
+
+The communication consists of *master* sending message requests and
+*slave* sending message replies. Most of the requests don't require
+replies. Here is a list of the ones that do:
+
+* ``VHOST_USER_GET_FEATURES``
+* ``VHOST_USER_GET_PROTOCOL_FEATURES``
+* ``VHOST_USER_GET_VRING_BASE``
+* ``VHOST_USER_SET_LOG_BASE`` (if ``VHOST_USER_PROTOCOL_F_LOG_SHMFD``)
+* ``VHOST_USER_GET_INFLIGHT_FD`` (if ``VHOST_USER_PROTOCOL_F_INFLIGHT_SHMFD``)
+
+.. seealso::
+
+ :ref:`REPLY_ACK <reply_ack>`
+ The section on ``REPLY_ACK`` protocol extension.
+
+There are several messages that the master sends with file descriptors passed
+in the ancillary data:
+
+* ``VHOST_USER_SET_MEM_TABLE``
+* ``VHOST_USER_SET_LOG_BASE`` (if ``VHOST_USER_PROTOCOL_F_LOG_SHMFD``)
+* ``VHOST_USER_SET_LOG_FD``
+* ``VHOST_USER_SET_VRING_KICK``
+* ``VHOST_USER_SET_VRING_CALL``
+* ``VHOST_USER_SET_VRING_ERR``
+* ``VHOST_USER_SET_SLAVE_REQ_FD``
+* ``VHOST_USER_SET_INFLIGHT_FD`` (if ``VHOST_USER_PROTOCOL_F_INFLIGHT_SHMFD``)
+
+If *master* is unable to send the full message or receives a wrong
+reply it will close the connection. An optional reconnection mechanism
+can be implemented.
+
+Any protocol extensions are gated by protocol feature bits, which
+allows full backwards compatibility on both master and slave. As
+older slaves don't support negotiating protocol features, a feature
+bit was dedicated for this purpose::
+
+ #define VHOST_USER_F_PROTOCOL_FEATURES 30
+
+Starting and stopping rings
+---------------------------
+
+Client must only process each ring when it is started.
+
+Client must only pass data between the ring and the backend, when the
+ring is enabled.
+
+If ring is started but disabled, client must process the ring without
+talking to the backend.
+
+For example, for a networking device, in the disabled state client
+must not supply any new RX packets, but must process and discard any
+TX packets.
+
+If ``VHOST_USER_F_PROTOCOL_FEATURES`` has not been negotiated, the
+ring is initialized in an enabled state.
+
+If ``VHOST_USER_F_PROTOCOL_FEATURES`` has been negotiated, the ring is
+initialized in a disabled state. Client must not pass data to/from the
+backend until ring is enabled by ``VHOST_USER_SET_VRING_ENABLE`` with
+parameter 1, or after it has been disabled by
+``VHOST_USER_SET_VRING_ENABLE`` with parameter 0.
+
+Each ring is initialized in a stopped state, client must not process
+it until ring is started, or after it has been stopped.
+
+Client must start ring upon receiving a kick (that is, detecting that
+file descriptor is readable) on the descriptor specified by
+``VHOST_USER_SET_VRING_KICK``, and stop ring upon receiving
+``VHOST_USER_GET_VRING_BASE``.
+
+While processing the rings (whether they are enabled or not), client
+must support changing some configuration aspects on the fly.
+
+Multiple queue support
+----------------------
+
+Multiple queue is treated as a protocol extension, hence the slave has
+to implement protocol features first. The multiple queues feature is
+supported only when the protocol feature ``VHOST_USER_PROTOCOL_F_MQ``
+(bit 0) is set.
+
+The max number of queue pairs the slave supports can be queried with
+message ``VHOST_USER_GET_QUEUE_NUM``. Master should stop when the
+number of requested queues is bigger than that.
+
+As all queues share one connection, the master uses a unique index for each
+queue in the sent message to identify a specified queue. One queue pair
+is enabled initially. More queues are enabled dynamically, by sending
+message ``VHOST_USER_SET_VRING_ENABLE``.
+
+Migration
+---------
+
+During live migration, the master may need to track the modifications
+the slave makes to the memory mapped regions. The client should mark
+the dirty pages in a log. Once it complies to this logging, it may
+declare the ``VHOST_F_LOG_ALL`` vhost feature.
+
+To start/stop logging of data/used ring writes, server may send
+messages ``VHOST_USER_SET_FEATURES`` with ``VHOST_F_LOG_ALL`` and
+``VHOST_USER_SET_VRING_ADDR`` with ``VHOST_VRING_F_LOG`` in ring's
+flags set to 1/0, respectively.
+
+All the modifications to memory pointed by vring "descriptor" should
+be marked. Modifications to "used" vring should be marked if
+``VHOST_VRING_F_LOG`` is part of ring's flags.
+
+Dirty pages are of size::
+
+ #define VHOST_LOG_PAGE 0x1000
+
+The log memory fd is provided in the ancillary data of
+``VHOST_USER_SET_LOG_BASE`` message when the slave has
+``VHOST_USER_PROTOCOL_F_LOG_SHMFD`` protocol feature.
+
+The size of the log is supplied as part of ``VhostUserMsg`` which
+should be large enough to cover all known guest addresses. Log starts
+at the supplied offset in the supplied file descriptor. The log
+covers from address 0 to the maximum of guest regions. In pseudo-code,
+to mark page at ``addr`` as dirty::
+
+ page = addr / VHOST_LOG_PAGE
+ log[page / 8] |= 1 << page % 8
+
+Where ``addr`` is the guest physical address.
+
+Use atomic operations, as the log may be concurrently manipulated.
+
+Note that when logging modifications to the used ring (when
+``VHOST_VRING_F_LOG`` is set for this ring), ``log_guest_addr`` should
+be used to calculate the log offset: the write to first byte of the
+used ring is logged at this offset from log start. Also note that this
+value might be outside the legal guest physical address range
+(i.e. does not have to be covered by the ``VhostUserMemory`` table), but
+the bit offset of the last byte of the ring must fall within the size
+supplied by ``VhostUserLog``.
+
+``VHOST_USER_SET_LOG_FD`` is an optional message with an eventfd in
+ancillary data, it may be used to inform the master that the log has
+been modified.
+
+Once the source has finished migration, rings will be stopped by the
+source. No further update must be done before rings are restarted.
+
+In postcopy migration the slave is started before all the memory has
+been received from the source host, and care must be taken to avoid
+accessing pages that have yet to be received. The slave opens a
+'userfault'-fd and registers the memory with it; this fd is then
+passed back over to the master. The master services requests on the
+userfaultfd for pages that are accessed and when the page is available
+it performs WAKE ioctl's on the userfaultfd to wake the stalled
+slave. The client indicates support for this via the
+``VHOST_USER_PROTOCOL_F_PAGEFAULT`` feature.
+
+Memory access
+-------------
+
+The master sends a list of vhost memory regions to the slave using the
+``VHOST_USER_SET_MEM_TABLE`` message. Each region has two base
+addresses: a guest address and a user address.
+
+Messages contain guest addresses and/or user addresses to reference locations
+within the shared memory. The mapping of these addresses works as follows.
+
+User addresses map to the vhost memory region containing that user address.
+
+When the ``VIRTIO_F_IOMMU_PLATFORM`` feature has not been negotiated:
+
+* Guest addresses map to the vhost memory region containing that guest
+ address.
+
+When the ``VIRTIO_F_IOMMU_PLATFORM`` feature has been negotiated:
+
+* Guest addresses are also called I/O virtual addresses (IOVAs). They are
+ translated to user addresses via the IOTLB.
+
+* The vhost memory region guest address is not used.
+
+IOMMU support
+-------------
+
+When the ``VIRTIO_F_IOMMU_PLATFORM`` feature has been negotiated, the
+master sends IOTLB entries update & invalidation by sending
+``VHOST_USER_IOTLB_MSG`` requests to the slave with a ``struct
+vhost_iotlb_msg`` as payload. For update events, the ``iotlb`` payload
+has to be filled with the update message type (2), the I/O virtual
+address, the size, the user virtual address, and the permissions
+flags. Addresses and size must be within vhost memory regions set via
+the ``VHOST_USER_SET_MEM_TABLE`` request. For invalidation events, the
+``iotlb`` payload has to be filled with the invalidation message type
+(3), the I/O virtual address and the size. On success, the slave is
+expected to reply with a zero payload, non-zero otherwise.
+
+The slave relies on the slave communcation channel (see :ref:`Slave
+communication <slave_communication>` section below) to send IOTLB miss
+and access failure events, by sending ``VHOST_USER_SLAVE_IOTLB_MSG``
+requests to the master with a ``struct vhost_iotlb_msg`` as
+payload. For miss events, the iotlb payload has to be filled with the
+miss message type (1), the I/O virtual address and the permissions
+flags. For access failure event, the iotlb payload has to be filled
+with the access failure message type (4), the I/O virtual address and
+the permissions flags. For synchronization purpose, the slave may
+rely on the reply-ack feature, so the master may send a reply when
+operation is completed if the reply-ack feature is negotiated and
+slaves requests a reply. For miss events, completed operation means
+either master sent an update message containing the IOTLB entry
+containing requested address and permission, or master sent nothing if
+the IOTLB miss message is invalid (invalid IOVA or permission).
+
+The master isn't expected to take the initiative to send IOTLB update
+messages, as the slave sends IOTLB miss messages for the guest virtual
+memory areas it needs to access.
+
+.. _slave_communication:
+
+Slave communication
+-------------------
+
+An optional communication channel is provided if the slave declares
+``VHOST_USER_PROTOCOL_F_SLAVE_REQ`` protocol feature, to allow the
+slave to make requests to the master.
+
+The fd is provided via ``VHOST_USER_SET_SLAVE_REQ_FD`` ancillary data.
+
+A slave may then send ``VHOST_USER_SLAVE_*`` messages to the master
+using this fd communication channel.
+
+If ``VHOST_USER_PROTOCOL_F_SLAVE_SEND_FD`` protocol feature is
+negotiated, slave can send file descriptors (at most 8 descriptors in
+each message) to master via ancillary data using this fd communication
+channel.
+
+Inflight I/O tracking
+---------------------
+
+To support reconnecting after restart or crash, slave may need to
+resubmit inflight I/Os. If virtqueue is processed in order, we can
+easily achieve that by getting the inflight descriptors from
+descriptor table (split virtqueue) or descriptor ring (packed
+virtqueue). However, it can't work when we process descriptors
+out-of-order because some entries which store the information of
+inflight descriptors in available ring (split virtqueue) or descriptor
+ring (packed virtqueue) might be overrided by new entries. To solve
+this problem, slave need to allocate an extra buffer to store this
+information of inflight descriptors and share it with master for
+persistent. ``VHOST_USER_GET_INFLIGHT_FD`` and
+``VHOST_USER_SET_INFLIGHT_FD`` are used to transfer this buffer
+between master and slave. And the format of this buffer is described
+below:
+
++---------------+---------------+-----+---------------+
+| queue0 region | queue1 region | ... | queueN region |
++---------------+---------------+-----+---------------+
+
+N is the number of available virtqueues. Slave could get it from num
+queues field of ``VhostUserInflight``.
+
+For split virtqueue, queue region can be implemented as:
+
+.. code:: c
+
+ typedef struct DescStateSplit {
+ /* Indicate whether this descriptor is inflight or not.
+ * Only available for head-descriptor. */
+ uint8_t inflight;
+
+ /* Padding */
+ uint8_t padding[5];
+
+ /* Maintain a list for the last batch of used descriptors.
+ * Only available when batching is used for submitting */
+ uint16_t next;
+
+ /* Used to preserve the order of fetching available descriptors.
+ * Only available for head-descriptor. */
+ uint64_t counter;
+ } DescStateSplit;
+
+ typedef struct QueueRegionSplit {
+ /* The feature flags of this region. Now it's initialized to 0. */
+ uint64_t features;
+
+ /* The version of this region. It's 1 currently.
+ * Zero value indicates an uninitialized buffer */
+ uint16_t version;
+
+ /* The size of DescStateSplit array. It's equal to the virtqueue
+ * size. Slave could get it from queue size field of VhostUserInflight. */
+ uint16_t desc_num;
+
+ /* The head of list that track the last batch of used descriptors. */
+ uint16_t last_batch_head;
+
+ /* Store the idx value of used ring */
+ uint16_t used_idx;
+
+ /* Used to track the state of each descriptor in descriptor table */
+ DescStateSplit desc[0];
+ } QueueRegionSplit;
+
+To track inflight I/O, the queue region should be processed as follows:
+
+When receiving available buffers from the driver:
+
+#. Get the next available head-descriptor index from available ring, ``i``
+
+#. Set ``desc[i].counter`` to the value of global counter
+
+#. Increase global counter by 1
+
+#. Set ``desc[i].inflight`` to 1
+
+When supplying used buffers to the driver:
+
+1. Get corresponding used head-descriptor index, i
+
+2. Set ``desc[i].next`` to ``last_batch_head``
+
+3. Set ``last_batch_head`` to ``i``
+
+#. Steps 1,2,3 may be performed repeatedly if batching is possible
+
+#. Increase the ``idx`` value of used ring by the size of the batch
+
+#. Set the ``inflight`` field of each ``DescStateSplit`` entry in the batch to 0
+
+#. Set ``used_idx`` to the ``idx`` value of used ring
+
+When reconnecting:
+
+#. If the value of ``used_idx`` does not match the ``idx`` value of
+ used ring (means the inflight field of ``DescStateSplit`` entries in
+ last batch may be incorrect),
+
+ a. Subtract the value of ``used_idx`` from the ``idx`` value of
+ used ring to get last batch size of ``DescStateSplit`` entries
+
+ #. Set the ``inflight`` field of each ``DescStateSplit`` entry to 0 in last batch
+ list which starts from ``last_batch_head``
+
+ #. Set ``used_idx`` to the ``idx`` value of used ring
+
+#. Resubmit inflight ``DescStateSplit`` entries in order of their
+ counter value
+
+For packed virtqueue, queue region can be implemented as:
+
+.. code:: c
+
+ typedef struct DescStatePacked {
+ /* Indicate whether this descriptor is inflight or not.
+ * Only available for head-descriptor. */
+ uint8_t inflight;
+
+ /* Padding */
+ uint8_t padding;
+
+ /* Link to the next free entry */
+ uint16_t next;
+
+ /* Link to the last entry of descriptor list.
+ * Only available for head-descriptor. */
+ uint16_t last;
+
+ /* The length of descriptor list.
+ * Only available for head-descriptor. */
+ uint16_t num;
+
+ /* Used to preserve the order of fetching available descriptors.
+ * Only available for head-descriptor. */
+ uint64_t counter;
+
+ /* The buffer id */
+ uint16_t id;
+
+ /* The descriptor flags */
+ uint16_t flags;
+
+ /* The buffer length */
+ uint32_t len;
+
+ /* The buffer address */
+ uint64_t addr;
+ } DescStatePacked;
+
+ typedef struct QueueRegionPacked {
+ /* The feature flags of this region. Now it's initialized to 0. */
+ uint64_t features;
+
+ /* The version of this region. It's 1 currently.
+ * Zero value indicates an uninitialized buffer */
+ uint16_t version;
+
+ /* The size of DescStatePacked array. It's equal to the virtqueue
+ * size. Slave could get it from queue size field of VhostUserInflight. */
+ uint16_t desc_num;
+
+ /* The head of free DescStatePacked entry list */
+ uint16_t free_head;
+
+ /* The old head of free DescStatePacked entry list */
+ uint16_t old_free_head;
+
+ /* The used index of descriptor ring */
+ uint16_t used_idx;
+
+ /* The old used index of descriptor ring */
+ uint16_t old_used_idx;
+
+ /* Device ring wrap counter */
+ uint8_t used_wrap_counter;
+
+ /* The old device ring wrap counter */
+ uint8_t old_used_wrap_counter;
+
+ /* Padding */
+ uint8_t padding[7];
+
+ /* Used to track the state of each descriptor fetched from descriptor ring */
+ DescStatePacked desc[0];
+ } QueueRegionPacked;
+
+To track inflight I/O, the queue region should be processed as follows:
+
+When receiving available buffers from the driver:
+
+#. Get the next available descriptor entry from descriptor ring, ``d``
+
+#. If ``d`` is head descriptor,
+
+ a. Set ``desc[old_free_head].num`` to 0
+
+ #. Set ``desc[old_free_head].counter`` to the value of global counter
+
+ #. Increase global counter by 1
+
+ #. Set ``desc[old_free_head].inflight`` to 1
+
+#. If ``d`` is last descriptor, set ``desc[old_free_head].last`` to
+ ``free_head``
+
+#. Increase ``desc[old_free_head].num`` by 1
+
+#. Set ``desc[free_head].addr``, ``desc[free_head].len``,
+ ``desc[free_head].flags``, ``desc[free_head].id`` to ``d.addr``,
+ ``d.len``, ``d.flags``, ``d.id``
+
+#. Set ``free_head`` to ``desc[free_head].next``
+
+#. If ``d`` is last descriptor, set ``old_free_head`` to ``free_head``
+
+When supplying used buffers to the driver:
+
+1. Get corresponding used head-descriptor entry from descriptor ring,
+ ``d``
+
+2. Get corresponding ``DescStatePacked`` entry, ``e``
+
+3. Set ``desc[e.last].next`` to ``free_head``
+
+4. Set ``free_head`` to the index of ``e``
+
+#. Steps 1,2,3,4 may be performed repeatedly if batching is possible
+
+#. Increase ``used_idx`` by the size of the batch and update
+ ``used_wrap_counter`` if needed
+
+#. Update ``d.flags``
+
+#. Set the ``inflight`` field of each head ``DescStatePacked`` entry
+ in the batch to 0
+
+#. Set ``old_free_head``, ``old_used_idx``, ``old_used_wrap_counter``
+ to ``free_head``, ``used_idx``, ``used_wrap_counter``
+
+When reconnecting:
+
+#. If ``used_idx`` does not match ``old_used_idx`` (means the
+ ``inflight`` field of ``DescStatePacked`` entries in last batch may
+ be incorrect),
+
+ a. Get the next descriptor ring entry through ``old_used_idx``, ``d``
+
+ #. Use ``old_used_wrap_counter`` to calculate the available flags
+
+ #. If ``d.flags`` is not equal to the calculated flags value (means
+ slave has submitted the buffer to guest driver before crash, so
+ it has to commit the in-progres update), set ``old_free_head``,
+ ``old_used_idx``, ``old_used_wrap_counter`` to ``free_head``,
+ ``used_idx``, ``used_wrap_counter``
+
+#. Set ``free_head``, ``used_idx``, ``used_wrap_counter`` to
+ ``old_free_head``, ``old_used_idx``, ``old_used_wrap_counter``
+ (roll back any in-progress update)
+
+#. Set the ``inflight`` field of each ``DescStatePacked`` entry in
+ free list to 0
+
+#. Resubmit inflight ``DescStatePacked`` entries in order of their
+ counter value
+
+Protocol features
+-----------------
+
+.. code:: c
+
+ #define VHOST_USER_PROTOCOL_F_MQ 0
+ #define VHOST_USER_PROTOCOL_F_LOG_SHMFD 1
+ #define VHOST_USER_PROTOCOL_F_RARP 2
+ #define VHOST_USER_PROTOCOL_F_REPLY_ACK 3
+ #define VHOST_USER_PROTOCOL_F_MTU 4
+ #define VHOST_USER_PROTOCOL_F_SLAVE_REQ 5
+ #define VHOST_USER_PROTOCOL_F_CROSS_ENDIAN 6
+ #define VHOST_USER_PROTOCOL_F_CRYPTO_SESSION 7
+ #define VHOST_USER_PROTOCOL_F_PAGEFAULT 8
+ #define VHOST_USER_PROTOCOL_F_CONFIG 9
+ #define VHOST_USER_PROTOCOL_F_SLAVE_SEND_FD 10
+ #define VHOST_USER_PROTOCOL_F_HOST_NOTIFIER 11
+ #define VHOST_USER_PROTOCOL_F_INFLIGHT_SHMFD 12
+
+Master message types
+--------------------
+
+``VHOST_USER_GET_FEATURES``
+ :id: 1
+ :equivalent ioctl: ``VHOST_GET_FEATURES``
+ :master payload: N/A
+ :slave payload: ``u64``
+
+ Get from the underlying vhost implementation the features bitmask.
+ Feature bit ``VHOST_USER_F_PROTOCOL_FEATURES`` signals slave support
+ for ``VHOST_USER_GET_PROTOCOL_FEATURES`` and
+ ``VHOST_USER_SET_PROTOCOL_FEATURES``.
+
+``VHOST_USER_SET_FEATURES``
+ :id: 2
+ :equivalent ioctl: ``VHOST_SET_FEATURES``
+ :master payload: ``u64``
+
+ Enable features in the underlying vhost implementation using a
+ bitmask. Feature bit ``VHOST_USER_F_PROTOCOL_FEATURES`` signals
+ slave support for ``VHOST_USER_GET_PROTOCOL_FEATURES`` and
+ ``VHOST_USER_SET_PROTOCOL_FEATURES``.
+
+``VHOST_USER_GET_PROTOCOL_FEATURES``
+ :id: 15
+ :equivalent ioctl: ``VHOST_GET_FEATURES``
+ :master payload: N/A
+ :slave payload: ``u64``
+
+ Get the protocol feature bitmask from the underlying vhost
+ implementation. Only legal if feature bit
+ ``VHOST_USER_F_PROTOCOL_FEATURES`` is present in
+ ``VHOST_USER_GET_FEATURES``.
+
+.. Note::
+ Slave that reported ``VHOST_USER_F_PROTOCOL_FEATURES`` must
+ support this message even before ``VHOST_USER_SET_FEATURES`` was
+ called.
+
+``VHOST_USER_SET_PROTOCOL_FEATURES``
+ :id: 16
+ :equivalent ioctl: ``VHOST_SET_FEATURES``
+ :master payload: ``u64``
+
+ Enable protocol features in the underlying vhost implementation.
+
+ Only legal if feature bit ``VHOST_USER_F_PROTOCOL_FEATURES`` is present in
+ ``VHOST_USER_GET_FEATURES``.
+
+.. Note::
+ Slave that reported ``VHOST_USER_F_PROTOCOL_FEATURES`` must support
+ this message even before ``VHOST_USER_SET_FEATURES`` was called.
+
+``VHOST_USER_SET_OWNER``
+ :id: 3
+ :equivalent ioctl: ``VHOST_SET_OWNER``
+ :master payload: N/A
+
+ Issued when a new connection is established. It sets the current
+ *master* as an owner of the session. This can be used on the *slave*
+ as a "session start" flag.
+
+``VHOST_USER_RESET_OWNER``
+ :id: 4
+ :master payload: N/A
+
+.. admonition:: Deprecated
+
+ This is no longer used. Used to be sent to request disabling all
+ rings, but some clients interpreted it to also discard connection
+ state (this interpretation would lead to bugs). It is recommended
+ that clients either ignore this message, or use it to disable all
+ rings.
+
+``VHOST_USER_SET_MEM_TABLE``
+ :id: 5
+ :equivalent ioctl: ``VHOST_SET_MEM_TABLE``
+ :master payload: memory regions description
+ :slave payload: (postcopy only) memory regions description
+
+ Sets the memory map regions on the slave so it can translate the
+ vring addresses. In the ancillary data there is an array of file
+ descriptors for each memory mapped region. The size and ordering of
+ the fds matches the number and ordering of memory regions.
+
+ When ``VHOST_USER_POSTCOPY_LISTEN`` has been received,
+ ``SET_MEM_TABLE`` replies with the bases of the memory mapped
+ regions to the master. The slave must have mmap'd the regions but
+ not yet accessed them and should not yet generate a userfault
+ event.
+
+.. Note::
+ ``NEED_REPLY_MASK`` is not set in this case. QEMU will then
+ reply back to the list of mappings with an empty
+ ``VHOST_USER_SET_MEM_TABLE`` as an acknowledgement; only upon
+ reception of this message may the guest start accessing the memory
+ and generating faults.
+
+``VHOST_USER_SET_LOG_BASE``
+ :id: 6
+ :equivalent ioctl: ``VHOST_SET_LOG_BASE``
+ :master payload: u64
+ :slave payload: N/A
+
+ Sets logging shared memory space.
+
+ When slave has ``VHOST_USER_PROTOCOL_F_LOG_SHMFD`` protocol feature,
+ the log memory fd is provided in the ancillary data of
+ ``VHOST_USER_SET_LOG_BASE`` message, the size and offset of shared
+ memory area provided in the message.
+
+``VHOST_USER_SET_LOG_FD``
+ :id: 7
+ :equivalent ioctl: ``VHOST_SET_LOG_FD``
+ :master payload: N/A
+
+ Sets the logging file descriptor, which is passed as ancillary data.
+
+``VHOST_USER_SET_VRING_NUM``
+ :id: 8
+ :equivalent ioctl: ``VHOST_SET_VRING_NUM``
+ :master payload: vring state description
+
+ Set the size of the queue.
+
+``VHOST_USER_SET_VRING_ADDR``
+ :id: 9
+ :equivalent ioctl: ``VHOST_SET_VRING_ADDR``
+ :master payload: vring address description
+ :slave payload: N/A
+
+ Sets the addresses of the different aspects of the vring.
+
+``VHOST_USER_SET_VRING_BASE``
+ :id: 10
+ :equivalent ioctl: ``VHOST_SET_VRING_BASE``
+ :master payload: vring state description
+
+ Sets the base offset in the available vring.
+
+``VHOST_USER_GET_VRING_BASE``
+ :id: 11
+ :equivalent ioctl: ``VHOST_USER_GET_VRING_BASE``
+ :master payload: vring state description
+ :slave payload: vring state description
+
+ Get the available vring base offset.
+
+``VHOST_USER_SET_VRING_KICK``
+ :id: 12
+ :equivalent ioctl: ``VHOST_SET_VRING_KICK``
+ :master payload: ``u64``
+
+ Set the event file descriptor for adding buffers to the vring. It is
+ passed in the ancillary data.
+
+ Bits (0-7) of the payload contain the vring index. Bit 8 is the
+ invalid FD flag. This flag is set when there is no file descriptor
+ in the ancillary data. This signals that polling should be used
+ instead of waiting for a kick.
+
+``VHOST_USER_SET_VRING_CALL``
+ :id: 13
+ :equivalent ioctl: ``VHOST_SET_VRING_CALL``
+ :master payload: ``u64``
+
+ Set the event file descriptor to signal when buffers are used. It is
+ passed in the ancillary data.
+
+ Bits (0-7) of the payload contain the vring index. Bit 8 is the
+ invalid FD flag. This flag is set when there is no file descriptor
+ in the ancillary data. This signals that polling will be used
+ instead of waiting for the call.
+
+``VHOST_USER_SET_VRING_ERR``
+ :id: 14
+ :equivalent ioctl: ``VHOST_SET_VRING_ERR``
+ :master payload: ``u64``
+
+ Set the event file descriptor to signal when error occurs. It is
+ passed in the ancillary data.
+
+ Bits (0-7) of the payload contain the vring index. Bit 8 is the
+ invalid FD flag. This flag is set when there is no file descriptor
+ in the ancillary data.
+
+``VHOST_USER_GET_QUEUE_NUM``
+ :id: 17
+ :equivalent ioctl: N/A
+ :master payload: N/A
+ :slave payload: u64
+
+ Query how many queues the backend supports.
+
+ This request should be sent only when ``VHOST_USER_PROTOCOL_F_MQ``
+ is set in queried protocol features by
+ ``VHOST_USER_GET_PROTOCOL_FEATURES``.
+
+``VHOST_USER_SET_VRING_ENABLE``
+ :id: 18
+ :equivalent ioctl: N/A
+ :master payload: vring state description
+
+ Signal slave to enable or disable corresponding vring.
+
+ This request should be sent only when
+ ``VHOST_USER_F_PROTOCOL_FEATURES`` has been negotiated.
+
+``VHOST_USER_SEND_RARP``
+ :id: 19
+ :equivalent ioctl: N/A
+ :master payload: ``u64``
+
+ Ask vhost user backend to broadcast a fake RARP to notify the migration
+ is terminated for guest that does not support GUEST_ANNOUNCE.
+
+ Only legal if feature bit ``VHOST_USER_F_PROTOCOL_FEATURES`` is
+ present in ``VHOST_USER_GET_FEATURES`` and protocol feature bit
+ ``VHOST_USER_PROTOCOL_F_RARP`` is present in
+ ``VHOST_USER_GET_PROTOCOL_FEATURES``. The first 6 bytes of the
+ payload contain the mac address of the guest to allow the vhost user
+ backend to construct and broadcast the fake RARP.
+
+``VHOST_USER_NET_SET_MTU``
+ :id: 20
+ :equivalent ioctl: N/A
+ :master payload: ``u64``
+
+ Set host MTU value exposed to the guest.
+
+ This request should be sent only when ``VIRTIO_NET_F_MTU`` feature
+ has been successfully negotiated, ``VHOST_USER_F_PROTOCOL_FEATURES``
+ is present in ``VHOST_USER_GET_FEATURES`` and protocol feature bit
+ ``VHOST_USER_PROTOCOL_F_NET_MTU`` is present in
+ ``VHOST_USER_GET_PROTOCOL_FEATURES``.
+
+ If ``VHOST_USER_PROTOCOL_F_REPLY_ACK`` is negotiated, slave must
+ respond with zero in case the specified MTU is valid, or non-zero
+ otherwise.
+
+``VHOST_USER_SET_SLAVE_REQ_FD``
+ :id: 21
+ :equivalent ioctl: N/A
+ :master payload: N/A
+
+ Set the socket file descriptor for slave initiated requests. It is passed
+ in the ancillary data.
+
+ This request should be sent only when
+ ``VHOST_USER_F_PROTOCOL_FEATURES`` has been negotiated, and protocol
+ feature bit ``VHOST_USER_PROTOCOL_F_SLAVE_REQ`` bit is present in
+ ``VHOST_USER_GET_PROTOCOL_FEATURES``. If
+ ``VHOST_USER_PROTOCOL_F_REPLY_ACK`` is negotiated, slave must
+ respond with zero for success, non-zero otherwise.
+
+``VHOST_USER_IOTLB_MSG``
+ :id: 22
+ :equivalent ioctl: N/A (equivalent to ``VHOST_IOTLB_MSG`` message type)
+ :master payload: ``struct vhost_iotlb_msg``
+ :slave payload: ``u64``
+
+ Send IOTLB messages with ``struct vhost_iotlb_msg`` as payload.
+
+ Master sends such requests to update and invalidate entries in the
+ device IOTLB. The slave has to acknowledge the request with sending
+ zero as ``u64`` payload for success, non-zero otherwise.
+
+ This request should be send only when ``VIRTIO_F_IOMMU_PLATFORM``
+ feature has been successfully negotiated.
+
+``VHOST_USER_SET_VRING_ENDIAN``
+ :id: 23
+ :equivalent ioctl: ``VHOST_SET_VRING_ENDIAN``
+ :master payload: vring state description
+
+ Set the endianness of a VQ for legacy devices. Little-endian is
+ indicated with state.num set to 0 and big-endian is indicated with
+ state.num set to 1. Other values are invalid.
+
+ This request should be sent only when
+ ``VHOST_USER_PROTOCOL_F_CROSS_ENDIAN`` has been negotiated.
+ Backends that negotiated this feature should handle both
+ endiannesses and expect this message once (per VQ) during device
+ configuration (ie. before the master starts the VQ).
+
+``VHOST_USER_GET_CONFIG``
+ :id: 24
+ :equivalent ioctl: N/A
+ :master payload: virtio device config space
+ :slave payload: virtio device config space
+
+ When ``VHOST_USER_PROTOCOL_F_CONFIG`` is negotiated, this message is
+ submitted by the vhost-user master to fetch the contents of the
+ virtio device configuration space, vhost-user slave's payload size
+ MUST match master's request, vhost-user slave uses zero length of
+ payload to indicate an error to vhost-user master. The vhost-user
+ master may cache the contents to avoid repeated
+ ``VHOST_USER_GET_CONFIG`` calls.
+
+``VHOST_USER_SET_CONFIG``
+ :id: 25
+ :equivalent ioctl: N/A
+ :master payload: virtio device config space
+ :slave payload: N/A
+
+ When ``VHOST_USER_PROTOCOL_F_CONFIG`` is negotiated, this message is
+ submitted by the vhost-user master when the Guest changes the virtio
+ device configuration space and also can be used for live migration
+ on the destination host. The vhost-user slave must check the flags
+ field, and slaves MUST NOT accept SET_CONFIG for read-only
+ configuration space fields unless the live migration bit is set.
+
+``VHOST_USER_CREATE_CRYPTO_SESSION``
+ :id: 26
+ :equivalent ioctl: N/A
+ :master payload: crypto session description
+ :slave payload: crypto session description
+
+ Create a session for crypto operation. The server side must return
+ the session id, 0 or positive for success, negative for failure.
+ This request should be sent only when
+ ``VHOST_USER_PROTOCOL_F_CRYPTO_SESSION`` feature has been
+ successfully negotiated. It's a required feature for crypto
+ devices.
+
+``VHOST_USER_CLOSE_CRYPTO_SESSION``
+ :id: 27
+ :equivalent ioctl: N/A
+ :master payload: ``u64``
+
+ Close a session for crypto operation which was previously
+ created by ``VHOST_USER_CREATE_CRYPTO_SESSION``.
+
+ This request should be sent only when
+ ``VHOST_USER_PROTOCOL_F_CRYPTO_SESSION`` feature has been
+ successfully negotiated. It's a required feature for crypto
+ devices.
+
+``VHOST_USER_POSTCOPY_ADVISE``
+ :id: 28
+ :master payload: N/A
+ :slave payload: userfault fd
+
+ When ``VHOST_USER_PROTOCOL_F_PAGEFAULT`` is supported, the master
+ advises slave that a migration with postcopy enabled is underway,
+ the slave must open a userfaultfd for later use. Note that at this
+ stage the migration is still in precopy mode.
+
+``VHOST_USER_POSTCOPY_LISTEN``
+ :id: 29
+ :master payload: N/A
+
+ Master advises slave that a transition to postcopy mode has
+ happened. The slave must ensure that shared memory is registered
+ with userfaultfd to cause faulting of non-present pages.
+
+ This is always sent sometime after a ``VHOST_USER_POSTCOPY_ADVISE``,
+ and thus only when ``VHOST_USER_PROTOCOL_F_PAGEFAULT`` is supported.
+
+``VHOST_USER_POSTCOPY_END``
+ :id: 30
+ :slave payload: ``u64``
+
+ Master advises that postcopy migration has now completed. The slave
+ must disable the userfaultfd. The response is an acknowledgement
+ only.
+
+ When ``VHOST_USER_PROTOCOL_F_PAGEFAULT`` is supported, this message
+ is sent at the end of the migration, after
+ ``VHOST_USER_POSTCOPY_LISTEN`` was previously sent.
+
+ The value returned is an error indication; 0 is success.
+
+``VHOST_USER_GET_INFLIGHT_FD``
+ :id: 31
+ :equivalent ioctl: N/A
+ :master payload: inflight description
+
+ When ``VHOST_USER_PROTOCOL_F_INFLIGHT_SHMFD`` protocol feature has
+ been successfully negotiated, this message is submitted by master to
+ get a shared buffer from slave. The shared buffer will be used to
+ track inflight I/O by slave. QEMU should retrieve a new one when vm
+ reset.
+
+``VHOST_USER_SET_INFLIGHT_FD``
+ :id: 32
+ :equivalent ioctl: N/A
+ :master payload: inflight description
+
+ When ``VHOST_USER_PROTOCOL_F_INFLIGHT_SHMFD`` protocol feature has
+ been successfully negotiated, this message is submitted by master to
+ send the shared inflight buffer back to slave so that slave could
+ get inflight I/O after a crash or restart.
+
+Slave message types
+-------------------
+
+``VHOST_USER_SLAVE_IOTLB_MSG``
+ :id: 1
+ :equivalent ioctl: N/A (equivalent to ``VHOST_IOTLB_MSG`` message type)
+ :slave payload: ``struct vhost_iotlb_msg``
+ :master payload: N/A
+
+ Send IOTLB messages with ``struct vhost_iotlb_msg`` as payload.
+ Slave sends such requests to notify of an IOTLB miss, or an IOTLB
+ access failure. If ``VHOST_USER_PROTOCOL_F_REPLY_ACK`` is
+ negotiated, and slave set the ``VHOST_USER_NEED_REPLY`` flag, master
+ must respond with zero when operation is successfully completed, or
+ non-zero otherwise. This request should be send only when
+ ``VIRTIO_F_IOMMU_PLATFORM`` feature has been successfully
+ negotiated.
+
+``VHOST_USER_SLAVE_CONFIG_CHANGE_MSG``
+ :id: 2
+ :equivalent ioctl: N/A
+ :slave payload: N/A
+ :master payload: N/A
+
+ When ``VHOST_USER_PROTOCOL_F_CONFIG`` is negotiated, vhost-user
+ slave sends such messages to notify that the virtio device's
+ configuration space has changed, for those host devices which can
+ support such feature, host driver can send ``VHOST_USER_GET_CONFIG``
+ message to slave to get the latest content. If
+ ``VHOST_USER_PROTOCOL_F_REPLY_ACK`` is negotiated, and slave set the
+ ``VHOST_USER_NEED_REPLY`` flag, master must respond with zero when
+ operation is successfully completed, or non-zero otherwise.
+
+``VHOST_USER_SLAVE_VRING_HOST_NOTIFIER_MSG``
+ :id: 3
+ :equivalent ioctl: N/A
+ :slave payload: vring area description
+ :master payload: N/A
+
+ Sets host notifier for a specified queue. The queue index is
+ contained in the ``u64`` field of the vring area description. The
+ host notifier is described by the file descriptor (typically it's a
+ VFIO device fd) which is passed as ancillary data and the size
+ (which is mmap size and should be the same as host page size) and
+ offset (which is mmap offset) carried in the vring area
+ description. QEMU can mmap the file descriptor based on the size and
+ offset to get a memory range. Registering a host notifier means
+ mapping this memory range to the VM as the specified queue's notify
+ MMIO region. Slave sends this request to tell QEMU to de-register
+ the existing notifier if any and register the new notifier if the
+ request is sent with a file descriptor.
+
+ This request should be sent only when
+ ``VHOST_USER_PROTOCOL_F_HOST_NOTIFIER`` protocol feature has been
+ successfully negotiated.
+
+.. _reply_ack:
+
+VHOST_USER_PROTOCOL_F_REPLY_ACK
+-------------------------------
+
+The original vhost-user specification only demands replies for certain
+commands. This differs from the vhost protocol implementation where
+commands are sent over an ``ioctl()`` call and block until the client
+has completed.
+
+With this protocol extension negotiated, the sender (QEMU) can set the
+``need_reply`` [Bit 3] flag to any command. This indicates that the
+client MUST respond with a Payload ``VhostUserMsg`` indicating success
+or failure. The payload should be set to zero on success or non-zero
+on failure, unless the message already has an explicit reply body.
+
+The response payload gives QEMU a deterministic indication of the result
+of the command. Today, QEMU is expected to terminate the main vhost-user
+loop upon receiving such errors. In future, qemu could be taught to be more
+resilient for selective requests.
+
+For the message types that already solicit a reply from the client,
+the presence of ``VHOST_USER_PROTOCOL_F_REPLY_ACK`` or need_reply bit
+being set brings no behavioural change. (See the Communication_
+section for details.)
+
+.. _backend_conventions:
+
+Backend program conventions
+===========================
+
+vhost-user backends can provide various devices & services and may
+need to be configured manually depending on the use case. However, it
+is a good idea to follow the conventions listed here when
+possible. Users, QEMU or libvirt, can then rely on some common
+behaviour to avoid heterogenous configuration and management of the
+backend programs and facilitate interoperability.
+
+Each backend installed on a host system should come with at least one
+JSON file that conforms to the vhost-user.json schema. Each file
+informs the management applications about the backend type, and binary
+location. In addition, it defines rules for management apps for
+picking the highest priority backend when multiple match the search
+criteria (see ``@VhostUserBackend`` documentation in the schema file).
+
+If the backend is not capable of enabling a requested feature on the
+host (such as 3D acceleration with virgl), or the initialization
+failed, the backend should fail to start early and exit with a status
+!= 0. It may also print a message to stderr for further details.
+
+The backend program must not daemonize itself, but it may be
+daemonized by the management layer. It may also have a restricted
+access to the system.
+
+File descriptors 0, 1 and 2 will exist, and have regular
+stdin/stdout/stderr usage (they may have been redirected to /dev/null
+by the management layer, or to a log handler).
+
+The backend program must end (as quickly and cleanly as possible) when
+the SIGTERM signal is received. Eventually, it may receive SIGKILL by
+the management layer after a few seconds.
+
+The following command line options have an expected behaviour. They
+are mandatory, unless explicitly said differently:
+
+--socket-path=PATH
+
+ This option specify the location of the vhost-user Unix domain socket.
+ It is incompatible with --fd.
+
+--fd=FDNUM
+
+ When this argument is given, the backend program is started with the
+ vhost-user socket as file descriptor FDNUM. It is incompatible with
+ --socket-path.
+
+--print-capabilities
+
+ Output to stdout the backend capabilities in JSON format, and then
+ exit successfully. Other options and arguments should be ignored, and
+ the backend program should not perform its normal function. The
+ capabilities can be reported dynamically depending on the host
+ capabilities.
+
+The JSON output is described in the ``vhost-user.json`` schema, by
+```@VHostUserBackendCapabilities``. Example:
+
+.. code:: json
+
+ {
+ "type": "foo",
+ "features": [
+ "feature-a",
+ "feature-b"
+ ]
+ }
+
+vhost-user-input
+----------------
+
+Command line options:
+
+--evdev-path=PATH
+
+ Specify the linux input device.
+
+ (optional)
+
+--no-grab
+
+ Do no request exclusive access to the input device.
+
+ (optional)
+
+vhost-user-gpu
+--------------
+
+Command line options:
+
+--render-node=PATH
+
+ Specify the GPU DRM render node.
+
+ (optional)
+
+--virgl
+
+ Enable virgl rendering support.
+
+ (optional)
--
MST
next prev parent reply other threads:[~2019-05-16 12:22 UTC|newest]
Thread overview: 289+ messages / expand[flat|nested] mbox.gz Atom feed top
2019-05-16 12:17 [Qemu-devel] [PULL 00/37] pci, pc, virtio: features, fixes Michael S. Tsirkin
2019-05-16 12:17 ` Michael S. Tsirkin [this message]
2019-05-20 23:10 ` [Qemu-devel] [PULL v2 03/36] docs: reST-ify vhost-user documentation Michael S. Tsirkin
2019-05-16 12:18 ` [Qemu-devel] [PULL 04/37] virtio: Introduce started flag to VirtioDevice Michael S. Tsirkin
2019-05-20 23:10 ` [Qemu-devel] [PULL v2 04/36] " Michael S. Tsirkin
2019-05-24 10:19 ` Greg Kurz
2019-05-24 11:56 ` Yongji Xie
2019-05-27 10:44 ` Greg Kurz
2019-05-27 13:04 ` Yongji Xie
2019-05-27 15:45 ` Greg Kurz
2019-05-27 18:53 ` Michael S. Tsirkin
2019-05-28 2:48 ` Yongji Xie
2019-05-31 19:36 ` Eduardo Habkost
2019-06-01 15:49 ` Greg Kurz
2019-06-24 17:54 ` Laurent Vivier
2019-07-05 13:45 ` Greg Kurz
2019-05-28 0:08 ` David Gibson
2019-05-28 6:39 ` Greg Kurz
2019-05-29 11:18 ` Dr. David Alan Gilbert
2019-05-29 11:54 ` Greg Kurz
2019-05-29 12:38 ` Dr. David Alan Gilbert
2019-05-29 13:02 ` Greg Kurz
2019-05-29 13:40 ` Dr. David Alan Gilbert
2019-05-29 14:35 ` Yongji Xie
2019-05-29 14:42 ` Dr. David Alan Gilbert
2019-05-30 0:39 ` Yongji Xie
2019-05-30 9:06 ` Dr. David Alan Gilbert
2019-05-30 9:26 ` Yongji Xie
2019-05-30 9:34 ` Dr. David Alan Gilbert
2019-05-29 13:57 ` Yongji Xie
2019-05-16 12:18 ` [Qemu-devel] [PULL 05/37] virtio: Use started flag in virtio_vmstate_change() Michael S. Tsirkin
2019-05-20 23:10 ` [Qemu-devel] [PULL v2 05/36] " Michael S. Tsirkin
2019-05-16 12:18 ` [Qemu-devel] [PULL 06/37] vhost-user-blk: Use started flag in vhost_user_blk_set_status() Michael S. Tsirkin
2019-05-20 23:10 ` [Qemu-devel] [PULL v2 06/36] " Michael S. Tsirkin
2019-05-16 12:18 ` [Qemu-devel] [PULL 07/37] vhost-user-blk: Only start vhost-user backend with the first kick Michael S. Tsirkin
2019-05-20 23:10 ` [Qemu-devel] [PULL v2 07/36] " Michael S. Tsirkin
2019-05-16 12:18 ` [Qemu-devel] [PULL 08/37] vhost-user-blk: Add return value for vhost_user_blk_start() Michael S. Tsirkin
2019-05-20 23:10 ` [Qemu-devel] [PULL v2 08/36] " Michael S. Tsirkin
2019-05-16 12:18 ` [Qemu-devel] [PULL 09/37] vhost-user-blk: Add support to reconnect backend Michael S. Tsirkin
2019-05-20 23:10 ` [Qemu-devel] [PULL v2 09/36] " Michael S. Tsirkin
2019-05-16 12:18 ` [Qemu-devel] [PULL 10/37] contrib/vhost-user-blk: enable inflight I/O tracking Michael S. Tsirkin
2019-05-20 23:10 ` [Qemu-devel] [PULL v2 10/36] " Michael S. Tsirkin
2019-05-16 16:04 ` [Qemu-devel] [PULL 00/37] pci, pc, virtio: features, fixes Peter Maydell
2019-05-16 18:33 ` Philippe Mathieu-Daudé
2019-05-16 18:53 ` Philippe Mathieu-Daudé
2019-05-17 1:48 ` Wei Yang
2019-05-17 8:12 ` Philippe Mathieu-Daudé
2019-05-17 8:18 ` Thomas Huth
2019-05-17 8:37 ` Philippe Mathieu-Daudé
2019-05-17 2:59 ` Wei Yang
2019-05-17 3:12 ` Wei Yang
2019-05-17 8:14 ` Philippe Mathieu-Daudé
2019-05-17 8:19 ` Wei Yang
2019-05-17 11:13 ` Igor Mammedov
2019-05-20 0:33 ` Wei Yang
2019-05-20 22:59 ` Michael S. Tsirkin
2019-05-20 22:57 ` Michael S. Tsirkin
-- strict thread matches above, loose matches on Subject: below --
2019-05-20 23:10 [Qemu-devel] [PULL v2 00/36] " Michael S. Tsirkin
2019-05-21 11:49 ` Peter Maydell
2019-05-21 13:26 ` Michael S. Tsirkin
2019-05-22 13:06 ` Igor Mammedov
2019-05-22 14:22 ` Laszlo Ersek
2019-05-22 21:15 ` Peter Maydell
2019-05-23 11:30 ` Laszlo Ersek
2019-05-23 0:51 ` Laszlo Ersek
2019-05-23 0:57 ` Laszlo Ersek
2019-05-23 8:37 ` Peter Maydell
2019-05-24 10:56 ` Laszlo Ersek
2019-05-21 13:35 ` Michael S. Tsirkin
2019-05-21 13:42 ` Michael S. Tsirkin
2019-05-21 14:56 ` Peter Maydell
2019-05-02 14:51 [Qemu-devel] [PATCH v4 00/15] tests: acpi: add UEFI (ARM) testing support Igor Mammedov
2019-05-02 14:51 ` Igor Mammedov
2019-05-02 14:51 ` [Qemu-devel] [PATCH v4 01/15] tests: acpi: rename acpi_parse_rsdp_table() into acpi_fetch_rsdp_table() Igor Mammedov
2019-05-20 23:11 ` [Qemu-devel] [PULL v2 22/36] " Michael S. Tsirkin
2019-05-16 12:20 ` [Qemu-devel] [PULL 23/37] " Michael S. Tsirkin
2019-05-02 14:51 ` [Qemu-devel] [PATCH v4 01/15] " Igor Mammedov
2019-05-12 18:19 ` Michael S. Tsirkin
2019-05-13 9:04 ` Igor Mammedov
2019-05-13 9:35 ` Igor Mammedov
2019-05-17 7:55 ` [Qemu-devel] [PULL 23/37] " Igor Mammedov
2019-05-17 8:04 ` Philippe Mathieu-Daudé
2019-05-02 14:51 ` [Qemu-devel] [PATCH v4 02/15] tests: acpi: make acpi_fetch_table() take size of fetched table pointer Igor Mammedov
2019-05-20 23:11 ` [Qemu-devel] [PULL v2 23/36] " Michael S. Tsirkin
2019-05-16 12:20 ` [Qemu-devel] [PULL 24/37] " Michael S. Tsirkin
2019-05-02 14:51 ` [Qemu-devel] [PATCH v4 02/15] " Igor Mammedov
2019-05-05 0:58 ` Wei Yang
2019-05-05 0:58 ` Wei Yang
2019-05-02 14:51 ` [Qemu-devel] [PATCH v4 03/15] tests: acpi: make RSDT test routine handle XSDT Igor Mammedov
2019-05-20 23:11 ` [Qemu-devel] [PULL v2 24/36] " Michael S. Tsirkin
2019-05-16 12:20 ` [Qemu-devel] [PULL 25/37] " Michael S. Tsirkin
2019-05-02 14:51 ` [Qemu-devel] [PATCH v4 03/15] " Igor Mammedov
2019-05-05 1:14 ` Wei Yang
2019-05-05 1:14 ` Wei Yang
2019-05-02 14:51 ` [Qemu-devel] [PATCH v4 04/15] tests: acpi: make pointer to RSDP 64bit Igor Mammedov
2019-05-20 23:11 ` [Qemu-devel] [PULL v2 25/36] " Michael S. Tsirkin
2019-05-16 12:20 ` [Qemu-devel] [PULL 26/37] " Michael S. Tsirkin
2019-05-02 14:51 ` [Qemu-devel] [PATCH v4 04/15] " Igor Mammedov
2019-05-05 1:18 ` Wei Yang
2019-05-05 1:18 ` Wei Yang
2019-05-08 6:15 ` Philippe Mathieu-Daudé
2019-05-02 14:51 ` [Qemu-devel] [PATCH v4 05/15] tests: acpi: fetch X_DSDT if pointer to DSDT is 0 Igor Mammedov
2019-05-20 23:11 ` [Qemu-devel] [PULL v2 26/36] " Michael S. Tsirkin
2019-05-16 12:20 ` [Qemu-devel] [PULL 27/37] " Michael S. Tsirkin
2019-05-02 14:51 ` [Qemu-devel] [PATCH v4 05/15] " Igor Mammedov
2019-05-05 1:27 ` Wei Yang
2019-05-05 1:27 ` Wei Yang
2019-05-07 10:04 ` Igor Mammedov
2019-05-08 5:51 ` Wei Yang
2019-05-02 14:51 ` [Qemu-devel] [PATCH v4 06/15] tests: acpi: skip FACS table if board uses hw reduced ACPI profile Igor Mammedov
2019-05-20 23:11 ` [Qemu-devel] [PULL v2 27/36] " Michael S. Tsirkin
2019-05-16 12:20 ` [Qemu-devel] [PULL 28/37] " Michael S. Tsirkin
2019-05-02 14:51 ` [Qemu-devel] [PATCH v4 06/15] " Igor Mammedov
2019-05-02 14:51 ` [Qemu-devel] [PATCH v4 07/15] tests: acpi: move boot_sector_init() into x86 tests branch Igor Mammedov
2019-05-20 23:11 ` [Qemu-devel] [PULL v2 28/36] " Michael S. Tsirkin
2019-05-16 12:20 ` [Qemu-devel] [PULL 29/37] " Michael S. Tsirkin
2019-05-02 14:51 ` [Qemu-devel] [PATCH v4 07/15] " Igor Mammedov
2019-05-08 6:13 ` Philippe Mathieu-Daudé
2019-05-02 14:51 ` [Qemu-devel] [PATCH v4 08/15] tests: acpi: add acpi_find_rsdp_address_uefi() helper Igor Mammedov
2019-05-20 23:11 ` [Qemu-devel] [PULL v2 29/36] " Michael S. Tsirkin
2019-05-16 12:20 ` [Qemu-devel] [PULL 30/37] " Michael S. Tsirkin
2019-05-02 14:51 ` [Qemu-devel] [PATCH v4 08/15] " Igor Mammedov
2019-05-02 14:51 ` [Qemu-devel] [PATCH v4 09/15] tests: acpi: add a way to start tests with UEFI firmware Igor Mammedov
2019-05-20 23:11 ` [Qemu-devel] [PULL v2 30/36] " Michael S. Tsirkin
2019-05-16 12:20 ` [Qemu-devel] [PULL 31/37] " Michael S. Tsirkin
2019-05-02 14:51 ` [Qemu-devel] [PATCH v4 09/15] " Igor Mammedov
2019-05-02 14:51 ` [Qemu-devel] [PATCH v4 10/15] tests: acpi: ignore SMBIOS tests when UEFI firmware is used Igor Mammedov
2019-05-20 23:11 ` [Qemu-devel] [PULL v2 31/36] " Michael S. Tsirkin
2019-05-16 12:20 ` [Qemu-devel] [PULL 32/37] " Michael S. Tsirkin
2019-05-02 14:51 ` [Qemu-devel] [PATCH v4 10/15] " Igor Mammedov
2019-05-08 6:12 ` Philippe Mathieu-Daudé
2019-05-02 14:51 ` [Qemu-devel] [PATCH v4 11/15] tests: acpi: allow to override default accelerator Igor Mammedov
2019-05-20 23:11 ` [Qemu-devel] [PULL v2 32/36] " Michael S. Tsirkin
2019-05-16 12:21 ` [Qemu-devel] [PULL 33/37] " Michael S. Tsirkin
2019-05-02 14:51 ` [Qemu-devel] [PATCH v4 11/15] " Igor Mammedov
2019-05-02 18:36 ` Laszlo Ersek
2019-05-02 14:52 ` [Qemu-devel] [PATCH v4 12/15] tests: add expected ACPI tables for arm/virt board Igor Mammedov
2019-05-20 23:11 ` [Qemu-devel] [PULL v2 33/36] " Michael S. Tsirkin
2019-05-16 12:21 ` [Qemu-devel] [PULL 34/37] " Michael S. Tsirkin
2019-05-02 14:52 ` [Qemu-devel] [PATCH v4 12/15] " Igor Mammedov
2019-05-02 14:52 ` [Qemu-devel] [PATCH v4 13/15] tests: acpi: add simple arm/virt testcase Igor Mammedov
2019-05-20 23:11 ` [Qemu-devel] [PULL v2 34/36] " Michael S. Tsirkin
2019-05-16 12:21 ` [Qemu-devel] [PULL 35/37] " Michael S. Tsirkin
2019-05-02 14:52 ` [Qemu-devel] [PATCH v4 13/15] " Igor Mammedov
2019-05-02 18:38 ` Laszlo Ersek
2019-05-02 14:52 ` [Qemu-devel] [PATCH v4 14/15] tests: acpi: refactor rebuild-expected-aml.sh to dump ACPI tables for a specified list of targets Igor Mammedov
2019-05-20 23:11 ` [Qemu-devel] [PULL v2 35/36] " Michael S. Tsirkin
2019-05-16 12:21 ` [Qemu-devel] [PULL 36/37] " Michael S. Tsirkin
2019-05-02 14:52 ` [Qemu-devel] [PATCH v4 14/15] " Igor Mammedov
2019-05-05 1:30 ` Wei Yang
2019-05-05 1:30 ` Wei Yang
2019-05-02 14:52 ` [Qemu-devel] [PATCH v4 15/15] tests: acpi: print error unable to dump ACPI table during rebuild Igor Mammedov
2019-05-20 23:11 ` [Qemu-devel] [PULL v2 36/36] " Michael S. Tsirkin
2019-05-16 12:21 ` [Qemu-devel] [PULL 37/37] " Michael S. Tsirkin
2019-05-02 14:52 ` [Qemu-devel] [PATCH v4 15/15] " Igor Mammedov
2019-05-02 15:16 ` [Qemu-devel] [PATCH v4 00/15] tests: acpi: add UEFI (ARM) testing support Wei Xu
2019-05-02 15:16 ` Wei Xu
2019-04-24 4:19 [Qemu-devel] [PATCH v2 0/3] Simplify some not-really-necessary PCI bus callbacks David Gibson
2019-04-24 4:19 ` David Gibson
2019-04-24 4:19 ` [Qemu-devel] [PATCH v2 1/3] pcie: Remove redundant test in pcie_mmcfg_data_{read, write}() David Gibson
2019-05-20 23:11 ` [Qemu-devel] [PULL v2 20/36] " Michael S. Tsirkin
2019-05-16 12:19 ` [Qemu-devel] [PULL 21/37] " Michael S. Tsirkin
2019-04-24 4:19 ` [Qemu-devel] [PATCH v2 1/3] " David Gibson
2019-04-24 16:04 ` Greg Kurz
2019-04-24 16:04 ` Greg Kurz
2019-04-24 4:19 ` [Qemu-devel] [PATCH v2 2/3] pci: Simplify pci_bus_is_root() David Gibson
2019-05-20 23:11 ` [Qemu-devel] [PULL v2 21/36] " Michael S. Tsirkin
2019-05-16 12:20 ` [Qemu-devel] [PULL 22/37] " Michael S. Tsirkin
2019-04-24 4:19 ` [Qemu-devel] [PATCH v2 2/3] " David Gibson
2019-05-22 6:11 ` [Qemu-devel] [PULL v2 21/36] " David Gibson
2019-04-24 4:19 ` [Qemu-devel] [PATCH v2 3/3] pcie: Simplify pci_adjust_config_limit() David Gibson
2019-04-24 4:19 ` David Gibson
2019-04-24 16:09 ` Greg Kurz
2019-04-24 16:09 ` Greg Kurz
2019-04-26 6:40 ` [Qemu-devel] [Qemu-ppc] " Alexey Kardashevskiy
2019-05-07 4:48 ` David Gibson
2019-05-12 18:13 ` Michael S. Tsirkin
2019-05-13 6:20 ` David Gibson
2019-04-20 9:10 [Qemu-devel] [PATCH] libvhost-user: fix bad vu_log_write Li Feng
2019-05-20 23:11 ` [Qemu-devel] [PULL v2 19/36] " Michael S. Tsirkin
2019-05-16 12:19 ` [Qemu-devel] [PULL 20/37] " Michael S. Tsirkin
2019-04-21 16:48 ` [Qemu-devel] [PATCH] " Marc-André Lureau
2019-04-21 16:48 ` Marc-André Lureau
2019-04-19 0:30 [Qemu-devel] [PATCH v4 0/6] Extract build_mcfg Wei Yang
2019-04-19 0:30 ` Wei Yang
2019-04-19 0:30 ` [Qemu-devel] [PATCH v4 1/6] q35: acpi: do not create dummy MCFG table Wei Yang
2019-04-19 0:30 ` Wei Yang
2019-04-19 0:30 ` [Qemu-devel] [PATCH v4 2/6] hw/arm/virt-acpi-build: remove unnecessary variable mcfg_start Wei Yang
2019-05-20 23:10 ` [Qemu-devel] [PULL v2 16/36] " Michael S. Tsirkin
2019-05-16 12:19 ` [Qemu-devel] [PULL 16/37] " Michael S. Tsirkin
2019-04-19 0:30 ` [Qemu-devel] [PATCH v4 2/6] " Wei Yang
2019-04-19 0:30 ` [Qemu-devel] [PATCH v4 3/6] i386, acpi: remove mcfg_ prefix in AcpiMcfgInfo members Wei Yang
2019-05-20 23:11 ` [Qemu-devel] [PULL v2 17/36] " Michael S. Tsirkin
2019-05-16 12:19 ` [Qemu-devel] [PULL 17/37] " Michael S. Tsirkin
2019-04-19 0:30 ` [Qemu-devel] [PATCH v4 3/6] " Wei Yang
2019-04-19 0:30 ` [Qemu-devel] [PATCH v4 4/6] hw/arm/virt-acpi-build: pass AcpiMcfgInfo to build_mcfg() Wei Yang
2019-05-20 23:11 ` [Qemu-devel] [PULL v2 18/36] " Michael S. Tsirkin
2019-05-16 12:19 ` [Qemu-devel] [PULL 18/37] " Michael S. Tsirkin
2019-04-19 0:30 ` [Qemu-devel] [PATCH v4 4/6] " Wei Yang
2019-04-19 0:30 ` [Qemu-devel] [PATCH v4 5/6] hw/acpi: Consolidate build_mcfg to pci.c Wei Yang
2019-05-16 12:19 ` [Qemu-devel] [PULL 19/37] " Michael S. Tsirkin
2019-04-19 0:30 ` [Qemu-devel] [PATCH v4 5/6] " Wei Yang
2019-05-16 18:35 ` Philippe Mathieu-Daudé
2019-05-17 0:33 ` Wei Yang
2019-04-19 0:30 ` [Qemu-devel] [PATCH v4 6/6] acpi: pci: use build_append_foo() API to construct MCFG Wei Yang
2019-04-19 0:30 ` Wei Yang
2019-05-15 1:10 ` Michael S. Tsirkin
2019-05-15 5:29 ` Philippe Mathieu-Daudé
2019-05-15 8:53 ` Wei Yang
2019-05-16 7:41 ` Wei Yang
2019-05-16 11:01 ` Philippe Mathieu-Daudé
2019-05-16 17:00 ` Igor Mammedov
2019-05-20 23:04 ` Michael S. Tsirkin
2019-05-15 8:46 ` Wei Yang
2019-05-10 21:22 ` [Qemu-devel] [PATCH v4 0/6] Extract build_mcfg Wei Yang
2019-05-10 23:59 ` Michael S. Tsirkin
2019-05-11 0:10 ` Wei Yang
2019-04-16 18:46 [Qemu-devel] [PATCH 0/2] vhost-user race condition on shutdown Dan Streetman
2019-04-16 18:46 ` [Qemu-devel] [PATCH 1/2] add VirtIONet vhost_stopped flag to prevent multiple stops Dan Streetman
2019-04-19 23:14 ` Michael S. Tsirkin
2019-04-22 20:31 ` Dan Streetman
2019-04-22 2:50 ` Jason Wang
2019-04-22 20:14 ` Dan Streetman
2019-04-22 20:14 ` Dan Streetman
2019-04-23 2:58 ` Jason Wang
2019-04-23 2:58 ` Jason Wang
2019-04-23 8:49 ` Dan Streetman
2019-04-23 8:49 ` Dan Streetman
2019-04-24 9:46 ` Jason Wang
2019-04-24 9:46 ` Jason Wang
2019-04-16 18:46 ` [Qemu-devel] [PATCH 2/2] do not call vhost_net_cleanup() on running net from char user event Dan Streetman
2019-05-20 23:10 ` [Qemu-devel] [PULL v2 15/36] " Michael S. Tsirkin
2019-05-16 12:19 ` [Qemu-devel] [PULL 15/37] " Michael S. Tsirkin
2019-04-19 23:12 ` [Qemu-devel] [PATCH 0/2] vhost-user race condition on shutdown Michael S. Tsirkin
2019-04-19 23:12 ` Michael S. Tsirkin
2019-04-09 15:00 [Qemu-devel] [PATCH for-4.1] q35: acpi: do not create dummy MCFG table Igor Mammedov
2019-05-20 23:10 ` [Qemu-devel] [PULL v2 14/36] " Michael S. Tsirkin
2019-05-16 12:19 ` [Qemu-devel] [PULL 14/37] " Michael S. Tsirkin
2019-04-09 15:00 ` [Qemu-devel] [PATCH for-4.1] " Igor Mammedov
2019-04-10 1:12 ` Wei Yang
2019-04-10 1:12 ` Wei Yang
2019-04-10 9:08 ` Igor Mammedov
2019-04-10 9:08 ` Igor Mammedov
2019-04-10 14:01 ` Wei Yang
2019-04-10 14:01 ` Wei Yang
2019-04-10 14:11 ` Igor Mammedov
2019-04-10 14:11 ` Igor Mammedov
2019-04-10 14:27 ` Wei Yang
2019-04-10 14:27 ` Wei Yang
2019-04-10 15:01 ` Igor Mammedov
2019-04-10 15:01 ` Igor Mammedov
2019-04-11 1:32 ` Wei Yang
2019-04-11 1:32 ` Wei Yang
2019-04-11 11:46 ` Igor Mammedov
2019-04-11 11:46 ` Igor Mammedov
2019-04-11 22:15 ` Wei Yang
2019-04-11 22:15 ` Wei Yang
2019-04-11 22:16 ` Wei Yang
2019-04-11 22:16 ` Wei Yang
2019-04-02 16:18 [Qemu-devel] [PATCH 0/3] acpi: More trace points Markus Armbruster
2019-04-02 16:18 ` [Qemu-devel] [PATCH 1/3] acpi/piix4: Convert debug printf()s to trace events Markus Armbruster
2019-05-20 23:10 ` [Qemu-devel] [PULL v2 11/36] " Michael S. Tsirkin
2019-05-16 12:19 ` [Qemu-devel] [PULL 11/37] " Michael S. Tsirkin
2019-04-04 10:07 ` [Qemu-devel] [PATCH 1/3] " Igor Mammedov
2019-04-02 16:18 ` [Qemu-devel] [PATCH 2/3] acpi/pcihp: " Markus Armbruster
2019-05-20 23:10 ` [Qemu-devel] [PULL v2 12/36] " Michael S. Tsirkin
2019-05-16 12:19 ` [Qemu-devel] [PULL 12/37] " Michael S. Tsirkin
2019-04-04 10:13 ` [Qemu-devel] [PATCH 2/3] " Igor Mammedov
2019-04-02 16:19 ` [Qemu-devel] [PATCH 3/3] acpi/pcihp: Add a few more trace points related to unplug Markus Armbruster
2019-05-20 23:10 ` [Qemu-devel] [PULL v2 13/36] " Michael S. Tsirkin
2019-05-16 12:19 ` [Qemu-devel] [PULL 13/37] " Michael S. Tsirkin
2019-04-04 10:14 ` [Qemu-devel] [PATCH 3/3] " Igor Mammedov
2019-04-04 12:54 ` Laszlo Ersek
2019-04-04 14:19 ` Igor Mammedov
2019-04-02 19:24 ` [Qemu-devel] [PATCH 0/3] acpi: More trace points Philippe Mathieu-Daudé
2019-05-08 11:19 ` Markus Armbruster
2019-05-08 16:30 ` Michael S. Tsirkin
2019-02-15 10:32 [Qemu-devel] [PATCH v2 0/2] hw: provide error checking of disable-legacy/modern property usage Daniel P. Berrangé
2019-02-15 10:32 ` [Qemu-devel] [PATCH v2 1/2] hw: report invalid disable-legacy|modern usage for virtio-1-only devs Daniel P. Berrangé
2019-05-20 23:10 ` [Qemu-devel] [PULL v2 01/36] " Michael S. Tsirkin
2019-05-16 12:17 ` [Qemu-devel] [PULL 01/37] " Michael S. Tsirkin
2019-05-17 19:01 ` [Qemu-devel] [PATCH v2 1/2] " Eduardo Habkost
2019-05-20 9:56 ` Daniel P. Berrangé
2019-05-20 20:59 ` Eduardo Habkost
2019-05-21 9:23 ` Daniel P. Berrangé
2019-02-15 10:32 ` [Qemu-devel] [PATCH v2 2/2] Revert "globals: Allow global properties to be optional" Daniel P. Berrangé
2019-05-20 23:10 ` [Qemu-devel] [PULL v2 02/36] " Michael S. Tsirkin
2019-05-16 12:17 ` [Qemu-devel] [PULL 02/37] " Michael S. Tsirkin
2019-05-20 21:00 ` [Qemu-devel] [PATCH v2 2/2] " Eduardo Habkost
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20190315180735.13096-1-marcandre.lureau@redhat.com \
--to=mst@redhat.com \
--cc=jfreimann@redhat.com \
--cc=marcandre.lureau@redhat.com \
--cc=peter.maydell@linaro.org \
--cc=qemu-devel@nongnu.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.