[virtio-dev] [PATCH v5 00/10] introduce virtio vhost-user backend device type

[virtio-dev] [PATCH v5 00/10] introduce virtio vhost-user backend device type

[Nikos Dragazis]

Hi everyone,

this patch presents an updated version of the virtio device spec for the
virtio-vhost-user device. The initial RFC implementation can be found
here: [1]. It’s been a long time since the last revision. I’m sending
this patch to kick off the discussion again. I'm Cc-ing Stefan and
Michael who had engaged with this in the past.

In this fifth revision of the patch I have made some minor and major
changes. The most important is the drop of the device-specific PCI
capabilities, as Michael suggested [2]. This guarantees portability
across transports, i.e., the device spec is not bound to the PCI
transport, as it used to be up to this point.

In more detail, the changes in this revision are outlined below. You can
find an HTML version here: [3]. Note that I have kept the commits
separate so that it is easy to track the changes from the previous
revision [4].

Looking forward to your comments.

Thanks,
Nikos

v5 changes:
 * Change device id from #28 to #32
 * Fix definitions of VIRTIO_VHOST_USER_STATUS_{MASTER,SLAVE}_UP bits
 * Remove the extra PCI capabilities and use existing virtio registers
   instead
 * Intercept slave's reply to VHOST_USER_GET_PROTOCOL_FEATURES
 * Some more minor fixes (typos, etc.)

v4 changes:
 * Add comformance targets for the virtio-vhost-user device.
 * Update the reference link for the vhost-user protocol specification.

v3 changes:
 * Device Requirements for the Notification Capability: point out the
   difference between the MSI-X Table Size stored in the Message Control
   register of the MSI-X capability structure and the actual MSI-X Table
   Size

v2 changes:
 * Change device id from #25 to #28


[1] https://lists.oasis-open.org/archives/virtio-dev/201801/msg00110.html
[2] https://lists.oasis-open.org/archives/virtio-dev/201908/msg00014.html
[3] https://ndragazis.github.io/virtio/vvu/v5/virtio-v1.1-cs01.html#x1-49600014
[4] https://lists.oasis-open.org/archives/virtio-dev/201906/msg00036.html

Nikos Dragazis (9):
  vhost-user: minor fixes
  vhost-user: add requirements for the notification capability
  vhost-user: update shared memory capability
  vhost-user: add conformance targets and clauses
  vhost-user: change the device id
  vhost-user: minor fix on status register
  vhost-user: remove the extra PCI capabilities
  vhost-user: intercept slave's reply to
    VHOST_USER_GET_PROTOCOL_FEATURES
  vhost-user: clarify that we are talking about slave-initiated messages

Stefan Hajnoczi (1):
  vhost-user: add vhost-user device type

 conformance.tex       |  11 +-
 content.tex           |   3 +
 introduction.tex      |   1 +
 virtio-vhost-user.tex | 282 ++++++++++++++++++++++++++++++++++++++++++
 4 files changed, 295 insertions(+), 2 deletions(-)
 create mode 100644 virtio-vhost-user.tex

-- 
2.17.1


---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

[virtio-dev] [PATCH v5 01/10] vhost-user: add vhost-user device type

[Nikos Dragazis]

From: Stefan Hajnoczi <stefanha@redhat.com>

The vhost-user device backend facilitates vhost-user device emulation
through vhost-user protocol exchanges and access to shared memory.
Software-defined networking, storage, and other I/O appliances can
provide services through this device.

This device is based on Wei Wang's vhost-pci work.  The virtio
vhost-user device differs from vhost-pci because it is a single virtio
device type that exposes the vhost-user protocol instead of a family of
new virtio device types, one for each vhost-user device type.

This device supports vhost-user slave and vhost-user master
reconnection.  It also contains a UUID so that vhost-user slave programs
can identify a specific device among many without using bus addresses.

It is somewhat unconventional for a virtio device because it makes use
of additional resources called doorbells, notifications, and shared
memory.  A mapping of these resources to the virtio PCI transport is
provided.  Other transports, such as CCW may not be able to support
this device.

Cc: Wei Wang <wei.w.wang@intel.com>
Cc: Michael S. Tsirkin <mst@redhat.com>
Cc: Maxime Coquelin <maxime.coquelin@redhat.com>
Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
---
 content.tex           |   3 +
 introduction.tex      |   1 +
 virtio-vhost-user.tex | 292 ++++++++++++++++++++++++++++++++++++++++++
 3 files changed, 296 insertions(+)
 create mode 100644 virtio-vhost-user.tex

diff --git a/content.tex b/content.tex
index 91735e3..9f3e86d 100644
--- a/content.tex
+++ b/content.tex
@@ -2801,6 +2801,8 @@ \chapter{Device Types}\label{sec:Device Types}
 \hline
 31         &   Video decoder device \\
 \hline
+28         &   vhost-user device backend \\
+\hline
 \end{tabular}
 
 Some of the devices above are unspecified by this document,
@@ -6062,6 +6064,7 @@ \subsubsection{Legacy Interface: Framing Requirements}\label{sec:Device
 \input{virtio-fs.tex}
 \input{virtio-rpmb.tex}
 \input{virtio-iommu.tex}
+\input{virtio-vhost-user.tex}
 
 \chapter{Reserved Feature Bits}\label{sec:Reserved Feature Bits}
 
diff --git a/introduction.tex b/introduction.tex
index 33da3ec..9ef0aa7 100644
--- a/introduction.tex
+++ b/introduction.tex
@@ -66,6 +66,7 @@ \section{Normative References}\label{sec:Normative References}
         \phantomsection\label{intro:eMMC}\textbf{[eMMC]} &
         eMMC Electrical Standard (5.1), JESD84-B51,
         \newline\url{http://www.jedec.org/sites/default/files/docs/JESD84-B51.pdf}\\
+	\phantomsection\label{intro:Vhost-user Protocol}\textbf{[Vhost-user Protocol]} & Vhost-user Protocol, \newline\url{https://git.qemu.org/?p=qemu.git;a=blob_plain;f=docs/interop/vhost-user.rst;hb=HEAD}, and any future revisions\\
 
 \end{longtable}
 
diff --git a/virtio-vhost-user.tex b/virtio-vhost-user.tex
new file mode 100644
index 0000000..ac96dc2
--- /dev/null
+++ b/virtio-vhost-user.tex
@@ -0,0 +1,292 @@
+\section{Vhost-user Device Backend}\label{sec:Device Types / Vhost-user Device Backend}
+
+The vhost-user device backend facilitates vhost-user device emulation through
+vhost-user protocol exchanges and access to shared memory.  Software-defined
+networking, storage, and other I/O appliances can provide services through this
+device.
+
+This section relies on definitions from the \hyperref[intro:Vhost-user
+Protocol]{Vhost-user Protocol}.  Knowledge of the vhost-user protocol is a
+prerequisite for understanding this device.
+
+The \hyperref[intro:Vhost-user Protocol]{Vhost-user Protocol} was originally
+designed for processes on a single system communicating over UNIX domain
+sockets.  The virtio vhost-user device backend allows the vhost-user slave to
+communicate with the vhost-user master over the device instead of a UNIX domain
+socket.  This allows the slave and master to run on two separate systems such
+as a virtual machine and a hypervisor.
+
+The vhost-user slave program exchanges vhost-user protocol messages with the
+vhost-user master through this device.  How the device implementation
+communicates with the vhost-user master is beyond the scope of this
+specification.  One possible device implementation uses a UNIX domain socket to
+relay messages to a vhost-user master process running on the same host.
+
+Existing vhost-user slave programs that communicate over UNIX domain sockets
+can support the virtio vhost-user device backend without invasive changes
+because the pre-existing vhost-user wire protocol is used.
+
+\subsection{Device ID}\label{sec:Device Types / Vhost-user Device Backend / Device ID}
+  28
+
+\subsection{Virtqueues}\label{sec:Device Types / Vhost-user Device Backend / Virtqueues}
+
+\begin{description}
+\item[0] rxq (device-to-driver vhost-user protocol messages)
+\item[1] txq (driver-to-device vhost-user protocol messages)
+\end{description}
+
+\subsection{Feature bits}\label{sec:Device Types / Vhost-user Device Backend / Feature bits}
+
+No feature bits are defined at this time.
+
+\subsection{Device configuration layout}\label{sec:Device Types / Vhost-user Device Backend / Device configuration layout}
+
+  All fields of this configuration are always available.
+
+\begin{lstlisting}
+struct virtio_vhost_user_config {
+        le32 status;
+#define VIRTIO_VHOST_USER_STATUS_SLAVE_UP 0
+#define VIRTIO_VHOST_USER_STATUS_MASTER_UP 1
+        le32 max_vhost_queues;
+        u8 uuid[16];
+};
+\end{lstlisting}
+
+\begin{description}
+\item[\field{status}] contains the vhost-user operational status.  The default
+    value of this field is 0.
+
+    The driver sets VIRTIO_VHOST_USER_STATUS_SLAVE_UP to indicate readiness for
+    the vhost-user master to connect.  The vhost-user master cannot connect
+    unless the driver has set this bit first.
+
+    When the driver clears VIRTIO_VHOST_USER_SLAVE_UP while the vhost-user
+    master is connected, the vhost-user master is disconnected.
+
+    When the vhost-user master disconnects, both
+    VIRTIO_VHOST_USER_STATUS_SLAVE_UP and VIRTIO_VHOST_USER_STATUS_MASTER_UP
+    are cleared by the device.  Communication can be restarted by the driver
+    setting VIRTIO_VHOST_USER_STATUS_SLAVE_UP again.
+
+    A configuration change notification is sent when the device changes
+    this field unless a write to the field by the driver caused the change.
+
+\item[\field{max_vhost_queues}] is the maximum number of vhost-user queues
+    supported by this device.  This field is always greater than 0.
+
+\item[\field{uuid}] is the Universally Unique Identifier (UUID) for this
+    device.  If the device has no UUID then this field contains the nil
+    UUID (all zeroes).  The UUID allows vhost-user slave programs to identify a
+    specific vhost-user device backend among many without relying on bus
+    addresses.
+\end{description}
+
+\drivernormative{\subsubsection}{Device configuration layout}{Device Types / Vhost-user Device Backend / Device configuration layout}
+
+The driver MUST NOT write to device configuration fields other than
+\field{status}.
+
+The driver MUST NOT set undefined bits in the \field{status} configuration field.
+
+\devicenormative{\subsection}{Device Initialization}{Device Types / Vhost-user Device Backend / Device Initialization}
+
+The driver SHOULD check the \field{max_vhost_queues} configuration field to
+determine how many queues the vhost-user slave will be able to support.
+
+The driver SHOULD fetch the \field{uuid} configuration field to allow
+vhost-user slave programs to identify a specific device among many.
+
+The driver SHOULD place at least one buffer in rxq before setting the
+VIRTIO_VHOST_USER_SLAVE_UP bit in the \field{status} configuration field.
+
+The driver MUST handle rxq virtqueue notifications that occur before the
+configuration change notification.  It is possible that a vhost-user protocol
+message from the vhost-user master arrives before the driver has seen the
+configuration change notification for the VIRTIO_VHOST_USER_STATUS_MASTER_UP
+\field{status} change.
+
+\subsection{Device Operation}\label{sec:Device Types / Vhost-user Device Backend / Device Operation}
+
+Device operation consists of operating request queues and response queues.
+
+\subsubsection{Device Operation: Request Queues}\label{sec:Device Types / Vhost-user Device Backend / Device Operation / Device Operation: Request Queues}
+
+The driver receives vhost-user protocol messages from the vhost-user master on
+rxq.  The driver sends responses to the vhost-user master on txq.
+
+The driver sends slave-initiated requests on txq.  The driver receives
+responses from the vhost-user master on rxq.
+
+All virtqueues offer in-order guaranteed delivery semantics for vhost-user
+protocol messages.
+
+Each buffer is a vhost-user protocol message as defined by the
+\hyperref[intro:Vhost-user Protocol]{Vhost-user Protocol}.  In order to enable
+cross-endian communication, all message fields are little-endian instead of the
+native byte order normally used by the protocol.
+
+The appropriate size of rxq buffers is at least as large as the largest message
+defined by the \hyperref[intro:Vhost-user Protocol]{Vhost-user Protocol}
+standard version that the driver supports.  If the vhost-user master sends a
+message that is too large for an rxq buffer then DEVICE_NEEDS_RESET is set and
+the driver must reset the device.
+
+File descriptor passing is handled differently by the vhost-user device
+backend.  When a message is received that carries one or more file descriptors
+according to the vhost-user protocol, additional device resources become
+available to the driver.
+
+\subsection{Additional Device Resources over PCI}\label{sec:Device Types / Vhost-user Device Backend / Additional Device Resources over PCI}
+
+The vhost-user device backend contains additional device resources beyond
+configuration space and virtqueues.  The nature of these resources is
+transport-specific and therefore only virtio transports that provide these
+resources support the vhost-user device backend.
+
+The following additional resources exist:
+\begin{description}
+  \item[Doorbells] The driver signals the vhost-user master through doorbells.  The signal does not carry any data, it is purely an event.
+  \item[Notifications] The vhost-user master signals the driver for events besides virtqueue activity and configuration changes by sending notifications.
+  \item[Shared memory] The vhost-user master gives access to memory that can be mapped by the driver.
+\end{description}
+
+\subsubsection{Doorbell Numbering}\label{sec:Device Types / Vhost-user Device Backend / Additional Device Resources over PCI / Doorbell Numbering}
+
+Doorbells are laid out as follows:
+
+\begin{description}
+\item[0] Vring call for vhost-user queue 0
+\item[\ldots]
+\item[N] Vring err for vhost-user queue 0
+\item[\ldots]
+\item[2N] Log
+\end{description}
+
+\subsubsection{Notifications}\label{sec:Device Types / Vhost-user Device Backend / Additional Device Resources over PCI / Notifications}
+
+Notifications are laid out as follows:
+
+\begin{description}
+\item[0] Vring kick for vhost-user queue 0
+\item[\ldots]
+\item[N-1] Vring kick for vhost-user queue N-1
+\end{description}
+
+\subsubsection{Shared Memory Layout}\label{sec:Device Types / Vhost-user Device Backend / Additional Device Resources over PCI / Shared Memory Layout}
+
+Shared memory is laid out as follows:
+
+\begin{description}
+\item[0] Vhost memory region 0
+\item[SIZE0] Vhost memory region 1
+\item[\ldots]
+\item[SIZE0 + SIZE1 + \ldots] Log
+\end{description}
+
+The size of vhost memory region 0 is \field{SIZE0}, the size of vhost memory
+region 1 is \field{SIZE1}, and so on.
+
+\subsubsection{Availability of Additional Resources}\label{sec:Device Types / Vhost-user Device Backend / Additional Device Resources over PCI / Availability of Additional Resources}
+
+The following vhost-user protocol messages convey access to additional device
+resources:
+
+\begin{description}
+\item[VHOST_USER_SET_MEM_TABLE] Contents of vhost memory regions are available to the driver in shared memory.  Region contents are laid out in the same order as the vhost memory region list.
+\item[VHOST_USER_SET_LOG_BASE] Contents of the log are available to the driver in shared memory.
+\item[VHOST_USER_SET_LOG_FD] The log doorbell is available to the driver.  Writes to the log doorbell before this message is received produce no effect.
+\item[VHOST_USER_SET_VRING_KICK] The vring kick notification for this queue is available to the driver.  The first notification may occur before the driver has processed this message.
+\item[VHOST_USER_SET_VRING_CALL] The vring call doorbell for this queue is available to the driver.  Writes to the vring call doorbell before this message is received produce no effect.
+\item[VHOST_USER_SET_VRING_ERR] The vring err doorbell for this queue is available to the driver.  Writes to the vring err doorbell before this message is received produce no effect.
+\item[VHOST_USER_SET_SLAVE_REQ_FD] The driver may send vhost-user protocol slave messages on txq.  Buffers put onto txq before this message is received are discarded by the device.
+\end{description}
+
+Additional resources are configured on the virtio PCI transport by the following \field{struct virtio_pci_cap.cfg_type} values:
+
+\begin{lstlisting}
+#define VIRTIO_PCI_CAP_DOORBELL_CFG 6
+#define VIRTIO_PCI_CAP_NOTIFICATION_CFG 7
+#define VIRTIO_PCI_CAP_SHARED_MEMORY_CFG 8
+\end{lstlisting}
+
+\subsubsection{Doorbell structure layout}\label{sec:Device Types / Vhost-user Device Backend / Additional Device Resources over PCI / Doorbell capability}
+
+The doorbell location is found using the VIRTIO_PCI_CAP_DOORBELL_CFG
+capability.  This capability is immediately followed by an additional
+field, like so:
+
+\begin{lstlisting}
+struct virtio_pci_doorbell_cap {
+        struct virtio_pci_cap cap;
+        le32 doorbell_off_multiplier;
+};
+\end{lstlisting}
+
+The doorbell address within a BAR is calculated as follows:
+
+\begin{lstlisting}
+        cap.offset + doorbell_idx * doorbell_off_multiplier
+\end{lstlisting}
+
+The \field{cap.offset} and \field{doorbell_off_multiplier} are taken from the
+notification capability structure above, and the \field{doorbell_idx} is the
+doorbell number.
+
+\devicenormative{\paragraph}{Doorbell capability}{Device Types / Vhost-user Device Backend / Additional Device Resources over PCI / Doorbell capability}
+The device MUST present at least one doorbell capability.
+
+The \field{cap.offset} MUST be 2-byte aligned.
+
+The device MUST either present \field{doorbell_off_multiplier} as an even power of 2,
+or present \field{doorbell_off_multiplier} as 0.
+
+The value \field{cap.length} presented by the device MUST be at least 2
+and MUST be large enough to support doorbell offsets for all supported
+doorbells in all possible configurations.
+
+The value \field{cap.length} presented by the device MUST satisfy:
+\begin{lstlisting}
+cap.length >= num_doorbells * doorbell_off_multiplier + 2
+\end{lstlisting}
+
+The number of doorbells is \field{num_doorbells} and is dependent on the
+device.
+
+\subsubsection{Notification structure layout}\label{sec:Device Types / Vhost-user Device Backend / Additional Device Resources over PCI / Notification capability}
+
+The notification structure allows MSI-X vectors to be configured for
+notification interrupts.  If MSI-X is not available, bit 2 of the ISR status
+indicates that a notification occurred.
+
+The notification structure is found using the VIRTIO_PCI_CAP_DOORBELL_CFG
+capability.
+
+\begin{lstlisting}
+struct virtio_pci_notification_cfg {
+        le16 notification_select;              /* read-write */
+        le16 notification_msix_vector;         /* read-write */
+};
+\end{lstlisting}
+
+The driver indicates which notification is of interest by writing the
+\field{notification_select} field.  The driver then writes the MSI-X vector or
+\field{VIRTIO_MSI_NO_VECTOR} to \field{notification_msix_vector} to change the
+MSI-X vector for that notification.
+
+\subsubsection{Shared memory capability}\label{sec:Device Types / Vhost-user Device Backend / Additional Device Resources over PCI / Shared Memory capability}
+
+The shared memory location is found using the VIRTIO_PCI_CAP_SHARED_MEMORY_CFG
+capability.
+
+\devicenormative{\paragraph}{Shared Memory capability}{Device Types / Vhost-user Device Backend / Additional Device Resources over PCI / Shared Memory capability}
+The device MUST present exactly one shared memory capability.
+
+The device MUST locate shared memory in a Memory Space BAR.
+
+The device SHOULD locate shared memory in a Prefetchable BAR.
+
+The \field{cap.offset} MUST be 4096-byte aligned.
+
+The value \field{cap.length} presented by the device MUST be non-zero and 4096-byte aligned.
-- 
2.17.1


---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

[virtio-dev] [PATCH v5 02/10] vhost-user: minor fixes

[Nikos Dragazis]

Signed-off-by: Nikos Dragazis <ndragazis@arrikto.com>
Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>
---
 virtio-vhost-user.tex | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/virtio-vhost-user.tex b/virtio-vhost-user.tex
index ac96dc2..13ecd41 100644
--- a/virtio-vhost-user.tex
+++ b/virtio-vhost-user.tex
@@ -231,7 +231,7 @@ \subsubsection{Doorbell structure layout}\label{sec:Device Types / Vhost-user De
 \end{lstlisting}
 
 The \field{cap.offset} and \field{doorbell_off_multiplier} are taken from the
-notification capability structure above, and the \field{doorbell_idx} is the
+doorbell capability structure above, and the \field{doorbell_idx} is the
 doorbell number.
 
 \devicenormative{\paragraph}{Doorbell capability}{Device Types / Vhost-user Device Backend / Additional Device Resources over PCI / Doorbell capability}
@@ -260,7 +260,7 @@ \subsubsection{Notification structure layout}\label{sec:Device Types / Vhost-use
 notification interrupts.  If MSI-X is not available, bit 2 of the ISR status
 indicates that a notification occurred.
 
-The notification structure is found using the VIRTIO_PCI_CAP_DOORBELL_CFG
+The notification structure is found using the VIRTIO_PCI_CAP_NOTIFICATION_CFG
 capability.
 
 \begin{lstlisting}
-- 
2.17.1


---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

[virtio-dev] [PATCH v5 03/10] vhost-user: add requirements for the notification capability

[Nikos Dragazis]

The configuration structure for the notification capability has certain
device/driver requirements, similar to those of the MSI-X vector
configuration. Add these requirements to the device spec.

Signed-off-by: Nikos Dragazis <ndragazis@arrikto.com>
---
 virtio-vhost-user.tex | 42 ++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 42 insertions(+)

diff --git a/virtio-vhost-user.tex b/virtio-vhost-user.tex
index 13ecd41..7a8cd9c 100644
--- a/virtio-vhost-user.tex
+++ b/virtio-vhost-user.tex
@@ -275,6 +275,48 @@ \subsubsection{Notification structure layout}\label{sec:Device Types / Vhost-use
 \field{VIRTIO_MSI_NO_VECTOR} to \field{notification_msix_vector} to change the
 MSI-X vector for that notification.
 
+\devicenormative{\paragraph}{Notification capability}{Device Types / Vhost-user Device Backend / Additional Device Resources over PCI / Notification capability}
+
+If MSI-X is available, device MUST support mapping any master queue
+event to any valid vector 0 to MSI-X \field{Table Size}. Here
+\field{Table Size} is the \field{N-1} encoded Table Size stored in the
+Message Control register of the MSI-X capability structure according to
+\hyperref[intro:PCI]{[PCI]}, where \field{N} is the actual MSI-X Table
+Size.
+
+Device MUST support unmapping any master queue event.
+
+The device MUST return vector mapped to a given master queue event,
+(\field{NO_VECTOR} if unmapped) on read of
+\field{notification_msix_vector}. The device MUST have all master queue
+events unmapped upon reset.
+
+Devices SHOULD NOT cause mapping an event to vector to fail unless it is
+impossible for the device to satisfy the mapping request. Devices MUST
+report mapping failures by returning the \field{NO_VECTOR} value when
+the relevant \field{notification_msix_vector} field is read.
+
+\drivernormative{\paragraph}{Notification capability}{Device Types /
+Vhost-user Device Backend / Additional Device Resources over PCI /
+Noification capability}
+
+Driver MUST support device with any MSI-X Table Size 0 to 0x7FF. Driver
+MAY fall back on using INT\#x interrupts for a device which only
+supports one MSI-X vector (MSI-X Table Size = 0).
+
+Driver MAY intepret the \field{Table Size} as a hint from the device for
+the suggested number of MSI-X vectors to use.
+
+Driver MUST NOT attempt to map an event to a vector outside the MSI-X
+Table supported by the device, as reported by \field{Table Size} in the
+MSI-X Capability.
+
+After mapping an event to vector, the driver MUST verify success by
+reading the Vector field value: on success, the previously written value
+is returned, and on failure, \field{NO_VECTOR} is returned. If a mapping
+failure is detected, the driver MAY retry mapping with fewer vectors,
+disable MSI-X or report device failure.
+
 \subsubsection{Shared memory capability}\label{sec:Device Types / Vhost-user Device Backend / Additional Device Resources over PCI / Shared Memory capability}
 
 The shared memory location is found using the VIRTIO_PCI_CAP_SHARED_MEMORY_CFG
-- 
2.17.1


---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

[virtio-dev] [PATCH v5 04/10] vhost-user: update shared memory capability

[Nikos Dragazis]

The virtio vhost-user device has shared memory resources. Therefore, it
uses the VIRTIO_PCI_CAP_SHARED_MEMORY_CFG capability to standardize
those resources through the PCI Configuration Space. This patch
synchronizes the virtio vhost-user device spec with the following shared
memory resources PATCH:

https://lists.oasis-open.org/archives/virtio-dev/201902/msg00142.html

Cc: Dr. David Alan Gilbert <dgilbert@redhat.com>
Signed-off-by: Nikos Dragazis <ndragazis@arrikto.com>
---
 virtio-vhost-user.tex | 17 ++++++++++++++++-
 1 file changed, 16 insertions(+), 1 deletion(-)

diff --git a/virtio-vhost-user.tex b/virtio-vhost-user.tex
index 7a8cd9c..d1305dd 100644
--- a/virtio-vhost-user.tex
+++ b/virtio-vhost-user.tex
@@ -320,7 +320,22 @@ \subsubsection{Notification structure layout}\label{sec:Device Types / Vhost-use
 \subsubsection{Shared memory capability}\label{sec:Device Types / Vhost-user Device Backend / Additional Device Resources over PCI / Shared Memory capability}
 
 The shared memory location is found using the VIRTIO_PCI_CAP_SHARED_MEMORY_CFG
-capability.
+capability. Using the additional \field{offset_hi} and \field{length_hi}
+fields following the SHARED_MEMORY_CFG capability structure, the shared
+memory region's 64-bit BAR offset is calculated as follows:
+
+\begin{lstlisting}
+        offset_hi << 32 + cap.offset
+\end{lstlisting}
+
+and the shared memory region's 64-bit length is calculated as follows:
+
+\begin{lstlisting}
+        length_hi << 32 + cap.length
+\end{lstlisting}
+
+The \field{cap.offset} and \field{cap.length} fields are taken from the
+SHARED_MEMORY_CFG capability structure.
 
 \devicenormative{\paragraph}{Shared Memory capability}{Device Types / Vhost-user Device Backend / Additional Device Resources over PCI / Shared Memory capability}
 The device MUST present exactly one shared memory capability.
-- 
2.17.1


---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

[virtio-dev] [PATCH v5 05/10] vhost-user: add conformance targets and clauses

[Nikos Dragazis]

Signed-off-by: Nikos Dragazis <ndragazis@arrikto.com>
---
 conformance.tex       | 25 ++++++++++++++++++++++---
 virtio-vhost-user.tex |  4 ++--
 2 files changed, 24 insertions(+), 5 deletions(-)

diff --git a/conformance.tex b/conformance.tex
index b6fdec0..dcd2199 100644
--- a/conformance.tex
+++ b/conformance.tex
@@ -15,7 +15,7 @@ \section{Conformance Targets}\label{sec:Conformance / Conformance Targets}
   \begin{itemize}
     \item Clause \ref{sec:Conformance / Driver Conformance}.
     \item One of clauses \ref{sec:Conformance / Driver Conformance / PCI Driver Conformance}, \ref{sec:Conformance / Driver Conformance / MMIO Driver Conformance} or \ref{sec:Conformance / Driver Conformance / Channel I/O Driver Conformance}.
-    \item One of clauses \ref{sec:Conformance / Driver Conformance / Network Driver Conformance}, \ref{sec:Conformance / Driver Conformance / Block Driver Conformance}, \ref{sec:Conformance / Driver Conformance / Console Driver Conformance}, \ref{sec:Conformance / Driver Conformance / Entropy Driver Conformance}, \ref{sec:Conformance / Driver Conformance / Traditional Memory Balloon Driver Conformance}, \ref{sec:Conformance / Driver Conformance / SCSI Host Driver Conformance}, \ref{sec:Conformance / Driver Conformance / Input Driver Conformance}, \ref{sec:Conformance / Driver Conformance / Crypto Driver Conformance}, \ref{sec:Conformance / Driver Conformance / Socket Driver Conformance} or \ref{sec:Conformance / Driver Conformance / IOMMU Driver Conformance}.
+    \item One of clauses \ref{sec:Conformance / Driver Conformance / Network Driver Conformance}, \ref{sec:Conformance / Driver Conformance / Block Driver Conformance}, \ref{sec:Conformance / Driver Conformance / Console Driver Conformance}, \ref{sec:Conformance / Driver Conformance / Entropy Driver Conformance}, \ref{sec:Conformance / Driver Conformance / Traditional Memory Balloon Driver Conformance}, \ref{sec:Conformance / Driver Conformance / SCSI Host Driver Conformance}, \ref{sec:Conformance / Driver Conformance / Input Driver Conformance}, \ref{sec:Conformance / Driver Conformance / Crypto Driver Conformance}, \ref{sec:Conformance / Driver Conformance / Socket Driver Conformance}, \ref{sec:Conformance / Driver Conformance / IOMMU Driver Conformance} or \ref{sec:Conformance / Driver Conformance / Vhost-user Backend Driver Conformance}.
     \item Clause \ref{sec:Conformance / Legacy Interface: Transitional Device and Transitional Driver Conformance}.
   \end{itemize}
 \item[Device] A device MUST conform to four conformance clauses:
@@ -32,8 +32,9 @@ \section{Conformance Targets}\label{sec:Conformance / Conformance Targets}
 \ref{sec:Conformance / Device Conformance / Input Device Conformance}, 
 \ref{sec:Conformance / Device Conformance / Crypto Device Conformance}, 
 \ref{sec:Conformance / Device Conformance / Socket Device Conformance}, 
-\ref{sec:Conformance / Device Conformance / RPMB Device Conformance} or 
-\ref{sec:Conformance / Device Conformance / IOMMU Device Conformance}.
+\ref{sec:Conformance / Device Conformance / RPMB Device Conformance}, 
+\ref{sec:Conformance / Device Conformance / IOMMU Device Conformance} or 
+\ref{sec:Conformance / Device Conformance / Vhost-user Backend Device Conformance}.
     \item Clause \ref{sec:Conformance / Legacy Interface: Transitional Device and Transitional Driver Conformance}.
   \end{itemize}
 \end{description}
@@ -219,6 +220,14 @@ \section{Conformance Targets}\label{sec:Conformance / Conformance Targets}
 \item \ref{drivernormative:Device Types / IOMMU Device / Device operations / PROBE properties / RESVMEM}
 \item \ref{drivernormative:Device Types / IOMMU Device / Device operations / Fault reporting}
 \end{itemize}
+\conformance{\subsection}{Vhost-user Backend Driver Conformance}\label{sec:Conformance / Driver Conformance / Vhost-user Backend Driver Conformance}
+
+A vhost-user backend driver MUST conform to the following normative statements:
+
+\begin{itemize}
+\item \ref{drivernormative:Device Types / Vhost-user Device Backend / Device configuration layout}
+\item \ref{drivernormative:Device Types / Vhost-user Device Backend / Additional Device Resources over PCI / Notification capability}
+\end{itemize}
 
 \conformance{\section}{Device Conformance}\label{sec:Conformance / Device Conformance}
 
@@ -407,6 +416,16 @@ \section{Conformance Targets}\label{sec:Conformance / Conformance Targets}
 \item \ref{devicenormative:Device Types / IOMMU Device / Device operations / PROBE properties / RESVMEM}
 \item \ref{devicenormative:Device Types / IOMMU Device / Device operations / Fault reporting}
 \end{itemize}
+\conformance{\subsection}{Vhost-user Backend Device Conformance}\label{sec:Conformance / Device Conformance / Vhost-user Backend Device Conformance}
+
+A vhost-user backend device MUST conform to the following normative statements:
+
+\begin{itemize}
+\item \ref{devicenormative:Device Types / Vhost-user Device Backend / Device Initialization}
+\item \ref{devicenormative:Device Types / Vhost-user Device Backend / Additional Device Resources over PCI / Doorbell capability}
+\item \ref{devicenormative:Device Types / Vhost-user Device Backend / Additional Device Resources over PCI / Notification capability}
+\item \ref{devicenormative:Device Types / Vhost-user Device Backend / Additional Device Resources over PCI / Shared memory capability}
+\end{itemize}
 
 \conformance{\section}{Legacy Interface: Transitional Device and Transitional Driver Conformance}\label{sec:Conformance / Legacy Interface: Transitional Device and Transitional Driver Conformance}
 A conformant implementation MUST be either transitional or
diff --git a/virtio-vhost-user.tex b/virtio-vhost-user.tex
index d1305dd..678d018 100644
--- a/virtio-vhost-user.tex
+++ b/virtio-vhost-user.tex
@@ -298,7 +298,7 @@ \subsubsection{Notification structure layout}\label{sec:Device Types / Vhost-use
 
 \drivernormative{\paragraph}{Notification capability}{Device Types /
 Vhost-user Device Backend / Additional Device Resources over PCI /
-Noification capability}
+Notification capability}
 
 Driver MUST support device with any MSI-X Table Size 0 to 0x7FF. Driver
 MAY fall back on using INT\#x interrupts for a device which only
@@ -337,7 +337,7 @@ \subsubsection{Shared memory capability}\label{sec:Device Types / Vhost-user Dev
 The \field{cap.offset} and \field{cap.length} fields are taken from the
 SHARED_MEMORY_CFG capability structure.
 
-\devicenormative{\paragraph}{Shared Memory capability}{Device Types / Vhost-user Device Backend / Additional Device Resources over PCI / Shared Memory capability}
+\devicenormative{\paragraph}{Shared memory capability}{Device Types / Vhost-user Device Backend / Additional Device Resources over PCI / Shared memory capability}
 The device MUST present exactly one shared memory capability.
 
 The device MUST locate shared memory in a Memory Space BAR.
-- 
2.17.1


---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

[virtio-dev] [PATCH v5 06/10] vhost-user: change the device id

[Nikos Dragazis]

Change the device id from 28 to 32.

Signed-off-by: Nikos Dragazis <ndragazis@arrikto.com>
---
 content.tex           | 2 +-
 virtio-vhost-user.tex | 2 +-
 2 files changed, 2 insertions(+), 2 deletions(-)

diff --git a/content.tex b/content.tex
index 9f3e86d..8b4aa88 100644
--- a/content.tex
+++ b/content.tex
@@ -2801,7 +2801,7 @@ \chapter{Device Types}\label{sec:Device Types}
 \hline
 31         &   Video decoder device \\
 \hline
-28         &   vhost-user device backend \\
+32         &   vhost-user device backend \\
 \hline
 \end{tabular}
 
diff --git a/virtio-vhost-user.tex b/virtio-vhost-user.tex
index 678d018..29e7323 100644
--- a/virtio-vhost-user.tex
+++ b/virtio-vhost-user.tex
@@ -27,7 +27,7 @@ \section{Vhost-user Device Backend}\label{sec:Device Types / Vhost-user Device B
 because the pre-existing vhost-user wire protocol is used.
 
 \subsection{Device ID}\label{sec:Device Types / Vhost-user Device Backend / Device ID}
-  28
+  32
 
 \subsection{Virtqueues}\label{sec:Device Types / Vhost-user Device Backend / Virtqueues}
 
-- 
2.17.1


---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

[virtio-dev] [PATCH v5 07/10] vhost-user: minor fix on status register

[Nikos Dragazis]

Change the definitions of VIRTIO_VHOST_USER_STATUS_MASTER_UP and
VIRTIO_VHOST_USER_STATUS_SLAVE_UP so that they refer to different bits.
This is necessary because these two flags are orthogonal, i.e., they are
unrelated and so they must be set/cleared individually.

Mention explicitly who sets the VIRTIO_VHOST_USER_STATUS_MASTER_UP
status bit.

Fix typo:

VIRTIO_VHOST_USER_SLAVE_UP -> VIRTIO_VHOST_USER_STATUS_SLAVE_UP

Signed-off-by: Nikos Dragazis <ndragazis@arrikto.com>
---
 virtio-vhost-user.tex | 15 +++++++++------
 1 file changed, 9 insertions(+), 6 deletions(-)

diff --git a/virtio-vhost-user.tex b/virtio-vhost-user.tex
index 29e7323..34f3126 100644
--- a/virtio-vhost-user.tex
+++ b/virtio-vhost-user.tex
@@ -42,13 +42,13 @@ \subsection{Feature bits}\label{sec:Device Types / Vhost-user Device Backend / F
 
 \subsection{Device configuration layout}\label{sec:Device Types / Vhost-user Device Backend / Device configuration layout}
 
-  All fields of this configuration are always available.
+All fields of this configuration are always available.
 
 \begin{lstlisting}
 struct virtio_vhost_user_config {
         le32 status;
-#define VIRTIO_VHOST_USER_STATUS_SLAVE_UP 0
-#define VIRTIO_VHOST_USER_STATUS_MASTER_UP 1
+#define VIRTIO_VHOST_USER_STATUS_SLAVE_UP (1 << 0)
+#define VIRTIO_VHOST_USER_STATUS_MASTER_UP (1 << 1)
         le32 max_vhost_queues;
         u8 uuid[16];
 };
@@ -62,8 +62,11 @@ \subsection{Device configuration layout}\label{sec:Device Types / Vhost-user Dev
     the vhost-user master to connect.  The vhost-user master cannot connect
     unless the driver has set this bit first.
 
-    When the driver clears VIRTIO_VHOST_USER_SLAVE_UP while the vhost-user
-    master is connected, the vhost-user master is disconnected.
+    The device sets VIRTIO_VHOST_USER_STATUS_MASTER_UP to indicate that the
+    vhost-user master is connected.
+
+    When the driver clears VIRTIO_VHOST_USER_STATUS_SLAVE_UP while the
+    vhost-user master is connected, the vhost-user master is disconnected.
 
     When the vhost-user master disconnects, both
     VIRTIO_VHOST_USER_STATUS_SLAVE_UP and VIRTIO_VHOST_USER_STATUS_MASTER_UP
@@ -99,7 +102,7 @@ \subsection{Device configuration layout}\label{sec:Device Types / Vhost-user Dev
 vhost-user slave programs to identify a specific device among many.
 
 The driver SHOULD place at least one buffer in rxq before setting the
-VIRTIO_VHOST_USER_SLAVE_UP bit in the \field{status} configuration field.
+VIRTIO_VHOST_USER_STATUS_SLAVE_UP bit in the \field{status} configuration field.
 
 The driver MUST handle rxq virtqueue notifications that occur before the
 configuration change notification.  It is possible that a vhost-user protocol
-- 
2.17.1


---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

[virtio-dev] [PATCH v5 08/10] vhost-user: remove the extra PCI capabilities

[Nikos Dragazis]

Remove the extra PCI capabilities and use the existing virtio registers
instead. This way we obtain portability across transports.

Add a reference to the "Shared Memory Regions" section.

Add some more minor changes (add missing commas, rename sections,
enhance "Device Initialization" section, adjust normative statements).

Signed-off-by: Nikos Dragazis <ndragazis@arrikto.com>
---
 conformance.tex       |  18 +---
 virtio-vhost-user.tex | 233 ++++++++++++++----------------------------
 2 files changed, 82 insertions(+), 169 deletions(-)

diff --git a/conformance.tex b/conformance.tex
index dcd2199..4f9ce43 100644
--- a/conformance.tex
+++ b/conformance.tex
@@ -32,9 +32,8 @@ \section{Conformance Targets}\label{sec:Conformance / Conformance Targets}
 \ref{sec:Conformance / Device Conformance / Input Device Conformance}, 
 \ref{sec:Conformance / Device Conformance / Crypto Device Conformance}, 
 \ref{sec:Conformance / Device Conformance / Socket Device Conformance}, 
-\ref{sec:Conformance / Device Conformance / RPMB Device Conformance}, 
-\ref{sec:Conformance / Device Conformance / IOMMU Device Conformance} or 
-\ref{sec:Conformance / Device Conformance / Vhost-user Backend Device Conformance}.
+\ref{sec:Conformance / Device Conformance / RPMB Device Conformance} or 
+\ref{sec:Conformance / Device Conformance / IOMMU Device Conformance}.
     \item Clause \ref{sec:Conformance / Legacy Interface: Transitional Device and Transitional Driver Conformance}.
   \end{itemize}
 \end{description}
@@ -226,7 +225,7 @@ \section{Conformance Targets}\label{sec:Conformance / Conformance Targets}
 
 \begin{itemize}
 \item \ref{drivernormative:Device Types / Vhost-user Device Backend / Device configuration layout}
-\item \ref{drivernormative:Device Types / Vhost-user Device Backend / Additional Device Resources over PCI / Notification capability}
+\item \ref{drivernormative:Device Types / Vhost-user Device Backend / Device Initialization}
 \end{itemize}
 
 \conformance{\section}{Device Conformance}\label{sec:Conformance / Device Conformance}
@@ -416,17 +415,6 @@ \section{Conformance Targets}\label{sec:Conformance / Conformance Targets}
 \item \ref{devicenormative:Device Types / IOMMU Device / Device operations / PROBE properties / RESVMEM}
 \item \ref{devicenormative:Device Types / IOMMU Device / Device operations / Fault reporting}
 \end{itemize}
-\conformance{\subsection}{Vhost-user Backend Device Conformance}\label{sec:Conformance / Device Conformance / Vhost-user Backend Device Conformance}
-
-A vhost-user backend device MUST conform to the following normative statements:
-
-\begin{itemize}
-\item \ref{devicenormative:Device Types / Vhost-user Device Backend / Device Initialization}
-\item \ref{devicenormative:Device Types / Vhost-user Device Backend / Additional Device Resources over PCI / Doorbell capability}
-\item \ref{devicenormative:Device Types / Vhost-user Device Backend / Additional Device Resources over PCI / Notification capability}
-\item \ref{devicenormative:Device Types / Vhost-user Device Backend / Additional Device Resources over PCI / Shared memory capability}
-\end{itemize}
-
 \conformance{\section}{Legacy Interface: Transitional Device and Transitional Driver Conformance}\label{sec:Conformance / Legacy Interface: Transitional Device and Transitional Driver Conformance}
 A conformant implementation MUST be either transitional or
 non-transitional, see \ref{intro:Legacy
diff --git a/virtio-vhost-user.tex b/virtio-vhost-user.tex
index 34f3126..a673526 100644
--- a/virtio-vhost-user.tex
+++ b/virtio-vhost-user.tex
@@ -74,13 +74,13 @@ \subsection{Device configuration layout}\label{sec:Device Types / Vhost-user Dev
     setting VIRTIO_VHOST_USER_STATUS_SLAVE_UP again.
 
     A configuration change notification is sent when the device changes
-    this field unless a write to the field by the driver caused the change.
+    this field, unless a write to the field by the driver caused the change.
 
 \item[\field{max_vhost_queues}] is the maximum number of vhost-user queues
     supported by this device.  This field is always greater than 0.
 
 \item[\field{uuid}] is the Universally Unique Identifier (UUID) for this
-    device.  If the device has no UUID then this field contains the nil
+    device.  If the device has no UUID, then this field contains the nil
     UUID (all zeroes).  The UUID allows vhost-user slave programs to identify a
     specific vhost-user device backend among many without relying on bus
     addresses.
@@ -93,7 +93,13 @@ \subsection{Device configuration layout}\label{sec:Device Types / Vhost-user Dev
 
 The driver MUST NOT set undefined bits in the \field{status} configuration field.
 
-\devicenormative{\subsection}{Device Initialization}{Device Types / Vhost-user Device Backend / Device Initialization}
+\subsection{Device Initialization}\label{sec:Device Types / Vhost-user Device Backend / Device Initialization}
+
+The driver initializes the rxq/txq virtqueues and then it sets
+VIRTIO_VHOST_USER_STATUS_SLAVE_UP to the \field{status} field of the device
+configuration structure.
+
+\drivernormative{\subsubsection}{Device Initialization}{Device Types / Vhost-user Device Backend / Device Initialization}
 
 The driver SHOULD check the \field{max_vhost_queues} configuration field to
 determine how many queues the vhost-user slave will be able to support.
@@ -114,7 +120,7 @@ \subsection{Device Operation}\label{sec:Device Types / Vhost-user Device Backend
 
 Device operation consists of operating request queues and response queues.
 
-\subsubsection{Device Operation: Request Queues}\label{sec:Device Types / Vhost-user Device Backend / Device Operation / Device Operation: Request Queues}
+\subsubsection{Device Operation: RX/TX Queues}\label{sec:Device Types / Vhost-user Device Backend / Device Operation / Device Operation: RX/TX Queues}
 
 The driver receives vhost-user protocol messages from the vhost-user master on
 rxq.  The driver sends responses to the vhost-user master on txq.
@@ -133,7 +139,7 @@ \subsubsection{Device Operation: Request Queues}\label{sec:Device Types / Vhost-
 The appropriate size of rxq buffers is at least as large as the largest message
 defined by the \hyperref[intro:Vhost-user Protocol]{Vhost-user Protocol}
 standard version that the driver supports.  If the vhost-user master sends a
-message that is too large for an rxq buffer then DEVICE_NEEDS_RESET is set and
+message that is too large for an rxq buffer, then DEVICE_NEEDS_RESET is set and
 the driver must reset the device.
 
 File descriptor passing is handled differently by the vhost-user device
@@ -141,12 +147,12 @@ \subsubsection{Device Operation: Request Queues}\label{sec:Device Types / Vhost-
 according to the vhost-user protocol, additional device resources become
 available to the driver.
 
-\subsection{Additional Device Resources over PCI}\label{sec:Device Types / Vhost-user Device Backend / Additional Device Resources over PCI}
+\subsection{Additional Device Resources}\label{sec:Device Types / Vhost-user Device Backend / Additional Device Resources}
 
-The vhost-user device backend contains additional device resources beyond
-configuration space and virtqueues.  The nature of these resources is
-transport-specific and therefore only virtio transports that provide these
-resources support the vhost-user device backend.
+The vhost-user device backend contains additional device resources, except for
+those specificied in \ref{sec:Basic Facilities of a Virtio Device}. The
+additional device resources allow the vhost-user master and slave to exchange
+notifications and data through the device.
 
 The following additional resources exist:
 \begin{description}
@@ -155,21 +161,26 @@ \subsection{Additional Device Resources over PCI}\label{sec:Device Types / Vhost
   \item[Shared memory] The vhost-user master gives access to memory that can be mapped by the driver.
 \end{description}
 
-\subsubsection{Doorbell Numbering}\label{sec:Device Types / Vhost-user Device Backend / Additional Device Resources over PCI / Doorbell Numbering}
+\subsubsection{Doorbells}\label{sec:Device Types / Vhost-user Device Backend / Additional Device Resources / Doorbells}
 
-Doorbells are laid out as follows:
+The vhost-user device backend provides all (or part) of the following doorbells:
 
 \begin{description}
 \item[0] Vring call for vhost-user queue 0
 \item[\ldots]
+\item[N-1] Vring call for vhost-user queue N-1
 \item[N] Vring err for vhost-user queue 0
 \item[\ldots]
+\item[2N-1] Vring err for vhost-user queue N-1
 \item[2N] Log
 \end{description}
 
-\subsubsection{Notifications}\label{sec:Device Types / Vhost-user Device Backend / Additional Device Resources over PCI / Notifications}
+where N is the number of the vhost-user virtqueues.
 
-Notifications are laid out as follows:
+\subsubsection{Notifications}\label{sec:Device Types / Vhost-user Device Backend / Additional Device Resources / Notifications}
+
+The vhost-user device backend provides all (or part) of the following
+notifications:
 
 \begin{description}
 \item[0] Vring kick for vhost-user queue 0
@@ -177,28 +188,29 @@ \subsubsection{Notifications}\label{sec:Device Types / Vhost-user Device Backend
 \item[N-1] Vring kick for vhost-user queue N-1
 \end{description}
 
-\subsubsection{Shared Memory Layout}\label{sec:Device Types / Vhost-user Device Backend / Additional Device Resources over PCI / Shared Memory Layout}
+where N is the number of the vhost-user virtqueues.
+
+\subsubsection{Shared Memory}\label{sec:Device Types / Vhost-user Device Backend / Additional Device Resources / Shared Memory}
 
-Shared memory is laid out as follows:
+The vhost-user device backend provides all (or part) of the following shared
+memory regions:
 
 \begin{description}
-\item[0] Vhost memory region 0
-\item[SIZE0] Vhost memory region 1
+\item[0] Vhost-user memory region 0
+\item[1] Vhost-user memory region 1
 \item[\ldots]
-\item[SIZE0 + SIZE1 + \ldots] Log
+\item[M-1] Vhost-user memory region M-1
+\item[M] Log memory region
 \end{description}
 
-The size of vhost memory region 0 is \field{SIZE0}, the size of vhost memory
-region 1 is \field{SIZE1}, and so on.
-
-\subsubsection{Availability of Additional Resources}\label{sec:Device Types / Vhost-user Device Backend / Additional Device Resources over PCI / Availability of Additional Resources}
+\subsubsection{Availability of Additional Resources}\label{sec:Device Types / Vhost-user Device Backend / Additional Device Resources / Availability of Additional Resources}
 
 The following vhost-user protocol messages convey access to additional device
 resources:
 
 \begin{description}
-\item[VHOST_USER_SET_MEM_TABLE] Contents of vhost memory regions are available to the driver in shared memory.  Region contents are laid out in the same order as the vhost memory region list.
-\item[VHOST_USER_SET_LOG_BASE] Contents of the log are available to the driver in shared memory.
+\item[VHOST_USER_SET_MEM_TABLE] Contents of vhost-user memory regions are available to the driver as device memory. Region contents are laid out in the same order as the vhost-user memory region list.
+\item[VHOST_USER_SET_LOG_BASE] Contents of the log memory region are available to the driver as device memory.
 \item[VHOST_USER_SET_LOG_FD] The log doorbell is available to the driver.  Writes to the log doorbell before this message is received produce no effect.
 \item[VHOST_USER_SET_VRING_KICK] The vring kick notification for this queue is available to the driver.  The first notification may occur before the driver has processed this message.
 \item[VHOST_USER_SET_VRING_CALL] The vring call doorbell for this queue is available to the driver.  Writes to the vring call doorbell before this message is received produce no effect.
@@ -206,147 +218,60 @@ \subsubsection{Availability of Additional Resources}\label{sec:Device Types / Vh
 \item[VHOST_USER_SET_SLAVE_REQ_FD] The driver may send vhost-user protocol slave messages on txq.  Buffers put onto txq before this message is received are discarded by the device.
 \end{description}
 
-Additional resources are configured on the virtio PCI transport by the following \field{struct virtio_pci_cap.cfg_type} values:
+\subsubsection{Doorbell layout}\label{sec:Device Types / Vhost-user Device Backend / Additional Device Resources / Doorbell layout}
 
-\begin{lstlisting}
-#define VIRTIO_PCI_CAP_DOORBELL_CFG 6
-#define VIRTIO_PCI_CAP_NOTIFICATION_CFG 7
-#define VIRTIO_PCI_CAP_SHARED_MEMORY_CFG 8
-\end{lstlisting}
+The device MUST reserve 2N+1 virtqueue indices that can be used by the driver to
+send doorbell notifications. The driver can use these indices to send doorbell
+notifications in the same way that it sends available buffer notifications
+\ref{sec:Basic Facilities of a Virtio Device / Notifications} for a virtqueue.
 
-\subsubsection{Doorbell structure layout}\label{sec:Device Types / Vhost-user Device Backend / Additional Device Resources over PCI / Doorbell capability}
+Specifically, the driver can find the locations and send notifications on the
+call and err doorbells by using the following special queue indices:
 
-The doorbell location is found using the VIRTIO_PCI_CAP_DOORBELL_CFG
-capability.  This capability is immediately followed by an additional
-field, like so:
+\begin{itemize}
+\item 2i+2: find call doorbell for vhost-user queue i
+\item 2i+3: find err doorbell for vhost-user queue i
+\end{itemize}
 
-\begin{lstlisting}
-struct virtio_pci_doorbell_cap {
-        struct virtio_pci_cap cap;
-        le32 doorbell_off_multiplier;
-};
-\end{lstlisting}
+where i is the vhost-user queue index (defined by the vhost-user master via the
+vhost-user messages). The indices 0 and 1 are reserved for the device's RX/TX
+virtqueues.
 
-The doorbell address within a BAR is calculated as follows:
-
-\begin{lstlisting}
-        cap.offset + doorbell_idx * doorbell_off_multiplier
-\end{lstlisting}
+For the Log doorbell, the driver can use the special queue index 2(N+1), where N
+is the number of vhost-user queues.
 
-The \field{cap.offset} and \field{doorbell_off_multiplier} are taken from the
-doorbell capability structure above, and the \field{doorbell_idx} is the
-doorbell number.
+\subsubsection{Notification layout}\label{sec:Device Types / Vhost-user Device Backend / Additional Device Resources / Notification layout}
 
-\devicenormative{\paragraph}{Doorbell capability}{Device Types / Vhost-user Device Backend / Additional Device Resources over PCI / Doorbell capability}
-The device MUST present at least one doorbell capability.
+If MSI-X is available, the driver can specify an MSI-X vector for a vhost-user
+queue in the same way that it specifies an MSI-X vector for a virtqueue. The
+driver MUST use the special index 2i+2 to specify an MSI-X vector for the
+vhost-user queue with index i. The vhost-user queue indices are defined by the
+vhost-user master via the vhost-user messages.
 
-The \field{cap.offset} MUST be 2-byte aligned.
+If MSI-X is not available, bit 2 (i.e., the third least significant bit) of the
+ISR status register indicates that a notification occured for one of the
+vhost-user queues.
 
-The device MUST either present \field{doorbell_off_multiplier} as an even power of 2,
-or present \field{doorbell_off_multiplier} as 0.
+\subsubsection{Shared Memory layout}\label{sec:Device Types / Vhost-user Device Backend / Additional Device Resources / Shared Memory layout}
 
-The value \field{cap.length} presented by the device MUST be at least 2
-and MUST be large enough to support doorbell offsets for all supported
-doorbells in all possible configurations.
-
-The value \field{cap.length} presented by the device MUST satisfy:
-\begin{lstlisting}
-cap.length >= num_doorbells * doorbell_off_multiplier + 2
-\end{lstlisting}
-
-The number of doorbells is \field{num_doorbells} and is dependent on the
-device.
-
-\subsubsection{Notification structure layout}\label{sec:Device Types / Vhost-user Device Backend / Additional Device Resources over PCI / Notification capability}
-
-The notification structure allows MSI-X vectors to be configured for
-notification interrupts.  If MSI-X is not available, bit 2 of the ISR status
-indicates that a notification occurred.
-
-The notification structure is found using the VIRTIO_PCI_CAP_NOTIFICATION_CFG
-capability.
-
-\begin{lstlisting}
-struct virtio_pci_notification_cfg {
-        le16 notification_select;              /* read-write */
-        le16 notification_msix_vector;         /* read-write */
-};
-\end{lstlisting}
+The device exports all memory regions reported by the vhost-user master as a
+single shared memory region \ref{sec:Basic Facilities of a Virtio Device /
+Shared Memory Regions}.
 
-The driver indicates which notification is of interest by writing the
-\field{notification_select} field.  The driver then writes the MSI-X vector or
-\field{VIRTIO_MSI_NO_VECTOR} to \field{notification_msix_vector} to change the
-MSI-X vector for that notification.
+The size of this shared memory region MUST be at least as much as the sum of the
+sizes of all the memory regions reported by the vhost-user master.
 
-\devicenormative{\paragraph}{Notification capability}{Device Types / Vhost-user Device Backend / Additional Device Resources over PCI / Notification capability}
+The memory regions MUST be laid out in the same order in which they are reported
+by the master with vhost-user messages.
 
-If MSI-X is available, device MUST support mapping any master queue
-event to any valid vector 0 to MSI-X \field{Table Size}. Here
-\field{Table Size} is the \field{N-1} encoded Table Size stored in the
-Message Control register of the MSI-X capability structure according to
-\hyperref[intro:PCI]{[PCI]}, where \field{N} is the actual MSI-X Table
-Size.
+The offsets in which the memory regions are mapped inside the shared memory
+region MUST be the following:
 
-Device MUST support unmapping any master queue event.
-
-The device MUST return vector mapped to a given master queue event,
-(\field{NO_VECTOR} if unmapped) on read of
-\field{notification_msix_vector}. The device MUST have all master queue
-events unmapped upon reset.
-
-Devices SHOULD NOT cause mapping an event to vector to fail unless it is
-impossible for the device to satisfy the mapping request. Devices MUST
-report mapping failures by returning the \field{NO_VECTOR} value when
-the relevant \field{notification_msix_vector} field is read.
-
-\drivernormative{\paragraph}{Notification capability}{Device Types /
-Vhost-user Device Backend / Additional Device Resources over PCI /
-Notification capability}
-
-Driver MUST support device with any MSI-X Table Size 0 to 0x7FF. Driver
-MAY fall back on using INT\#x interrupts for a device which only
-supports one MSI-X vector (MSI-X Table Size = 0).
-
-Driver MAY intepret the \field{Table Size} as a hint from the device for
-the suggested number of MSI-X vectors to use.
-
-Driver MUST NOT attempt to map an event to a vector outside the MSI-X
-Table supported by the device, as reported by \field{Table Size} in the
-MSI-X Capability.
-
-After mapping an event to vector, the driver MUST verify success by
-reading the Vector field value: on success, the previously written value
-is returned, and on failure, \field{NO_VECTOR} is returned. If a mapping
-failure is detected, the driver MAY retry mapping with fewer vectors,
-disable MSI-X or report device failure.
-
-\subsubsection{Shared memory capability}\label{sec:Device Types / Vhost-user Device Backend / Additional Device Resources over PCI / Shared Memory capability}
-
-The shared memory location is found using the VIRTIO_PCI_CAP_SHARED_MEMORY_CFG
-capability. Using the additional \field{offset_hi} and \field{length_hi}
-fields following the SHARED_MEMORY_CFG capability structure, the shared
-memory region's 64-bit BAR offset is calculated as follows:
-
-\begin{lstlisting}
-        offset_hi << 32 + cap.offset
-\end{lstlisting}
-
-and the shared memory region's 64-bit length is calculated as follows:
-
-\begin{lstlisting}
-        length_hi << 32 + cap.length
-\end{lstlisting}
-
-The \field{cap.offset} and \field{cap.length} fields are taken from the
-SHARED_MEMORY_CFG capability structure.
-
-\devicenormative{\paragraph}{Shared memory capability}{Device Types / Vhost-user Device Backend / Additional Device Resources over PCI / Shared memory capability}
-The device MUST present exactly one shared memory capability.
-
-The device MUST locate shared memory in a Memory Space BAR.
-
-The device SHOULD locate shared memory in a Prefetchable BAR.
-
-The \field{cap.offset} MUST be 4096-byte aligned.
+\begin{description}
+\item[0] Offset for vhost-user memory region 0
+\item[SIZE0] Offset for vhost-user memory region 1
+\item[\ldots]
+\item[SIZE0 + SIZE1 + \ldots] Offset for vhost-user memory region M
+\end{description}
 
-The value \field{cap.length} presented by the device MUST be non-zero and 4096-byte aligned.
+where SIZEi is the size of the vhost-user memory region i.
-- 
2.17.1


---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

[virtio-dev] [PATCH v5 09/10] vhost-user: intercept slave's reply to VHOST_USER_GET_PROTOCOL_FEATURES

[Nikos Dragazis]

Signed-off-by: Nikos Dragazis <ndragazis@arrikto.com>
---
 virtio-vhost-user.tex | 13 +++++++++----
 1 file changed, 9 insertions(+), 4 deletions(-)

diff --git a/virtio-vhost-user.tex b/virtio-vhost-user.tex
index a673526..328baec 100644
--- a/virtio-vhost-user.tex
+++ b/virtio-vhost-user.tex
@@ -142,10 +142,15 @@ \subsubsection{Device Operation: RX/TX Queues}\label{sec:Device Types / Vhost-us
 message that is too large for an rxq buffer, then DEVICE_NEEDS_RESET is set and
 the driver must reset the device.
 
-File descriptor passing is handled differently by the vhost-user device
-backend.  When a message is received that carries one or more file descriptors
-according to the vhost-user protocol, additional device resources become
-available to the driver.
+File descriptor passing is handled differently by the vhost-user device backend.
+When a master-initiated message is received that carries one or more file
+descriptors according to the vhost-user protocol, additional device resources
+become available to the driver.
+
+On the contrary, the slave cannot pass file descriptors to the master. For this
+reason, the vhost-user device backend MUST be intercepting the slave's reply to
+the VHOST_USER_GET_PROTOCOL_FEATURES vhost-user message and clearing these
+feature bits that allow the slave to send messages that pass file descriptors.
 
 \subsection{Additional Device Resources}\label{sec:Device Types / Vhost-user Device Backend / Additional Device Resources}
 
-- 
2.17.1


---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

[virtio-dev] [PATCH v5 10/10] vhost-user: clarify that we are talking about slave-initiated messages

[Nikos Dragazis]

Signed-off-by: Nikos Dragazis <ndragazis@arrikto.com>
---
 virtio-vhost-user.tex | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/virtio-vhost-user.tex b/virtio-vhost-user.tex
index 328baec..6bcc6af 100644
--- a/virtio-vhost-user.tex
+++ b/virtio-vhost-user.tex
@@ -220,7 +220,7 @@ \subsubsection{Availability of Additional Resources}\label{sec:Device Types / Vh
 \item[VHOST_USER_SET_VRING_KICK] The vring kick notification for this queue is available to the driver.  The first notification may occur before the driver has processed this message.
 \item[VHOST_USER_SET_VRING_CALL] The vring call doorbell for this queue is available to the driver.  Writes to the vring call doorbell before this message is received produce no effect.
 \item[VHOST_USER_SET_VRING_ERR] The vring err doorbell for this queue is available to the driver.  Writes to the vring err doorbell before this message is received produce no effect.
-\item[VHOST_USER_SET_SLAVE_REQ_FD] The driver may send vhost-user protocol slave messages on txq.  Buffers put onto txq before this message is received are discarded by the device.
+\item[VHOST_USER_SET_SLAVE_REQ_FD] The driver may send vhost-user protocol slave messages on txq.  Slave-initiated messages put onto txq before this message is received are discarded by the device.
 \end{description}
 
 \subsubsection{Doorbell layout}\label{sec:Device Types / Vhost-user Device Backend / Additional Device Resources / Doorbell layout}
-- 
2.17.1


---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

Re: [virtio-dev] [PATCH v5 00/10] introduce virtio vhost-user backend device type

[Nikos Dragazis]

On 18/5/20 11:37 μ.μ., Nikos Dragazis wrote:

> Hi everyone,
>
> this patch presents an updated version of the virtio device spec for the
> virtio-vhost-user device. The initial RFC implementation can be found
> here: [1]. It’s been a long time since the last revision. I’m sending
> this patch to kick off the discussion again. I'm Cc-ing Stefan and
> Michael who had engaged with this in the past.
>
> In this fifth revision of the patch I have made some minor and major
> changes. The most important is the drop of the device-specific PCI
> capabilities, as Michael suggested [2]. This guarantees portability
> across transports, i.e., the device spec is not bound to the PCI
> transport, as it used to be up to this point.
>
> In more detail, the changes in this revision are outlined below. You can
> find an HTML version here: [3]. Note that I have kept the commits
> separate so that it is easy to track the changes from the previous
> revision [4].
>
> Looking forward to your comments.
>
> Thanks,
> Nikos
>
> v5 changes:
>   * Change device id from #28 to #32
>   * Fix definitions of VIRTIO_VHOST_USER_STATUS_{MASTER,SLAVE}_UP bits
>   * Remove the extra PCI capabilities and use existing virtio registers
>     instead
>   * Intercept slave's reply to VHOST_USER_GET_PROTOCOL_FEATURES
>   * Some more minor fixes (typos, etc.)
>
> v4 changes:
>   * Add comformance targets for the virtio-vhost-user device.
>   * Update the reference link for the vhost-user protocol specification.
>
> v3 changes:
>   * Device Requirements for the Notification Capability: point out the
>     difference between the MSI-X Table Size stored in the Message Control
>     register of the MSI-X capability structure and the actual MSI-X Table
>     Size
>
> v2 changes:
>   * Change device id from #25 to #28
>
>
> [1] https://lists.oasis-open.org/archives/virtio-dev/201801/msg00110.html
> [2] https://lists.oasis-open.org/archives/virtio-dev/201908/msg00014.html
> [3] https://ndragazis.github.io/virtio/vvu/v5/virtio-v1.1-cs01.html#x1-49600014
> [4] https://lists.oasis-open.org/archives/virtio-dev/201906/msg00036.html
>
> Nikos Dragazis (9):
>    vhost-user: minor fixes
>    vhost-user: add requirements for the notification capability
>    vhost-user: update shared memory capability
>    vhost-user: add conformance targets and clauses
>    vhost-user: change the device id
>    vhost-user: minor fix on status register
>    vhost-user: remove the extra PCI capabilities
>    vhost-user: intercept slave's reply to
>      VHOST_USER_GET_PROTOCOL_FEATURES
>    vhost-user: clarify that we are talking about slave-initiated messages
>
> Stefan Hajnoczi (1):
>    vhost-user: add vhost-user device type
>
>   conformance.tex       |  11 +-
>   content.tex           |   3 +
>   introduction.tex      |   1 +
>   virtio-vhost-user.tex | 282 ++++++++++++++++++++++++++++++++++++++++++
>   4 files changed, 295 insertions(+), 2 deletions(-)
>   create mode 100644 virtio-vhost-user.tex
>

Kind reminder on this.

Did anyone have the chance to look at this patchset? Is there anything
that seems off or doesn't make sense? Is there anything else I can do to
help?

Any feedback is more than welcome.

Thanks,
Nikos


---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

Re: [virtio-dev] [PATCH v5 01/10] vhost-user: add vhost-user device type

[Alex Bennée]

Nikos Dragazis <ndragazis@arrikto.com> writes:

> From: Stefan Hajnoczi <stefanha@redhat.com>
>
> The vhost-user device backend facilitates vhost-user device emulation
> through vhost-user protocol exchanges and access to shared memory.
> Software-defined networking, storage, and other I/O appliances can
> provide services through this device.
>
> This device is based on Wei Wang's vhost-pci work.  The virtio
> vhost-user device differs from vhost-pci because it is a single virtio
> device type that exposes the vhost-user protocol instead of a family of
> new virtio device types, one for each vhost-user device type.
>
> This device supports vhost-user slave and vhost-user master
> reconnection.  It also contains a UUID so that vhost-user slave programs
> can identify a specific device among many without using bus addresses.
>
> It is somewhat unconventional for a virtio device because it makes use
> of additional resources called doorbells, notifications, and shared
> memory.  A mapping of these resources to the virtio PCI transport is
> provided.  Other transports, such as CCW may not be able to support
> this device.
>
> Cc: Wei Wang <wei.w.wang@intel.com>
> Cc: Michael S. Tsirkin <mst@redhat.com>
> Cc: Maxime Coquelin <maxime.coquelin@redhat.com>
> Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
> ---
>  content.tex           |   3 +
>  introduction.tex      |   1 +
>  virtio-vhost-user.tex | 292 ++++++++++++++++++++++++++++++++++++++++++
>  3 files changed, 296 insertions(+)
>  create mode 100644 virtio-vhost-user.tex
>
> diff --git a/content.tex b/content.tex
> index 91735e3..9f3e86d 100644
> --- a/content.tex
> +++ b/content.tex
> @@ -2801,6 +2801,8 @@ \chapter{Device Types}\label{sec:Device Types}
>  \hline
>  31         &   Video decoder device \\
>  \hline
> +28         &   vhost-user device backend \\
> +\hline
>  \end{tabular}
>  
>  Some of the devices above are unspecified by this document,
> @@ -6062,6 +6064,7 @@ \subsubsection{Legacy Interface: Framing Requirements}\label{sec:Device
>  \input{virtio-fs.tex}
>  \input{virtio-rpmb.tex}
>  \input{virtio-iommu.tex}
> +\input{virtio-vhost-user.tex}
>  
>  \chapter{Reserved Feature Bits}\label{sec:Reserved Feature Bits}
>  
> diff --git a/introduction.tex b/introduction.tex
> index 33da3ec..9ef0aa7 100644
> --- a/introduction.tex
> +++ b/introduction.tex
> @@ -66,6 +66,7 @@ \section{Normative References}\label{sec:Normative References}
>          \phantomsection\label{intro:eMMC}\textbf{[eMMC]} &
>          eMMC Electrical Standard (5.1), JESD84-B51,
>          \newline\url{http://www.jedec.org/sites/default/files/docs/JESD84-B51.pdf}\\
> +	\phantomsection\label{intro:Vhost-user Protocol}\textbf{[Vhost-user Protocol]} & Vhost-user Protocol, \newline\url{https://git.qemu.org/?p=qemu.git;a=blob_plain;f=docs/interop/vhost-user.rst;hb=HEAD}, and any future revisions\\
>  
>  \end{longtable}
>  
> diff --git a/virtio-vhost-user.tex b/virtio-vhost-user.tex
> new file mode 100644
> index 0000000..ac96dc2
> --- /dev/null
> +++ b/virtio-vhost-user.tex
> @@ -0,0 +1,292 @@
> +\section{Vhost-user Device Backend}\label{sec:Device Types / Vhost-user Device Backend}
> +
> +The vhost-user device backend facilitates vhost-user device emulation through
> +vhost-user protocol exchanges and access to shared memory.  Software-defined
> +networking, storage, and other I/O appliances can provide services through this
> +device.
> +
> +This section relies on definitions from the \hyperref[intro:Vhost-user
> +Protocol]{Vhost-user Protocol}.  Knowledge of the vhost-user protocol is a
> +prerequisite for understanding this device.
> +
> +The \hyperref[intro:Vhost-user Protocol]{Vhost-user Protocol} was originally
> +designed for processes on a single system communicating over UNIX domain
> +sockets.  The virtio vhost-user device backend allows the vhost-user slave to
> +communicate with the vhost-user master over the device instead of a UNIX domain
> +socket.  This allows the slave and master to run on two separate
> systems such

I realise we already have the terms master/slave baked into the
vhost-user spec but perhaps we could find better wording? The vhost
documentation describes thing in terms of who owns the virtqueues (the
drive) and who processes the requests (the device). There may be better
terminology to use.

> +as a virtual machine and a hypervisor.

This implies type-2 setups, depending on where you define the
hypervisor. Could the language be extended: " or device in one virtual
machine with the driver operating in another"?

> +
> +The vhost-user slave program exchanges vhost-user protocol messages with the
> +vhost-user master through this device.  How the device implementation
> +communicates with the vhost-user master is beyond the scope of this
> +specification.  One possible device implementation uses a UNIX domain socket to
> +relay messages to a vhost-user master process running on the same host.
> +
> +Existing vhost-user slave programs that communicate over UNIX domain sockets
> +can support the virtio vhost-user device backend without invasive changes
> +because the pre-existing vhost-user wire protocol is used.
> +
> +\subsection{Device ID}\label{sec:Device Types / Vhost-user Device Backend / Device ID}
> +  28
> +
> +\subsection{Virtqueues}\label{sec:Device Types / Vhost-user Device Backend / Virtqueues}
> +
> +\begin{description}
> +\item[0] rxq (device-to-driver vhost-user protocol messages)
> +\item[1] txq (driver-to-device vhost-user protocol messages)
> +\end{description}
> +
> +\subsection{Feature bits}\label{sec:Device Types / Vhost-user Device Backend / Feature bits}
> +
> +No feature bits are defined at this time.
> +
> +\subsection{Device configuration layout}\label{sec:Device Types / Vhost-user Device Backend / Device configuration layout}
> +
> +  All fields of this configuration are always available.
> +
> +\begin{lstlisting}
> +struct virtio_vhost_user_config {
> +        le32 status;
> +#define VIRTIO_VHOST_USER_STATUS_SLAVE_UP 0
> +#define VIRTIO_VHOST_USER_STATUS_MASTER_UP 1
> +        le32 max_vhost_queues;
> +        u8 uuid[16];
> +};
> +\end{lstlisting}
> +
> +\begin{description}
> +\item[\field{status}] contains the vhost-user operational status.  The default
> +    value of this field is 0.
> +
> +    The driver sets VIRTIO_VHOST_USER_STATUS_SLAVE_UP to indicate readiness for
> +    the vhost-user master to connect.  The vhost-user master cannot connect
> +    unless the driver has set this bit first.

I suspect some deployment diagrams are going to help here. Does this
imply that there is something in userspace connected to the slave kernel
ready to process messages or just that the driver in the kernel is ready
to accept messages?

> +
> +    When the driver clears VIRTIO_VHOST_USER_SLAVE_UP while the vhost-user
> +    master is connected, the vhost-user master is disconnected.

What happens to messages in flight? Do they stay in the queues until
there is a re-connection?

> +
> +    When the vhost-user master disconnects, both
> +    VIRTIO_VHOST_USER_STATUS_SLAVE_UP and VIRTIO_VHOST_USER_STATUS_MASTER_UP
> +    are cleared by the device.  Communication can be restarted by the driver
> +    setting VIRTIO_VHOST_USER_STATUS_SLAVE_UP again.
> +
> +    A configuration change notification is sent when the device changes
> +    this field unless a write to the field by the driver caused the change.
> +
> +\item[\field{max_vhost_queues}] is the maximum number of vhost-user queues
> +    supported by this device.  This field is always greater than 0.
> +
> +\item[\field{uuid}] is the Universally Unique Identifier (UUID) for this
> +    device.  If the device has no UUID then this field contains the nil
> +    UUID (all zeroes).  The UUID allows vhost-user slave programs to identify a
> +    specific vhost-user device backend among many without relying on bus
> +    addresses.
> +\end{description}
> +
> +\drivernormative{\subsubsection}{Device configuration layout}{Device Types / Vhost-user Device Backend / Device configuration layout}
> +
> +The driver MUST NOT write to device configuration fields other than
> +\field{status}.
> +
> +The driver MUST NOT set undefined bits in the \field{status} configuration field.
> +
> +\devicenormative{\subsection}{Device Initialization}{Device Types / Vhost-user Device Backend / Device Initialization}
> +
> +The driver SHOULD check the \field{max_vhost_queues} configuration field to
> +determine how many queues the vhost-user slave will be able to support.
> +
> +The driver SHOULD fetch the \field{uuid} configuration field to allow
> +vhost-user slave programs to identify a specific device among many.
> +
> +The driver SHOULD place at least one buffer in rxq before setting the
> +VIRTIO_VHOST_USER_SLAVE_UP bit in the \field{status} configuration
> field.

This is a buffer for use - not an initial message?

> +
> +The driver MUST handle rxq virtqueue notifications that occur before the
> +configuration change notification.  It is possible that a vhost-user protocol
> +message from the vhost-user master arrives before the driver has seen the
> +configuration change notification for the VIRTIO_VHOST_USER_STATUS_MASTER_UP
> +\field{status} change.
> +
> +\subsection{Device Operation}\label{sec:Device Types / Vhost-user Device Backend / Device Operation}
> +
> +Device operation consists of operating request queues and response queues.
> +
> +\subsubsection{Device Operation: Request Queues}\label{sec:Device Types / Vhost-user Device Backend / Device Operation / Device Operation: Request Queues}
> +
> +The driver receives vhost-user protocol messages from the vhost-user master on
> +rxq.  The driver sends responses to the vhost-user master on txq.
> +
> +The driver sends slave-initiated requests on txq.  The driver receives
> +responses from the vhost-user master on rxq.
> +
> +All virtqueues offer in-order guaranteed delivery semantics for vhost-user
> +protocol messages.
> +
> +Each buffer is a vhost-user protocol message as defined by the
> +\hyperref[intro:Vhost-user Protocol]{Vhost-user Protocol}.  In order to enable
> +cross-endian communication, all message fields are little-endian instead of the
> +native byte order normally used by the protocol.
> +
> +The appropriate size of rxq buffers is at least as large as the largest message
> +defined by the \hyperref[intro:Vhost-user Protocol]{Vhost-user Protocol}
> +standard version that the driver supports.  If the vhost-user master sends a
> +message that is too large for an rxq buffer then DEVICE_NEEDS_RESET is set and
> +the driver must reset the device.
> +
> +File descriptor passing is handled differently by the vhost-user device
> +backend.  When a message is received that carries one or more file descriptors
> +according to the vhost-user protocol, additional device resources become
> +available to the driver.
> +
> +\subsection{Additional Device Resources over PCI}\label{sec:Device Types / Vhost-user Device Backend / Additional Device Resources over PCI}
> +
> +The vhost-user device backend contains additional device resources beyond
> +configuration space and virtqueues.  The nature of these resources is
> +transport-specific and therefore only virtio transports that provide these
> +resources support the vhost-user device backend.
> +
> +The following additional resources exist:
> +\begin{description}
> +  \item[Doorbells] The driver signals the vhost-user master through doorbells.  The signal does not carry any data, it is purely an event.
> +  \item[Notifications] The vhost-user master signals the driver for events besides virtqueue activity and configuration changes by sending notifications.

What is the difference between a doorbell and a notification?

> +  \item[Shared memory] The vhost-user master gives access to memory that can be mapped by the driver.
> +\end{description}
> +
> +\subsubsection{Doorbell Numbering}\label{sec:Device Types / Vhost-user Device Backend / Additional Device Resources over PCI / Doorbell Numbering}
> +
> +Doorbells are laid out as follows:
> +
> +\begin{description}
> +\item[0] Vring call for vhost-user queue 0
> +\item[\ldots]
> +\item[N] Vring err for vhost-user queue 0
> +\item[\ldots]
> +\item[2N] Log
> +\end{description}
> +
> +\subsubsection{Notifications}\label{sec:Device Types / Vhost-user Device Backend / Additional Device Resources over PCI / Notifications}
> +
> +Notifications are laid out as follows:
> +
> +\begin{description}
> +\item[0] Vring kick for vhost-user queue 0
> +\item[\ldots]
> +\item[N-1] Vring kick for vhost-user queue N-1
> +\end{description}
> +
> +\subsubsection{Shared Memory Layout}\label{sec:Device Types /
> Vhost-user Device Backend / Additional Device Resources over PCI /
> Shared Memory Layout}

These subsections seem to get renamed later in the series.

> +
> +Shared memory is laid out as follows:
> +
> +\begin{description}
> +\item[0] Vhost memory region 0
> +\item[SIZE0] Vhost memory region 1
> +\item[\ldots]
> +\item[SIZE0 + SIZE1 + \ldots] Log
> +\end{description}
> +
> +The size of vhost memory region 0 is \field{SIZE0}, the size of vhost memory
> +region 1 is \field{SIZE1}, and so on.
> +
> +\subsubsection{Availability of Additional Resources}\label{sec:Device Types / Vhost-user Device Backend / Additional Device Resources over PCI / Availability of Additional Resources}
> +
> +The following vhost-user protocol messages convey access to additional device
> +resources:
> +
> +\begin{description}
> +\item[VHOST_USER_SET_MEM_TABLE] Contents of vhost memory regions are available to the driver in shared memory.  Region contents are laid out in the same order as the vhost memory region list.
> +\item[VHOST_USER_SET_LOG_BASE] Contents of the log are available to the driver in shared memory.
> +\item[VHOST_USER_SET_LOG_FD] The log doorbell is available to the driver.  Writes to the log doorbell before this message is received produce no effect.
> +\item[VHOST_USER_SET_VRING_KICK] The vring kick notification for this queue is available to the driver.  The first notification may occur before the driver has processed this message.
> +\item[VHOST_USER_SET_VRING_CALL] The vring call doorbell for this queue is available to the driver.  Writes to the vring call doorbell before this message is received produce no effect.
> +\item[VHOST_USER_SET_VRING_ERR] The vring err doorbell for this queue is available to the driver.  Writes to the vring err doorbell before this message is received produce no effect.
> +\item[VHOST_USER_SET_SLAVE_REQ_FD] The driver may send vhost-user protocol slave messages on txq.  Buffers put onto txq before this message is received are discarded by the device.
> +\end{description}
> +
> +Additional resources are configured on the virtio PCI transport by the following \field{struct virtio_pci_cap.cfg_type} values:
> +
> +\begin{lstlisting}
> +#define VIRTIO_PCI_CAP_DOORBELL_CFG 6
> +#define VIRTIO_PCI_CAP_NOTIFICATION_CFG 7
> +#define VIRTIO_PCI_CAP_SHARED_MEMORY_CFG 8
> +\end{lstlisting}
> +
> +\subsubsection{Doorbell structure layout}\label{sec:Device Types / Vhost-user Device Backend / Additional Device Resources over PCI / Doorbell capability}
> +
> +The doorbell location is found using the VIRTIO_PCI_CAP_DOORBELL_CFG
> +capability.  This capability is immediately followed by an additional
> +field, like so:
> +
> +\begin{lstlisting}
> +struct virtio_pci_doorbell_cap {
> +        struct virtio_pci_cap cap;
> +        le32 doorbell_off_multiplier;
> +};
> +\end{lstlisting}

OK stuff is disappearing in later patches. Maybe it shouldn't be
introduced in the first place?

-- 
Alex Bennée

---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

Re: [virtio-dev] [PATCH v5 02/10] vhost-user: minor fixes

[Alex Bennée]

Nikos Dragazis <ndragazis@arrikto.com> writes:

> Signed-off-by: Nikos Dragazis <ndragazis@arrikto.com>
> Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>
> ---
>  virtio-vhost-user.tex | 4 ++--
>  1 file changed, 2 insertions(+), 2 deletions(-)
>
> diff --git a/virtio-vhost-user.tex b/virtio-vhost-user.tex
> index ac96dc2..13ecd41 100644
> --- a/virtio-vhost-user.tex
> +++ b/virtio-vhost-user.tex
> @@ -231,7 +231,7 @@ \subsubsection{Doorbell structure layout}\label{sec:Device Types / Vhost-user De
>  \end{lstlisting}
>  
>  The \field{cap.offset} and \field{doorbell_off_multiplier} are taken from the
> -notification capability structure above, and the \field{doorbell_idx} is the
> +doorbell capability structure above, and the \field{doorbell_idx} is the
>  doorbell number.

I think it would be helpful to add what we mean by doorbell and
notification into the terminology section at the front of the document?

>  
>  \devicenormative{\paragraph}{Doorbell capability}{Device Types / Vhost-user Device Backend / Additional Device Resources over PCI / Doorbell capability}
> @@ -260,7 +260,7 @@ \subsubsection{Notification structure layout}\label{sec:Device Types / Vhost-use
>  notification interrupts.  If MSI-X is not available, bit 2 of the ISR status
>  indicates that a notification occurred.
>  
> -The notification structure is found using the VIRTIO_PCI_CAP_DOORBELL_CFG
> +The notification structure is found using the VIRTIO_PCI_CAP_NOTIFICATION_CFG
>  capability.
>  
>  \begin{lstlisting}


-- 
Alex Bennée

---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

Re: [virtio-dev] [PATCH v5 01/10] vhost-user: add vhost-user device type

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 7365 bytes --]

On Thu, Jul 16, 2020 at 05:45:47PM +0100, Alex Bennée wrote:
> Nikos Dragazis <ndragazis@arrikto.com> writes:
> > diff --git a/virtio-vhost-user.tex b/virtio-vhost-user.tex
> > new file mode 100644
> > index 0000000..ac96dc2
> > --- /dev/null
> > +++ b/virtio-vhost-user.tex
> > @@ -0,0 +1,292 @@
> > +\section{Vhost-user Device Backend}\label{sec:Device Types / Vhost-user Device Backend}
> > +
> > +The vhost-user device backend facilitates vhost-user device emulation through
> > +vhost-user protocol exchanges and access to shared memory.  Software-defined
> > +networking, storage, and other I/O appliances can provide services through this
> > +device.
> > +
> > +This section relies on definitions from the \hyperref[intro:Vhost-user
> > +Protocol]{Vhost-user Protocol}.  Knowledge of the vhost-user protocol is a
> > +prerequisite for understanding this device.
> > +
> > +The \hyperref[intro:Vhost-user Protocol]{Vhost-user Protocol} was originally
> > +designed for processes on a single system communicating over UNIX domain
> > +sockets.  The virtio vhost-user device backend allows the vhost-user slave to
> > +communicate with the vhost-user master over the device instead of a UNIX domain
> > +socket.  This allows the slave and master to run on two separate
> > systems such
> 
> I realise we already have the terms master/slave baked into the
> vhost-user spec but perhaps we could find better wording? The vhost
> documentation describes thing in terms of who owns the virtqueues (the
> drive) and who processes the requests (the device). There may be better
> terminology to use.

"backend" is now commonly used instead of "slave". There is no new term
for "master" yet. I suggest replacing "slave" with "backend" in this
patch.

Using "device"/"driver" would cause confusion here because
both virtio-vhost-user itself and the vhost device being emulated
already use those terms. We need to be able to differentiate between the
vhost-level master/backend concept and the VIRTIO driver/device concept.

> > +as a virtual machine and a hypervisor.
> 
> This implies type-2 setups, depending on where you define the
> hypervisor. Could the language be extended: " or device in one virtual
> machine with the driver operating in another"?

Traditional vhost-user looks like this:

       VM
   virtio-net

       |

      VMM                     vhost-user-net process
vhost-user master  ------------ vhost-user backend

The vhost-user protocol communication is not visible to the VM. It just
sees a virtio-net device.

With virtio-vhost-user it looks like this:

    Driver VM                      Device VM
virtio-net driver             virtio-vhost-user driver

       |                               |

   Driver VMM                      Device VMM
vhost-user master  ------------ vhost-user backend

Here the master is running on the "hypervisor" (it's the Driver VMM) and
the backend is running inside the Device VM.

This spec does not require that the Driver VMM and Device VMM
communicate over the traditional vhost-user UNIX domain socket.

I'm not sure what "device in one virtual machine with the driver
operating in another" means. The main point of the paragraph is that a
VIRTIO device for vhost-user allows the master and backend to run on
separate systems (no longer tied to a UNIX domain socket).

Can you think of a rewording that captures this better?

> > +\begin{description}
> > +\item[\field{status}] contains the vhost-user operational status.  The default
> > +    value of this field is 0.
> > +
> > +    The driver sets VIRTIO_VHOST_USER_STATUS_SLAVE_UP to indicate readiness for
> > +    the vhost-user master to connect.  The vhost-user master cannot connect
> > +    unless the driver has set this bit first.
> 
> I suspect some deployment diagrams are going to help here. Does this
> imply that there is something in userspace connected to the slave kernel
> ready to process messages or just that the driver in the kernel is ready
> to accept messages?

That is beyond the scope of the spec. There is no requirement for
implementing the virtio-vhost-user driver in the kernel or in userspace.

The existing implementation in DPDK/SPDK is a userspace VFIO PCI
implementation. The guest kernel in the Device VM does not touch the
virtio-vhost-user device.

Maybe someone will come up with a use-case where the device emulation
needs to happen in the guest kernel (e.g. the in-kernel SCSI target).

I think a kernel driver for virtio-vhost-user is not very useful since
the actual behavior happens in userspace and involves shared memory.
There is already an API for that VFIO. A userspace library would make it
nicer to use though.

But these are just my thoughts on how the Device VM's software stack
should look. This spec allows all approaches.

I do think it would be helpful to include an diagram/description in the
beginning of the spec with a concrete example of how the Device VM's
virtio-vhost-user software stack could look.

> > +The driver SHOULD place at least one buffer in rxq before setting the
> > +VIRTIO_VHOST_USER_SLAVE_UP bit in the \field{status} configuration
> > field.
> 
> This is a buffer for use - not an initial message?

Yes, an empty buffer. The rxq needs to be populated with buffers so that
messages can be received from the master. Vhost-user messages are
initiated by the master so the backend does not send an initial message.

> > +The following additional resources exist:
> > +\begin{description}
> > +  \item[Doorbells] The driver signals the vhost-user master through doorbells.  The signal does not carry any data, it is purely an event.
> > +  \item[Notifications] The vhost-user master signals the driver for events besides virtqueue activity and configuration changes by sending notifications.
> 
> What is the difference between a doorbell and a notification?

Doorbells allow the driver to signal the device (i.e. device hardware
registers that the driver writes to).

Notifications allow the driver to be signalled by the device (i.e. MSI-X
interrupts that the driver handles).

The more abstract "doorbell" and "notification" terms are used instead
of hardware registers and interrupts because transports other than PCI
may want to support these features too.

> > +\subsubsection{Doorbell structure layout}\label{sec:Device Types / Vhost-user Device Backend / Additional Device Resources over PCI / Doorbell capability}
> > +
> > +The doorbell location is found using the VIRTIO_PCI_CAP_DOORBELL_CFG
> > +capability.  This capability is immediately followed by an additional
> > +field, like so:
> > +
> > +\begin{lstlisting}
> > +struct virtio_pci_doorbell_cap {
> > +        struct virtio_pci_cap cap;
> > +        le32 doorbell_off_multiplier;
> > +};
> > +\end{lstlisting}
> 
> OK stuff is disappearing in later patches. Maybe it shouldn't be
> introduced in the first place?

I think Nikos sent the patches in this order to keep my authorship.

Nikos: Feel free to squash everything into one patch and add your
Signed-off-by. You can also set yourself as the author of the git
commit. As long as my Signed-off-by stays on the patch it indicates I
have given permission to use this text.

Stefan

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: [virtio-dev] [PATCH v5 02/10] vhost-user: minor fixes

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 316 bytes --]

On Mon, May 18, 2020 at 11:37:13PM +0300, Nikos Dragazis wrote:
> Signed-off-by: Nikos Dragazis <ndragazis@arrikto.com>
> Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>
> ---
>  virtio-vhost-user.tex | 4 ++--
>  1 file changed, 2 insertions(+), 2 deletions(-)

Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: [virtio-dev] [PATCH v5 03/10] vhost-user: add requirements for the notification capability

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 2398 bytes --]

On Mon, May 18, 2020 at 11:37:14PM +0300, Nikos Dragazis wrote:
> The configuration structure for the notification capability has certain
> device/driver requirements, similar to those of the MSI-X vector
> configuration. Add these requirements to the device spec.
> 
> Signed-off-by: Nikos Dragazis <ndragazis@arrikto.com>
> ---
>  virtio-vhost-user.tex | 42 ++++++++++++++++++++++++++++++++++++++++++
>  1 file changed, 42 insertions(+)
> 
> diff --git a/virtio-vhost-user.tex b/virtio-vhost-user.tex
> index 13ecd41..7a8cd9c 100644
> --- a/virtio-vhost-user.tex
> +++ b/virtio-vhost-user.tex
> @@ -275,6 +275,48 @@ \subsubsection{Notification structure layout}\label{sec:Device Types / Vhost-use
>  \field{VIRTIO_MSI_NO_VECTOR} to \field{notification_msix_vector} to change the
>  MSI-X vector for that notification.
>  
> +\devicenormative{\paragraph}{Notification capability}{Device Types / Vhost-user Device Backend / Additional Device Resources over PCI / Notification capability}
> +
> +If MSI-X is available, device MUST support mapping any master queue
> +event to any valid vector 0 to MSI-X \field{Table Size}. Here

This section combines two things:
1. vhost-user concepts -> VIRTIO notifications
2. VIRTIO notifications -> VIRTIO PCI transport MSI-X implementation of
   notifications

I think this should be split:
1. The virtio-vhost-user device spec should explain that vhost-user queue
   events are mapped to VIRTIO notifications
2. The PCI transport spec should explain how notifications are mapped to
   MSI-X

This way other VIRTIO device types will be able to use notifications too
without everyone explaining MSI-X specifics. Also it won't tie
virtio-vhost-user to VIRTIO PCI.

> +\field{Table Size} is the \field{N-1} encoded Table Size stored in the
> +Message Control register of the MSI-X capability structure according to
> +\hyperref[intro:PCI]{[PCI]}, where \field{N} is the actual MSI-X Table
> +Size.
> +
> +Device MUST support unmapping any master queue event.
> +
> +The device MUST return vector mapped to a given master queue event,
> +(\field{NO_VECTOR} if unmapped) on read of
> +\field{notification_msix_vector}. The device MUST have all master queue
> +events unmapped upon reset.

"The device MUST unmap all master queue events upon reset" is clearer in
indicating that the device must take this action.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: [virtio-dev] [PATCH v5 04/10] vhost-user: update shared memory capability

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 1511 bytes --]

On Mon, May 18, 2020 at 11:37:15PM +0300, Nikos Dragazis wrote:
> The virtio vhost-user device has shared memory resources. Therefore, it
> uses the VIRTIO_PCI_CAP_SHARED_MEMORY_CFG capability to standardize
> those resources through the PCI Configuration Space. This patch
> synchronizes the virtio vhost-user device spec with the following shared
> memory resources PATCH:
> 
> https://lists.oasis-open.org/archives/virtio-dev/201902/msg00142.html
> 
> Cc: Dr. David Alan Gilbert <dgilbert@redhat.com>
> Signed-off-by: Nikos Dragazis <ndragazis@arrikto.com>
> ---
>  virtio-vhost-user.tex | 17 ++++++++++++++++-
>  1 file changed, 16 insertions(+), 1 deletion(-)
> 
> diff --git a/virtio-vhost-user.tex b/virtio-vhost-user.tex
> index 7a8cd9c..d1305dd 100644
> --- a/virtio-vhost-user.tex
> +++ b/virtio-vhost-user.tex
> @@ -320,7 +320,22 @@ \subsubsection{Notification structure layout}\label{sec:Device Types / Vhost-use
>  \subsubsection{Shared memory capability}\label{sec:Device Types / Vhost-user Device Backend / Additional Device Resources over PCI / Shared Memory capability}
>  
>  The shared memory location is found using the VIRTIO_PCI_CAP_SHARED_MEMORY_CFG

Now that Shared Memory Resources are part of the VIRTIO device model the
virtio-vhost-user spec should not refer to PCI-specific details anymore.
It should just explain which Shared Memory Resources are available on
the device. The mapping to PCI is already covered in the VIRTIO PCI
transport spec section.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: [virtio-dev] [PATCH v5 05/10] vhost-user: add conformance targets and clauses

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 322 bytes --]

On Mon, May 18, 2020 at 11:37:16PM +0300, Nikos Dragazis wrote:
> Signed-off-by: Nikos Dragazis <ndragazis@arrikto.com>
> ---
>  conformance.tex       | 25 ++++++++++++++++++++++---
>  virtio-vhost-user.tex |  4 ++--
>  2 files changed, 24 insertions(+), 5 deletions(-)

Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: [virtio-dev] [PATCH v5 07/10] vhost-user: minor fix on status register

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 741 bytes --]

On Mon, May 18, 2020 at 11:37:18PM +0300, Nikos Dragazis wrote:
> Change the definitions of VIRTIO_VHOST_USER_STATUS_MASTER_UP and
> VIRTIO_VHOST_USER_STATUS_SLAVE_UP so that they refer to different bits.
> This is necessary because these two flags are orthogonal, i.e., they are
> unrelated and so they must be set/cleared individually.
> 
> Mention explicitly who sets the VIRTIO_VHOST_USER_STATUS_MASTER_UP
> status bit.
> 
> Fix typo:
> 
> VIRTIO_VHOST_USER_SLAVE_UP -> VIRTIO_VHOST_USER_STATUS_SLAVE_UP
> 
> Signed-off-by: Nikos Dragazis <ndragazis@arrikto.com>
> ---
>  virtio-vhost-user.tex | 15 +++++++++------
>  1 file changed, 9 insertions(+), 6 deletions(-)

Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: [virtio-dev] [PATCH v5 08/10] vhost-user: remove the extra PCI capabilities

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 1077 bytes --]

On Mon, May 18, 2020 at 11:37:19PM +0300, Nikos Dragazis wrote:
> -Additional resources are configured on the virtio PCI transport by the following \field{struct virtio_pci_cap.cfg_type} values:
> +\subsubsection{Doorbell layout}\label{sec:Device Types / Vhost-user Device Backend / Additional Device Resources / Doorbell layout}
>  
> -\begin{lstlisting}
> -#define VIRTIO_PCI_CAP_DOORBELL_CFG 6
> -#define VIRTIO_PCI_CAP_NOTIFICATION_CFG 7
> -#define VIRTIO_PCI_CAP_SHARED_MEMORY_CFG 8
> -\end{lstlisting}
> +The device MUST reserve 2N+1 virtqueue indices that can be used by the driver to
> +send doorbell notifications. The driver can use these indices to send doorbell
> +notifications in the same way that it sends available buffer notifications
> +\ref{sec:Basic Facilities of a Virtio Device / Notifications} for a virtqueue.

Notifications and doorbells should be first-class VIRTIO device model
concepts, like Shared Memory resources. It should not be necessary to
reserve dummy virtqueues (a hack for getting MSI-X vectors and doorbell
registers).

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: [virtio-dev] [PATCH v5 09/10] vhost-user: intercept slave's reply to VHOST_USER_GET_PROTOCOL_FEATURES

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 1789 bytes --]

On Mon, May 18, 2020 at 11:37:20PM +0300, Nikos Dragazis wrote:
> Signed-off-by: Nikos Dragazis <ndragazis@arrikto.com>
> ---
>  virtio-vhost-user.tex | 13 +++++++++----
>  1 file changed, 9 insertions(+), 4 deletions(-)
> 
> diff --git a/virtio-vhost-user.tex b/virtio-vhost-user.tex
> index a673526..328baec 100644
> --- a/virtio-vhost-user.tex
> +++ b/virtio-vhost-user.tex
> @@ -142,10 +142,15 @@ \subsubsection{Device Operation: RX/TX Queues}\label{sec:Device Types / Vhost-us
>  message that is too large for an rxq buffer, then DEVICE_NEEDS_RESET is set and
>  the driver must reset the device.
>  
> -File descriptor passing is handled differently by the vhost-user device
> -backend.  When a message is received that carries one or more file descriptors
> -according to the vhost-user protocol, additional device resources become
> -available to the driver.
> +File descriptor passing is handled differently by the vhost-user device backend.
> +When a master-initiated message is received that carries one or more file
> +descriptors according to the vhost-user protocol, additional device resources
> +become available to the driver.
> +
> +On the contrary, the slave cannot pass file descriptors to the master. For this
> +reason, the vhost-user device backend MUST be intercepting the slave's reply to
> +the VHOST_USER_GET_PROTOCOL_FEATURES vhost-user message and clearing these
> +feature bits that allow the slave to send messages that pass file descriptors.

Please be specific about which vhost-user protocol bits are not
supported.

Why does the virtio-vhost-user device implementation need to silently
clear those feature bits? Is there a security impact or some other
reason why the VIRTIO spec should specify this behavior?

Stefan

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: [virtio-dev] [PATCH v5 10/10] vhost-user: clarify that we are talking about slave-initiated messages

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 259 bytes --]

On Mon, May 18, 2020 at 11:37:21PM +0300, Nikos Dragazis wrote:
> Signed-off-by: Nikos Dragazis <ndragazis@arrikto.com>
> ---
>  virtio-vhost-user.tex | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)

Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: [virtio-dev] [PATCH v5 01/10] vhost-user: add vhost-user device type

[Alex Bennée]

Stefan Hajnoczi <stefanha@redhat.com> writes:

> On Thu, Jul 16, 2020 at 05:45:47PM +0100, Alex Bennée wrote:
>> Nikos Dragazis <ndragazis@arrikto.com> writes:
<snip>
>> 
>> OK stuff is disappearing in later patches. Maybe it shouldn't be
>> introduced in the first place?
>
> I think Nikos sent the patches in this order to keep my authorship.
>
> Nikos: Feel free to squash everything into one patch and add your
> Signed-off-by. You can also set yourself as the author of the git
> commit. As long as my Signed-off-by stays on the patch it indicates I
> have given permission to use this text.

FWIW I think building up a document in discreet patches is still
worthwhile from an ease of review and following the "story" of a feature
development - just that if blocks of text are deleted later in that
story you might as well squash those deletions into the earlier patch.

-- 
Alex Bennée

---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

Re: [virtio-dev] [PATCH v5 01/10] vhost-user: add vhost-user device type

[Nikos Dragazis]

On 17/7/20 12:26 μ.μ., Stefan Hajnoczi wrote:

> On Thu, Jul 16, 2020 at 05:45:47PM +0100, Alex Bennée wrote:
>> Nikos Dragazis <ndragazis@arrikto.com> writes:
>>> diff --git a/virtio-vhost-user.tex b/virtio-vhost-user.tex
>>> new file mode 100644
>>> index 0000000..ac96dc2
>>> --- /dev/null
>>> +++ b/virtio-vhost-user.tex
>>> @@ -0,0 +1,292 @@
>>> +\section{Vhost-user Device Backend}\label{sec:Device Types / Vhost-user Device Backend}
>>> +
>>> +The vhost-user device backend facilitates vhost-user device emulation through
>>> +vhost-user protocol exchanges and access to shared memory.  Software-defined
>>> +networking, storage, and other I/O appliances can provide services through this
>>> +device.
>>> +
>>> +This section relies on definitions from the \hyperref[intro:Vhost-user
>>> +Protocol]{Vhost-user Protocol}.  Knowledge of the vhost-user protocol is a
>>> +prerequisite for understanding this device.
>>> +
>>> +The \hyperref[intro:Vhost-user Protocol]{Vhost-user Protocol} was originally
>>> +designed for processes on a single system communicating over UNIX domain
>>> +sockets.  The virtio vhost-user device backend allows the vhost-user slave to
>>> +communicate with the vhost-user master over the device instead of a UNIX domain
>>> +socket.  This allows the slave and master to run on two separate
>>> systems such
>> I realise we already have the terms master/slave baked into the
>> vhost-user spec but perhaps we could find better wording? The vhost
>> documentation describes thing in terms of who owns the virtqueues (the
>> drive) and who processes the requests (the device). There may be better
>> terminology to use.
> "backend" is now commonly used instead of "slave". There is no new term
> for "master" yet. I suggest replacing "slave" with "backend" in this
> patch.

Makes sense. Some observations:

1. Since "backend" is used instead of "slave", why "frontend" is not
    used instead of "master"? Also, why does the vhost-user spec use the
    terms"slave" and "backend" interchangeably and doesn't just drop the
    term"slave"completely?

2. The term "driver" cannot replace the term "master" for one more
    reason: they refer todifferent things. The master is the hypervisor,
    not the driver. The driver doesn't even know that the device is
    implemented via the vhost-user mechanism.

>
> Using "device"/"driver" would cause confusion here because
> both virtio-vhost-user itself and the vhost device being emulated
> already use those terms. We need to be able to differentiate between the
> vhost-level master/backend concept and the VIRTIO driver/device concept.
>
>>> +as a virtual machine and a hypervisor.
>> This implies type-2 setups, depending on where you define the
>> hypervisor. Could the language be extended: " or device in one virtual
>> machine with the driver operating in another"?
> Traditional vhost-user looks like this:
>
>         VM
>     virtio-net
>
>         |
>
>        VMM                     vhost-user-net process
> vhost-user master  ------------ vhost-user backend
>
> The vhost-user protocol communication is not visible to the VM. It just
> sees a virtio-net device.
>
> With virtio-vhost-user it looks like this:
>
>      Driver VM                      Device VM
> virtio-net driver             virtio-vhost-user driver
>
>         |                               |
>
>     Driver VMM                      Device VMM
> vhost-user master  ------------ vhost-user backend
>
> Here the master is running on the "hypervisor" (it's the Driver VMM) and
> the backend is running inside the Device VM.
>
> This spec does not require that the Driver VMM and Device VMM
> communicate over the traditional vhost-user UNIX domain socket.
>
> I'm not sure what "device in one virtual machine with the driver
> operating in another" means. The main point of the paragraph is that a
> VIRTIO device for vhost-user allows the master and backend to run on
> separate systems (no longer tied to a UNIX domain socket).
>
> Can you think of a rewording that captures this better?
>
>>> +\begin{description}
>>> +\item[\field{status}] contains the vhost-user operational status.  The default
>>> +    value of this field is 0.
>>> +
>>> +    The driver sets VIRTIO_VHOST_USER_STATUS_SLAVE_UP to indicate readiness for
>>> +    the vhost-user master to connect.  The vhost-user master cannot connect
>>> +    unless the driver has set this bit first.
>> I suspect some deployment diagrams are going to help here. Does this
>> imply that there is something in userspace connected to the slave kernel
>> ready to process messages or just that the driver in the kernel is ready
>> to accept messages?
> That is beyond the scope of the spec. There is no requirement for
> implementing the virtio-vhost-user driver in the kernel or in userspace.
>
> The existing implementation in DPDK/SPDK is a userspace VFIO PCI
> implementation. The guest kernel in the Device VM does not touch the
> virtio-vhost-user device.
>
> Maybe someone will come up with a use-case where the device emulation
> needs to happen in the guest kernel (e.g. the in-kernel SCSI target).
>
> I think a kernel driver for virtio-vhost-user is not very useful since
> the actual behavior happens in userspace and involves shared memory.
> There is already an API for that VFIO. A userspace library would make it
> nicer to use though.
>
> But these are just my thoughts on how the Device VM's software stack
> should look. This spec allows all approaches.
>
> I do think it would be helpful to include an diagram/description in the
> beginning of the spec with a concrete example of how the Device VM's
> virtio-vhost-user software stack could look.

Totally agree. Though virtio does not contain diagrams, in this
particular case, a diagram would really help. Will add one in the next
revision.

>
>>> +The driver SHOULD place at least one buffer in rxq before setting the
>>> +VIRTIO_VHOST_USER_SLAVE_UP bit in the \field{status} configuration
>>> field.
>> This is a buffer for use - not an initial message?
> Yes, an empty buffer. The rxq needs to be populated with buffers so that
> messages can be received from the master. Vhost-user messages are
> initiated by the master so the backend does not send an initial message.
>
>>> +The following additional resources exist:
>>> +\begin{description}
>>> +  \item[Doorbells] The driver signals the vhost-user master through doorbells.  The signal does not carry any data, it is purely an event.
>>> +  \item[Notifications] The vhost-user master signals the driver for events besides virtqueue activity and configuration changes by sending notifications.
>> What is the difference between a doorbell and a notification?
> Doorbells allow the driver to signal the device (i.e. device hardware
> registers that the driver writes to).
>
> Notifications allow the driver to be signalled by the device (i.e. MSI-X
> interrupts that the driver handles).
>
> The more abstract "doorbell" and "notification" terms are used instead
> of hardware registers and interrupts because transports other than PCI
> may want to support these features too.

Let me try to make this a little bit more clear.

A doorbell is a device register that is hooked up to a vhost-user
callfd.  There is one doorbell for each callfd. When the driver (i.e.,
the virtio-vhost-user driver) kicks a doorbell, the device (i.e., the
virtio-vhost-user device) kicks the corresponding callfd. So, doorbells
are the means for the device emulation software (e.g., the
vhost-user-net process in the above diagram), running in the Device VM,
to notify the device driver (e.g., the virtio-net driver in the above
diagram), running in the Driver VM, for I/O completions.

A notification is an interrupt. Notifications are hooked up to
vhost-user kickds. When a kickfd is kicked by the vhost-user master, the
virtio-vhost-user device sends the corresponding interrupt to the
virtio-vhost-user driver. So, notifications are the means for the device
driver, running in the Driver VM, to notify the device emulation
software, running in the Device VM, for I/O submissions.

>
>>> +\subsubsection{Doorbell structure layout}\label{sec:Device Types / Vhost-user Device Backend / Additional Device Resources over PCI / Doorbell capability}
>>> +
>>> +The doorbell location is found using the VIRTIO_PCI_CAP_DOORBELL_CFG
>>> +capability.  This capability is immediately followed by an additional
>>> +field, like so:
>>> +
>>> +\begin{lstlisting}
>>> +struct virtio_pci_doorbell_cap {
>>> +        struct virtio_pci_cap cap;
>>> +        le32 doorbell_off_multiplier;
>>> +};
>>> +\end{lstlisting}
>> OK stuff is disappearing in later patches. Maybe it shouldn't be
>> introduced in the first place?
> I think Nikos sent the patches in this order to keep my authorship.

That's right.

>
> Nikos: Feel free to squash everything into one patch and add your
> Signed-off-by. You can also set yourself as the author of the git
> commit. As long as my Signed-off-by stays on the patch it indicates I
> have given permission to use this text.

OK. Will do.

>
> Stefan

---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

Re: [virtio-dev] [PATCH v5 04/10] vhost-user: update shared memory capability

[Nikos Dragazis]

On 17/7/20 12:36 μ.μ., Stefan Hajnoczi wrote:

> On Mon, May 18, 2020 at 11:37:15PM +0300, Nikos Dragazis wrote:
>> The virtio vhost-user device has shared memory resources. Therefore, it
>> uses the VIRTIO_PCI_CAP_SHARED_MEMORY_CFG capability to standardize
>> those resources through the PCI Configuration Space. This patch
>> synchronizes the virtio vhost-user device spec with the following shared
>> memory resources PATCH:
>>
>> https://lists.oasis-open.org/archives/virtio-dev/201902/msg00142.html
>>
>> Cc: Dr. David Alan Gilbert <dgilbert@redhat.com>
>> Signed-off-by: Nikos Dragazis <ndragazis@arrikto.com>
>> ---
>>   virtio-vhost-user.tex | 17 ++++++++++++++++-
>>   1 file changed, 16 insertions(+), 1 deletion(-)
>>
>> diff --git a/virtio-vhost-user.tex b/virtio-vhost-user.tex
>> index 7a8cd9c..d1305dd 100644
>> --- a/virtio-vhost-user.tex
>> +++ b/virtio-vhost-user.tex
>> @@ -320,7 +320,22 @@ \subsubsection{Notification structure layout}\label{sec:Device Types / Vhost-use
>>   \subsubsection{Shared memory capability}\label{sec:Device Types / Vhost-user Device Backend / Additional Device Resources over PCI / Shared Memory capability}
>>   
>>   The shared memory location is found using the VIRTIO_PCI_CAP_SHARED_MEMORY_CFG
> Now that Shared Memory Resources are part of the VIRTIO device model the
> virtio-vhost-user spec should not refer to PCI-specific details anymore.
> It should just explain which Shared Memory Resources are available on
> the device. The mapping to PCI is already covered in the VIRTIO PCI
> transport spec section.

Right. Actually, I have removed the PCI-specific details in the commit
08/10. I see that having non-incremental commits creates confusion, so
I will not be doing this anymore.

---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

Re: [virtio-dev] [PATCH v5 08/10] vhost-user: remove the extra PCI capabilities

[Nikos Dragazis]

On 17/7/20 12:48 μ.μ., Stefan Hajnoczi wrote:

> On Mon, May 18, 2020 at 11:37:19PM +0300, Nikos Dragazis wrote:
>> -Additional resources are configured on the virtio PCI transport by the following \field{struct virtio_pci_cap.cfg_type} values:
>> +\subsubsection{Doorbell layout}\label{sec:Device Types / Vhost-user Device Backend / Additional Device Resources / Doorbell layout}
>>   
>> -\begin{lstlisting}
>> -#define VIRTIO_PCI_CAP_DOORBELL_CFG 6
>> -#define VIRTIO_PCI_CAP_NOTIFICATION_CFG 7
>> -#define VIRTIO_PCI_CAP_SHARED_MEMORY_CFG 8
>> -\end{lstlisting}
>> +The device MUST reserve 2N+1 virtqueue indices that can be used by the driver to
>> +send doorbell notifications. The driver can use these indices to send doorbell
>> +notifications in the same way that it sends available buffer notifications
>> +\ref{sec:Basic Facilities of a Virtio Device / Notifications} for a virtqueue.
> Notifications and doorbells should be first-class VIRTIO device model
> concepts, like Shared Memory resources. It should not be necessary to
> reserve dummy virtqueues (a hack for getting MSI-X vectors and doorbell
> registers).

Well, it's not exactly equivalent to the shared memory resource
because, in case of doorbells and notifications, the functionality is
already there. But I see your point. It is a design issue. We could
definitely standardize the doorbell and notification resources, but I
don't see why this solution is preferable. Do you have something in
mind?

---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

Re: [virtio-dev] [PATCH v5 09/10] vhost-user: intercept slave's reply to VHOST_USER_GET_PROTOCOL_FEATURES

[Nikos Dragazis]

On 17/7/20 12:57 μ.μ., Stefan Hajnoczi wrote:

> On Mon, May 18, 2020 at 11:37:20PM +0300, Nikos Dragazis wrote:
>> Signed-off-by: Nikos Dragazis <ndragazis@arrikto.com>
>> ---
>>   virtio-vhost-user.tex | 13 +++++++++----
>>   1 file changed, 9 insertions(+), 4 deletions(-)
>>
>> diff --git a/virtio-vhost-user.tex b/virtio-vhost-user.tex
>> index a673526..328baec 100644
>> --- a/virtio-vhost-user.tex
>> +++ b/virtio-vhost-user.tex
>> @@ -142,10 +142,15 @@ \subsubsection{Device Operation: RX/TX Queues}\label{sec:Device Types / Vhost-us
>>   message that is too large for an rxq buffer, then DEVICE_NEEDS_RESET is set and
>>   the driver must reset the device.
>>   
>> -File descriptor passing is handled differently by the vhost-user device
>> -backend.  When a message is received that carries one or more file descriptors
>> -according to the vhost-user protocol, additional device resources become
>> -available to the driver.
>> +File descriptor passing is handled differently by the vhost-user device backend.
>> +When a master-initiated message is received that carries one or more file
>> +descriptors according to the vhost-user protocol, additional device resources
>> +become available to the driver.
>> +
>> +On the contrary, the slave cannot pass file descriptors to the master. For this
>> +reason, the vhost-user device backend MUST be intercepting the slave's reply to
>> +the VHOST_USER_GET_PROTOCOL_FEATURES vhost-user message and clearing these
>> +feature bits that allow the slave to send messages that pass file descriptors.
> Please be specific about which vhost-user protocol bits are not
> supported.

Currently, this goes for the VHOST_USER_PROTOCOL_F_SLAVE_SEND_FD and
VHOST_USER_PROTOCOL_F_PAGEFAULT feature bits.

>
> Why does the virtio-vhost-user device implementation need to silently
> clear those feature bits? Is there a security impact or some other
> reason why the VIRTIO spec should specify this behavior?

In our setup, the slave cannot pass file descriptors to the master.
The easiest way to enforce this restriction is during feature
negotiation. The virtio-vhost-user device will mask the unsupported
feature bits and, therefore, the master will not enable these feature
bits on the slave.

>
> Stefan

---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

Re: [virtio-dev] [PATCH v5 03/10] vhost-user: add requirements for the notification capability

[Nikos Dragazis]

On 17/7/20 12:34 μ.μ., Stefan Hajnoczi wrote:

> On Mon, May 18, 2020 at 11:37:14PM +0300, Nikos Dragazis wrote:
>> The configuration structure for the notification capability has certain
>> device/driver requirements, similar to those of the MSI-X vector
>> configuration. Add these requirements to the device spec.
>>
>> Signed-off-by: Nikos Dragazis <ndragazis@arrikto.com>
>> ---
>>   virtio-vhost-user.tex | 42 ++++++++++++++++++++++++++++++++++++++++++
>>   1 file changed, 42 insertions(+)
>>
>> diff --git a/virtio-vhost-user.tex b/virtio-vhost-user.tex
>> index 13ecd41..7a8cd9c 100644
>> --- a/virtio-vhost-user.tex
>> +++ b/virtio-vhost-user.tex
>> @@ -275,6 +275,48 @@ \subsubsection{Notification structure layout}\label{sec:Device Types / Vhost-use
>>   \field{VIRTIO_MSI_NO_VECTOR} to \field{notification_msix_vector} to change the
>>   MSI-X vector for that notification.
>>   
>> +\devicenormative{\paragraph}{Notification capability}{Device Types / Vhost-user Device Backend / Additional Device Resources over PCI / Notification capability}
>> +
>> +If MSI-X is available, device MUST support mapping any master queue
>> +event to any valid vector 0 to MSI-X \field{Table Size}. Here
> This section combines two things:
> 1. vhost-user concepts -> VIRTIO notifications
> 2. VIRTIO notifications -> VIRTIO PCI transport MSI-X implementation of
>     notifications
>
> I think this should be split:
> 1. The virtio-vhost-user device spec should explain that vhost-user queue
>     events are mapped to VIRTIO notifications
> 2. The PCI transport spec should explain how notifications are mapped to
>     MSI-X
>
> This way other VIRTIO device types will be able to use notifications too
> without everyone explaining MSI-X specifics. Also it won't tie
> virtio-vhost-user to VIRTIO PCI.

If I understand correctly, you are suggesting that we should standardize
the notification resource, similarly to the shared memory resource, so
that any device can use it. Is that so?

>
>> +\field{Table Size} is the \field{N-1} encoded Table Size stored in the
>> +Message Control register of the MSI-X capability structure according to
>> +\hyperref[intro:PCI]{[PCI]}, where \field{N} is the actual MSI-X Table
>> +Size.
>> +
>> +Device MUST support unmapping any master queue event.
>> +
>> +The device MUST return vector mapped to a given master queue event,
>> +(\field{NO_VECTOR} if unmapped) on read of
>> +\field{notification_msix_vector}. The device MUST have all master queue
>> +events unmapped upon reset.
> "The device MUST unmap all master queue events upon reset" is clearer in
> indicating that the device must take this action.

ACK


---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

Re: [virtio-dev] [PATCH v5 01/10] vhost-user: add vhost-user device type

[Nikos Dragazis]

On 16/7/20 7:45 μ.μ., Alex Bennée wrote:

> Nikos Dragazis <ndragazis@arrikto.com> writes:
>
>> +
>> +    When the driver clears VIRTIO_VHOST_USER_SLAVE_UP while the vhost-user
>> +    master is connected, the vhost-user master is disconnected.
> What happens to messages in flight? Do they stay in the queues until
> there is a re-connection?

After the reconnection, the master-slave communication starts from
scratch, i.e., there is no state. At least, this is what I can infer
from the vhost-user spec.  So, in-flight vhost-user messages have to be
discarded. We should mention it explicitly in the spec.

---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

Re: [virtio-dev] [PATCH v5 01/10] vhost-user: add vhost-user device type

[Alex Bennée]

Nikos Dragazis <ndragazis@arrikto.com> writes:

> On 17/7/20 12:26 μ.μ., Stefan Hajnoczi wrote:
>
>> On Thu, Jul 16, 2020 at 05:45:47PM +0100, Alex Bennée wrote:
>>> Nikos Dragazis <ndragazis@arrikto.com> writes:
>>>> diff --git a/virtio-vhost-user.tex b/virtio-vhost-user.tex
>>>> new file mode 100644
>>>> index 0000000..ac96dc2
>>>> --- /dev/null
>>>> +++ b/virtio-vhost-user.tex
>>>> @@ -0,0 +1,292 @@
>>>> +\section{Vhost-user Device Backend}\label{sec:Device Types / Vhost-user Device Backend}
>>>> +
>>>> +The vhost-user device backend facilitates vhost-user device emulation through
>>>> +vhost-user protocol exchanges and access to shared memory.  Software-defined
>>>> +networking, storage, and other I/O appliances can provide services through this
>>>> +device.
>>>> +
>>>> +This section relies on definitions from the \hyperref[intro:Vhost-user
>>>> +Protocol]{Vhost-user Protocol}.  Knowledge of the vhost-user protocol is a
>>>> +prerequisite for understanding this device.
>>>> +
>>>> +The \hyperref[intro:Vhost-user Protocol]{Vhost-user Protocol} was originally
>>>> +designed for processes on a single system communicating over UNIX domain
>>>> +sockets.  The virtio vhost-user device backend allows the vhost-user slave to
>>>> +communicate with the vhost-user master over the device instead of a UNIX domain
>>>> +socket.  This allows the slave and master to run on two separate
>>>> systems such
>>> I realise we already have the terms master/slave baked into the
>>> vhost-user spec but perhaps we could find better wording? The vhost
>>> documentation describes thing in terms of who owns the virtqueues (the
>>> drive) and who processes the requests (the device). There may be better
>>> terminology to use.
>> "backend" is now commonly used instead of "slave". There is no new term
>> for "master" yet. I suggest replacing "slave" with "backend" in this
>> patch.
>
> Makes sense. Some observations:
>
> 1. Since "backend" is used instead of "slave", why "frontend" is not
>     used instead of "master"? Also, why does the vhost-user spec use the
>     terms"slave" and "backend" interchangeably and doesn't just drop the
>     term"slave"completely?
>
> 2. The term "driver" cannot replace the term "master" for one more
>     reason: they refer todifferent things. The master is the hypervisor,
>     not the driver. The driver doesn't even know that the device is
>     implemented via the vhost-user mechanism.
>
>>
>> Using "device"/"driver" would cause confusion here because
>> both virtio-vhost-user itself and the vhost device being emulated
>> already use those terms. We need to be able to differentiate between the
>> vhost-level master/backend concept and the VIRTIO driver/device concept.
>>
>>>> +as a virtual machine and a hypervisor.
>>> This implies type-2 setups, depending on where you define the
>>> hypervisor. Could the language be extended: " or device in one virtual
>>> machine with the driver operating in another"?
>> Traditional vhost-user looks like this:
>>
>>         VM
>>     virtio-net
>>
>>         |
>>
>>        VMM                     vhost-user-net process
>> vhost-user master  ------------ vhost-user backend
>>
>> The vhost-user protocol communication is not visible to the VM. It just
>> sees a virtio-net device.
>>
>> With virtio-vhost-user it looks like this:
>>
>>      Driver VM                      Device VM
>> virtio-net driver             virtio-vhost-user driver
>>
>>         |                               |
>>
>>     Driver VMM                      Device VMM
>> vhost-user master  ------------ vhost-user backend
>>
>> Here the master is running on the "hypervisor" (it's the Driver VMM) and
>> the backend is running inside the Device VM.
>>
>> This spec does not require that the Driver VMM and Device VMM
>> communicate over the traditional vhost-user UNIX domain socket.
>>
>> I'm not sure what "device in one virtual machine with the driver
>> operating in another" means. The main point of the paragraph is that a
>> VIRTIO device for vhost-user allows the master and backend to run on
>> separate systems (no longer tied to a UNIX domain socket).
>>
>> Can you think of a rewording that captures this better?
>>
>>>> +\begin{description}
>>>> +\item[\field{status}] contains the vhost-user operational status.  The default
>>>> +    value of this field is 0.
>>>> +
>>>> +    The driver sets VIRTIO_VHOST_USER_STATUS_SLAVE_UP to indicate readiness for
>>>> +    the vhost-user master to connect.  The vhost-user master cannot connect
>>>> +    unless the driver has set this bit first.
>>> I suspect some deployment diagrams are going to help here. Does this
>>> imply that there is something in userspace connected to the slave kernel
>>> ready to process messages or just that the driver in the kernel is ready
>>> to accept messages?
>> That is beyond the scope of the spec. There is no requirement for
>> implementing the virtio-vhost-user driver in the kernel or in userspace.
>>
>> The existing implementation in DPDK/SPDK is a userspace VFIO PCI
>> implementation. The guest kernel in the Device VM does not touch the
>> virtio-vhost-user device.
>>
>> Maybe someone will come up with a use-case where the device emulation
>> needs to happen in the guest kernel (e.g. the in-kernel SCSI target).
>>
>> I think a kernel driver for virtio-vhost-user is not very useful since
>> the actual behavior happens in userspace and involves shared memory.
>> There is already an API for that VFIO. A userspace library would make it
>> nicer to use though.
>>
>> But these are just my thoughts on how the Device VM's software stack
>> should look. This spec allows all approaches.
>>
>> I do think it would be helpful to include an diagram/description in the
>> beginning of the spec with a concrete example of how the Device VM's
>> virtio-vhost-user software stack could look.
>
> Totally agree. Though virtio does not contain diagrams, in this
> particular case, a diagram would really help. Will add one in the next
> revision.
>
>>
>>>> +The driver SHOULD place at least one buffer in rxq before setting the
>>>> +VIRTIO_VHOST_USER_SLAVE_UP bit in the \field{status} configuration
>>>> field.
>>> This is a buffer for use - not an initial message?
>> Yes, an empty buffer. The rxq needs to be populated with buffers so that
>> messages can be received from the master. Vhost-user messages are
>> initiated by the master so the backend does not send an initial message.
>>
>>>> +The following additional resources exist:
>>>> +\begin{description}
>>>> +  \item[Doorbells] The driver signals the vhost-user master through doorbells.  The signal does not carry any data, it is purely an event.
>>>> +  \item[Notifications] The vhost-user master signals the driver for events besides virtqueue activity and configuration changes by sending notifications.
>>> What is the difference between a doorbell and a notification?
>> Doorbells allow the driver to signal the device (i.e. device hardware
>> registers that the driver writes to).
>>
>> Notifications allow the driver to be signalled by the device (i.e. MSI-X
>> interrupts that the driver handles).
>>
>> The more abstract "doorbell" and "notification" terms are used instead
>> of hardware registers and interrupts because transports other than PCI
>> may want to support these features too.
>
> Let me try to make this a little bit more clear.
>
> A doorbell is a device register that is hooked up to a vhost-user
> callfd.  There is one doorbell for each callfd. When the driver (i.e.,
> the virtio-vhost-user driver) kicks a doorbell, the device (i.e., the
> virtio-vhost-user device) kicks the corresponding callfd. So, doorbells
> are the means for the device emulation software (e.g., the
> vhost-user-net process in the above diagram), running in the Device VM,
> to notify the device driver (e.g., the virtio-net driver in the above
> diagram), running in the Driver VM, for I/O completions.

OK - I think I follow that although it's tricky because a
virtio-vhost-user driver is not like a virtio-device driver - being at
opposite ends of each other. Another good argument for having clear
terminology descriptions. I'll try and come up with some words and a
patch for the spec so we can make it clearer.

> A notification is an interrupt. Notifications are hooked up to
> vhost-user kickds. When a kickfd is kicked by the vhost-user master, the
> virtio-vhost-user device sends the corresponding interrupt to the
> virtio-vhost-user driver. So, notifications are the means for the device
> driver, running in the Driver VM, to notify the device emulation
> software, running in the Device VM, for I/O submissions.

Isn't that the wrong way around, surely a device driver sees a IRQ as
it's notification? Or are we talking about notifications going both ways
here?

-- 
Alex Bennée

---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

Re: [virtio-dev] [PATCH v5 05/10] vhost-user: add conformance targets and clauses

[Alex Bennée]

Nikos Dragazis <ndragazis@arrikto.com> writes:

> Signed-off-by: Nikos Dragazis <ndragazis@arrikto.com>

Reviewed-by: Alex Bennée <alex.bennee@linaro.org>
<snip>

-- 
Alex Bennée

---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

Re: [virtio-dev] [PATCH v5 06/10] vhost-user: change the device id

[Alex Bennée]

Nikos Dragazis <ndragazis@arrikto.com> writes:

> Change the device id from 28 to 32.

Yeah this can squash merge with it's original introduction.

-- 
Alex Bennée

---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

Re: [virtio-dev] [PATCH v5 01/10] vhost-user: add vhost-user device type

[Nikos Dragazis]

On 20/7/20 12:37 μ.μ., Alex Bennée wrote:

> Nikos Dragazis <ndragazis@arrikto.com> writes:
>
>> On 17/7/20 12:26 μ.μ., Stefan Hajnoczi wrote:
>>
>>> On Thu, Jul 16, 2020 at 05:45:47PM +0100, Alex Bennée wrote:
>>>> Nikos Dragazis <ndragazis@arrikto.com> writes:
>>>>> diff --git a/virtio-vhost-user.tex b/virtio-vhost-user.tex
>>>>> new file mode 100644
>>>>> index 0000000..ac96dc2
>>>>> --- /dev/null
>>>>> +++ b/virtio-vhost-user.tex
>>>>> @@ -0,0 +1,292 @@
>>>>> +\section{Vhost-user Device Backend}\label{sec:Device Types / Vhost-user Device Backend}
>>>>> +
>>>>> +The vhost-user device backend facilitates vhost-user device emulation through
>>>>> +vhost-user protocol exchanges and access to shared memory.  Software-defined
>>>>> +networking, storage, and other I/O appliances can provide services through this
>>>>> +device.
>>>>> +
>>>>> +This section relies on definitions from the \hyperref[intro:Vhost-user
>>>>> +Protocol]{Vhost-user Protocol}.  Knowledge of the vhost-user protocol is a
>>>>> +prerequisite for understanding this device.
>>>>> +
>>>>> +The \hyperref[intro:Vhost-user Protocol]{Vhost-user Protocol} was originally
>>>>> +designed for processes on a single system communicating over UNIX domain
>>>>> +sockets.  The virtio vhost-user device backend allows the vhost-user slave to
>>>>> +communicate with the vhost-user master over the device instead of a UNIX domain
>>>>> +socket.  This allows the slave and master to run on two separate
>>>>> systems such
>>>> I realise we already have the terms master/slave baked into the
>>>> vhost-user spec but perhaps we could find better wording? The vhost
>>>> documentation describes thing in terms of who owns the virtqueues (the
>>>> drive) and who processes the requests (the device). There may be better
>>>> terminology to use.
>>> "backend" is now commonly used instead of "slave". There is no new term
>>> for "master" yet. I suggest replacing "slave" with "backend" in this
>>> patch.
>> Makes sense. Some observations:
>>
>> 1. Since "backend" is used instead of "slave", why "frontend" is not
>>      used instead of "master"? Also, why does the vhost-user spec use the
>>      terms"slave" and "backend" interchangeably and doesn't just drop the
>>      term"slave"completely?
>>
>> 2. The term "driver" cannot replace the term "master" for one more
>>      reason: they refer todifferent things. The master is the hypervisor,
>>      not the driver. The driver doesn't even know that the device is
>>      implemented via the vhost-user mechanism.
>>
>>> Using "device"/"driver" would cause confusion here because
>>> both virtio-vhost-user itself and the vhost device being emulated
>>> already use those terms. We need to be able to differentiate between the
>>> vhost-level master/backend concept and the VIRTIO driver/device concept.
>>>
>>>>> +as a virtual machine and a hypervisor.
>>>> This implies type-2 setups, depending on where you define the
>>>> hypervisor. Could the language be extended: " or device in one virtual
>>>> machine with the driver operating in another"?
>>> Traditional vhost-user looks like this:
>>>
>>>          VM
>>>      virtio-net
>>>
>>>          |
>>>
>>>         VMM                     vhost-user-net process
>>> vhost-user master  ------------ vhost-user backend
>>>
>>> The vhost-user protocol communication is not visible to the VM. It just
>>> sees a virtio-net device.
>>>
>>> With virtio-vhost-user it looks like this:
>>>
>>>       Driver VM                      Device VM
>>> virtio-net driver             virtio-vhost-user driver
>>>
>>>          |                               |
>>>
>>>      Driver VMM                      Device VMM
>>> vhost-user master  ------------ vhost-user backend
>>>
>>> Here the master is running on the "hypervisor" (it's the Driver VMM) and
>>> the backend is running inside the Device VM.
>>>
>>> This spec does not require that the Driver VMM and Device VMM
>>> communicate over the traditional vhost-user UNIX domain socket.
>>>
>>> I'm not sure what "device in one virtual machine with the driver
>>> operating in another" means. The main point of the paragraph is that a
>>> VIRTIO device for vhost-user allows the master and backend to run on
>>> separate systems (no longer tied to a UNIX domain socket).
>>>
>>> Can you think of a rewording that captures this better?
>>>
>>>>> +\begin{description}
>>>>> +\item[\field{status}] contains the vhost-user operational status.  The default
>>>>> +    value of this field is 0.
>>>>> +
>>>>> +    The driver sets VIRTIO_VHOST_USER_STATUS_SLAVE_UP to indicate readiness for
>>>>> +    the vhost-user master to connect.  The vhost-user master cannot connect
>>>>> +    unless the driver has set this bit first.
>>>> I suspect some deployment diagrams are going to help here. Does this
>>>> imply that there is something in userspace connected to the slave kernel
>>>> ready to process messages or just that the driver in the kernel is ready
>>>> to accept messages?
>>> That is beyond the scope of the spec. There is no requirement for
>>> implementing the virtio-vhost-user driver in the kernel or in userspace.
>>>
>>> The existing implementation in DPDK/SPDK is a userspace VFIO PCI
>>> implementation. The guest kernel in the Device VM does not touch the
>>> virtio-vhost-user device.
>>>
>>> Maybe someone will come up with a use-case where the device emulation
>>> needs to happen in the guest kernel (e.g. the in-kernel SCSI target).
>>>
>>> I think a kernel driver for virtio-vhost-user is not very useful since
>>> the actual behavior happens in userspace and involves shared memory.
>>> There is already an API for that VFIO. A userspace library would make it
>>> nicer to use though.
>>>
>>> But these are just my thoughts on how the Device VM's software stack
>>> should look. This spec allows all approaches.
>>>
>>> I do think it would be helpful to include an diagram/description in the
>>> beginning of the spec with a concrete example of how the Device VM's
>>> virtio-vhost-user software stack could look.
>> Totally agree. Though virtio does not contain diagrams, in this
>> particular case, a diagram would really help. Will add one in the next
>> revision.
>>
>>>>> +The driver SHOULD place at least one buffer in rxq before setting the
>>>>> +VIRTIO_VHOST_USER_SLAVE_UP bit in the \field{status} configuration
>>>>> field.
>>>> This is a buffer for use - not an initial message?
>>> Yes, an empty buffer. The rxq needs to be populated with buffers so that
>>> messages can be received from the master. Vhost-user messages are
>>> initiated by the master so the backend does not send an initial message.
>>>
>>>>> +The following additional resources exist:
>>>>> +\begin{description}
>>>>> +  \item[Doorbells] The driver signals the vhost-user master through doorbells.  The signal does not carry any data, it is purely an event.
>>>>> +  \item[Notifications] The vhost-user master signals the driver for events besides virtqueue activity and configuration changes by sending notifications.
>>>> What is the difference between a doorbell and a notification?
>>> Doorbells allow the driver to signal the device (i.e. device hardware
>>> registers that the driver writes to).
>>>
>>> Notifications allow the driver to be signalled by the device (i.e. MSI-X
>>> interrupts that the driver handles).
>>>
>>> The more abstract "doorbell" and "notification" terms are used instead
>>> of hardware registers and interrupts because transports other than PCI
>>> may want to support these features too.
>> Let me try to make this a little bit more clear.
>>
>> A doorbell is a device register that is hooked up to a vhost-user
>> callfd.  There is one doorbell for each callfd. When the driver (i.e.,
>> the virtio-vhost-user driver) kicks a doorbell, the device (i.e., the
>> virtio-vhost-user device) kicks the corresponding callfd. So, doorbells
>> are the means for the device emulation software (e.g., the
>> vhost-user-net process in the above diagram), running in the Device VM,
>> to notify the device driver (e.g., the virtio-net driver in the above
>> diagram), running in the Driver VM, for I/O completions.
> OK - I think I follow that although it's tricky because a
> virtio-vhost-user driver is not like a virtio-device driver - being at
> opposite ends of each other. Another good argument for having clear
> terminology descriptions. I'll try and come up with some words and a
> patch for the spec so we can make it clearer.
>
>> A notification is an interrupt. Notifications are hooked up to
>> vhost-user kickds. When a kickfd is kicked by the vhost-user master, the
>> virtio-vhost-user device sends the corresponding interrupt to the
>> virtio-vhost-user driver. So, notifications are the means for the device
>> driver, running in the Driver VM, to notify the device emulation
>> software, running in the Device VM, for I/O submissions.
> Isn't that the wrong way around, surely a device driver sees a IRQ as
> it's notification? Or are we talking about notifications going both ways
> here?
>

We are talking about notifications from the vhost-user frontend to the
backend. The virtio-vhost-user device uses an interrupt to signal the
vhost-user backend in the Device VM for events on the vhost-user kickfd.


In more detail, the signaling path goes like this:

1. The virtio-net driver (in the above diagram) adds new descriptors in
    the virtqueue and kicks the corresponding doorbell/register of the
    vhost-user-net device.

2. The VMM in the Driver VM kicks the vhost-user kickfd that is
    associated with this virtqueue in order to notify the backend.

3. The virtio-vhost-user device, which is monitoring this kickfd,
    notices the event and, in response, it sends an interrupt to the
    guest in order to notify the vhost-user backend.


 From your comment, I can't really tell for sure what troubles you. Yes,
the device driver sees an IRQ as its notification. Is there any other
way that a device can notify a driver for something? Also, the
meaning of a device-to-driver notification is device-specific. In this
particular case, we are reserving specific MSI vectors to designate
events on the vhost-user virtqueues.

Could you please explain in more detail what troubles you? Is it about
the word "notification"? In the virtio spec [1], this word refers to
both driver-to-device and device-to-driver signaling.

[1]https://docs.oasis-open.org/virtio/virtio/v1.1/csprd01/virtio-v1.1-csprd01.html#x1-170003

---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

Re: [virtio-dev] [PATCH v5 01/10] vhost-user: add vhost-user device type

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 2573 bytes --]

On Fri, Jul 17, 2020 at 02:17:13PM +0300, Nikos Dragazis wrote:
> On 17/7/20 12:26 μ.μ., Stefan Hajnoczi wrote:
> 
> > On Thu, Jul 16, 2020 at 05:45:47PM +0100, Alex Bennée wrote:
> > > Nikos Dragazis <ndragazis@arrikto.com> writes:
> > > > diff --git a/virtio-vhost-user.tex b/virtio-vhost-user.tex
> > > > new file mode 100644
> > > > index 0000000..ac96dc2
> > > > --- /dev/null
> > > > +++ b/virtio-vhost-user.tex
> > > > @@ -0,0 +1,292 @@
> > > > +\section{Vhost-user Device Backend}\label{sec:Device Types / Vhost-user Device Backend}
> > > > +
> > > > +The vhost-user device backend facilitates vhost-user device emulation through
> > > > +vhost-user protocol exchanges and access to shared memory.  Software-defined
> > > > +networking, storage, and other I/O appliances can provide services through this
> > > > +device.
> > > > +
> > > > +This section relies on definitions from the \hyperref[intro:Vhost-user
> > > > +Protocol]{Vhost-user Protocol}.  Knowledge of the vhost-user protocol is a
> > > > +prerequisite for understanding this device.
> > > > +
> > > > +The \hyperref[intro:Vhost-user Protocol]{Vhost-user Protocol} was originally
> > > > +designed for processes on a single system communicating over UNIX domain
> > > > +sockets.  The virtio vhost-user device backend allows the vhost-user slave to
> > > > +communicate with the vhost-user master over the device instead of a UNIX domain
> > > > +socket.  This allows the slave and master to run on two separate
> > > > systems such
> > > I realise we already have the terms master/slave baked into the
> > > vhost-user spec but perhaps we could find better wording? The vhost
> > > documentation describes thing in terms of who owns the virtqueues (the
> > > drive) and who processes the requests (the device). There may be better
> > > terminology to use.
> > "backend" is now commonly used instead of "slave". There is no new term
> > for "master" yet. I suggest replacing "slave" with "backend" in this
> > patch.
> 
> Makes sense. Some observations:
> 
> 1. Since "backend" is used instead of "slave", why "frontend" is not
>    used instead of "master"? Also, why does the vhost-user spec use the
>    terms"slave" and "backend" interchangeably and doesn't just drop the
>    term"slave"completely?

Using "frontend" and "backend" sounds good. Do you want to send a patch
to qemu.git docs/interop/vhost-user.rst?

The vhost-user spec grew organically and few people have spent time on
polishing it, making it consistent, etc.

Stefan

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: [virtio-dev] [PATCH v5 03/10] vhost-user: add requirements for the notification capability

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 2335 bytes --]

On Fri, Jul 17, 2020 at 07:02:18PM +0300, Nikos Dragazis wrote:
> On 17/7/20 12:34 μ.μ., Stefan Hajnoczi wrote:
> 
> > On Mon, May 18, 2020 at 11:37:14PM +0300, Nikos Dragazis wrote:
> > > The configuration structure for the notification capability has certain
> > > device/driver requirements, similar to those of the MSI-X vector
> > > configuration. Add these requirements to the device spec.
> > > 
> > > Signed-off-by: Nikos Dragazis <ndragazis@arrikto.com>
> > > ---
> > >   virtio-vhost-user.tex | 42 ++++++++++++++++++++++++++++++++++++++++++
> > >   1 file changed, 42 insertions(+)
> > > 
> > > diff --git a/virtio-vhost-user.tex b/virtio-vhost-user.tex
> > > index 13ecd41..7a8cd9c 100644
> > > --- a/virtio-vhost-user.tex
> > > +++ b/virtio-vhost-user.tex
> > > @@ -275,6 +275,48 @@ \subsubsection{Notification structure layout}\label{sec:Device Types / Vhost-use
> > >   \field{VIRTIO_MSI_NO_VECTOR} to \field{notification_msix_vector} to change the
> > >   MSI-X vector for that notification.
> > > +\devicenormative{\paragraph}{Notification capability}{Device Types / Vhost-user Device Backend / Additional Device Resources over PCI / Notification capability}
> > > +
> > > +If MSI-X is available, device MUST support mapping any master queue
> > > +event to any valid vector 0 to MSI-X \field{Table Size}. Here
> > This section combines two things:
> > 1. vhost-user concepts -> VIRTIO notifications
> > 2. VIRTIO notifications -> VIRTIO PCI transport MSI-X implementation of
> >     notifications
> > 
> > I think this should be split:
> > 1. The virtio-vhost-user device spec should explain that vhost-user queue
> >     events are mapped to VIRTIO notifications
> > 2. The PCI transport spec should explain how notifications are mapped to
> >     MSI-X
> > 
> > This way other VIRTIO device types will be able to use notifications too
> > without everyone explaining MSI-X specifics. Also it won't tie
> > virtio-vhost-user to VIRTIO PCI.
> 
> If I understand correctly, you are suggesting that we should standardize
> the notification resource, similarly to the shared memory resource, so
> that any device can use it. Is that so?

Yes. Notifications and doorbells should become part of the core VIRTIO
device model. That way other devices can use them too.

Stefan

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: [virtio-dev] [PATCH v5 08/10] vhost-user: remove the extra PCI capabilities

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 2297 bytes --]

On Fri, Jul 17, 2020 at 06:03:57PM +0300, Nikos Dragazis wrote:
> On 17/7/20 12:48 μ.μ., Stefan Hajnoczi wrote:
> 
> > On Mon, May 18, 2020 at 11:37:19PM +0300, Nikos Dragazis wrote:
> > > -Additional resources are configured on the virtio PCI transport by the following \field{struct virtio_pci_cap.cfg_type} values:
> > > +\subsubsection{Doorbell layout}\label{sec:Device Types / Vhost-user Device Backend / Additional Device Resources / Doorbell layout}
> > > -\begin{lstlisting}
> > > -#define VIRTIO_PCI_CAP_DOORBELL_CFG 6
> > > -#define VIRTIO_PCI_CAP_NOTIFICATION_CFG 7
> > > -#define VIRTIO_PCI_CAP_SHARED_MEMORY_CFG 8
> > > -\end{lstlisting}
> > > +The device MUST reserve 2N+1 virtqueue indices that can be used by the driver to
> > > +send doorbell notifications. The driver can use these indices to send doorbell
> > > +notifications in the same way that it sends available buffer notifications
> > > +\ref{sec:Basic Facilities of a Virtio Device / Notifications} for a virtqueue.
> > Notifications and doorbells should be first-class VIRTIO device model
> > concepts, like Shared Memory resources. It should not be necessary to
> > reserve dummy virtqueues (a hack for getting MSI-X vectors and doorbell
> > registers).
> 
> Well, it's not exactly equivalent to the shared memory resource
> because, in case of doorbells and notifications, the functionality is
> already there. But I see your point. It is a design issue. We could
> definitely standardize the doorbell and notification resources, but I
> don't see why this solution is preferable. Do you have something in
> mind?

Yes, there are advantages to defining doorbells and notifications as
separate concepts:

 * Doorbells and notifications may be implemented differently from
   virtqueue notifications on some the transports.
 * Virtqueues may consume more resources on some transports.
 * Existing drivers may have issues distinguishing between normal
   virtqueues and dummy virtqueues.

Overall I think using dummy virtqueues is a hacky approach. It's
something that would be necessary if we couldn't extend the core VIRTIO
device model. Since we do have the opportunity of extending the core
VIRTIO device model cleanly I think doing so will avoid headaches later.

Stefan

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: [virtio-dev] [PATCH v5 09/10] vhost-user: intercept slave's reply to VHOST_USER_GET_PROTOCOL_FEATURES

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 988 bytes --]

On Fri, Jul 17, 2020 at 06:37:11PM +0300, Nikos Dragazis wrote:
> On 17/7/20 12:57 μ.μ., Stefan Hajnoczi wrote:
> > On Mon, May 18, 2020 at 11:37:20PM +0300, Nikos Dragazis wrote:
> > Why does the virtio-vhost-user device implementation need to silently
> > clear those feature bits? Is there a security impact or some other
> > reason why the VIRTIO spec should specify this behavior?
> 
> In our setup, the slave cannot pass file descriptors to the master.
> The easiest way to enforce this restriction is during feature
> negotiation. The virtio-vhost-user device will mask the unsupported
> feature bits and, therefore, the master will not enable these feature
> bits on the slave.

I see. It stood out because the spec could simply forbid using those
bits instead of mandating that they be silently clearer at runtime.

I think there's no harm in clearing them at runtime and it may make
device emulation slightly simpler. So this sounds fine.

Thanks,
Stefan

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: [virtio-dev] [PATCH v5 01/10] vhost-user: add vhost-user device type

[Nikos Dragazis]

On 23/7/20 9:15 π.μ., Stefan Hajnoczi wrote:

> On Fri, Jul 17, 2020 at 02:17:13PM +0300, Nikos Dragazis wrote:
>> On 17/7/20 12:26 μ.μ., Stefan Hajnoczi wrote:
>>
>>> On Thu, Jul 16, 2020 at 05:45:47PM +0100, Alex Bennée wrote:
>>>> Nikos Dragazis <ndragazis@arrikto.com> writes:
>>>>> diff --git a/virtio-vhost-user.tex b/virtio-vhost-user.tex
>>>>> new file mode 100644
>>>>> index 0000000..ac96dc2
>>>>> --- /dev/null
>>>>> +++ b/virtio-vhost-user.tex
>>>>> @@ -0,0 +1,292 @@
>>>>> +\section{Vhost-user Device Backend}\label{sec:Device Types / Vhost-user Device Backend}
>>>>> +
>>>>> +The vhost-user device backend facilitates vhost-user device emulation through
>>>>> +vhost-user protocol exchanges and access to shared memory.  Software-defined
>>>>> +networking, storage, and other I/O appliances can provide services through this
>>>>> +device.
>>>>> +
>>>>> +This section relies on definitions from the \hyperref[intro:Vhost-user
>>>>> +Protocol]{Vhost-user Protocol}.  Knowledge of the vhost-user protocol is a
>>>>> +prerequisite for understanding this device.
>>>>> +
>>>>> +The \hyperref[intro:Vhost-user Protocol]{Vhost-user Protocol} was originally
>>>>> +designed for processes on a single system communicating over UNIX domain
>>>>> +sockets.  The virtio vhost-user device backend allows the vhost-user slave to
>>>>> +communicate with the vhost-user master over the device instead of a UNIX domain
>>>>> +socket.  This allows the slave and master to run on two separate
>>>>> systems such
>>>> I realise we already have the terms master/slave baked into the
>>>> vhost-user spec but perhaps we could find better wording? The vhost
>>>> documentation describes thing in terms of who owns the virtqueues (the
>>>> drive) and who processes the requests (the device). There may be better
>>>> terminology to use.
>>> "backend" is now commonly used instead of "slave". There is no new term
>>> for "master" yet. I suggest replacing "slave" with "backend" in this
>>> patch.
>> Makes sense. Some observations:
>>
>> 1. Since "backend" is used instead of "slave", why "frontend" is not
>>     used instead of "master"? Also, why does the vhost-user spec use the
>>     terms"slave" and "backend" interchangeably and doesn't just drop the
>>     term"slave"completely?
> Using "frontend" and "backend" sounds good. Do you want to send a patch
> to qemu.git docs/interop/vhost-user.rst?

OK, I will.

>
> The vhost-user spec grew organically and few people have spent time on
> polishing it, making it consistent, etc.
>
> Stefan

---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

Re: [virtio-dev] [PATCH v5 08/10] vhost-user: remove the extra PCI capabilities

[Nikos Dragazis]

On 23/7/20 9:29 π.μ., Stefan Hajnoczi wrote:

> On Fri, Jul 17, 2020 at 06:03:57PM +0300, Nikos Dragazis wrote:
>> On 17/7/20 12:48 μ.μ., Stefan Hajnoczi wrote:
>>
>>> On Mon, May 18, 2020 at 11:37:19PM +0300, Nikos Dragazis wrote:
>>>> -Additional resources are configured on the virtio PCI transport by the following \field{struct virtio_pci_cap.cfg_type} values:
>>>> +\subsubsection{Doorbell layout}\label{sec:Device Types / Vhost-user Device Backend / Additional Device Resources / Doorbell layout}
>>>> -\begin{lstlisting}
>>>> -#define VIRTIO_PCI_CAP_DOORBELL_CFG 6
>>>> -#define VIRTIO_PCI_CAP_NOTIFICATION_CFG 7
>>>> -#define VIRTIO_PCI_CAP_SHARED_MEMORY_CFG 8
>>>> -\end{lstlisting}
>>>> +The device MUST reserve 2N+1 virtqueue indices that can be used by the driver to
>>>> +send doorbell notifications. The driver can use these indices to send doorbell
>>>> +notifications in the same way that it sends available buffer notifications
>>>> +\ref{sec:Basic Facilities of a Virtio Device / Notifications} for a virtqueue.
>>> Notifications and doorbells should be first-class VIRTIO device model
>>> concepts, like Shared Memory resources. It should not be necessary to
>>> reserve dummy virtqueues (a hack for getting MSI-X vectors and doorbell
>>> registers).
>> Well, it's not exactly equivalent to the shared memory resource
>> because, in case of doorbells and notifications, the functionality is
>> already there. But I see your point. It is a design issue. We could
>> definitely standardize the doorbell and notification resources, but I
>> don't see why this solution is preferable. Do you have something in
>> mind?
> Yes, there are advantages to defining doorbells and notifications as
> separate concepts:
>
>   * Doorbells and notifications may be implemented differently from
>     virtqueue notifications on some the transports.
>   * Virtqueues may consume more resources on some transports.
>   * Existing drivers may have issues distinguishing between normal
>     virtqueues and dummy virtqueues.
>
> Overall I think using dummy virtqueues is a hacky approach. It's
> something that would be necessary if we couldn't extend the core VIRTIO
> device model. Since we do have the opportunity of extending the core
> VIRTIO device model cleanly I think doing so will avoid headaches later.
>
> Stefan

OK. Let's standardize them. I will send a separate patch for this. I
can't think of all the details right now, so, I'll come back if anything
troubles me.

Nikos

---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

Re: [virtio-dev] [PATCH v5 01/10] vhost-user: add vhost-user device type

[Michael S. Tsirkin]

On Fri, Jul 17, 2020 at 10:26:35AM +0100, Stefan Hajnoczi wrote:
> > > +The following additional resources exist:
> > > +\begin{description}
> > > +  \item[Doorbells] The driver signals the vhost-user master through doorbells.  The signal does not carry any data, it is purely an event.
> > > +  \item[Notifications] The vhost-user master signals the driver for events besides virtqueue activity and configuration changes by sending notifications.
> > 
> > What is the difference between a doorbell and a notification?
> 
> Doorbells allow the driver to signal the device (i.e. device hardware
> registers that the driver writes to).
> 
> Notifications allow the driver to be signalled by the device (i.e. MSI-X
> interrupts that the driver handles).
> 
> The more abstract "doorbell" and "notification" terms are used instead
> of hardware registers and interrupts because transports other than PCI
> may want to support these features too.

I think notification and doorbell are a bit too generic of a name.

What we have is:
available buffer notifications, used buffer
notifications, configuration change notifications.


So how about XXX notifications for both of these things?
Where XXX explains what is the reason for triggering each
type of notification.

It says "besides virtqueue activity and configuration changes" -
what exactly is there?



---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

Re: [virtio-dev] [PATCH v5 01/10] vhost-user: add vhost-user device type

[Nikos Dragazis]

On 24/7/20 5:41 μ.μ., Michael S. Tsirkin wrote:

> On Fri, Jul 17, 2020 at 10:26:35AM +0100, Stefan Hajnoczi wrote:
>>>> +The following additional resources exist:
>>>> +\begin{description}
>>>> +  \item[Doorbells] The driver signals the vhost-user master through doorbells.  The signal does not carry any data, it is purely an event.
>>>> +  \item[Notifications] The vhost-user master signals the driver for events besides virtqueue activity and configuration changes by sending notifications.
>>> What is the difference between a doorbell and a notification?
>> Doorbells allow the driver to signal the device (i.e. device hardware
>> registers that the driver writes to).
>>
>> Notifications allow the driver to be signalled by the device (i.e. MSI-X
>> interrupts that the driver handles).
>>
>> The more abstract "doorbell" and "notification" terms are used instead
>> of hardware registers and interrupts because transports other than PCI
>> may want to support these features too.
> I think notification and doorbell are a bit too generic of a name.
>
> What we have is:
> available buffer notifications, used buffer
> notifications, configuration change notifications.

The problem with the virtio-vhost-user device is that it must be able to
exchange notifications (with the driver) that have nothing to do with
the notification types that you mentioned. Specifically, the
virtio-vhost-user device must be able to send/receive notifications
regarding activity on the vhost-user virtqueues.

The vhost-user virtqueues are not processed by the virtio-vhost-user
device and they have nothing to do with the RX/TX virtqueues of the
virtio-vhost-user device. I think this is the main reason why we don't
want to use the existing registers that we use for virtqueue
notifications.

So, my point is that we could extend the notion of a "notification".
A notification could be either one of the following:

1. available buffer notification
2. used buffer notification
3. configuration change notification
4. driver-to-device notification for other purposes
5. device-to-driver notification for other purposes

>
>
> So how about XXX notifications for both of these things?
> Where XXX explains what is the reason for triggering each
> type of notification.

I guess Stefan introduced the term "doorbell" because it is a widely
used term (e.g., in NVMe devices) and I think it is a pretty good
term. However, I agree that the vague term "notification" may cause
confusion.

Replacing "doorbells" and "notifications" with "XXX notifications"
sounds good. I can't think of some good terms for XXX right now.
Feel free to propose.

>
> It says "besides virtqueue activity and configuration changes" -
> what exactly is there?
>

As I said above, it is about available buffers on the vhost-user
virtqueues. But the vhost-user virtqueues are not related by any means
with the virtio-vhost-user device, so a notification resource in the
virtio-vhost-user device does not fit in the current definition of a
notification in virtio. It is something completely different.

---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

Re: [virtio-dev] [PATCH v5 01/10] vhost-user: add vhost-user device type

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 979 bytes --]

On Fri, Jul 24, 2020 at 08:31:48PM +0300, Nikos Dragazis wrote:
> On 24/7/20 5:41 μ.μ., Michael S. Tsirkin wrote:
> > On Fri, Jul 17, 2020 at 10:26:35AM +0100, Stefan Hajnoczi wrote:
> > So how about XXX notifications for both of these things?
> > Where XXX explains what is the reason for triggering each
> > type of notification.
> 
> I guess Stefan introduced the term "doorbell" because it is a widely
> used term (e.g., in NVMe devices) and I think it is a pretty good
> term. However, I agree that the vague term "notification" may cause
> confusion.
> 
> Replacing "doorbells" and "notifications" with "XXX notifications"
> sounds good. I can't think of some good terms for XXX right now.
> Feel free to propose.

Device-specific notifications and device-specific doorbells?

I think these concepts should be part of the core VIRTIO device model so
that other devices can also define device-specific
notifications/doorbells in the future.

Stefan

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: [virtio-dev] [PATCH v5 01/10] vhost-user: add vhost-user device type

[Nikos Dragazis]

On 27/7/20 3:25 μ.μ., Stefan Hajnoczi wrote:

> On Fri, Jul 24, 2020 at 08:31:48PM +0300, Nikos Dragazis wrote:
>> On 24/7/20 5:41 μ.μ., Michael S. Tsirkin wrote:
>>> On Fri, Jul 17, 2020 at 10:26:35AM +0100, Stefan Hajnoczi wrote:
>>> So how about XXX notifications for both of these things?
>>> Where XXX explains what is the reason for triggering each
>>> type of notification.
>> I guess Stefan introduced the term "doorbell" because it is a widely
>> used term (e.g., in NVMe devices) and I think it is a pretty good
>> term. However, I agree that the vague term "notification" may cause
>> confusion.
>>
>> Replacing "doorbells" and "notifications" with "XXX notifications"
>> sounds good. I can't think of some good terms for XXX right now.
>> Feel free to propose.
> Device-specific notifications and device-specific doorbells?
>
> I think these concepts should be part of the core VIRTIO device model so
> that other devices can also define device-specific
> notifications/doorbells in the future.
>
> Stefan

I think that Michael is OK with standardizing these resources, but he is
not really happy with the term "doorbell".

 From the one hand, using the term "XXX notification" would be more
consistent with the virtio spec, since virtio already uses the term
"notification" for driver-to-device signaling regarding the virtqueues.

On the other hand, I can't find something for XXX that would sound
better than "doorbell".

But let's see what Michael has to say.

Nikos

---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

Re: [virtio-dev] [PATCH v5 01/10] vhost-user: add vhost-user device type

[Michael S. Tsirkin]

On Mon, Jul 27, 2020 at 09:59:36PM +0300, Nikos Dragazis wrote:
> On 27/7/20 3:25 μ.μ., Stefan Hajnoczi wrote:
> 
> > On Fri, Jul 24, 2020 at 08:31:48PM +0300, Nikos Dragazis wrote:
> > > On 24/7/20 5:41 μ.μ., Michael S. Tsirkin wrote:
> > > > On Fri, Jul 17, 2020 at 10:26:35AM +0100, Stefan Hajnoczi wrote:
> > > > So how about XXX notifications for both of these things?
> > > > Where XXX explains what is the reason for triggering each
> > > > type of notification.
> > > I guess Stefan introduced the term "doorbell" because it is a widely
> > > used term (e.g., in NVMe devices) and I think it is a pretty good
> > > term. However, I agree that the vague term "notification" may cause
> > > confusion.
> > > 
> > > Replacing "doorbells" and "notifications" with "XXX notifications"
> > > sounds good. I can't think of some good terms for XXX right now.
> > > Feel free to propose.
> > Device-specific notifications and device-specific doorbells?

Um. What does this mean exactly?
Really I am guessing you are notifying about some kind of event.
So let's call it according to this event then.

> > 
> > I think these concepts should be part of the core VIRTIO device model so
> > that other devices can also define device-specific
> > notifications/doorbells in the future.
> > 
> > Stefan
> 
> I think that Michael is OK with standardizing these resources, but he is
> not really happy with the term "doorbell".
> 
> From the one hand, using the term "XXX notification" would be more
> consistent with the virtio spec, since virtio already uses the term
> "notification" for driver-to-device signaling regarding the virtqueues.
> 
> On the other hand, I can't find something for XXX that would sound
> better than "doorbell".
> 
> But let's see what Michael has to say.
> 
> Nikos

"doorbells" is generally like available buffer notifications that we
have in virtio. I am still not sure what does it mean in this context.

-- 
MST


---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

Re: [virtio-dev] [PATCH v5 01/10] vhost-user: add vhost-user device type

[Nikos Dragazis]

On 29/7/20 4:52 μ.μ., Michael S. Tsirkin wrote:

> On Mon, Jul 27, 2020 at 09:59:36PM +0300, Nikos Dragazis wrote:
>> On 27/7/20 3:25 μ.μ., Stefan Hajnoczi wrote:
>>
>>> On Fri, Jul 24, 2020 at 08:31:48PM +0300, Nikos Dragazis wrote:
>>>> On 24/7/20 5:41 μ.μ., Michael S. Tsirkin wrote:
>>>>> On Fri, Jul 17, 2020 at 10:26:35AM +0100, Stefan Hajnoczi wrote:
>>>>> So how about XXX notifications for both of these things?
>>>>> Where XXX explains what is the reason for triggering each
>>>>> type of notification.
>>>> I guess Stefan introduced the term "doorbell" because it is a widely
>>>> used term (e.g., in NVMe devices) and I think it is a pretty good
>>>> term. However, I agree that the vague term "notification" may cause
>>>> confusion.
>>>>
>>>> Replacing "doorbells" and "notifications" with "XXX notifications"
>>>> sounds good. I can't think of some good terms for XXX right now.
>>>> Feel free to propose.
>>> Device-specific notifications and device-specific doorbells?
> Um. What does this mean exactly?
> Really I am guessing you are notifying about some kind of event.
> So let's call it according to this event then.

It's for available and used buffer notifications on the vhost-user
virtqueues. We don't want to use this name because it refers to a very
specific use case and, hence, it is too restrictive. What we want is to
introduce new types of notifications that will be generic enough so that
other future devices will be able to use them as well.

>>> I think these concepts should be part of the core VIRTIO device model so
>>> that other devices can also define device-specific
>>> notifications/doorbells in the future.
>>>
>>> Stefan
>> I think that Michael is OK with standardizing these resources, but he is
>> not really happy with the term "doorbell".
>>
>>  From the one hand, using the term "XXX notification" would be more
>> consistent with the virtio spec, since virtio already uses the term
>> "notification" for driver-to-device signaling regarding the virtqueues.
>>
>> On the other hand, I can't find something for XXX that would sound
>> better than "doorbell".
>>
>> But let's see what Michael has to say.
>>
>> Nikos
> "doorbells" is generally like available buffer notifications that we
> have in virtio. I am still not sure what does it mean in this context.
>

"doorbells" is more generic than available buffer notifications. It will
refer to any type of driver-to-device notification that will not be an
available buffer notification.

Think about the vendor-specific capabilities in PCI. They are there so
that a device manufacturer can implement non-standardized capabilities.
Similarly, doorbells will be there so that a device can support
driver-to-device notifications for other things except for available
buffers.

As an alternative to:

- device-specific doorbells
- device-specific notifications

I would also put the following names on the table:

- device-specific driver-to-device notifications
- device-specific device-to-driver notifications

but I like Stefan's better.

Nikos

---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

Re: [virtio-dev] [PATCH v5 01/10] vhost-user: add vhost-user device type

[Nikos Dragazis]

On 30/7/20 12:05 π.μ., Nikos Dragazis wrote:

> On 29/7/20 4:52 μ.μ., Michael S. Tsirkin wrote:
>
>> On Mon, Jul 27, 2020 at 09:59:36PM +0300, Nikos Dragazis wrote:
>>> On 27/7/20 3:25 μ.μ., Stefan Hajnoczi wrote:
>>>
>>>> On Fri, Jul 24, 2020 at 08:31:48PM +0300, Nikos Dragazis wrote:
>>>>> On 24/7/20 5:41 μ.μ., Michael S. Tsirkin wrote:
>>>>>> On Fri, Jul 17, 2020 at 10:26:35AM +0100, Stefan Hajnoczi wrote:
>>>>>> So how about XXX notifications for both of these things?
>>>>>> Where XXX explains what is the reason for triggering each
>>>>>> type of notification.
>>>>> I guess Stefan introduced the term "doorbell" because it is a widely
>>>>> used term (e.g., in NVMe devices) and I think it is a pretty good
>>>>> term. However, I agree that the vague term "notification" may cause
>>>>> confusion.
>>>>>
>>>>> Replacing "doorbells" and "notifications" with "XXX notifications"
>>>>> sounds good. I can't think of some good terms for XXX right now.
>>>>> Feel free to propose.
>>>> Device-specific notifications and device-specific doorbells?
>> Um. What does this mean exactly?
>> Really I am guessing you are notifying about some kind of event.
>> So let's call it according to this event then.
>
> It's for available and used buffer notifications on the vhost-user
> virtqueues. We don't want to use this name because it refers to a very
> specific use case and, hence, it is too restrictive. What we want is to
> introduce new types of notifications that will be generic enough so that
> other future devices will be able to use them as well.
>
>>>> I think these concepts should be part of the core VIRTIO device model so
>>>> that other devices can also define device-specific
>>>> notifications/doorbells in the future.
>>>>
>>>> Stefan
>>> I think that Michael is OK with standardizing these resources, but he is
>>> not really happy with the term "doorbell".
>>>
>>>  From the one hand, using the term "XXX notification" would be more
>>> consistent with the virtio spec, since virtio already uses the term
>>> "notification" for driver-to-device signaling regarding the virtqueues.
>>>
>>> On the other hand, I can't find something for XXX that would sound
>>> better than "doorbell".
>>>
>>> But let's see what Michael has to say.
>>>
>>> Nikos
>> "doorbells" is generally like available buffer notifications that we
>> have in virtio. I am still not sure what does it mean in this context.
>>
>
> "doorbells" is more generic than available buffer notifications. It will
> refer to any type of driver-to-device notification that will not be an
> available buffer notification.
>
> Think about the vendor-specific capabilities in PCI. They are there so
> that a device manufacturer can implement non-standardized capabilities.
> Similarly, doorbells will be there so that a device can support
> driver-to-device notifications for other things except for available
> buffers.
>
> As an alternative to:
>
> - device-specific doorbells
> - device-specific notifications
>
> I would also put the following names on the table:
>
> - device-specific driver-to-device notifications
> - device-specific device-to-driver notifications
>
> but I like Stefan's better.
>
> Nikos
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
> For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

I sent a separate patch for doorbells and notifications. Let's move the
discussion on that thread.

Nikos


---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org