[virtio-dev] [PATCH 0/4] Introduce aux. notifications and virtio-vhost-user

[virtio-dev] [PATCH 0/4] Introduce aux. notifications and virtio-vhost-user

[Usama Arif]

Hi,

This patch series introduces device and driver auxiliary notifications
as a new set of virtio device resources, as well as vhost-user device
backend that uses these resources.

Driver auxiliary notifications allow the device to send notifications
other than configuration changes and used buffer notifications to the
driver, these are optional and their meaning is **device-specific**.

Device auxiliary notifcations allow the driver to send notifcations
other than available buffer notifications to the device for example
through a device register, these are optional and their meaning is
**device-specific**.

These resources are used in the last patch by the virtio-vhost-user
device in order to send/receive notifications to/from the driver
regarding activity on the vhost-user virtqueues. By standardizing
these resources as standalone virtio device resources, other future
devices will be able to use them as well.

The last patch introduces the vhost-user device backend which facilitates
vhost-user device emulation through vhost-user protocol exchanges and
access to shared memory. Software-defined networking, storage, and other
I/O appliances can provide services through this device.

This device is based on Wei Wang's vhost-pci work. The virtio-vhost-user
device differs from vhost-pci because it is a single virtio
device type that exposes the vhost-user protocol instead of a family of
new virtio device types, one for each vhost-user device type.

A HTML version with the changes is available at [1].

For more information about virtio-vhost-user, see [2].

These patches are based on the work initially done by Stefan Hajnoczi [2]
and continued by Nikos Dragazis [3]. A lot of the content is from patches
by Stefan and Nikos earlier [4]. (Hence, i have added their sign-offs in
the patches, I hope thats ok)

A working prototype implementing this spec can be reproduced using
instructions in [5] utilizing work done on top of latest (at time of
writing these patches) QEMU [6] and DPDK [7].
This is also based on the work initially done by Stefan
and continued later by Nikos. The prototype code uses the terms
"doorbell"/"device-specific notification" and "master"/"slave" instead of
"device auxiliary notification"/"driver auxiliary notification" and
"frontend"/"backend". This is based on older work, however, their
functionality is the same. If these virtio-spec changes get approved,
I will refractor the patches for QEMU/DPDK and send them for review
according to the final patches in the respective mailing list.

Please let me know if there is something that can be done from my side to
make sure vhost-user is allowed to be referred to in virtio-spec.

Thanks and looking forward to your response!
Usama

[1] https://uarif1.github.io/vvu/virtio-v1.1-cs01
[2] https://wiki.qemu.org/Features/VirtioVhostUser
[3] https://ndragazis.github.io/dpdk-vhost-vvu-demo.html
[4] https://lists.oasis-open.org/archives/virtio-dev/202005/msg00132.html
[5] https://uarif1.github.io/vvu/dpdk-vvu-instructions
[6] https://github.com/uarif1/qemu/tree/vvu
[7] https://github.com/uarif1/dpdk/tree/vvu


Usama Arif (4):
  content: Introduce driver/device auxiliary notifications
  content: Introduce driver/device aux. notification cfg type for PCI
  content: Introduce driver/device auxiliary notifications for MMIO
  vhost-user: add vhost-user device type

 conformance.tex       |  29 ++++-
 content.tex           | 195 +++++++++++++++++++++++++------
 introduction.tex      |   3 +
 virtio-vhost-user.tex | 259 ++++++++++++++++++++++++++++++++++++++++++
 4 files changed, 448 insertions(+), 38 deletions(-)
 create mode 100644 virtio-vhost-user.tex

-- 
2.25.1


---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

[virtio-dev] [PATCH 1/4] content: Introduce driver/device auxiliary notifications

[Usama Arif]

Driver auxiliary notifications allow the device to send notifications
other than configuration changes and used buffer notifications to the
driver, these are optional and their meaning is device-specific.

Device auxiliary notifcations allow the driver to send notifcations
other than available buffer notifications to the device for example
through a device register, these are optional and their meaning is
device-specific.

These device-specific notifications are needed later when adding support
for virtio-vhost-user device.

Signed-off-by: Usama Arif <usama.arif@bytedance.com>
Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
Signed-off-by: Nikos Dragazis <ndragazis@arrikto.com>
---
 content.tex | 35 ++++++++++++++++++++++-------------
 1 file changed, 22 insertions(+), 13 deletions(-)

diff --git a/content.tex b/content.tex
index c6f116c..85980ac 100644
--- a/content.tex
+++ b/content.tex
@@ -160,29 +160,38 @@ \subsection{Legacy Interface: A Note on Feature
 Specification text within these sections generally does not apply
 to non-transitional devices.
 
-\section{Notifications}\label{sec:Basic Facilities of a Virtio Device
-/ Notifications}
+\section{Notifications}\label{sec:Basic Facilities of a Virtio Device / Notifications}
 
 The notion of sending a notification (driver to device or device
 to driver) plays an important role in this specification. The
 modus operandi of the notifications is transport specific.
 
-There are three types of notifications: 
+There are five types of notifications:
 \begin{itemize}
 \item configuration change notification
 \item available buffer notification
-\item used buffer notification. 
+\item used buffer notification
+\item driver auxiliary notification
+\item device auxiliary notification
 \end{itemize}
 
-Configuration change notifications and used buffer notifications are sent
-by the device, the recipient is the driver. A configuration change
-notification indicates that the device configuration space has changed; a
-used buffer notification indicates that a buffer may have been made used
-on the virtqueue designated by the notification.
-
-Available buffer notifications are sent by the driver, the recipient is
-the device. This type of notification indicates that a buffer may have
-been made available on the virtqueue designated by the notification.
+Configuration change notifications, used buffer notifications and
+driver auxiliary notifications are sent by the device,
+the recipient is the driver. A configuration change notification indicates
+that the device configuration space has changed; a used buffer notification
+indicates that a buffer may have been made used on the virtqueue designated
+by the notification; driver auxiliary notifications allow the
+device to send notifications other than configuration changes and used
+buffer notifications to the driver, these are optional and their meaning
+is device-specific.
+
+Available buffer notifications and device auxiliary notifications
+are sent by the driver, the recipient is the device. Available buffer
+notifications indicate that a buffer may have been made available on the
+virtqueue designated by the notification; device auxiliary
+notifcations allow the driver to send notifcations other than available
+buffer notifications to the device for example through a device register, these
+are optional and their meaning is device-specific.
 
 The semantics, the transport-specific implementations, and other
 important aspects of the different notifications are specified in detail
-- 
2.25.1


---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

[virtio-dev] [PATCH 2/4] content: Introduce driver/device aux. notification cfg type for PCI

[Usama Arif]

This includes the PCI device conformances for these notification
capabilities.

Signed-off-by: Usama Arif <usama.arif@bytedance.com>
Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
Signed-off-by: Nikos Dragazis <ndragazis@arrikto.com>
---
 conformance.tex |   2 +
 content.tex     | 134 +++++++++++++++++++++++++++++++++++++++++-------
 2 files changed, 117 insertions(+), 19 deletions(-)

diff --git a/conformance.tex b/conformance.tex
index 42f8537..cddaf75 100644
--- a/conformance.tex
+++ b/conformance.tex
@@ -356,6 +356,8 @@ \section{Conformance Targets}\label{sec:Conformance / Conformance Targets}
 \item \ref{devicenormative:Virtio Transport Options / Virtio Over PCI Bus / PCI Device Layout / Notification capability}
 \item \ref{devicenormative:Virtio Transport Options / Virtio Over PCI Bus / PCI Device Layout / ISR status capability}
 \item \ref{devicenormative:Virtio Transport Options / Virtio Over PCI Bus / PCI Device Layout / Device-specific configuration}
+\item \ref{devicenormative:Virtio Transport Options / Virtio Over PCI Bus / PCI Device Layout / Device auxiliary notification capability}
+\item \ref{devicenormative:Virtio Transport Options / Virtio Over PCI Bus / PCI Device Layout / Driver auxiliary notification capability}
 \item \ref{devicenormative:Virtio Transport Options / Virtio Over PCI Bus / PCI Device Layout / Shared memory capability}
 \item \ref{devicenormative:Virtio Transport Options / Virtio Over PCI Bus / PCI Device Layout / PCI configuration access capability}
 \item \ref{devicenormative:Virtio Transport Options / Virtio Over PCI Bus / PCI-specific Initialization And Device Operation / Device Initialization / Non-transitional Device With Legacy Driver}
diff --git a/content.tex b/content.tex
index 85980ac..5430324 100644
--- a/content.tex
+++ b/content.tex
@@ -719,6 +719,8 @@ \subsection{Virtio Structure PCI Capabilities}\label{sec:Virtio Transport Option
 \item ISR Status
 \item Device-specific configuration (optional)
 \item PCI configuration access
+\item Driver auxiliary notifications (optional)
+\item Device auxiliary notifications (optional)
 \end{itemize}
 
 Each structure can be mapped by a Base Address register (BAR) belonging to
@@ -765,19 +767,23 @@ \subsection{Virtio Structure PCI Capabilities}\label{sec:Virtio Transport Option
 
 \begin{lstlisting}
 /* Common configuration */
-#define VIRTIO_PCI_CAP_COMMON_CFG        1
+#define VIRTIO_PCI_CAP_COMMON_CFG             1
 /* Notifications */
-#define VIRTIO_PCI_CAP_NOTIFY_CFG        2
+#define VIRTIO_PCI_CAP_NOTIFY_CFG             2
 /* ISR Status */
-#define VIRTIO_PCI_CAP_ISR_CFG           3
+#define VIRTIO_PCI_CAP_ISR_CFG                3
 /* Device specific configuration */
-#define VIRTIO_PCI_CAP_DEVICE_CFG        4
+#define VIRTIO_PCI_CAP_DEVICE_CFG             4
 /* PCI configuration access */
-#define VIRTIO_PCI_CAP_PCI_CFG           5
+#define VIRTIO_PCI_CAP_PCI_CFG                5
+/* Device auxiliary notification */
+#define VIRTIO_PCI_CAP_DEVICE_AUX_NOTIFY_CFG  6
+/* Driver auxiliary notification */
+#define VIRTIO_PCI_CAP_DRIVER_AUX_NOTIFY_CFG  7
 /* Shared memory region */
-#define VIRTIO_PCI_CAP_SHARED_MEMORY_CFG 8
+#define VIRTIO_PCI_CAP_SHARED_MEMORY_CFG      8
 /* Vendor-specific data */
-#define VIRTIO_PCI_CAP_VENDOR_CFG        9
+#define VIRTIO_PCI_CAP_VENDOR_CFG             9
 \end{lstlisting}
 
         Any other value is reserved for future use.
@@ -1212,6 +1218,92 @@ \subsubsection{Device-specific configuration}\label{sec:Virtio Transport Options
 
 The \field{offset} for the device-specific configuration MUST be 4-byte aligned.
 
+\subsubsection{Device auxiliary notification capability}\label{sec:Virtio Transport Options / Virtio Over PCI Bus / PCI Device Layout / Device auxiliary notification capability}
+
+The device auxiliary notification \ref{sec:Basic Facilities of a Virtio Device / Notifications}
+location is found using the VIRTIO_PCI_CAP_DEVICE_AUX_NOTIFY_CFG capability. This
+capability is immediately followed by an additional field, like so:
+
+\begin{lstlisting}
+struct virtio_pci_dev_aux_notification_cap {
+        struct virtio_pci_cap cap;
+        le32 dev_aux_notification_off_multiplier;
+};
+\end{lstlisting}
+
+The device auxiliary notification address within a BAR is calculated as follows:
+
+\begin{lstlisting}
+cap.offset + dev_aux_notification_idx * dev_aux_notification_off_multiplier
+\end{lstlisting}
+
+The \field{cap.offset} and \field{dev_aux_notification_off_multiplier} are taken
+from the device auxiliary notification capability structure above, and the
+\field{dev_aux_notification_idx} is the device auxiliary notification index. There is no restriction for
+the mapping between device auxiliary notifications and dev_aux_notification_idx. The mapping is
+device-specific. One possible mapping would be to use the integers 0 to
+N-1 as the device auxiliary notification indices for a total of N device auxiliary notifications.
+The total number of device auxiliary notifications exposed by the device is also device-specific.
+
+\devicenormative{\paragraph}{Device auxiliary notification capability}{Virtio Transport Options / Virtio Over PCI Bus / PCI Device Layout / Device auxiliary notification capability}
+
+The \field{cap.offset} MUST be 2-byte aligned.
+
+The device MUST either present \field{dev_aux_notification_off_multiplier} as an
+even power of 2, or present \field{dev_aux_notification_off_multiplier} as 0.
+
+The value \field{cap.length} presented by the device MUST be at least 2
+and MUST be large enough to support device auxiliary notification offsets for all supported
+device auxiliary notifications in all possible configurations.
+
+The value \field{cap.length} presented by the device MUST satisfy:
+
+\begin{lstlisting}
+cap.length >= max_dev_aux_notification_idx * dev_aux_notification_off_multiplier + 2
+\end{lstlisting}
+
+where \field{max_dev_aux_notification_idx} is the maximum device auxiliary notification index and is
+dependent on the device.
+
+\subsubsection{Driver auxiliary notification capability}\label{sec:Virtio Transport Options / Virtio Over PCI Bus / PCI Device Layout / Driver auxiliary notification capability}
+
+The Driver auxiliary notification
+\ref{sec:Basic Facilities of a Virtio Device / Notifications} location
+is found using the VIRTIO_PCI_CAP_DRIVER_AUX_NOTIFY_CFG capability. The
+driver auxiliary notification structure allows MSI-X vectors to be
+configured for notification interrupts. If MSI-X is not available, bit 2
+of the ISR status \ref{sec:Virtio Transport Options / Virtio Over PCI
+Bus / PCI Device Layout / ISR status capability} indicates that a
+driver auxiliary notification occurred.
+
+The driver auxiliary notification structure is the following:
+
+\begin{lstlisting}
+struct virtio_pci_driver_aux_notification_cfg {
+	le16 driver_aux_notification_select;              /* read-write */
+	le16 driver_aux_notification_msix_vector;         /* read-write */
+};
+\end{lstlisting}
+
+The driver indicates which notification is of interest by writing the
+\field{driver_aux_notification_select} field. The driver then writes the MSI-X
+vector or VIRTIO_MSI_NO_VECTOR to \field{driver_aux_notification_msix_vector} to
+change the MSI-X vector for that notification.
+
+The mapping between notifications and notification indices is
+device-specific. The total number of notifications is also
+device-specific.
+
+\devicenormative{\paragraph}{Driver auxiliary notification capability}{Virtio Transport Options / Virtio Over PCI Bus / PCI Device Layout / Driver auxiliary notification capability}
+
+Device MUST ignore writes to \field{driver_aux_notification_msix_vector} if the
+value written to \field{driver_aux_notification_select} is not a valid notification
+index.
+
+Device MUST return VIRTIO_MSI_NO_VECTOR for reads from
+\field{driver_aux_notification_msix_vector} if the value written to
+\field{driver_aux_notification_select} is not a valid notification index.
+
 \subsubsection{Shared memory capability}\label{sec:Virtio Transport Options / Virtio Over PCI Bus / PCI Device Layout / Shared memory capability}
 
 Shared memory regions \ref{sec:Basic Facilities of a Virtio
@@ -1523,15 +1615,17 @@ \subsubsection{Device Initialization}\label{sec:Virtio Transport Options / Virti
 \paragraph{MSI-X Vector Configuration}\label{sec:Virtio Transport Options / Virtio Over PCI Bus / PCI-specific Initialization And Device Operation / Device Initialization / MSI-X Vector Configuration}
 
 When MSI-X capability is present and enabled in the device
-(through standard PCI configuration space) \field{config_msix_vector} and \field{queue_msix_vector} are used to map configuration change and queue
-interrupts to MSI-X vectors. In this case, the ISR Status is unused.
+(through standard PCI configuration space) \field{config_msix_vector},
+\field{queue_msix_vector} and \field{driver_aux_notification_msix_vector} are used
+to map configuration change, queue and device-specific interrupts to
+MSI-X vectors respectively. In this case, the ISR Status is unused.
 
 Writing a valid MSI-X Table entry number, 0 to 0x7FF, to
-\field{config_msix_vector}/\field{queue_msix_vector} maps interrupts triggered
-by the configuration change/selected queue events respectively to
-the corresponding MSI-X vector. To disable interrupts for an
-event type, the driver unmaps this event by writing a special NO_VECTOR
-value:
+\field{config_msix_vector}/\field{queue_msix_vector}/\field{driver_aux_notification_msix_vector}
+maps interrupts triggered by the configuration change/selected
+queue/device-specific events respectively to the corresponding MSI-X
+vector. To disable interrupts for an event type, the driver unmaps this
+event by writing a special NO_VECTOR value:
 
 \begin{lstlisting}
 /* Vector value used to disable MSI for queue */
@@ -1558,16 +1652,18 @@ \subsubsection{Device Initialization}\label{sec:Virtio Transport Options / Virti
 vector 0 to MSI-X \field{Table Size}.
 Device MUST support unmapping any event type.
 
-The device MUST return vector mapped to a given event,
-(NO_VECTOR if unmapped) on read of \field{config_msix_vector}/\field{queue_msix_vector}.
-The device MUST have all queue and configuration change
-events are unmapped upon reset.
+The device MUST return the vector mapped to a given event,
+(NO_VECTOR if unmapped) on read of
+\field{config_msix_vector}/\field{queue_msix_vector}/\field{driver_aux_notification_msix_vector}.
+The device MUST have all queue/configuration change/device-specific
+events unmapped upon reset.
 
 Devices SHOULD NOT cause mapping an event to vector to fail
 unless it is impossible for the device to satisfy the mapping
 request.  Devices MUST report mapping
 failures by returning the NO_VECTOR value when the relevant
-\field{config_msix_vector}/\field{queue_msix_vector} field is read. 
+\field{config_msix_vector}/\field{queue_msix_vector}/\field{driver_aux_notification_msix_vector}
+field is read.
 
 \drivernormative{\subparagraph}{MSI-X Vector Configuration}{Virtio Transport Options / Virtio Over PCI Bus / PCI-specific Initialization And Device Operation / Device Initialization / MSI-X Vector Configuration}
 
-- 
2.25.1


---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

[virtio-dev] [PATCH 3/4] content: Introduce driver/device auxiliary notifications for MMIO

[Usama Arif]

This includes the additions to the corresponding device and driver
conformances.

Signed-off-by: Usama Arif <usama.arif@bytedance.com>
Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
Signed-off-by: Nikos Dragazis <ndragazis@arrikto.com>
---
 content.tex | 23 +++++++++++++++++++++--
 1 file changed, 21 insertions(+), 2 deletions(-)

diff --git a/content.tex b/content.tex
index 5430324..0fc50c4 100644
--- a/content.tex
+++ b/content.tex
@@ -2002,6 +2002,9 @@ \subsection{MMIO Device Register Layout}\label{sec:Virtio Transport Options / Vi
         in at least one of the active virtual queues.
       \item [Configuration Change Notification] - bit 1 - the interrupt was
         asserted because the configuration of the device has changed.
+      \item [Device-specific Driver Auxiliary Notification] - bit 2 - the interrupt was
+        asserted because a device-specific event occurred to notify the driver. See
+        \ref{sec:Basic Facilities of a Virtio Device / Notifications}.
     \end{description}
   }
   \hline 
@@ -2074,6 +2077,16 @@ \subsection{MMIO Device Register Layout}\label{sec:Virtio Transport Options / Vi
     apply to the queue selected by writing to \field{QueueSel}.
   }
   \hline
+  \mmioreg{DeviceAuxiliaryNotification}{Device Auxiliary Notifier}{0xd0}{W}{
+    Writing a value to this register triggers a
+    notification to the device. The value written to this register is
+    interpreted as a device auxiliary notification index. The mapping of device auxiliary notifications to
+    device auxiliary notification indices is device-specific. One possible mapping would be
+    to use the integers 0 to N-1 as the device auxiliary notification indices for a total of
+    N device auxiliary notifications. The total number of device auxiliary notifications exposed by the device is
+    also device-specific.
+  }
+  \hline
   \mmioreg{ConfigGeneration}{Configuration atomicity value}{0x0fc}{R}{
     Reading from this register returns a value describing a version of the device-specific configuration space (see \field{Config}).
     The driver can then access the configuration space and, when finished, read \field{ConfigGeneration} again.
@@ -2109,6 +2122,9 @@ \subsection{MMIO Device Register Layout}\label{sec:Virtio Transport Options / Vi
 
 The device MUST NOT access virtual queue contents when \field{QueueReady} is zero (0x0).
 
+The device MUST ignore device auxiliary notifications for invalid device auxiliary notification
+indices.
+
 If VIRTIO_F_RING_RESET has been negotiated, the device MUST present a 0 in
 \field{QueueReady} after the driver has reset the virtqueue via
 \field{QueueReset}.
@@ -2191,6 +2207,8 @@ \subsubsection{Device Initialization}\label{sec:Virtio Transport Options / Virti
 Drivers not expecting shared memory MUST NOT use the shared
 memory registers.
 
+Drivers not expecting device auxiliary notifications MUST NOT use the device auxiliary notification register.
+
 Further initialization MUST follow the procedure described in
 \ref{sec:General Initialization And Device Operation / Device Initialization}~\nameref{sec:General Initialization And Device Operation / Device Initialization}.
 
@@ -2244,8 +2262,9 @@ \subsubsection{Notifications From The Device}\label{sec:Virtio Transport Options
 The memory mapped virtio device is using a single, dedicated
 interrupt signal, which is asserted when at least one of the
 bits described in the description of \field{InterruptStatus}
-is set. This is how the device sends a used buffer notification
-or a configuration change notification to the device.
+is set. This is how the device sends a
+used buffer/configuration change/device-specific notification to the
+device.
 
 \drivernormative{\paragraph}{Notifications From The Device}{Virtio Transport Options / Virtio Over MMIO / MMIO-specific Initialization And Device Operation / Notifications From The Device}
 After receiving an interrupt, the driver MUST read
-- 
2.25.1


---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

[virtio-dev] [PATCH 4/4] vhost-user: add vhost-user device type

[Usama Arif]

The vhost-user device backend facilitates vhost-user device emulation
through vhost-user protocol exchanges and access to shared memory.
Software-defined networking, storage, and other I/O appliances can
provide services through this device.

This device is based on Wei Wang's vhost-pci work.  The virtio
vhost-user device differs from vhost-pci because it is a single virtio
device type that exposes the vhost-user protocol instead of a family of
new virtio device types, one for each vhost-user device type.

This device supports vhost-user backend and vhost-user frontend
reconnection. It also contains a UUID so that vhost-user backend programs
can identify a specific device among many without using bus addresses.

virtio-vhost-user makes use of additional resources introduced in earlier
patches including device aux. notifications, driver aux. notifications,
as well as shared memory.

Signed-off-by: Usama Arif <usama.arif@bytedance.com>
Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
Signed-off-by: Nikos Dragazis <ndragazis@arrikto.com>
---
 conformance.tex       |  27 ++++-
 content.tex           |   3 +
 introduction.tex      |   3 +
 virtio-vhost-user.tex | 259 ++++++++++++++++++++++++++++++++++++++++++
 4 files changed, 288 insertions(+), 4 deletions(-)
 create mode 100644 virtio-vhost-user.tex

diff --git a/conformance.tex b/conformance.tex
index cddaf75..fab49c3 100644
--- a/conformance.tex
+++ b/conformance.tex
@@ -32,8 +32,9 @@ \section{Conformance Targets}\label{sec:Conformance / Conformance Targets}
 \ref{sec:Conformance / Driver Conformance / Memory Driver Conformance},
 \ref{sec:Conformance / Driver Conformance / I2C Adapter Driver Conformance},
 \ref{sec:Conformance / Driver Conformance / SCMI Driver Conformance},
-\ref{sec:Conformance / Driver Conformance / GPIO Driver Conformance} or
-\ref{sec:Conformance / Driver Conformance / PMEM Driver Conformance}.
+\ref{sec:Conformance / Driver Conformance / GPIO Driver Conformance},
+\ref{sec:Conformance / Driver Conformance / PMEM Driver Conformance} or
+\ref{sec:Conformance / Driver Conformance / Vhost-user Backend Driver Conformance}.
 
     \item Clause \ref{sec:Conformance / Legacy Interface: Transitional Device and Transitional Driver Conformance}.
   \end{itemize}
@@ -58,8 +59,9 @@ \section{Conformance Targets}\label{sec:Conformance / Conformance Targets}
 \ref{sec:Conformance / Device Conformance / Memory Device Conformance},
 \ref{sec:Conformance / Device Conformance / I2C Adapter Device Conformance},
 \ref{sec:Conformance / Device Conformance / SCMI Device Conformance},
-\ref{sec:Conformance / Device Conformance / GPIO Device Conformance} or
-\ref{sec:Conformance / Device Conformance / PMEM Device Conformance}.
+\ref{sec:Conformance / Device Conformance / GPIO Device Conformance},
+\ref{sec:Conformance / Device Conformance / PMEM Device Conformance} or
+\ref{sec:Conformance / Device Conformance / Vhost-user Backend Device Conformance}.
 
     \item Clause \ref{sec:Conformance / Legacy Interface: Transitional Device and Transitional Driver Conformance}.
   \end{itemize}
@@ -324,6 +326,15 @@ \section{Conformance Targets}\label{sec:Conformance / Conformance Targets}
 \item \ref{drivernormative:Device Types / PMEM Device / Device Initialization}
 \end{itemize}
 
+\conformance{\subsection}{Vhost-user Backend Driver Conformance}\label{sec:Conformance / Driver Conformance / Vhost-user Backend Driver Conformance}
+
+A vhost-user backend driver MUST conform to the following normative statements:
+
+\begin{itemize}
+\item \ref{drivernormative:Device Types / Vhost-user Device Backend / Device configuration layout}
+\item \ref{drivernormative:Device Types / Vhost-user Device Backend / Device Initialization}
+\end{itemize}
+
 \conformance{\section}{Device Conformance}\label{sec:Conformance / Device Conformance}
 
 A device MUST conform to the following normative statements:
@@ -595,6 +606,14 @@ \section{Conformance Targets}\label{sec:Conformance / Conformance Targets}
 \item \ref{devicenormative:Device Types / PMEM Device / Device Operation / Virtqueue return}
 \end{itemize}
 
+\conformance{\subsection}{Vhost-user Backend Device Conformance}\label{sec:Conformance / Device Conformance / Vhost-user Backend Device Conformance}
+
+A Vhost-user backend device MUST conform to the following normative statements:
+
+\begin{itemize}
+\item \ref{devicenormative:Device Types / Vhost-user Device Backend / Additional Device Resources / Shared Memory layout}
+\end{itemize}
+
 \conformance{\section}{Legacy Interface: Transitional Device and Transitional Driver Conformance}\label{sec:Conformance / Legacy Interface: Transitional Device and Transitional Driver Conformance}
 A conformant implementation MUST be either transitional or
 non-transitional, see \ref{intro:Legacy
diff --git a/content.tex b/content.tex
index 0fc50c4..8bf114d 100644
--- a/content.tex
+++ b/content.tex
@@ -3122,6 +3122,8 @@ \chapter{Device Types}\label{sec:Device Types}
 \hline
 42         &   RDMA device \\
 \hline
+43         &   vhost-user device backend \ \\
+\hline
 \end{tabular}
 
 Some of the devices above are unspecified by this document,
@@ -6878,6 +6880,7 @@ \subsubsection{Legacy Interface: Framing Requirements}\label{sec:Device
 \input{virtio-scmi.tex}
 \input{virtio-gpio.tex}
 \input{virtio-pmem.tex}
+\input{virtio-vhost-user.tex}
 
 \chapter{Reserved Feature Bits}\label{sec:Reserved Feature Bits}
 
diff --git a/introduction.tex b/introduction.tex
index 6d52717..5bd1b95 100644
--- a/introduction.tex
+++ b/introduction.tex
@@ -79,6 +79,9 @@ \section{Normative References}\label{sec:Normative References}
 	\phantomsection\label{intro:SCMI}\textbf{[SCMI]} &
 	Arm System Control and Management Interface, DEN0056,
 	\newline\url{https://developer.arm.com/docs/den0056/c}, version C and any future revisions\\
+	\phantomsection\label{intro:Vhost-user Protocol}\textbf{[Vhost-user Protocol]}
+	& Vhost-user Protocol,
+	\newline\url{https://qemu.readthedocs.io/en/latest/interop/vhost-user.html}\\
 
 \end{longtable}
 
diff --git a/virtio-vhost-user.tex b/virtio-vhost-user.tex
new file mode 100644
index 0000000..303054f
--- /dev/null
+++ b/virtio-vhost-user.tex
@@ -0,0 +1,259 @@
+\section{Vhost-user Device Backend}\label{sec:Device Types / Vhost-user Device
+Backend}
+
+The vhost-user device backend facilitates vhost-user device emulation through
+vhost-user protocol exchanges and access to shared memory. Software-defined
+networking, storage, and other I/O appliances can provide services through this
+device.
+
+This section relies on definitions from the \hyperref[intro:Vhost-user
+Protocol]{Vhost-user Protocol}.  Knowledge of the vhost-user protocol is a
+prerequisite for understanding this device.
+
+The \hyperref[intro:Vhost-user Protocol]{Vhost-user Protocol} was originally
+designed for processes on a single system communicating over UNIX domain
+sockets. The virtio vhost-user device backend allows the vhost-user backend to
+communicate with the vhost-user frontend over the device instead of a UNIX domain
+socket. This allows the backend and frontend to run on two separate systems such
+as a virtual machine and a hypervisor.
+
+The vhost-user backend program exchanges vhost-user protocol messages with the
+vhost-user frontend through this device. How the device implementation
+communicates with the vhost-user frontend is beyond the scope of this
+specification.  One possible device implementation uses a UNIX domain socket to
+relay messages to a vhost-user frontend process running on the same host.
+
+Existing vhost-user backend programs that communicate over UNIX domain sockets
+can support the virtio vhost-user device backend without invasive changes
+because the pre-existing vhost-user wire protocol is used.
+
+\subsection{Device ID}\label{sec:Device Types / Vhost-user Device Backend / Device ID}
+  43
+
+\subsection{Virtqueues}\label{sec:Device Types / Vhost-user Device Backend / Virtqueues}
+
+\begin{description}
+\item[0] rxq (device-to-driver vhost-user protocol messages)
+\item[1] txq (driver-to-device vhost-user protocol messages)
+\end{description}
+
+\subsection{Feature bits}\label{sec:Device Types / Vhost-user Device Backend / Feature bits}
+
+No feature bits are defined at this time.
+
+\subsection{Device configuration layout}\label{sec:Device Types / Vhost-user Device Backend / Device configuration layout}
+
+  All fields of this configuration are always available.
+
+\begin{lstlisting}
+struct virtio_vhost_user_config {
+        le32 status;
+#define VIRTIO_VHOST_USER_STATUS_BACKEND_UP (1 << 0)
+#define VIRTIO_VHOST_USER_STATUS_FRONTEND_UP (1 << 1)
+        le32 max_vhost_queues;
+        u8 uuid[16];
+};
+\end{lstlisting}
+
+\begin{description}
+\item[\field{status}] contains the vhost-user operational status.  The default
+    value of this field is 0.
+
+    The driver sets VIRTIO_VHOST_USER_STATUS_BACKEND_UP to indicate readiness for
+    the vhost-user frontend to connect.  The vhost-user frontend cannot connect
+    unless the driver has set this bit first.
+
+    The device sets VIRTIO_VHOST_USER_STATUS_FRONTEND_UP to indicate that the
+    vhost-user frontend is connected.
+
+    When the driver clears VIRTIO_VHOST_USER_STATUS_BACKEND_UP while the
+    vhost-user frontend is connected, the vhost-user frontend is disconnected.
+
+    When the vhost-user frontend disconnects, both
+    VIRTIO_VHOST_USER_STATUS_BACKEND_UP and VIRTIO_VHOST_USER_STATUS_FRONTEND_UP
+    are cleared by the device.  Communication can be restarted by the driver
+    setting VIRTIO_VHOST_USER_STATUS_BACKEND_UP again.
+
+    A configuration change notification is sent when the device changes
+    this field, unless a write to the field by the driver caused the change.
+
+\item[\field{max_vhost_queues}] is the maximum number of vhost-user queues
+    supported by this device.  This field is always greater than 0.
+
+\item[\field{uuid}] is the Universally Unique Identifier (UUID) for this
+    device. If the device has no UUID then this field contains the nil
+    UUID (all zeroes).  The UUID allows vhost-user backend programs to identify a
+    specific vhost-user device backend among many without relying on bus
+    addresses.
+\end{description}
+
+\drivernormative{\subsubsection}{Device configuration layout}{Device Types / Vhost-user Device Backend / Device configuration layout}
+
+The driver MUST NOT write to device configuration fields other than \field{status}.
+
+The driver MUST NOT set undefined bits in the \field{status} configuration field.
+
+\subsection{Device Initialization}\label{sec:Device Types / Vhost-user Device Backend / Device Initialization}
+
+The driver initializes the rxq/txq virtqueues and then it sets
+VIRTIO_VHOST_USER_STATUS_BACKEND_UP to the \field{status} field of the device
+configuration structure.
+
+\drivernormative{\subsubsection}{Device Initialization}{Device Types / Vhost-user Device Backend / Device Initialization}
+
+The driver SHOULD check the \field{max_vhost_queues} configuration field to
+determine how many queues the vhost-user backend will be able to support.
+
+The driver SHOULD fetch the \field{uuid} configuration field to allow
+vhost-user backend programs to identify a specific device among many.
+
+The driver SHOULD place at least one buffer in rxq before setting the
+VIRTIO_VHOST_USER_STATUS_BACKEND_UP bit in the \field{status} configuration field.
+
+The driver MUST handle rxq virtqueue notifications that occur before the
+configuration change notification.  It is possible that a vhost-user protocol
+message from the vhost-user frontend arrives before the driver has seen the
+configuration change notification for the VIRTIO_VHOST_USER_STATUS_FRONTEND_UP
+\field{status} change.
+
+\subsection{Device Operation}\label{sec:Device Types / Vhost-user Device Backend / Device Operation}
+
+Device operation consists of operating request queues and response queues.
+
+\subsubsection{Device Operation: Request Queues}\label{sec:Device Types / Vhost-user Device Backend / Device Operation / Device Operation: RX/TX Queues}
+
+The driver receives vhost-user protocol messages from the vhost-user frontend on
+rxq. The driver sends responses to the vhost-user frontend on txq.
+
+The driver sends backend-initiated requests on txq. The driver receives
+responses from the vhost-user frontend on rxq.
+
+All virtqueues offer in-order guaranteed delivery semantics for vhost-user
+protocol messages.
+
+Each buffer is a vhost-user protocol message as defined by the
+\hyperref[intro:Vhost-user Protocol]{Vhost-user Protocol}.  In order to enable
+cross-endian communication, all message fields are little-endian instead of the
+native byte order normally used by the protocol.
+
+The appropriate size of rxq buffers is at least as large as the largest message
+defined by the \hyperref[intro:Vhost-user Protocol]{Vhost-user Protocol}
+standard version that the driver supports.  If the vhost-user frontend sends a
+message that is too large for an rxq buffer, then DEVICE_NEEDS_RESET is set and
+the driver must reset the device.
+
+File descriptor passing is handled differently by the vhost-user device
+backend. When a frontend-initiated message is received that carries one or more file
+descriptors according to the vhost-user protocol, additional device resources
+become available to the driver.
+
+\subsection{Additional Device Resources}\label{sec:Device Types / Vhost-user Device Backend / Additional Device Resources}
+
+The vhost-user device backend uses the following facilities from virtio device
+\ref{sec:Basic Facilities of a Virtio Device} for the vhost-user frontend and
+backend to exchange notifications and data through the device:
+
+\begin{description}
+  \item[Device auxiliary notification] \ref{sec:Basic Facilities of a Virtio Device / Notifications}
+The driver signals the vhost-user frontend through device auxiliary notifications. The signal does not
+carry any data, it is purely an event.
+  \item[Driver auxiliary notification] \ref{sec:Basic Facilities of a Virtio Device / Notifications}
+The vhost-user frontend signals the driver for events besides virtqueue activity
+and configuration changes by sending driver auxiliary notification.
+  \item[Shared memory] \ref{sec:Basic Facilities of a Virtio Device / Shared Memory Regions}
+The vhost-user frontend gives access to memory that can be mapped by the driver.
+\end{description}
+
+\subsubsection{Device auxiliary notifications}\label{sec:Device Types / Vhost-user Device Backend / Additional Device Resources / Device auxiliary notifications}
+
+The vhost-user device backend provides all (or part) of the following device auxiliary notifications:
+
+\begin{description}
+\item[0] Vring call for vhost-user queue 0
+\item[\ldots]
+\item[N-1] Vring call for vhost-user queue N-1
+\item[N] Vring err for vhost-user queue 0
+\item[\ldots]
+\item[2N-1] Vring err for vhost-user queue N-1
+\item[2N] Log
+\end{description}
+
+where N is the number of the vhost-user virtqueues.
+
+\subsubsection{Driver auxiliary notifications}\label{sec:Device Types / Vhost-user Device Backend / Additional Device Resources / Driver auxiliary notifications}
+
+The vhost-user device backend provides all (or part) of the following driver auxiliary notifications:
+
+\begin{description}
+\item[0] Vring kick for vhost-user queue 0
+\item[\ldots]
+\item[N-1] Vring kick for vhost-user queue N-1
+\end{description}
+
+where N is the number of the vhost-user virtqueues.
+
+\subsubsection{Shared Memory}\label{sec:Device Types / Vhost-user Device Backend / Additional Device Resources / Shared Memory}
+
+The vhost-user device backend provides all (or part) of the following shared memory regions:
+
+\begin{description}
+\item[0] Vhost-user memory region 0
+\item[1] Vhost-user memory region 1
+\item[\ldots]
+\item[M-1] Vhost-user memory region M-1
+\item[M] Log memory region
+\end{description}
+
+where M is the total number of memory regions shared.
+
+\devicenormative{\paragraph}{Shared Memory layout}{Device Types / Vhost-user Device Backend / Additional Device Resources / Shared Memory layout}
+
+The device exports all memory regions reported by the vhost-user frontend as a
+single shared memory region \ref{sec:Basic Facilities of a Virtio Device /
+Shared Memory Regions}.
+
+The size of this shared memory region exported by the device MUST be at least
+as much as the sum of the sizes of all the memory regions reported by the
+vhost-user frontend.
+
+The memory regions exported by the device MUST be laid out in the same order
+in which they are reported by the frontend with vhost-user messages.
+
+The offsets in which the memory regions are mapped inside the shared memory
+region MUST be the following:
+
+\begin{description}
+\item[0] Offset for vhost-user memory region 0
+\item[SIZE0] Offset for vhost-user memory region 1
+\item[\ldots]
+\item[SIZE0 + SIZE1 + \ldots] Offset for vhost-user memory region M
+\end{description}
+    
+where SIZEi is the size of the vhost-user memory region i.
+
+\subsubsection{Availability of Additional Resources}\label{sec:Device Types / Vhost-user Device Backend / Additional Device Resources / Availability of Additional Resources}
+
+The following vhost-user protocol messages convey access to additional device
+resources:
+
+\begin{description}
+\item[VHOST_USER_SET_MEM_TABLE] Contents of vhost-user memory regions are
+available to the driver as device memory. Region contents are laid out in the
+same order as the vhost-user memory region list.
+\item[VHOST_USER_SET_LOG_BASE] Contents of the log memory region are available
+to the driver as device memory.
+\item[VHOST_USER_SET_LOG_FD] The log device auxiliary notification is available to the driver.
+Writes to the log device auxiliary notification before this message is received produce no effect.
+\item[VHOST_USER_SET_VRING_KICK] The vring kick notification for this queue is
+available to the driver. The first notification may occur before the driver has
+processed this message.
+\item[VHOST_USER_SET_VRING_CALL] The vring call device auxiliary notification for this queue is
+available to the driver. Writes to the vring call device auxiliary notification before this message
+is received produce no effect.
+\item[VHOST_USER_SET_VRING_ERR] The vring err device auxiliary notification for this queue is
+available to the driver. Writes to the vring err device auxiliary notification before this message
+is received produce no effect.
+\item[VHOST_USER_SET_SLAVE_REQ_FD] The driver may send vhost-user protocol
+backend messages on txq. Backend-initiated messages put onto txq before this
+message is received are discarded by the device.
+\end{description}
-- 
2.25.1


---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

[virtio-dev] Re: [PATCH 1/4] content: Introduce driver/device auxiliary notifications

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 481 bytes --]

On Wed, Mar 30, 2022 at 04:26:56PM +0100, Usama Arif wrote:
> +Available buffer notifications and device auxiliary notifications
> +are sent by the driver, the recipient is the device. Available buffer
> +notifications indicate that a buffer may have been made available on the
> +virtqueue designated by the notification; device auxiliary
> +notifcations allow the driver to send notifcations other than available

s/notifcations/notifications/

Please spell-check these commits.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

[virtio-dev] Re: [PATCH 2/4] content: Introduce driver/device aux. notification cfg type for PCI

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 4631 bytes --]

On Wed, Mar 30, 2022 at 04:26:57PM +0100, Usama Arif wrote:
> +The \field{cap.offset} and \field{dev_aux_notification_off_multiplier} are taken
> +from the device auxiliary notification capability structure above, and the
> +\field{dev_aux_notification_idx} is the device auxiliary notification index. There is no restriction for
> +the mapping between device auxiliary notifications and dev_aux_notification_idx. The mapping is
> +device-specific. One possible mapping would be to use the integers 0 to
> +N-1 as the device auxiliary notification indices for a total of N device auxiliary notifications.
> +The total number of device auxiliary notifications exposed by the device is also device-specific.

The second half of the paragraph could be shortened to "The number of
device auxiliary notifications and their purpose is device-specific". I
don't think discussing the device-specific mapping and that it may be
0 to N-1 or something else adds anything.

> +
> +\devicenormative{\paragraph}{Device auxiliary notification capability}{Virtio Transport Options / Virtio Over PCI Bus / PCI Device Layout / Device auxiliary notification capability}
> +
> +The \field{cap.offset} MUST be 2-byte aligned.
> +
> +The device MUST either present \field{dev_aux_notification_off_multiplier} as an
> +even power of 2, or present \field{dev_aux_notification_off_multiplier} as 0.
> +
> +The value \field{cap.length} presented by the device MUST be at least 2
> +and MUST be large enough to support device auxiliary notification offsets for all supported
> +device auxiliary notifications in all possible configurations.
> +
> +The value \field{cap.length} presented by the device MUST satisfy:
> +
> +\begin{lstlisting}
> +cap.length >= max_dev_aux_notification_idx * dev_aux_notification_off_multiplier + 2
> +\end{lstlisting}
> +
> +where \field{max_dev_aux_notification_idx} is the maximum device auxiliary notification index and is
> +dependent on the device.

One thought about dev aux notifications: the size is limited to 2 bytes.
Sometimes it's useful to communicate information (i.e. 1, 2, or 4 bytes)
as part of the notification (virtio-vhost-user may not need it but other
users might). Perhaps the width should be device-specific and not fixed
to 2?

> +
> +\subsubsection{Driver auxiliary notification capability}\label{sec:Virtio Transport Options / Virtio Over PCI Bus / PCI Device Layout / Driver auxiliary notification capability}
> +
> +The Driver auxiliary notification
> +\ref{sec:Basic Facilities of a Virtio Device / Notifications} location
> +is found using the VIRTIO_PCI_CAP_DRIVER_AUX_NOTIFY_CFG capability. The
> +driver auxiliary notification structure allows MSI-X vectors to be
> +configured for notification interrupts. If MSI-X is not available, bit 2
> +of the ISR status \ref{sec:Virtio Transport Options / Virtio Over PCI
> +Bus / PCI Device Layout / ISR status capability} indicates that a
> +driver auxiliary notification occurred.

Is there any way to determine which driver auxiliary notification
occurred when MSI-X is not available? Please make this clear in the
text.

> +
> +The driver auxiliary notification structure is the following:
> +
> +\begin{lstlisting}
> +struct virtio_pci_driver_aux_notification_cfg {
> +	le16 driver_aux_notification_select;              /* read-write */
> +	le16 driver_aux_notification_msix_vector;         /* read-write */
> +};
> +\end{lstlisting}
> +
> +The driver indicates which notification is of interest by writing the
> +\field{driver_aux_notification_select} field. The driver then writes the MSI-X
> +vector or VIRTIO_MSI_NO_VECTOR to \field{driver_aux_notification_msix_vector} to
> +change the MSI-X vector for that notification.
> +
> +The mapping between notifications and notification indices is
> +device-specific. The total number of notifications is also
> +device-specific.
> +
> +\devicenormative{\paragraph}{Driver auxiliary notification capability}{Virtio Transport Options / Virtio Over PCI Bus / PCI Device Layout / Driver auxiliary notification capability}
> +
> +Device MUST ignore writes to \field{driver_aux_notification_msix_vector} if the
> +value written to \field{driver_aux_notification_select} is not a valid notification
> +index.
> +
> +Device MUST return VIRTIO_MSI_NO_VECTOR for reads from
> +\field{driver_aux_notification_msix_vector} if the value written to
> +\field{driver_aux_notification_select} is not a valid notification index.

Maybe the spec should say something about the minimum number of MSI-X
vectors presented by the device?

  num_vectors >= 1 /* config change */ + num_vqs + num_driver_aux_notifications

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

[virtio-dev] Re: [PATCH 3/4] content: Introduce driver/device auxiliary notifications for MMIO

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 2043 bytes --]

On Wed, Mar 30, 2022 at 04:26:58PM +0100, Usama Arif wrote:
> @@ -2074,6 +2077,16 @@ \subsection{MMIO Device Register Layout}\label{sec:Virtio Transport Options / Vi
>      apply to the queue selected by writing to \field{QueueSel}.
>    }
>    \hline
> +  \mmioreg{DeviceAuxiliaryNotification}{Device Auxiliary Notifier}{0xd0}{W}{
> +    Writing a value to this register triggers a
> +    notification to the device. The value written to this register is
> +    interpreted as a device auxiliary notification index. The mapping of device auxiliary notifications to
> +    device auxiliary notification indices is device-specific. One possible mapping would be
> +    to use the integers 0 to N-1 as the device auxiliary notification indices for a total of
> +    N device auxiliary notifications. The total number of device auxiliary notifications exposed by the device is
> +    also device-specific.
> +  }

Hmm...this is slightly different from the PCI transport's approach.
There each dev aux notification has its own hardware register. That has
advantages:
1. The notification may carry a value with it because the driver writes
   a value to the register and the device can interpret that value. That
   can be used as a performance optimization to reduce the number of
   hardware register writes performed by the driver or DMA transfers
   initiated by the device.
2. The dev aux notification registers can be laid out on separate memory
   pages. This is useful in passthrough scenarios like nested
   virtualization or slicing up a device in software and passing a
   subset to an untrusted process. The MMU only offers page-level
   protection, so MMIO registers that are colocated within the same
   memory page cannot be selectively passed through to untrusted
   software.

It would be nice if PCI and MMIO exposed dev aux notifications in a
similar way.

I haven't followed the history of virtio-mmio closely, so I'm not sure
if the same approach that was used for PCI is feasible here. Maybe
someone else can comment?

Stefan

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

[virtio-dev] Re: [PATCH 4/4] vhost-user: add vhost-user device type

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 20180 bytes --]

On Wed, Mar 30, 2022 at 04:26:59PM +0100, Usama Arif wrote:
> The vhost-user device backend facilitates vhost-user device emulation
> through vhost-user protocol exchanges and access to shared memory.
> Software-defined networking, storage, and other I/O appliances can
> provide services through this device.
> 
> This device is based on Wei Wang's vhost-pci work.  The virtio
> vhost-user device differs from vhost-pci because it is a single virtio
> device type that exposes the vhost-user protocol instead of a family of
> new virtio device types, one for each vhost-user device type.
> 
> This device supports vhost-user backend and vhost-user frontend
> reconnection. It also contains a UUID so that vhost-user backend programs
> can identify a specific device among many without using bus addresses.
> 
> virtio-vhost-user makes use of additional resources introduced in earlier
> patches including device aux. notifications, driver aux. notifications,
> as well as shared memory.
> 
> Signed-off-by: Usama Arif <usama.arif@bytedance.com>
> Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
> Signed-off-by: Nikos Dragazis <ndragazis@arrikto.com>
> ---
>  conformance.tex       |  27 ++++-
>  content.tex           |   3 +
>  introduction.tex      |   3 +
>  virtio-vhost-user.tex | 259 ++++++++++++++++++++++++++++++++++++++++++
>  4 files changed, 288 insertions(+), 4 deletions(-)
>  create mode 100644 virtio-vhost-user.tex
> 
> diff --git a/conformance.tex b/conformance.tex
> index cddaf75..fab49c3 100644
> --- a/conformance.tex
> +++ b/conformance.tex
> @@ -32,8 +32,9 @@ \section{Conformance Targets}\label{sec:Conformance / Conformance Targets}
>  \ref{sec:Conformance / Driver Conformance / Memory Driver Conformance},
>  \ref{sec:Conformance / Driver Conformance / I2C Adapter Driver Conformance},
>  \ref{sec:Conformance / Driver Conformance / SCMI Driver Conformance},
> -\ref{sec:Conformance / Driver Conformance / GPIO Driver Conformance} or
> -\ref{sec:Conformance / Driver Conformance / PMEM Driver Conformance}.
> +\ref{sec:Conformance / Driver Conformance / GPIO Driver Conformance},
> +\ref{sec:Conformance / Driver Conformance / PMEM Driver Conformance} or
> +\ref{sec:Conformance / Driver Conformance / Vhost-user Backend Driver Conformance}.
>  
>      \item Clause \ref{sec:Conformance / Legacy Interface: Transitional Device and Transitional Driver Conformance}.
>    \end{itemize}
> @@ -58,8 +59,9 @@ \section{Conformance Targets}\label{sec:Conformance / Conformance Targets}
>  \ref{sec:Conformance / Device Conformance / Memory Device Conformance},
>  \ref{sec:Conformance / Device Conformance / I2C Adapter Device Conformance},
>  \ref{sec:Conformance / Device Conformance / SCMI Device Conformance},
> -\ref{sec:Conformance / Device Conformance / GPIO Device Conformance} or
> -\ref{sec:Conformance / Device Conformance / PMEM Device Conformance}.
> +\ref{sec:Conformance / Device Conformance / GPIO Device Conformance},
> +\ref{sec:Conformance / Device Conformance / PMEM Device Conformance} or
> +\ref{sec:Conformance / Device Conformance / Vhost-user Backend Device Conformance}.
>  
>      \item Clause \ref{sec:Conformance / Legacy Interface: Transitional Device and Transitional Driver Conformance}.
>    \end{itemize}
> @@ -324,6 +326,15 @@ \section{Conformance Targets}\label{sec:Conformance / Conformance Targets}
>  \item \ref{drivernormative:Device Types / PMEM Device / Device Initialization}
>  \end{itemize}
>  
> +\conformance{\subsection}{Vhost-user Backend Driver Conformance}\label{sec:Conformance / Driver Conformance / Vhost-user Backend Driver Conformance}
> +
> +A vhost-user backend driver MUST conform to the following normative statements:
> +
> +\begin{itemize}
> +\item \ref{drivernormative:Device Types / Vhost-user Device Backend / Device configuration layout}
> +\item \ref{drivernormative:Device Types / Vhost-user Device Backend / Device Initialization}
> +\end{itemize}
> +
>  \conformance{\section}{Device Conformance}\label{sec:Conformance / Device Conformance}
>  
>  A device MUST conform to the following normative statements:
> @@ -595,6 +606,14 @@ \section{Conformance Targets}\label{sec:Conformance / Conformance Targets}
>  \item \ref{devicenormative:Device Types / PMEM Device / Device Operation / Virtqueue return}
>  \end{itemize}
>  
> +\conformance{\subsection}{Vhost-user Backend Device Conformance}\label{sec:Conformance / Device Conformance / Vhost-user Backend Device Conformance}
> +
> +A Vhost-user backend device MUST conform to the following normative statements:
> +
> +\begin{itemize}
> +\item \ref{devicenormative:Device Types / Vhost-user Device Backend / Additional Device Resources / Shared Memory layout}
> +\end{itemize}
> +
>  \conformance{\section}{Legacy Interface: Transitional Device and Transitional Driver Conformance}\label{sec:Conformance / Legacy Interface: Transitional Device and Transitional Driver Conformance}
>  A conformant implementation MUST be either transitional or
>  non-transitional, see \ref{intro:Legacy
> diff --git a/content.tex b/content.tex
> index 0fc50c4..8bf114d 100644
> --- a/content.tex
> +++ b/content.tex
> @@ -3122,6 +3122,8 @@ \chapter{Device Types}\label{sec:Device Types}
>  \hline
>  42         &   RDMA device \\
>  \hline
> +43         &   vhost-user device backend \ \\
> +\hline
>  \end{tabular}
>  
>  Some of the devices above are unspecified by this document,
> @@ -6878,6 +6880,7 @@ \subsubsection{Legacy Interface: Framing Requirements}\label{sec:Device
>  \input{virtio-scmi.tex}
>  \input{virtio-gpio.tex}
>  \input{virtio-pmem.tex}
> +\input{virtio-vhost-user.tex}
>  
>  \chapter{Reserved Feature Bits}\label{sec:Reserved Feature Bits}
>  
> diff --git a/introduction.tex b/introduction.tex
> index 6d52717..5bd1b95 100644
> --- a/introduction.tex
> +++ b/introduction.tex
> @@ -79,6 +79,9 @@ \section{Normative References}\label{sec:Normative References}
>  	\phantomsection\label{intro:SCMI}\textbf{[SCMI]} &
>  	Arm System Control and Management Interface, DEN0056,
>  	\newline\url{https://developer.arm.com/docs/den0056/c}, version C and any future revisions\\
> +	\phantomsection\label{intro:Vhost-user Protocol}\textbf{[Vhost-user Protocol]}
> +	& Vhost-user Protocol,
> +	\newline\url{https://qemu.readthedocs.io/en/latest/interop/vhost-user.html}\\
>  
>  \end{longtable}
>  
> diff --git a/virtio-vhost-user.tex b/virtio-vhost-user.tex
> new file mode 100644
> index 0000000..303054f
> --- /dev/null
> +++ b/virtio-vhost-user.tex
> @@ -0,0 +1,259 @@
> +\section{Vhost-user Device Backend}\label{sec:Device Types / Vhost-user Device
> +Backend}
> +
> +The vhost-user device backend facilitates vhost-user device emulation through
> +vhost-user protocol exchanges and access to shared memory. Software-defined
> +networking, storage, and other I/O appliances can provide services through this
> +device.
> +
> +This section relies on definitions from the \hyperref[intro:Vhost-user
> +Protocol]{Vhost-user Protocol}.  Knowledge of the vhost-user protocol is a
> +prerequisite for understanding this device.
> +
> +The \hyperref[intro:Vhost-user Protocol]{Vhost-user Protocol} was originally
> +designed for processes on a single system communicating over UNIX domain
> +sockets. The virtio vhost-user device backend allows the vhost-user backend to
> +communicate with the vhost-user frontend over the device instead of a UNIX domain
> +socket. This allows the backend and frontend to run on two separate systems such
> +as a virtual machine and a hypervisor.
> +
> +The vhost-user backend program exchanges vhost-user protocol messages with the
> +vhost-user frontend through this device. How the device implementation
> +communicates with the vhost-user frontend is beyond the scope of this
> +specification.  One possible device implementation uses a UNIX domain socket to
> +relay messages to a vhost-user frontend process running on the same host.
> +
> +Existing vhost-user backend programs that communicate over UNIX domain sockets
> +can support the virtio vhost-user device backend without invasive changes
> +because the pre-existing vhost-user wire protocol is used.
> +
> +\subsection{Device ID}\label{sec:Device Types / Vhost-user Device Backend / Device ID}
> +  43
> +
> +\subsection{Virtqueues}\label{sec:Device Types / Vhost-user Device Backend / Virtqueues}
> +
> +\begin{description}
> +\item[0] rxq (device-to-driver vhost-user protocol messages)
> +\item[1] txq (driver-to-device vhost-user protocol messages)
> +\end{description}
> +
> +\subsection{Feature bits}\label{sec:Device Types / Vhost-user Device Backend / Feature bits}
> +
> +No feature bits are defined at this time.
> +
> +\subsection{Device configuration layout}\label{sec:Device Types / Vhost-user Device Backend / Device configuration layout}
> +
> +  All fields of this configuration are always available.
> +
> +\begin{lstlisting}
> +struct virtio_vhost_user_config {
> +        le32 status;
> +#define VIRTIO_VHOST_USER_STATUS_BACKEND_UP (1 << 0)
> +#define VIRTIO_VHOST_USER_STATUS_FRONTEND_UP (1 << 1)
> +        le32 max_vhost_queues;
> +        u8 uuid[16];
> +};
> +\end{lstlisting}
> +
> +\begin{description}
> +\item[\field{status}] contains the vhost-user operational status.  The default
> +    value of this field is 0.
> +
> +    The driver sets VIRTIO_VHOST_USER_STATUS_BACKEND_UP to indicate readiness for
> +    the vhost-user frontend to connect.  The vhost-user frontend cannot connect
> +    unless the driver has set this bit first.
> +
> +    The device sets VIRTIO_VHOST_USER_STATUS_FRONTEND_UP to indicate that the
> +    vhost-user frontend is connected.
> +
> +    When the driver clears VIRTIO_VHOST_USER_STATUS_BACKEND_UP while the
> +    vhost-user frontend is connected, the vhost-user frontend is disconnected.
> +
> +    When the vhost-user frontend disconnects, both
> +    VIRTIO_VHOST_USER_STATUS_BACKEND_UP and VIRTIO_VHOST_USER_STATUS_FRONTEND_UP
> +    are cleared by the device.  Communication can be restarted by the driver
> +    setting VIRTIO_VHOST_USER_STATUS_BACKEND_UP again.
> +
> +    A configuration change notification is sent when the device changes
> +    this field, unless a write to the field by the driver caused the change.
> +
> +\item[\field{max_vhost_queues}] is the maximum number of vhost-user queues
> +    supported by this device.  This field is always greater than 0.
> +
> +\item[\field{uuid}] is the Universally Unique Identifier (UUID) for this
> +    device. If the device has no UUID then this field contains the nil
> +    UUID (all zeroes).  The UUID allows vhost-user backend programs to identify a
> +    specific vhost-user device backend among many without relying on bus
> +    addresses.
> +\end{description}
> +
> +\drivernormative{\subsubsection}{Device configuration layout}{Device Types / Vhost-user Device Backend / Device configuration layout}
> +
> +The driver MUST NOT write to device configuration fields other than \field{status}.
> +
> +The driver MUST NOT set undefined bits in the \field{status} configuration field.
> +
> +\subsection{Device Initialization}\label{sec:Device Types / Vhost-user Device Backend / Device Initialization}
> +
> +The driver initializes the rxq/txq virtqueues and then it sets
> +VIRTIO_VHOST_USER_STATUS_BACKEND_UP to the \field{status} field of the device
> +configuration structure.
> +
> +\drivernormative{\subsubsection}{Device Initialization}{Device Types / Vhost-user Device Backend / Device Initialization}
> +
> +The driver SHOULD check the \field{max_vhost_queues} configuration field to
> +determine how many queues the vhost-user backend will be able to support.
> +
> +The driver SHOULD fetch the \field{uuid} configuration field to allow
> +vhost-user backend programs to identify a specific device among many.
> +
> +The driver SHOULD place at least one buffer in rxq before setting the
> +VIRTIO_VHOST_USER_STATUS_BACKEND_UP bit in the \field{status} configuration field.
> +
> +The driver MUST handle rxq virtqueue notifications that occur before the
> +configuration change notification.  It is possible that a vhost-user protocol
> +message from the vhost-user frontend arrives before the driver has seen the
> +configuration change notification for the VIRTIO_VHOST_USER_STATUS_FRONTEND_UP
> +\field{status} change.
> +
> +\subsection{Device Operation}\label{sec:Device Types / Vhost-user Device Backend / Device Operation}
> +
> +Device operation consists of operating request queues and response queues.
> +
> +\subsubsection{Device Operation: Request Queues}\label{sec:Device Types / Vhost-user Device Backend / Device Operation / Device Operation: RX/TX Queues}
> +
> +The driver receives vhost-user protocol messages from the vhost-user frontend on
> +rxq. The driver sends responses to the vhost-user frontend on txq.
> +
> +The driver sends backend-initiated requests on txq. The driver receives
> +responses from the vhost-user frontend on rxq.
> +
> +All virtqueues offer in-order guaranteed delivery semantics for vhost-user
> +protocol messages.
> +
> +Each buffer is a vhost-user protocol message as defined by the
> +\hyperref[intro:Vhost-user Protocol]{Vhost-user Protocol}.  In order to enable
> +cross-endian communication, all message fields are little-endian instead of the
> +native byte order normally used by the protocol.
> +
> +The appropriate size of rxq buffers is at least as large as the largest message
> +defined by the \hyperref[intro:Vhost-user Protocol]{Vhost-user Protocol}
> +standard version that the driver supports.  If the vhost-user frontend sends a
> +message that is too large for an rxq buffer, then DEVICE_NEEDS_RESET is set and
> +the driver must reset the device.
> +
> +File descriptor passing is handled differently by the vhost-user device
> +backend. When a frontend-initiated message is received that carries one or more file
> +descriptors according to the vhost-user protocol, additional device resources
> +become available to the driver.
> +
> +\subsection{Additional Device Resources}\label{sec:Device Types / Vhost-user Device Backend / Additional Device Resources}
> +
> +The vhost-user device backend uses the following facilities from virtio device
> +\ref{sec:Basic Facilities of a Virtio Device} for the vhost-user frontend and
> +backend to exchange notifications and data through the device:
> +
> +\begin{description}
> +  \item[Device auxiliary notification] \ref{sec:Basic Facilities of a Virtio Device / Notifications}
> +The driver signals the vhost-user frontend through device auxiliary notifications. The signal does not
> +carry any data, it is purely an event.
> +  \item[Driver auxiliary notification] \ref{sec:Basic Facilities of a Virtio Device / Notifications}
> +The vhost-user frontend signals the driver for events besides virtqueue activity
> +and configuration changes by sending driver auxiliary notification.
> +  \item[Shared memory] \ref{sec:Basic Facilities of a Virtio Device / Shared Memory Regions}
> +The vhost-user frontend gives access to memory that can be mapped by the driver.
> +\end{description}
> +
> +\subsubsection{Device auxiliary notifications}\label{sec:Device Types / Vhost-user Device Backend / Additional Device Resources / Device auxiliary notifications}
> +
> +The vhost-user device backend provides all (or part) of the following device auxiliary notifications:
> +
> +\begin{description}
> +\item[0] Vring call for vhost-user queue 0
> +\item[\ldots]
> +\item[N-1] Vring call for vhost-user queue N-1
> +\item[N] Vring err for vhost-user queue 0
> +\item[\ldots]
> +\item[2N-1] Vring err for vhost-user queue N-1
> +\item[2N] Log
> +\end{description}
> +
> +where N is the number of the vhost-user virtqueues.
> +
> +\subsubsection{Driver auxiliary notifications}\label{sec:Device Types / Vhost-user Device Backend / Additional Device Resources / Driver auxiliary notifications}
> +
> +The vhost-user device backend provides all (or part) of the following driver auxiliary notifications:
> +
> +\begin{description}
> +\item[0] Vring kick for vhost-user queue 0
> +\item[\ldots]
> +\item[N-1] Vring kick for vhost-user queue N-1
> +\end{description}
> +
> +where N is the number of the vhost-user virtqueues.
> +
> +\subsubsection{Shared Memory}\label{sec:Device Types / Vhost-user Device Backend / Additional Device Resources / Shared Memory}
> +
> +The vhost-user device backend provides all (or part) of the following shared memory regions:
> +
> +\begin{description}
> +\item[0] Vhost-user memory region 0
> +\item[1] Vhost-user memory region 1
> +\item[\ldots]
> +\item[M-1] Vhost-user memory region M-1
> +\item[M] Log memory region
> +\end{description}
> +
> +where M is the total number of memory regions shared.
> +
> +\devicenormative{\paragraph}{Shared Memory layout}{Device Types / Vhost-user Device Backend / Additional Device Resources / Shared Memory layout}
> +
> +The device exports all memory regions reported by the vhost-user frontend as a
> +single shared memory region \ref{sec:Basic Facilities of a Virtio Device /
> +Shared Memory Regions}.

This seems to contradict the list above where it shows "Vhost-user memory
region 0", "Vhost-user memory region 1", etc as separate shared memory
regions. Is it a single shared memory region or not?

> +
> +The size of this shared memory region exported by the device MUST be at least
> +as much as the sum of the sizes of all the memory regions reported by the
> +vhost-user frontend.
> +
> +The memory regions exported by the device MUST be laid out in the same order
> +in which they are reported by the frontend with vhost-user messages.
> +
> +The offsets in which the memory regions are mapped inside the shared memory
> +region MUST be the following:
> +
> +\begin{description}
> +\item[0] Offset for vhost-user memory region 0
> +\item[SIZE0] Offset for vhost-user memory region 1
> +\item[\ldots]
> +\item[SIZE0 + SIZE1 + \ldots] Offset for vhost-user memory region M
> +\end{description}
> +    
> +where SIZEi is the size of the vhost-user memory region i.

It's unclear to me how vhost-user's
VHOST_USER_ADD_MEM_REG/VHOST_USER_REM_MEM_REG messages are handled. They
are dynamic.

> +
> +\subsubsection{Availability of Additional Resources}\label{sec:Device Types / Vhost-user Device Backend / Additional Device Resources / Availability of Additional Resources}
> +
> +The following vhost-user protocol messages convey access to additional device
> +resources:
> +
> +\begin{description}
> +\item[VHOST_USER_SET_MEM_TABLE] Contents of vhost-user memory regions are
> +available to the driver as device memory. Region contents are laid out in the

s/as device memory/in Shared Memory Regions/ here and throughout this
section. Let's use the spec's terminology instead of the more vague
"device memory".

> +same order as the vhost-user memory region list.

This refers to VHOST_USER_SET_MEM_TABLE but doesn't cover the
VHOST_USER_ADD_MEM_REG/VHOST_USER_REM_MEM_REG messages.

> +\item[VHOST_USER_SET_LOG_BASE] Contents of the log memory region are available
> +to the driver as device memory.
> +\item[VHOST_USER_SET_LOG_FD] The log device auxiliary notification is available to the driver.
> +Writes to the log device auxiliary notification before this message is received produce no effect.
> +\item[VHOST_USER_SET_VRING_KICK] The vring kick notification for this queue is
> +available to the driver. The first notification may occur before the driver has
> +processed this message.

Maybe the spec can suggest how drivers should handle kicks that arrive
before the corresponding VHOST_USER_SET_VRING_KICK message has been
processed? Perhaps the vhost-user backend should ignore the unknown kick
and peek at the vring when VHOST_USER_SET_VRING_ENABLE is processed.
That way no kicks are lost.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

[virtio-dev] Re: [PATCH 2/4] content: Introduce driver/device aux. notification cfg type for PCI

[Usama Arif]

Hi,

Thanks for the reviews! I have tried to address the comments in the v2 
of the patch.
Also replying inline in this and other comment mails to further discuss 
them.

On 04/04/2022 11:27, Stefan Hajnoczi wrote:
> On Wed, Mar 30, 2022 at 04:26:57PM +0100, Usama Arif wrote:
>> +The \field{cap.offset} and \field{dev_aux_notification_off_multiplier} are taken
>> +from the device auxiliary notification capability structure above, and the
>> +\field{dev_aux_notification_idx} is the device auxiliary notification index. There is no restriction for
>> +the mapping between device auxiliary notifications and dev_aux_notification_idx. The mapping is
>> +device-specific. One possible mapping would be to use the integers 0 to
>> +N-1 as the device auxiliary notification indices for a total of N device auxiliary notifications.
>> +The total number of device auxiliary notifications exposed by the device is also device-specific.
> 
> The second half of the paragraph could be shortened to "The number of
> device auxiliary notifications and their purpose is device-specific". I
> don't think discussing the device-specific mapping and that it may be
> 0 to N-1 or something else adds anything.
>

Thanks, part of v2. Also have shortened it for MMIO.

>> +
>> +\devicenormative{\paragraph}{Device auxiliary notification capability}{Virtio Transport Options / Virtio Over PCI Bus / PCI Device Layout / Device auxiliary notification capability}
>> +
>> +The \field{cap.offset} MUST be 2-byte aligned.
>> +
>> +The device MUST either present \field{dev_aux_notification_off_multiplier} as an
>> +even power of 2, or present \field{dev_aux_notification_off_multiplier} as 0.
>> +
>> +The value \field{cap.length} presented by the device MUST be at least 2
>> +and MUST be large enough to support device auxiliary notification offsets for all supported
>> +device auxiliary notifications in all possible configurations.
>> +
>> +The value \field{cap.length} presented by the device MUST satisfy:
>> +
>> +\begin{lstlisting}
>> +cap.length >= max_dev_aux_notification_idx * dev_aux_notification_off_multiplier + 2
>> +\end{lstlisting}
>> +
>> +where \field{max_dev_aux_notification_idx} is the maximum device auxiliary notification index and is
>> +dependent on the device.
> 
> One thought about dev aux notifications: the size is limited to 2 bytes.
> Sometimes it's useful to communicate information (i.e. 1, 2, or 4 bytes)
> as part of the notification (virtio-vhost-user may not need it but other
> users might). Perhaps the width should be device-specific and not fixed
> to 2?
> 

Yes sounds better not to limit the data width. I have changed this in 
v2. The
normatives for data width 2 and 4 in v2 are similar to
Notification capability device normative with the substituted terms for
device aux. notification, i.e. dev_aux_notification_off_multiplier 
instead of
notify_off_multiplier, etc.


>> +
>> +\subsubsection{Driver auxiliary notification capability}\label{sec:Virtio Transport Options / Virtio Over PCI Bus / PCI Device Layout / Driver auxiliary notification capability}
>> +
>> +The Driver auxiliary notification
>> +\ref{sec:Basic Facilities of a Virtio Device / Notifications} location
>> +is found using the VIRTIO_PCI_CAP_DRIVER_AUX_NOTIFY_CFG capability. The
>> +driver auxiliary notification structure allows MSI-X vectors to be
>> +configured for notification interrupts. If MSI-X is not available, bit 2
>> +of the ISR status \ref{sec:Virtio Transport Options / Virtio Over PCI
>> +Bus / PCI Device Layout / ISR status capability} indicates that a
>> +driver auxiliary notification occurred.
> 
> Is there any way to determine which driver auxiliary notification
> occurred when MSI-X is not available? Please make this clear in the
> text.
> 

The only thing that comes to mind is using additional ISR bits to indicate
which driver aux. notification occured, but that probably is a bad idea?

So it means that only a single driver aux. notification
can be supported if MSI-X is not available as its not possible to 
distinguish
between them.

The same situation arises in vritio-mmio as well, so only a single 
driver aux. notification
can be supported there as well i think.

I have updated this in v2, but just wanted to check if you have an idea on
how to support multiple types of driver aux. notifications in non MSI-X 
and virtio-mmio case?
>> +
>> +The driver auxiliary notification structure is the following:
>> +
>> +\begin{lstlisting}
>> +struct virtio_pci_driver_aux_notification_cfg {
>> +	le16 driver_aux_notification_select;              /* read-write */
>> +	le16 driver_aux_notification_msix_vector;         /* read-write */
>> +};
>> +\end{lstlisting}
>> +
>> +The driver indicates which notification is of interest by writing the
>> +\field{driver_aux_notification_select} field. The driver then writes the MSI-X
>> +vector or VIRTIO_MSI_NO_VECTOR to \field{driver_aux_notification_msix_vector} to
>> +change the MSI-X vector for that notification.
>> +
>> +The mapping between notifications and notification indices is
>> +device-specific. The total number of notifications is also
>> +device-specific.
>> +
>> +\devicenormative{\paragraph}{Driver auxiliary notification capability}{Virtio Transport Options / Virtio Over PCI Bus / PCI Device Layout / Driver auxiliary notification capability}
>> +
>> +Device MUST ignore writes to \field{driver_aux_notification_msix_vector} if the
>> +value written to \field{driver_aux_notification_select} is not a valid notification
>> +index.
>> +
>> +Device MUST return VIRTIO_MSI_NO_VECTOR for reads from
>> +\field{driver_aux_notification_msix_vector} if the value written to
>> +\field{driver_aux_notification_select} is not a valid notification index.
> 
> Maybe the spec should say something about the minimum number of MSI-X
> vectors presented by the device?
> 
>    num_vectors >= 1 /* config change */ + num_vqs + num_driver_aux_notifications

I have added this in v2 in MSI-X Vector Configuration device normative.

---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

[virtio-dev] Re: [PATCH 3/4] content: Introduce driver/device auxiliary notifications for MMIO

[Usama Arif]

On 04/04/2022 13:36, Stefan Hajnoczi wrote:
> On Wed, Mar 30, 2022 at 04:26:58PM +0100, Usama Arif wrote:
>> @@ -2074,6 +2077,16 @@ \subsection{MMIO Device Register Layout}\label{sec:Virtio Transport Options / Vi
>>       apply to the queue selected by writing to \field{QueueSel}.
>>     }
>>     \hline
>> +  \mmioreg{DeviceAuxiliaryNotification}{Device Auxiliary Notifier}{0xd0}{W}{
>> +    Writing a value to this register triggers a
>> +    notification to the device. The value written to this register is
>> +    interpreted as a device auxiliary notification index. The mapping of device auxiliary notifications to
>> +    device auxiliary notification indices is device-specific. One possible mapping would be
>> +    to use the integers 0 to N-1 as the device auxiliary notification indices for a total of
>> +    N device auxiliary notifications. The total number of device auxiliary notifications exposed by the device is
>> +    also device-specific.
>> +  }
> 
> Hmm...this is slightly different from the PCI transport's approach.
> There each dev aux notification has its own hardware register. That has
> advantages:
> 1. The notification may carry a value with it because the driver writes
>     a value to the register and the device can interpret that value. That
>     can be used as a performance optimization to reduce the number of
>     hardware register writes performed by the driver or DMA transfers
>     initiated by the device.
> 2. The dev aux notification registers can be laid out on separate memory
>     pages. This is useful in passthrough scenarios like nested
>     virtualization or slicing up a device in software and passing a
>     subset to an untrusted process. The MMU only offers page-level
>     protection, so MMIO registers that are colocated within the same
>     memory page cannot be selectively passed through to untrusted
>     software.
> 
> It would be nice if PCI and MMIO exposed dev aux notifications in a
> similar way.
> 
> I haven't followed the history of virtio-mmio closely, so I'm not sure
> if the same approach that was used for PCI is feasible here. Maybe
> someone else can comment?
> 
> Stefan


Agree with the points above. I have split the mmio implementation into 2 
registers in v2. DeviceAuxNotificationIndex to indicate the device 
auxiliary notification index and DeviceAuxNotificationData writing to 
which actually does the notification and carries the data which is 
device specific.

---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

[virtio-dev] Re: [PATCH 4/4] vhost-user: add vhost-user device type

[Usama Arif]

On 04/04/2022 14:05, Stefan Hajnoczi wrote:
> On Wed, Mar 30, 2022 at 04:26:59PM +0100, Usama Arif wrote:
>> The vhost-user device backend facilitates vhost-user device emulation
>> through vhost-user protocol exchanges and access to shared memory.
>> Software-defined networking, storage, and other I/O appliances can
>> provide services through this device.
>>
>> This device is based on Wei Wang's vhost-pci work.  The virtio
>> vhost-user device differs from vhost-pci because it is a single virtio
>> device type that exposes the vhost-user protocol instead of a family of
>> new virtio device types, one for each vhost-user device type.
>>
>> This device supports vhost-user backend and vhost-user frontend
>> reconnection. It also contains a UUID so that vhost-user backend programs
>> can identify a specific device among many without using bus addresses.
>>
>> virtio-vhost-user makes use of additional resources introduced in earlier
>> patches including device aux. notifications, driver aux. notifications,
>> as well as shared memory.
>>
>> Signed-off-by: Usama Arif <usama.arif@bytedance.com>
>> Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
>> Signed-off-by: Nikos Dragazis <ndragazis@arrikto.com>
>> ---
>>   conformance.tex       |  27 ++++-
>>   content.tex           |   3 +
>>   introduction.tex      |   3 +
>>   virtio-vhost-user.tex | 259 ++++++++++++++++++++++++++++++++++++++++++
>>   4 files changed, 288 insertions(+), 4 deletions(-)
>>   create mode 100644 virtio-vhost-user.tex
>>
>> diff --git a/conformance.tex b/conformance.tex
>> index cddaf75..fab49c3 100644
>> --- a/conformance.tex
>> +++ b/conformance.tex
>> @@ -32,8 +32,9 @@ \section{Conformance Targets}\label{sec:Conformance / Conformance Targets}
>>   \ref{sec:Conformance / Driver Conformance / Memory Driver Conformance},
>>   \ref{sec:Conformance / Driver Conformance / I2C Adapter Driver Conformance},
>>   \ref{sec:Conformance / Driver Conformance / SCMI Driver Conformance},
>> -\ref{sec:Conformance / Driver Conformance / GPIO Driver Conformance} or
>> -\ref{sec:Conformance / Driver Conformance / PMEM Driver Conformance}.
>> +\ref{sec:Conformance / Driver Conformance / GPIO Driver Conformance},
>> +\ref{sec:Conformance / Driver Conformance / PMEM Driver Conformance} or
>> +\ref{sec:Conformance / Driver Conformance / Vhost-user Backend Driver Conformance}.
>>   
>>       \item Clause \ref{sec:Conformance / Legacy Interface: Transitional Device and Transitional Driver Conformance}.
>>     \end{itemize}
>> @@ -58,8 +59,9 @@ \section{Conformance Targets}\label{sec:Conformance / Conformance Targets}
>>   \ref{sec:Conformance / Device Conformance / Memory Device Conformance},
>>   \ref{sec:Conformance / Device Conformance / I2C Adapter Device Conformance},
>>   \ref{sec:Conformance / Device Conformance / SCMI Device Conformance},
>> -\ref{sec:Conformance / Device Conformance / GPIO Device Conformance} or
>> -\ref{sec:Conformance / Device Conformance / PMEM Device Conformance}.
>> +\ref{sec:Conformance / Device Conformance / GPIO Device Conformance},
>> +\ref{sec:Conformance / Device Conformance / PMEM Device Conformance} or
>> +\ref{sec:Conformance / Device Conformance / Vhost-user Backend Device Conformance}.
>>   
>>       \item Clause \ref{sec:Conformance / Legacy Interface: Transitional Device and Transitional Driver Conformance}.
>>     \end{itemize}
>> @@ -324,6 +326,15 @@ \section{Conformance Targets}\label{sec:Conformance / Conformance Targets}
>>   \item \ref{drivernormative:Device Types / PMEM Device / Device Initialization}
>>   \end{itemize}
>>   
>> +\conformance{\subsection}{Vhost-user Backend Driver Conformance}\label{sec:Conformance / Driver Conformance / Vhost-user Backend Driver Conformance}
>> +
>> +A vhost-user backend driver MUST conform to the following normative statements:
>> +
>> +\begin{itemize}
>> +\item \ref{drivernormative:Device Types / Vhost-user Device Backend / Device configuration layout}
>> +\item \ref{drivernormative:Device Types / Vhost-user Device Backend / Device Initialization}
>> +\end{itemize}
>> +
>>   \conformance{\section}{Device Conformance}\label{sec:Conformance / Device Conformance}
>>   
>>   A device MUST conform to the following normative statements:
>> @@ -595,6 +606,14 @@ \section{Conformance Targets}\label{sec:Conformance / Conformance Targets}
>>   \item \ref{devicenormative:Device Types / PMEM Device / Device Operation / Virtqueue return}
>>   \end{itemize}
>>   
>> +\conformance{\subsection}{Vhost-user Backend Device Conformance}\label{sec:Conformance / Device Conformance / Vhost-user Backend Device Conformance}
>> +
>> +A Vhost-user backend device MUST conform to the following normative statements:
>> +
>> +\begin{itemize}
>> +\item \ref{devicenormative:Device Types / Vhost-user Device Backend / Additional Device Resources / Shared Memory layout}
>> +\end{itemize}
>> +
>>   \conformance{\section}{Legacy Interface: Transitional Device and Transitional Driver Conformance}\label{sec:Conformance / Legacy Interface: Transitional Device and Transitional Driver Conformance}
>>   A conformant implementation MUST be either transitional or
>>   non-transitional, see \ref{intro:Legacy
>> diff --git a/content.tex b/content.tex
>> index 0fc50c4..8bf114d 100644
>> --- a/content.tex
>> +++ b/content.tex
>> @@ -3122,6 +3122,8 @@ \chapter{Device Types}\label{sec:Device Types}
>>   \hline
>>   42         &   RDMA device \\
>>   \hline
>> +43         &   vhost-user device backend \ \\
>> +\hline
>>   \end{tabular}
>>   
>>   Some of the devices above are unspecified by this document,
>> @@ -6878,6 +6880,7 @@ \subsubsection{Legacy Interface: Framing Requirements}\label{sec:Device
>>   \input{virtio-scmi.tex}
>>   \input{virtio-gpio.tex}
>>   \input{virtio-pmem.tex}
>> +\input{virtio-vhost-user.tex}
>>   
>>   \chapter{Reserved Feature Bits}\label{sec:Reserved Feature Bits}
>>   
>> diff --git a/introduction.tex b/introduction.tex
>> index 6d52717..5bd1b95 100644
>> --- a/introduction.tex
>> +++ b/introduction.tex
>> @@ -79,6 +79,9 @@ \section{Normative References}\label{sec:Normative References}
>>   	\phantomsection\label{intro:SCMI}\textbf{[SCMI]} &
>>   	Arm System Control and Management Interface, DEN0056,
>>   	\newline\url{https://developer.arm.com/docs/den0056/c}, version C and any future revisions\\
>> +	\phantomsection\label{intro:Vhost-user Protocol}\textbf{[Vhost-user Protocol]}
>> +	& Vhost-user Protocol,
>> +	\newline\url{https://qemu.readthedocs.io/en/latest/interop/vhost-user.html}\\
>>   
>>   \end{longtable}
>>   
>> diff --git a/virtio-vhost-user.tex b/virtio-vhost-user.tex
>> new file mode 100644
>> index 0000000..303054f
>> --- /dev/null
>> +++ b/virtio-vhost-user.tex
>> @@ -0,0 +1,259 @@
>> +\section{Vhost-user Device Backend}\label{sec:Device Types / Vhost-user Device
>> +Backend}
>> +
>> +The vhost-user device backend facilitates vhost-user device emulation through
>> +vhost-user protocol exchanges and access to shared memory. Software-defined
>> +networking, storage, and other I/O appliances can provide services through this
>> +device.
>> +
>> +This section relies on definitions from the \hyperref[intro:Vhost-user
>> +Protocol]{Vhost-user Protocol}.  Knowledge of the vhost-user protocol is a
>> +prerequisite for understanding this device.
>> +
>> +The \hyperref[intro:Vhost-user Protocol]{Vhost-user Protocol} was originally
>> +designed for processes on a single system communicating over UNIX domain
>> +sockets. The virtio vhost-user device backend allows the vhost-user backend to
>> +communicate with the vhost-user frontend over the device instead of a UNIX domain
>> +socket. This allows the backend and frontend to run on two separate systems such
>> +as a virtual machine and a hypervisor.
>> +
>> +The vhost-user backend program exchanges vhost-user protocol messages with the
>> +vhost-user frontend through this device. How the device implementation
>> +communicates with the vhost-user frontend is beyond the scope of this
>> +specification.  One possible device implementation uses a UNIX domain socket to
>> +relay messages to a vhost-user frontend process running on the same host.
>> +
>> +Existing vhost-user backend programs that communicate over UNIX domain sockets
>> +can support the virtio vhost-user device backend without invasive changes
>> +because the pre-existing vhost-user wire protocol is used.
>> +
>> +\subsection{Device ID}\label{sec:Device Types / Vhost-user Device Backend / Device ID}
>> +  43
>> +
>> +\subsection{Virtqueues}\label{sec:Device Types / Vhost-user Device Backend / Virtqueues}
>> +
>> +\begin{description}
>> +\item[0] rxq (device-to-driver vhost-user protocol messages)
>> +\item[1] txq (driver-to-device vhost-user protocol messages)
>> +\end{description}
>> +
>> +\subsection{Feature bits}\label{sec:Device Types / Vhost-user Device Backend / Feature bits}
>> +
>> +No feature bits are defined at this time.
>> +
>> +\subsection{Device configuration layout}\label{sec:Device Types / Vhost-user Device Backend / Device configuration layout}
>> +
>> +  All fields of this configuration are always available.
>> +
>> +\begin{lstlisting}
>> +struct virtio_vhost_user_config {
>> +        le32 status;
>> +#define VIRTIO_VHOST_USER_STATUS_BACKEND_UP (1 << 0)
>> +#define VIRTIO_VHOST_USER_STATUS_FRONTEND_UP (1 << 1)
>> +        le32 max_vhost_queues;
>> +        u8 uuid[16];
>> +};
>> +\end{lstlisting}
>> +
>> +\begin{description}
>> +\item[\field{status}] contains the vhost-user operational status.  The default
>> +    value of this field is 0.
>> +
>> +    The driver sets VIRTIO_VHOST_USER_STATUS_BACKEND_UP to indicate readiness for
>> +    the vhost-user frontend to connect.  The vhost-user frontend cannot connect
>> +    unless the driver has set this bit first.
>> +
>> +    The device sets VIRTIO_VHOST_USER_STATUS_FRONTEND_UP to indicate that the
>> +    vhost-user frontend is connected.
>> +
>> +    When the driver clears VIRTIO_VHOST_USER_STATUS_BACKEND_UP while the
>> +    vhost-user frontend is connected, the vhost-user frontend is disconnected.
>> +
>> +    When the vhost-user frontend disconnects, both
>> +    VIRTIO_VHOST_USER_STATUS_BACKEND_UP and VIRTIO_VHOST_USER_STATUS_FRONTEND_UP
>> +    are cleared by the device.  Communication can be restarted by the driver
>> +    setting VIRTIO_VHOST_USER_STATUS_BACKEND_UP again.
>> +
>> +    A configuration change notification is sent when the device changes
>> +    this field, unless a write to the field by the driver caused the change.
>> +
>> +\item[\field{max_vhost_queues}] is the maximum number of vhost-user queues
>> +    supported by this device.  This field is always greater than 0.
>> +
>> +\item[\field{uuid}] is the Universally Unique Identifier (UUID) for this
>> +    device. If the device has no UUID then this field contains the nil
>> +    UUID (all zeroes).  The UUID allows vhost-user backend programs to identify a
>> +    specific vhost-user device backend among many without relying on bus
>> +    addresses.
>> +\end{description}
>> +
>> +\drivernormative{\subsubsection}{Device configuration layout}{Device Types / Vhost-user Device Backend / Device configuration layout}
>> +
>> +The driver MUST NOT write to device configuration fields other than \field{status}.
>> +
>> +The driver MUST NOT set undefined bits in the \field{status} configuration field.
>> +
>> +\subsection{Device Initialization}\label{sec:Device Types / Vhost-user Device Backend / Device Initialization}
>> +
>> +The driver initializes the rxq/txq virtqueues and then it sets
>> +VIRTIO_VHOST_USER_STATUS_BACKEND_UP to the \field{status} field of the device
>> +configuration structure.
>> +
>> +\drivernormative{\subsubsection}{Device Initialization}{Device Types / Vhost-user Device Backend / Device Initialization}
>> +
>> +The driver SHOULD check the \field{max_vhost_queues} configuration field to
>> +determine how many queues the vhost-user backend will be able to support.
>> +
>> +The driver SHOULD fetch the \field{uuid} configuration field to allow
>> +vhost-user backend programs to identify a specific device among many.
>> +
>> +The driver SHOULD place at least one buffer in rxq before setting the
>> +VIRTIO_VHOST_USER_STATUS_BACKEND_UP bit in the \field{status} configuration field.
>> +
>> +The driver MUST handle rxq virtqueue notifications that occur before the
>> +configuration change notification.  It is possible that a vhost-user protocol
>> +message from the vhost-user frontend arrives before the driver has seen the
>> +configuration change notification for the VIRTIO_VHOST_USER_STATUS_FRONTEND_UP
>> +\field{status} change.
>> +
>> +\subsection{Device Operation}\label{sec:Device Types / Vhost-user Device Backend / Device Operation}
>> +
>> +Device operation consists of operating request queues and response queues.
>> +
>> +\subsubsection{Device Operation: Request Queues}\label{sec:Device Types / Vhost-user Device Backend / Device Operation / Device Operation: RX/TX Queues}
>> +
>> +The driver receives vhost-user protocol messages from the vhost-user frontend on
>> +rxq. The driver sends responses to the vhost-user frontend on txq.
>> +
>> +The driver sends backend-initiated requests on txq. The driver receives
>> +responses from the vhost-user frontend on rxq.
>> +
>> +All virtqueues offer in-order guaranteed delivery semantics for vhost-user
>> +protocol messages.
>> +
>> +Each buffer is a vhost-user protocol message as defined by the
>> +\hyperref[intro:Vhost-user Protocol]{Vhost-user Protocol}.  In order to enable
>> +cross-endian communication, all message fields are little-endian instead of the
>> +native byte order normally used by the protocol.
>> +
>> +The appropriate size of rxq buffers is at least as large as the largest message
>> +defined by the \hyperref[intro:Vhost-user Protocol]{Vhost-user Protocol}
>> +standard version that the driver supports.  If the vhost-user frontend sends a
>> +message that is too large for an rxq buffer, then DEVICE_NEEDS_RESET is set and
>> +the driver must reset the device.
>> +
>> +File descriptor passing is handled differently by the vhost-user device
>> +backend. When a frontend-initiated message is received that carries one or more file
>> +descriptors according to the vhost-user protocol, additional device resources
>> +become available to the driver.
>> +
>> +\subsection{Additional Device Resources}\label{sec:Device Types / Vhost-user Device Backend / Additional Device Resources}
>> +
>> +The vhost-user device backend uses the following facilities from virtio device
>> +\ref{sec:Basic Facilities of a Virtio Device} for the vhost-user frontend and
>> +backend to exchange notifications and data through the device:
>> +
>> +\begin{description}
>> +  \item[Device auxiliary notification] \ref{sec:Basic Facilities of a Virtio Device / Notifications}
>> +The driver signals the vhost-user frontend through device auxiliary notifications. The signal does not
>> +carry any data, it is purely an event.
>> +  \item[Driver auxiliary notification] \ref{sec:Basic Facilities of a Virtio Device / Notifications}
>> +The vhost-user frontend signals the driver for events besides virtqueue activity
>> +and configuration changes by sending driver auxiliary notification.
>> +  \item[Shared memory] \ref{sec:Basic Facilities of a Virtio Device / Shared Memory Regions}
>> +The vhost-user frontend gives access to memory that can be mapped by the driver.
>> +\end{description}
>> +
>> +\subsubsection{Device auxiliary notifications}\label{sec:Device Types / Vhost-user Device Backend / Additional Device Resources / Device auxiliary notifications}
>> +
>> +The vhost-user device backend provides all (or part) of the following device auxiliary notifications:
>> +
>> +\begin{description}
>> +\item[0] Vring call for vhost-user queue 0
>> +\item[\ldots]
>> +\item[N-1] Vring call for vhost-user queue N-1
>> +\item[N] Vring err for vhost-user queue 0
>> +\item[\ldots]
>> +\item[2N-1] Vring err for vhost-user queue N-1
>> +\item[2N] Log
>> +\end{description}
>> +
>> +where N is the number of the vhost-user virtqueues.
>> +
>> +\subsubsection{Driver auxiliary notifications}\label{sec:Device Types / Vhost-user Device Backend / Additional Device Resources / Driver auxiliary notifications}
>> +
>> +The vhost-user device backend provides all (or part) of the following driver auxiliary notifications:
>> +
>> +\begin{description}
>> +\item[0] Vring kick for vhost-user queue 0
>> +\item[\ldots]
>> +\item[N-1] Vring kick for vhost-user queue N-1
>> +\end{description}
>> +
>> +where N is the number of the vhost-user virtqueues.
>> +
>> +\subsubsection{Shared Memory}\label{sec:Device Types / Vhost-user Device Backend / Additional Device Resources / Shared Memory}
>> +
>> +The vhost-user device backend provides all (or part) of the following shared memory regions:
>> +
>> +\begin{description}
>> +\item[0] Vhost-user memory region 0
>> +\item[1] Vhost-user memory region 1
>> +\item[\ldots]
>> +\item[M-1] Vhost-user memory region M-1
>> +\item[M] Log memory region
>> +\end{description}
>> +
>> +where M is the total number of memory regions shared.
>> +
>> +\devicenormative{\paragraph}{Shared Memory layout}{Device Types / Vhost-user Device Backend / Additional Device Resources / Shared Memory layout}
>> +
>> +The device exports all memory regions reported by the vhost-user frontend as a
>> +single shared memory region \ref{sec:Basic Facilities of a Virtio Device /
>> +Shared Memory Regions} >
> This seems to contradict the list above where it shows "Vhost-user memory
> region 0", "Vhost-user memory region 1", etc as separate shared memory
> regions. Is it a single shared memory region or not?
> 


The shared memory section in v1 was unclear, I have hopefully improved 
the description in v2.
VHOST_USER_SET_MEM_TABLE has an array of memory regions that need to be 
mapped. In the case of VHOST_USER_SET_MEM_TABLE a single Shared Memory 
region exists in which the M-1 vhost-user memory region are 
consecutively mmaped.

I thought about the approach of using multiple shared memories, i.e. a 
shared-memory per vhost-user memory region and if it might be better to 
switch to that. However, I think this could not be practically 
implemented in PCI. Just adding the reasons here incase needed in future:
As the config space is 0x100, out of which, 0x40 is header space, it 
doesn't leave enough space for having a shared-memory capability per 
vhost-user memory region. After taking into account all the capabilities 
excluding shared-memory, i.e. ISR, device/driver aux. notification, 
device configuration, etc, there is only space for 4 virtio pci 
capabilities in the config space. This is much less than the 
VHOST_MEMORY_MAX_NREGIONS (8) and VHOST_USER_MAX_RAM_SLOTS (32) values 
currently in qemu.



>> +
>> +The size of this shared memory region exported by the device MUST be at least
>> +as much as the sum of the sizes of all the memory regions reported by the
>> +vhost-user frontend.
>> +
>> +The memory regions exported by the device MUST be laid out in the same order
>> +in which they are reported by the frontend with vhost-user messages.
>> +
>> +The offsets in which the memory regions are mapped inside the shared memory
>> +region MUST be the following:
>> +
>> +\begin{description}
>> +\item[0] Offset for vhost-user memory region 0
>> +\item[SIZE0] Offset for vhost-user memory region 1
>> +\item[\ldots]
>> +\item[SIZE0 + SIZE1 + \ldots] Offset for vhost-user memory region M
>> +\end{description}
>> +
>> +where SIZEi is the size of the vhost-user memory region i.
> 
> It's unclear to me how vhost-user's
> VHOST_USER_ADD_MEM_REG/VHOST_USER_REM_MEM_REG messages are handled. They
> are dynamic.
> 


I have made an attempt on adding about 
VHOST_USER_ADD_MEM_REG/VHOST_USER_REM_MEM_REG in v2.

I have tried to split it into when when 
VHOST_USER_PROTOCOL_F_CONFIGURE_MEM_SLOTS is present and is not present:
- When the memory regions are mapped into the backend using the single 
VHOST_USER_SET_MEM_TABLE message, i.e. 
VHOST_USER_PROTOCOL_F_CONFIGURE_MEM_SLOTS feature is not present, the 
same approach as v1 is used, i.e. the memory regions are consecutively 
available to the driver in the format

\begin{description}
     \item[0] Offset for vhost-user memory region 0
     \item[SIZE0] Offset for vhost-user memory region 1
     \item[\ldots]
     \item[SIZE0 + SIZE1 + \ldots] Offset for vhost-user memory region M - 1
     \end{description}

- When the memory regions are being mapped/unmapped using 
VHOST_USER_ADD_MEM_REG/VHOST_USER_REM_MEM_REG message, i.e. 
VHOST_USER_PROTOCOL_F_CONFIGURE_MEM_SLOTS feature is negotiated a single 
  shared memory region is still used, however, the size as you said 
would  be dynamic.
I think there possibly could be multiple ways that the device and driver 
could deal with this?
I have mentioned in v2 how the device and driver would operate the 
shared memory in memory compaction. I have added this in the shared 
memory subsection but not in the devicenormative as maybe there might be 
multiple ways of dealing with fragmentation
that comes up and how its dealt with shouldn't be enforced in spec?

>> +
>> +\subsubsection{Availability of Additional Resources}\label{sec:Device Types / Vhost-user Device Backend / Additional Device Resources / Availability of Additional Resources}
>> +
>> +The following vhost-user protocol messages convey access to additional device
>> +resources:
>> +
>> +\begin{description}
>> +\item[VHOST_USER_SET_MEM_TABLE] Contents of vhost-user memory regions are
>> +available to the driver as device memory. Region contents are laid out in the
> 
> s/as device memory/in Shared Memory Regions/ here and throughout this
> section. Let's use the spec's terminology instead of the more vague
> "device memory".
> 

Thanks, changed in v2 for both VHOST_USER_SET_MEM_TABLE and 
VHOST_USER_SET_LOG_BASE.

>> +same order as the vhost-user memory region list.
> 
> This refers to VHOST_USER_SET_MEM_TABLE but doesn't cover the
> VHOST_USER_ADD_MEM_REG/VHOST_USER_REM_MEM_REG messages >
I have added infromation about this in v2.

>> +\item[VHOST_USER_SET_LOG_BASE] Contents of the log memory region are available
>> +to the driver as device memory.
>> +\item[VHOST_USER_SET_LOG_FD] The log device auxiliary notification is available to the driver.
>> +Writes to the log device auxiliary notification before this message is received produce no effect.
>> +\item[VHOST_USER_SET_VRING_KICK] The vring kick notification for this queue is
>> +available to the driver. The first notification may occur before the driver has
>> +processed this message.
> 
> Maybe the spec can suggest how drivers should handle kicks that arrive
> before the corresponding VHOST_USER_SET_VRING_KICK message has been
> processed? Perhaps the vhost-user backend should ignore the unknown kick
> and peek at the vring when VHOST_USER_SET_VRING_ENABLE is processed.
> That way no kicks are lost.

Thanks! added this in v2.


Regards,
Usama

---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

Re: [virtio-dev] Re: [PATCH 1/4] content: Introduce driver/device auxiliary notifications

[Halil Pasic]

On Thu, 11 Aug 2022 11:12:06 +0200
Cornelia Huck <cohuck@redhat.com> wrote:

> On Wed, Aug 10 2022, "Michael S. Tsirkin" <mst@redhat.com> wrote:
> 
> > On Wed, Aug 10, 2022 at 07:41:08PM +0200, Halil Pasic wrote:  
> >> On Wed, 10 Aug 2022 11:54:35 +0200
> >> Cornelia Huck <cohuck@redhat.com> wrote:  
> >> > This highlights another problem, however: When we introduce new features
> >> > that require a transport-specific implementation, we often end up with a
> >> > PCI implementation, but sometimes MMIO and more often ccw are left
> >> > behind -- which is understandable, as PCI is what most people use, and
> >> > ccw is something only a very few people are familiar with. This sadly
> >> > means that we have a backlog of features supported in PCI, but not in
> >> > ccw... requiring implementations for ccw would put an undue burden on
> >> > contributors, as most of them are unlikely to write anything for a
> >> > mainframe, ever. On the flip side, I do not have enough bandwith to deal
> >> > with all of this.  
> >> 
> >> I'm completely with you in a sense that I see the same problem. I think
> >> we have to get these resolved on a case by case basis. In my opinion at
> >> least in theory it would make a big difference, whether the new feature
> >> obligatory or not. But since VIRTIO is big on compatibility, and also
> >> cares about the initial investment required, in practice, I think, we
> >> are mostly good with the transports delivering features on their own
> >> schedule. What I mean here is: it is kind of difficult to make a new
> >> facility (like shm, or aux notifications) mandatory, because stuff
> >> that conform to a previous incarnation of the spec would become
> >> non-conform.  
> 
> I don't think there's a big case for making new things mandatory;
> everything should be guarded by a feature bit or similar.

Yes! I tried to say the same, just differently :)

> 
> >> 
> >> And the people who care about the particular transport, and the users
> >> of the transport (indirectly also platforms) should make up their own
> >> mind with regards to whether and when to invest into the new facilities
> >> and the new tech and opportunities associated with those.  
> 
> PCI will probably satisfy the needs of the vast majority of users, and
> MMIO is not too alien to just change at the same time. ccw is the big
> problem. Is IBM still spending resources on virtio-ccw? [My own
> involvement with s390x has dwindled a lot, so it would be great to see
> some of it picked up by others. Certainly not trying to pin everything
> on Halil, though.]

Yes! IBM is definitely (still) invested in virtio-ccw! I don't work on
Virtio on s390x full time any more, but I'm totally committed to
fulfilling my duties as a TC member and as the virtio-ccw
sub-maintainer. IBM is looking for a solution to at least replace what
the Virtio community has lost with me gaining new responsibilities not
closely related to Virtio.


BTW, should I not show up in time on some discussion, mentioning my
name in the body of the email should help. I auto-tag my emails,
and I have a separate tag for that. When I'm completely under water
I check for that tag to avoid not showing up when my name is called
:D

Thanks for calling my name occasionally!

Regards,
Halil

Re: [virtio-dev] Re: [PATCH 1/4] content: Introduce driver/device auxiliary notifications

[David Hildenbrand]

On 11.08.22 16:09, Halil Pasic wrote:
> On Thu, 11 Aug 2022 10:53:35 +0200
> Cornelia Huck <cohuck@redhat.com> wrote:
> 
>> On Wed, Aug 10 2022, Halil Pasic <pasic@linux.ibm.com> wrote:
>>
>>> On Wed, 10 Aug 2022 11:54:35 +0200
>>> Cornelia Huck <cohuck@redhat.com> wrote:
>>>  
>>>>>> These device-specific notifications are needed later when adding support
>>>>>> for virtio-vhost-user device.
>>>>>>
>>>>>> Signed-off-by: Usama Arif <usama.arif@bytedance.com>
>>>>>> Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
>>>>>> Signed-off-by: Nikos Dragazis <ndragazis@arrikto.com>    
>>>>>
>>>>> I see ccw is missing. Cornelia, any suggestions?    
>>>>
>>>> Hmm... I seem to be really behind on ccw things :(
>>>>
>>>> We can probably use the following:
>>>>
>>>> - for device->driver notification, use the next bit in the secondary
>>>>   indicators (bit 0 is used for config change notification)
>>>> - for driver->device notification, maybe use a new subcode for diagnose
>>>>   0x500 (4 is probably the next free one?)
>>>>   
>>>
>>> Sounds reasonable! I will have to double check the DIAG stuff though. I'm
>>> not sure where what needs to be reserved and documented.   
>>
>> There's
>> https://gitlab.com/davidhildenbrand/s390x-os-virt-spec/-/merge_requests/1,
>> but nothing much has happened there recently. David?

Heh, I'd like to say it has high priority on my todo list ... but there
is a lot of stuff with even higher priority. I'll definetly come back to
this soonish.

> 
> Hm, that seems to be private. I land on a sign-in page.

Oh, it was private. It's public now.


-- 
Thanks,

David / dhildenb

Re: [virtio-dev] Re: [PATCH 1/4] content: Introduce driver/device auxiliary notifications

[Halil Pasic]

On Thu, 11 Aug 2022 11:12:06 +0200
Cornelia Huck <cohuck@redhat.com> wrote:

> >  
> >> I think making the people not motivated to do the design and write the
> >> spec for all the platforms add to that list is a reasonable middle
> >> ground. It would also make the differences very clear, and the same goes
> >> for the intention (i.e. not omitted by mistake).  
> 
> Hm. On the one hand, I like that it would add a laundry list for
> transports regarding features that might be implemented. 

I believe, all three of us are in agreement on this one. I could hack up
a spec-patch for ccw at some point in time which rectifies things for
the features that are already in the spec.

> In the other
> hand, I think the main problem is not enough people with enough
> understanding and bandwidth to add new features everywhere... but I
> suppose that needs to be fixed in a different place anyway.

I kind of agree, especially with the part that this is not the right
place to fix that problem, if it is a problem. Actually I see this
rather as a strength of virtio: a large amount of freedom in deciding
how much and where to invest. I would rather not specify stuff without
having some sort of a reference implementation, so the ability to
delay specifying how some optional features are to work with virtio-ccw
is very welcome form my perspective.

Regards,
Halil

Re: [virtio-dev] Re: [PATCH 1/4] content: Introduce driver/device auxiliary notifications

[Halil Pasic]

On Thu, 11 Aug 2022 10:53:35 +0200
Cornelia Huck <cohuck@redhat.com> wrote:

> On Wed, Aug 10 2022, Halil Pasic <pasic@linux.ibm.com> wrote:
> 
> > On Wed, 10 Aug 2022 11:54:35 +0200
> > Cornelia Huck <cohuck@redhat.com> wrote:
> >  
> >> >> These device-specific notifications are needed later when adding support
> >> >> for virtio-vhost-user device.
> >> >> 
> >> >> Signed-off-by: Usama Arif <usama.arif@bytedance.com>
> >> >> Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
> >> >> Signed-off-by: Nikos Dragazis <ndragazis@arrikto.com>    
> >> >
> >> > I see ccw is missing. Cornelia, any suggestions?    
> >> 
> >> Hmm... I seem to be really behind on ccw things :(
> >> 
> >> We can probably use the following:
> >> 
> >> - for device->driver notification, use the next bit in the secondary
> >>   indicators (bit 0 is used for config change notification)
> >> - for driver->device notification, maybe use a new subcode for diagnose
> >>   0x500 (4 is probably the next free one?)
> >>   
> >
> > Sounds reasonable! I will have to double check the DIAG stuff though. I'm
> > not sure where what needs to be reserved and documented.   
> 
> There's
> https://gitlab.com/davidhildenbrand/s390x-os-virt-spec/-/merge_requests/1,
> but nothing much has happened there recently. David?

Hm, that seems to be private. I land on a sign-in page.

Regards,
Halil

Re: [virtio-dev] Re: [PATCH 1/4] content: Introduce driver/device auxiliary notifications

[Michael S. Tsirkin]

On Thu, Aug 11, 2022 at 11:12:06AM +0200, Cornelia Huck wrote:
> >> OTOH when reading the spec, it my strike one as strange, that for example
> >> CCW does not mention aux notifications at all. One idea: maybe we could
> >> add a note, or a subsection, or something, which states the list of
> >> general optional virtio facilities or features not supported by the given
> >> transport on the spec level for a given incarnation of the spec.
> >
> > Yes I think each transport should list features it does not
> > support, and a feature specific to some transports must
> > also require that other transports disable it and
> > that drivers do not ack it.
> > Otherwise it's too easy for devices to offer the feature bit
> > by mistake.
> 
> Do we need to add a requirement for every transport-specific feature, or
> would it be sufficient to add a statement like "if a feature requires a
> transport-specific implementation, a device using that transport MUST
> NOT offer that feature"?


I'd prefer to list the features explicitly.

In particular is kind of forces whoever is proposing
the feature into at least be aware that other transports
exist and that the proposal is incomplete.


> >
> >> I think making the people not motivated to do the design and write the
> >> spec for all the platforms add to that list is a reasonable middle
> >> ground. It would also make the differences very clear, and the same goes
> >> for the intention (i.e. not omitted by mistake).
> 
> Hm. On the one hand, I like that it would add a laundry list for
> transports regarding features that might be implemented. On the other
> hand, I think the main problem is not enough people with enough
> understanding and bandwidth to add new features everywhere... but I
> suppose that needs to be fixed in a different place anyway.

Re: [virtio-dev] Re: [PATCH 1/4] content: Introduce driver/device auxiliary notifications

[Cornelia Huck]

On Wed, Aug 10 2022, "Michael S. Tsirkin" <mst@redhat.com> wrote:

> On Wed, Aug 10, 2022 at 07:41:08PM +0200, Halil Pasic wrote:
>> On Wed, 10 Aug 2022 11:54:35 +0200
>> Cornelia Huck <cohuck@redhat.com> wrote:
>> > This highlights another problem, however: When we introduce new features
>> > that require a transport-specific implementation, we often end up with a
>> > PCI implementation, but sometimes MMIO and more often ccw are left
>> > behind -- which is understandable, as PCI is what most people use, and
>> > ccw is something only a very few people are familiar with. This sadly
>> > means that we have a backlog of features supported in PCI, but not in
>> > ccw... requiring implementations for ccw would put an undue burden on
>> > contributors, as most of them are unlikely to write anything for a
>> > mainframe, ever. On the flip side, I do not have enough bandwith to deal
>> > with all of this.
>> 
>> I'm completely with you in a sense that I see the same problem. I think
>> we have to get these resolved on a case by case basis. In my opinion at
>> least in theory it would make a big difference, whether the new feature
>> obligatory or not. But since VIRTIO is big on compatibility, and also
>> cares about the initial investment required, in practice, I think, we
>> are mostly good with the transports delivering features on their own
>> schedule. What I mean here is: it is kind of difficult to make a new
>> facility (like shm, or aux notifications) mandatory, because stuff
>> that conform to a previous incarnation of the spec would become
>> non-conform.

I don't think there's a big case for making new things mandatory;
everything should be guarded by a feature bit or similar.

>> 
>> And the people who care about the particular transport, and the users
>> of the transport (indirectly also platforms) should make up their own
>> mind with regards to whether and when to invest into the new facilities
>> and the new tech and opportunities associated with those.

PCI will probably satisfy the needs of the vast majority of users, and
MMIO is not too alien to just change at the same time. ccw is the big
problem. Is IBM still spending resources on virtio-ccw? [My own
involvement with s390x has dwindled a lot, so it would be great to see
some of it picked up by others. Certainly not trying to pin everything
on Halil, though.]

>> 
>> OTOH when reading the spec, it my strike one as strange, that for example
>> CCW does not mention aux notifications at all. One idea: maybe we could
>> add a note, or a subsection, or something, which states the list of
>> general optional virtio facilities or features not supported by the given
>> transport on the spec level for a given incarnation of the spec.
>
> Yes I think each transport should list features it does not
> support, and a feature specific to some transports must
> also require that other transports disable it and
> that drivers do not ack it.
> Otherwise it's too easy for devices to offer the feature bit
> by mistake.

Do we need to add a requirement for every transport-specific feature, or
would it be sufficient to add a statement like "if a feature requires a
transport-specific implementation, a device using that transport MUST
NOT offer that feature"?

>
>> I think making the people not motivated to do the design and write the
>> spec for all the platforms add to that list is a reasonable middle
>> ground. It would also make the differences very clear, and the same goes
>> for the intention (i.e. not omitted by mistake).

Hm. On the one hand, I like that it would add a laundry list for
transports regarding features that might be implemented. On the other
hand, I think the main problem is not enough people with enough
understanding and bandwidth to add new features everywhere... but I
suppose that needs to be fixed in a different place anyway.


---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

Re: [virtio-dev] Re: [PATCH 1/4] content: Introduce driver/device auxiliary notifications

[Cornelia Huck]

On Wed, Aug 10 2022, Halil Pasic <pasic@linux.ibm.com> wrote:

> On Wed, 10 Aug 2022 11:54:35 +0200
> Cornelia Huck <cohuck@redhat.com> wrote:
>
>> >> These device-specific notifications are needed later when adding support
>> >> for virtio-vhost-user device.
>> >> 
>> >> Signed-off-by: Usama Arif <usama.arif@bytedance.com>
>> >> Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
>> >> Signed-off-by: Nikos Dragazis <ndragazis@arrikto.com>  
>> >
>> > I see ccw is missing. Cornelia, any suggestions?  
>> 
>> Hmm... I seem to be really behind on ccw things :(
>> 
>> We can probably use the following:
>> 
>> - for device->driver notification, use the next bit in the secondary
>>   indicators (bit 0 is used for config change notification)
>> - for driver->device notification, maybe use a new subcode for diagnose
>>   0x500 (4 is probably the next free one?)
>> 
>
> Sounds reasonable! I will have to double check the DIAG stuff though. I'm
> not sure where what needs to be reserved and documented. 

There's
https://gitlab.com/davidhildenbrand/s390x-os-virt-spec/-/merge_requests/1,
but nothing much has happened there recently. David?


---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

Re: [virtio-dev] Re: [PATCH 1/4] content: Introduce driver/device auxiliary notifications

[Michael S. Tsirkin]

On Wed, Aug 10, 2022 at 07:41:08PM +0200, Halil Pasic wrote:
> On Wed, 10 Aug 2022 11:54:35 +0200
> Cornelia Huck <cohuck@redhat.com> wrote:
> 
> > >> These device-specific notifications are needed later when adding support
> > >> for virtio-vhost-user device.
> > >> 
> > >> Signed-off-by: Usama Arif <usama.arif@bytedance.com>
> > >> Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
> > >> Signed-off-by: Nikos Dragazis <ndragazis@arrikto.com>  
> > >
> > > I see ccw is missing. Cornelia, any suggestions?  
> > 
> > Hmm... I seem to be really behind on ccw things :(
> > 
> > We can probably use the following:
> > 
> > - for device->driver notification, use the next bit in the secondary
> >   indicators (bit 0 is used for config change notification)
> > - for driver->device notification, maybe use a new subcode for diagnose
> >   0x500 (4 is probably the next free one?)
> > 
> 
> Sounds reasonable! I will have to double check the DIAG stuff though. I'm
> not sure where what needs to be reserved and documented. 
> 
> > I have not looked at the requirements deeply, though.
> > 
> > This highlights another problem, however: When we introduce new features
> > that require a transport-specific implementation, we often end up with a
> > PCI implementation, but sometimes MMIO and more often ccw are left
> > behind -- which is understandable, as PCI is what most people use, and
> > ccw is something only a very few people are familiar with. This sadly
> > means that we have a backlog of features supported in PCI, but not in
> > ccw... requiring implementations for ccw would put an undue burden on
> > contributors, as most of them are unlikely to write anything for a
> > mainframe, ever. On the flip side, I do not have enough bandwith to deal
> > with all of this.
> 
> I'm completely with you in a sense that I see the same problem. I think
> we have to get these resolved on a case by case basis. In my opinion at
> least in theory it would make a big difference, whether the new feature
> obligatory or not. But since VIRTIO is big on compatibility, and also
> cares about the initial investment required, in practice, I think, we
> are mostly good with the transports delivering features on their own
> schedule. What I mean here is: it is kind of difficult to make a new
> facility (like shm, or aux notifications) mandatory, because stuff
> that conform to a previous incarnation of the spec would become
> non-conform.
> 
> And the people who care about the particular transport, and the users
> of the transport (indirectly also platforms) should make up their own
> mind with regards to whether and when to invest into the new facilities
> and the new tech and opportunities associated with those.
> 
> OTOH when reading the spec, it my strike one as strange, that for example
> CCW does not mention aux notifications at all. One idea: maybe we could
> add a note, or a subsection, or something, which states the list of
> general optional virtio facilities or features not supported by the given
> transport on the spec level for a given incarnation of the spec.

Yes I think each transport should list features it does not
support, and a feature specific to some transports must
also require that other transports disable it and
that drivers do not ack it.
Otherwise it's too easy for devices to offer the feature bit
by mistake.

> I think making the people not motivated to do the design and write the
> spec for all the platforms add to that list is a reasonable middle
> ground. It would also make the differences very clear, and the same goes
> for the intention (i.e. not omitted by mistake).
> 
> > 
> > Halil, any thoughts (on any of the above)?
> 
> Yes, definitely! See above. :) Thanks for drawing my attention to this.
> I'm very interested in your opinion.
> 
> Regards,
> Halil

Re: [virtio-dev] Re: [PATCH 1/4] content: Introduce driver/device auxiliary notifications

[Halil Pasic]

On Wed, 10 Aug 2022 08:45:25 -0400
"Michael S. Tsirkin" <mst@redhat.com> wrote:

> On Wed, Aug 10, 2022 at 11:54:35AM +0200, Cornelia Huck wrote:
> > On Tue, Aug 09 2022, "Michael S. Tsirkin" <mst@redhat.com> wrote:
> >   
> > > On Wed, Mar 30, 2022 at 04:21:02PM +0100, Usama Arif wrote:  
> > >> Driver auxiliary notifications allow the device to send notifications
> > >> other than configuration changes and used buffer notifications to the
> > >> driver, these are optional and their meaning is device-specific.
> > >> 
> > >> Device auxiliary notifcations allow the driver to send notifcations
> > >> other than available buffer notifications to the device for example
> > >> through a device register, these are optional and their meaning is
> > >> device-specific.
> > >> 
> > >> These device-specific notifications are needed later when adding support
> > >> for virtio-vhost-user device.
> > >> 
> > >> Signed-off-by: Usama Arif <usama.arif@bytedance.com>
> > >> Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
> > >> Signed-off-by: Nikos Dragazis <ndragazis@arrikto.com>  
> > >
> > > I see ccw is missing. Cornelia, any suggestions?  
> > 
> > Hmm... I seem to be really behind on ccw things :(
> > 
> > We can probably use the following:
> > 
> > - for device->driver notification, use the next bit in the secondary
> >   indicators (bit 0 is used for config change notification)
> > - for driver->device notification, maybe use a new subcode for diagnose
> >   0x500 (4 is probably the next free one?)
> > 
> > I have not looked at the requirements deeply, though.
> > 
> > This highlights another problem, however: When we introduce new features
> > that require a transport-specific implementation, we often end up with a
> > PCI implementation, but sometimes MMIO and more often ccw are left
> > behind -- which is understandable, as PCI is what most people use, and
> > ccw is something only a very few people are familiar with. This sadly
> > means that we have a backlog of features supported in PCI, but not in
> > ccw... requiring implementations for ccw would put an undue burden on
> > contributors, as most of them are unlikely to write anything for a
> > mainframe, ever. On the flip side, I do not have enough bandwith to deal
> > with all of this.
> > 
> > Halil, any thoughts (on any of the above)?  
> 
> Kind of depends. We Do we want to add a "universal config"
> structure shared between transports?
> Will help with some use-cases though not this one.

I'm for "unversal config"! Regarding this use case I have to dig a
little deeper to really understand!

Regards,
Halil 

Re: [virtio-dev] Re: [PATCH 1/4] content: Introduce driver/device auxiliary notifications

[Halil Pasic]

On Wed, 10 Aug 2022 11:54:35 +0200
Cornelia Huck <cohuck@redhat.com> wrote:

> >> These device-specific notifications are needed later when adding support
> >> for virtio-vhost-user device.
> >> 
> >> Signed-off-by: Usama Arif <usama.arif@bytedance.com>
> >> Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
> >> Signed-off-by: Nikos Dragazis <ndragazis@arrikto.com>  
> >
> > I see ccw is missing. Cornelia, any suggestions?  
> 
> Hmm... I seem to be really behind on ccw things :(
> 
> We can probably use the following:
> 
> - for device->driver notification, use the next bit in the secondary
>   indicators (bit 0 is used for config change notification)
> - for driver->device notification, maybe use a new subcode for diagnose
>   0x500 (4 is probably the next free one?)
> 

Sounds reasonable! I will have to double check the DIAG stuff though. I'm
not sure where what needs to be reserved and documented. 

> I have not looked at the requirements deeply, though.
> 
> This highlights another problem, however: When we introduce new features
> that require a transport-specific implementation, we often end up with a
> PCI implementation, but sometimes MMIO and more often ccw are left
> behind -- which is understandable, as PCI is what most people use, and
> ccw is something only a very few people are familiar with. This sadly
> means that we have a backlog of features supported in PCI, but not in
> ccw... requiring implementations for ccw would put an undue burden on
> contributors, as most of them are unlikely to write anything for a
> mainframe, ever. On the flip side, I do not have enough bandwith to deal
> with all of this.

I'm completely with you in a sense that I see the same problem. I think
we have to get these resolved on a case by case basis. In my opinion at
least in theory it would make a big difference, whether the new feature
obligatory or not. But since VIRTIO is big on compatibility, and also
cares about the initial investment required, in practice, I think, we
are mostly good with the transports delivering features on their own
schedule. What I mean here is: it is kind of difficult to make a new
facility (like shm, or aux notifications) mandatory, because stuff
that conform to a previous incarnation of the spec would become
non-conform.

And the people who care about the particular transport, and the users
of the transport (indirectly also platforms) should make up their own
mind with regards to whether and when to invest into the new facilities
and the new tech and opportunities associated with those.

OTOH when reading the spec, it my strike one as strange, that for example
CCW does not mention aux notifications at all. One idea: maybe we could
add a note, or a subsection, or something, which states the list of
general optional virtio facilities or features not supported by the given
transport on the spec level for a given incarnation of the spec.

I think making the people not motivated to do the design and write the
spec for all the platforms add to that list is a reasonable middle
ground. It would also make the differences very clear, and the same goes
for the intention (i.e. not omitted by mistake).

> 
> Halil, any thoughts (on any of the above)?

Yes, definitely! See above. :) Thanks for drawing my attention to this.
I'm very interested in your opinion.

Regards,
Halil

[virtio-dev] Re: [PATCH 1/4] content: Introduce driver/device auxiliary notifications

[Cornelia Huck]

On Wed, Aug 10 2022, "Michael S. Tsirkin" <mst@redhat.com> wrote:

> On Wed, Aug 10, 2022 at 11:54:35AM +0200, Cornelia Huck wrote:
>> On Tue, Aug 09 2022, "Michael S. Tsirkin" <mst@redhat.com> wrote:
>> 
>> > On Wed, Mar 30, 2022 at 04:21:02PM +0100, Usama Arif wrote:
>> >> Driver auxiliary notifications allow the device to send notifications
>> >> other than configuration changes and used buffer notifications to the
>> >> driver, these are optional and their meaning is device-specific.
>> >> 
>> >> Device auxiliary notifcations allow the driver to send notifcations
>> >> other than available buffer notifications to the device for example
>> >> through a device register, these are optional and their meaning is
>> >> device-specific.
>> >> 
>> >> These device-specific notifications are needed later when adding support
>> >> for virtio-vhost-user device.
>> >> 
>> >> Signed-off-by: Usama Arif <usama.arif@bytedance.com>
>> >> Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
>> >> Signed-off-by: Nikos Dragazis <ndragazis@arrikto.com>
>> >
>> > I see ccw is missing. Cornelia, any suggestions?
>> 
>> Hmm... I seem to be really behind on ccw things :(
>> 
>> We can probably use the following:
>> 
>> - for device->driver notification, use the next bit in the secondary
>>   indicators (bit 0 is used for config change notification)
>> - for driver->device notification, maybe use a new subcode for diagnose
>>   0x500 (4 is probably the next free one?)
>> 
>> I have not looked at the requirements deeply, though.
>> 
>> This highlights another problem, however: When we introduce new features
>> that require a transport-specific implementation, we often end up with a
>> PCI implementation, but sometimes MMIO and more often ccw are left
>> behind -- which is understandable, as PCI is what most people use, and
>> ccw is something only a very few people are familiar with. This sadly
>> means that we have a backlog of features supported in PCI, but not in
>> ccw... requiring implementations for ccw would put an undue burden on
>> contributors, as most of them are unlikely to write anything for a
>> mainframe, ever. On the flip side, I do not have enough bandwith to deal
>> with all of this.
>> 
>> Halil, any thoughts (on any of the above)?
>
> Kind of depends. We Do we want to add a "universal config"
> structure shared between transports?
> Will help with some use-cases though not this one.

A "universal config" structure seems like a good idea; but, as you said,
that will not help in all cases.

>
>
>> >
>> >> ---
>> >>  content.tex | 35 ++++++++++++++++++++++-------------
>> >>  1 file changed, 22 insertions(+), 13 deletions(-)
>> >> 
>> >> diff --git a/content.tex b/content.tex
>> >> index c6f116c..85980ac 100644
>> >> --- a/content.tex
>> >> +++ b/content.tex
>> >> @@ -160,29 +160,38 @@ \subsection{Legacy Interface: A Note on Feature
>> >>  Specification text within these sections generally does not apply
>> >>  to non-transitional devices.
>> >>  
>> >> -\section{Notifications}\label{sec:Basic Facilities of a Virtio Device
>> >> -/ Notifications}
>> >> +\section{Notifications}\label{sec:Basic Facilities of a Virtio Device / Notifications}
>> >>  
>> >>  The notion of sending a notification (driver to device or device
>> >>  to driver) plays an important role in this specification. The
>> >>  modus operandi of the notifications is transport specific.
>> >>  
>> >> -There are three types of notifications: 
>> >> +There are five types of notifications:
>> >>  \begin{itemize}
>> >>  \item configuration change notification
>> >>  \item available buffer notification
>> >> -\item used buffer notification. 
>> >> +\item used buffer notification
>> >> +\item driver auxiliary notification
>> >> +\item device auxiliary notification
>> >>  \end{itemize}
>> >>  
>> >> -Configuration change notifications and used buffer notifications are sent
>> >> -by the device, the recipient is the driver. A configuration change
>> >> -notification indicates that the device configuration space has changed; a
>> >> -used buffer notification indicates that a buffer may have been made used
>> >> -on the virtqueue designated by the notification.
>> >> -
>> >> -Available buffer notifications are sent by the driver, the recipient is
>> >> -the device. This type of notification indicates that a buffer may have
>> >> -been made available on the virtqueue designated by the notification.
>> >> +Configuration change notifications, used buffer notifications and
>> >> +driver auxiliary notifications are sent by the device,
>> >> +the recipient is the driver. A configuration change notification indicates
>> >> +that the device configuration space has changed; a used buffer notification
>> >> +indicates that a buffer may have been made used on the virtqueue designated
>> >> +by the notification; driver auxiliary notifications allow the
>> >> +device to send notifications other than configuration changes and used
>> >> +buffer notifications to the driver, these are optional and their meaning
>> >> +is device-specific.
>> >> +
>> >> +Available buffer notifications and device auxiliary notifications
>> >> +are sent by the driver, the recipient is the device. Available buffer
>> >> +notifications indicate that a buffer may have been made available on the
>> >> +virtqueue designated by the notification; device auxiliary
>> >> +notifcations allow the driver to send notifcations other than available
>> >> +buffer notifications to the device for example through a device register, these
>> >> +are optional and their meaning is device-specific.
>> >>  
>> >>  The semantics, the transport-specific implementations, and other
>> >>  important aspects of the different notifications are specified in detail
>> >> -- 
>> >> 2.25.1


---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

[virtio-dev] Re: [PATCH 1/4] content: Introduce driver/device auxiliary notifications

[Cornelia Huck]

On Tue, Aug 09 2022, "Michael S. Tsirkin" <mst@redhat.com> wrote:

> On Wed, Mar 30, 2022 at 04:21:02PM +0100, Usama Arif wrote:
>> Driver auxiliary notifications allow the device to send notifications
>> other than configuration changes and used buffer notifications to the
>> driver, these are optional and their meaning is device-specific.
>> 
>> Device auxiliary notifcations allow the driver to send notifcations
>> other than available buffer notifications to the device for example
>> through a device register, these are optional and their meaning is
>> device-specific.
>> 
>> These device-specific notifications are needed later when adding support
>> for virtio-vhost-user device.
>> 
>> Signed-off-by: Usama Arif <usama.arif@bytedance.com>
>> Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
>> Signed-off-by: Nikos Dragazis <ndragazis@arrikto.com>
>
> I see ccw is missing. Cornelia, any suggestions?

Hmm... I seem to be really behind on ccw things :(

We can probably use the following:

- for device->driver notification, use the next bit in the secondary
  indicators (bit 0 is used for config change notification)
- for driver->device notification, maybe use a new subcode for diagnose
  0x500 (4 is probably the next free one?)

I have not looked at the requirements deeply, though.

This highlights another problem, however: When we introduce new features
that require a transport-specific implementation, we often end up with a
PCI implementation, but sometimes MMIO and more often ccw are left
behind -- which is understandable, as PCI is what most people use, and
ccw is something only a very few people are familiar with. This sadly
means that we have a backlog of features supported in PCI, but not in
ccw... requiring implementations for ccw would put an undue burden on
contributors, as most of them are unlikely to write anything for a
mainframe, ever. On the flip side, I do not have enough bandwith to deal
with all of this.

Halil, any thoughts (on any of the above)?

>
>> ---
>>  content.tex | 35 ++++++++++++++++++++++-------------
>>  1 file changed, 22 insertions(+), 13 deletions(-)
>> 
>> diff --git a/content.tex b/content.tex
>> index c6f116c..85980ac 100644
>> --- a/content.tex
>> +++ b/content.tex
>> @@ -160,29 +160,38 @@ \subsection{Legacy Interface: A Note on Feature
>>  Specification text within these sections generally does not apply
>>  to non-transitional devices.
>>  
>> -\section{Notifications}\label{sec:Basic Facilities of a Virtio Device
>> -/ Notifications}
>> +\section{Notifications}\label{sec:Basic Facilities of a Virtio Device / Notifications}
>>  
>>  The notion of sending a notification (driver to device or device
>>  to driver) plays an important role in this specification. The
>>  modus operandi of the notifications is transport specific.
>>  
>> -There are three types of notifications: 
>> +There are five types of notifications:
>>  \begin{itemize}
>>  \item configuration change notification
>>  \item available buffer notification
>> -\item used buffer notification. 
>> +\item used buffer notification
>> +\item driver auxiliary notification
>> +\item device auxiliary notification
>>  \end{itemize}
>>  
>> -Configuration change notifications and used buffer notifications are sent
>> -by the device, the recipient is the driver. A configuration change
>> -notification indicates that the device configuration space has changed; a
>> -used buffer notification indicates that a buffer may have been made used
>> -on the virtqueue designated by the notification.
>> -
>> -Available buffer notifications are sent by the driver, the recipient is
>> -the device. This type of notification indicates that a buffer may have
>> -been made available on the virtqueue designated by the notification.
>> +Configuration change notifications, used buffer notifications and
>> +driver auxiliary notifications are sent by the device,
>> +the recipient is the driver. A configuration change notification indicates
>> +that the device configuration space has changed; a used buffer notification
>> +indicates that a buffer may have been made used on the virtqueue designated
>> +by the notification; driver auxiliary notifications allow the
>> +device to send notifications other than configuration changes and used
>> +buffer notifications to the driver, these are optional and their meaning
>> +is device-specific.
>> +
>> +Available buffer notifications and device auxiliary notifications
>> +are sent by the driver, the recipient is the device. Available buffer
>> +notifications indicate that a buffer may have been made available on the
>> +virtqueue designated by the notification; device auxiliary
>> +notifcations allow the driver to send notifcations other than available
>> +buffer notifications to the device for example through a device register, these
>> +are optional and their meaning is device-specific.
>>  
>>  The semantics, the transport-specific implementations, and other
>>  important aspects of the different notifications are specified in detail
>> -- 
>> 2.25.1


---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org